[dev.icinga.com #9927] Figure out how to sync dynamically created objects inside the cluster #3278

icinga-migration · 2015-08-17T08:55:48Z

This issue has been migrated from Redmine: https://dev.icinga.com/issues/9927

Created by mfriedrich on 2015-08-17 08:55:48 +00:00

Assignee: mfriedrich
Status: Resolved (closed on 2015-09-17 13:59:33 +00:00)
Target Version: 2.4.0
Last Update: 2015-11-11 14:48:55 +00:00 (in Redmine)

Backport?: No
Include in Changelog: 0

Config objects including comments and downtimes - will be dealt with in Doc: Troubleshooting: recommend reducing concurrency for more speed #9777
Initial sync (/)
Object version (/)
Object origin (/)
Permissions! (/)

API Handler
- UpdateConfig (create (/), modify_attribute (/))
- DeleteConfig (depends on cascading delete) (/)

Changesets

2015-09-10 13:49:56 +00:00 by (unknown) 8306373

Implement config object sync

Adds object version.

refs #9927

2015-09-10 14:54:05 +00:00 by mfriedrich 86b8c00

Implement config sync permissions and modified attributes based on version

refs #9927

2015-09-10 15:03:06 +00:00 by mfriedrich f0eafa7

Implement object config sync permissions and modified attributes based on version

refs #9927

2015-09-10 15:48:06 +00:00 by mfriedrich 2983e0d

Implement deleting api created objects in the cluster

refs #9927

2015-09-11 12:09:46 +00:00 by mfriedrich cc6673e

Config Sync: Properly modify attributes and object version

refs #9927

2015-09-15 09:59:09 +00:00 by mfriedrich cb7bbd6

Update DB IDO on changed object version (create, update, delete)

refs #9927

2015-09-15 14:09:56 +00:00 by mfriedrich df5dde4

Implement initial api object sync for newly connected endpoints

TODO: Figure out how to deal with Shutdown() deactivating and
therefore deleting all api created objects.

refs #9927

2015-09-15 15:56:34 +00:00 by mfriedrich 47f0aad

Prevent object deletion on shutdown when checking !IsActive()

This is fairly ugly and sets an extension for the ConfigObjectUtility
delete handler to signal the OnActiveChanged handler inside the cluster
configsync to send a delete event to the other nodes.

refs #9927

2015-09-17 11:54:09 +00:00 by mfriedrich 20aa36d

Add zone attribute influencing cluster config sync for API objects

1) No zone defined. The object will only be synced in the local zone for HA purposes.
2) Zone is set to 'master'. Only nodes in the master zone will get this object and updates synced.
3) Zone is set to 'satellite'. Only nodes in the satellite zone, or in parent zones above will get this object and updates synced.
4) Zone is set to 'client'. Only nodes in the client zone, and in parent zones (satellite, master) will get object updates.

Furthermore this commit adds a bit more security measures for syncing object
config bottom-up which is clearly restricted at this time. Clients cannot
send their config to the top, but yet we only support the top-down thing we
also have with the cluster file config sync.

The initial sync will also take the zone relation model into account
and only allow object syncs only when the same conditions apply as written
above.

refs #9927
refs #9100

2015-09-17 12:20:43 +00:00 by (unknown) 6fa58a5

Implement config object sync

Adds object version.

refs #9927

2015-09-17 12:20:43 +00:00 by mfriedrich f9c058e

Implement object config sync permissions and modified attributes based on version

refs #9927

2015-09-17 12:20:44 +00:00 by mfriedrich 81a0bc6

Implement deleting api created objects in the cluster

refs #9927

2015-09-17 12:20:44 +00:00 by mfriedrich b271594

Config Sync: Properly modify attributes and object version

refs #9927

2015-09-17 12:20:44 +00:00 by mfriedrich a6d8cea

Update DB IDO on changed object version (create, update, delete)

refs #9927

2015-09-17 12:20:44 +00:00 by mfriedrich 4955c28

Implement initial api object sync for newly connected endpoints

TODO: Figure out how to deal with Shutdown() deactivating and
therefore deleting all api created objects.

refs #9927

2015-09-17 12:20:44 +00:00 by mfriedrich 0fd9d34

Prevent object deletion on shutdown when checking !IsActive()

This is fairly ugly and sets an extension for the ConfigObjectUtility
delete handler to signal the OnActiveChanged handler inside the cluster
configsync to send a delete event to the other nodes.

refs #9927

2015-09-17 12:20:44 +00:00 by mfriedrich 18d645e

Add zone attribute influencing cluster config sync for API objects

1) No zone defined. The object will only be synced in the local zone for HA purposes.
2) Zone is set to 'master'. Only nodes in the master zone will get this object and updates synced.
3) Zone is set to 'satellite'. Only nodes in the satellite zone, or in parent zones above will get this object and updates synced.
4) Zone is set to 'client'. Only nodes in the client zone, and in parent zones (satellite, master) will get object updates.

Furthermore this commit adds a bit more security measures for syncing object
config bottom-up which is clearly restricted at this time. Clients cannot
send their config to the top, but yet we only support the top-down thing we
also have with the cluster file config sync.

The initial sync will also take the zone relation model into account
and only allow object syncs only when the same conditions apply as written
above.

refs #9927
refs #9100

2015-09-17 13:52:54 +00:00 by mfriedrich 7dc3d28

Docs: Add a chapter on cluster config sync for the API

refs #9927
refs #9100

2015-09-18 08:07:13 +00:00 by mfriedrich f2c3bff

Sync cluster config before replaying the logs

If there were objects added at runtime (either through direct api
creation or by using the config file management api) the newly
created objects must be synced first, and then the stored historical
data should be synced.

refs #9927
refs #9100

2015-09-18 10:49:38 +00:00 by mfriedrich 57179f3

Only sync objects actually belonging to a cluster zone

refs #9927
refs #9100

2015-09-29 12:24:39 +00:00 by mfriedrich bb3b724

Fix object sync for modified attributes

refs #9851
refs #9927
refs #9081

2015-09-30 08:04:37 +00:00 by mfriedrich 19e7524

Fix problem with non-existing objects in config sync updates

refs #9851
refs #9927
refs #9081

2015-10-26 05:53:36 +00:00 by (unknown) 43bbbfc

Undo accidental edit in icinga.cpp from 6fa58a52

refs #9927

2015-11-08 20:22:06 +00:00 by (unknown) 4bb9bed

Don't send cluster config messages for objects with version 0

refs #9927

Parent Task: #9082

Relations:

relates #9927
relates #9927
relates #9927
relates #9927
relates #9927
relates #9927
relates #9777
relates #9927
relates #9927
relates #9927

The text was updated successfully, but these errors were encountered:

icinga-migration · 2015-08-17T13:02:03Z

Updated by mfriedrich on 2015-08-17 13:02:03 +00:00

Description updated

icinga-migration · 2015-08-18T14:58:36Z

Updated by mfriedrich on 2015-08-18 14:58:36 +00:00

Relates set to 9100

icinga-migration · 2015-08-20T14:38:52Z

Updated by mfriedrich on 2015-08-20 14:38:52 +00:00

Description updated

icinga-migration · 2015-08-20T14:41:32Z

Updated by mfriedrich on 2015-08-20 14:41:32 +00:00

Description updated

icinga-migration · 2015-09-04T11:57:12Z

Updated by mfriedrich on 2015-09-04 11:57:12 +00:00

Priority changed from Normal to High

icinga-migration · 2015-09-05T15:33:40Z

Updated by mfriedrich on 2015-09-05 15:33:40 +00:00

Relates set to 7342

icinga-migration · 2015-09-10T16:15:59Z

Updated by mfriedrich on 2015-09-10 16:15:59 +00:00

Description updated
Status changed from New to Assigned
Assigned to set to mfriedrich
Done % changed from 0 to 50

rebased against master (broken at several steps)
added modified attributes support (missing vars.os still)
added delete object handlers

TODOs

permissions - same zone works, but do we sync top-down / bottom-up?
initial sync when there no objects on a new instance
downtimes and comments as config objects (separate issue)

icinga-migration · 2015-09-11T08:19:53Z

Updated by mfriedrich on 2015-09-11 08:19:53 +00:00

Relates set to 9850

icinga-migration · 2015-09-11T09:05:48Z

Updated by mfriedrich on 2015-09-11 09:05:48 +00:00

Created objects trigger both: OnActiveChanged and then OnVersionChanged which results in two updates.

icinga-migration · 2015-09-11T13:31:42Z

Updated by mfriedrich on 2015-09-11 13:31:42 +00:00

Ok nvm. Each modified attribute will trigger OnVersionChanged and therefore sending a request on its own.

$ curl -k -s -u root:icinga -X POST -d '{ "attrs": { "address": "8.8.8.2", "check_command": "ping4" } }' 'https://localhost:7000/v1/hosts/google.com2' | python -m json.tool

[2015-09-11 15:27:34 +0200] information/ApiListener: New client connection (no client certificate)
[2015-09-11 15:27:34 +0200] information/ApiListener: New HTTP client
[2015-09-11 15:27:34 +0200] debug/HttpRequest: line: POST /v1/hosts/google.com2 HTTP/1.1, tokens: 3
[2015-09-11 15:27:34 +0200] information/HttpServerConnection: Processing Http message
[2015-09-11 15:27:34 +0200] warning/ApiListener: Sent update for object: {"config":"object Host \"google.com2\" {\n\taddress = \"8.8.8.8\"\n\tcheck_command = \"hostalive\"\n}\n","modified_attributes":{"address":"8.8.8.2"},"name":"google.com2","type":"Host","version":2.0}
[2015-09-11 15:27:34 +0200] notice/ApiListener: Relaying 'config::UpdateObject' message
[2015-09-11 15:27:34 +0200] notice/ApiListener: Sending message to 'icinga2b'
[2015-09-11 15:27:34 +0200] notice/ApiListener: Relaying 'event::SetCheckCommand' message
[2015-09-11 15:27:34 +0200] notice/ApiListener: Sending message to 'icinga2b'
[2015-09-11 15:27:34 +0200] warning/ApiListener: Sent update for object: {"config":"object Host \"google.com2\" {\n\taddress = \"8.8.8.8\"\n\tcheck_command = \"hostalive\"\n}\n","modified_attributes":{"address":"8.8.8.2","check_command":"ping4"},"name":"google.com2","type":"Host","version":3.0}
[2015-09-11 15:27:34 +0200] notice/ApiListener: Relaying 'config::UpdateObject' message
[2015-09-11 15:27:34 +0200] notice/ApiListener: Sending message to 'icinga2b'
[2015-09-11 15:27:34 +0200] notice/JsonRpcConnection: Received 'config::UpdateObject' message from 'icinga2b'
[2015-09-11 15:27:34 +0200] warning/ApiListener: Received update for object: {"config":"object Host \"google.com2\" {\n\taddress = \"8.8.8.8\"\n\tcheck_command = \"hostalive\"\n}\n","modified_attributes":{"address":"8.8.8.2"},"name":"google.com2","type":"Host","version":2.0}
[2015-09-11 15:27:34 +0200] warning/ApiListener: Discarding config update for object 'google.com2': Object version 3 is more recent than the received version 2.
[2015-09-11 15:27:34 +0200] notice/JsonRpcConnection: Received 'event::SetCheckCommand' message from 'icinga2b'
[2015-09-11 15:27:34 +0200] notice/JsonRpcConnection: Received 'config::UpdateObject' message from 'icinga2b'
[2015-09-11 15:27:34 +0200] warning/ApiListener: Received update for object: {"config":"object Host \"google.com2\" {\n\taddress = \"8.8.8.8\"\n\tcheck_command = \"hostalive\"\n}\n","modified_attributes":{"address":"8.8.8.2","check_command":"ping4"},"name":"google.com2","type":"Host","version":3.0}
[2015-09-11 15:27:34 +0200] warning/ApiListener: Discarding config update for object 'google.com2': Object version 3 is more recent than the received version 3.

The other cluster node will update its own local object and set the object version, which will send back a new message but being discarded thanks to the object version comparison inside the api handler. Since this message is relayed and just an asynchronous answer, I don't see a legit way to suppress these messages (everything else is handled by the cluster protocol already).

icinga-migration · 2015-09-12T09:05:21Z

Updated by mfriedrich on 2015-09-12 09:05:21 +00:00

Relates set to 8213

icinga-migration · 2015-09-12T09:08:31Z

Updated by mfriedrich on 2015-09-12 09:08:31 +00:00

Relates set to 8897

icinga-migration · 2015-09-15T10:05:11Z

Updated by mfriedrich on 2015-09-15 10:05:11 +00:00

Relates set to 9098

icinga-migration · 2015-09-15T14:26:30Z

Updated by mfriedrich on 2015-09-15 14:26:30 +00:00

Initial sync works the same way as the cluster config sync, except that it just calls UpdateConfigObject for all api created objects, and sends them over directly to the connected endpoint as client. Instead of iterating over all objects and only selecting the "_api" package this could be enhanced performance-wise by just glob'ing the types and object names from disk? (in memory is certainly better, requires tests).

One thing I've found still not working properly - when the application is shut down, OnActiveChanged is triggered, object->IsActive() is false and the DeleteObject handler is called. Then the object silently gets removed on shutdown, even relaying it to the other node.

icinga-migration · 2015-09-15T16:03:23Z

Updated by mfriedrich on 2015-09-15 16:03:23 +00:00

It's origin is in the async ping-pong for cluster events on create/update/delete. Sending the delete request over to the first node must not happen at all, and then the event comes back deleting it locally, even worse.

^C[2015-09-15 17:56:02 +0200] information/Application: Received request to shut down.
[2015-09-15 17:56:02 +0200] information/Application: Shutting down...
[2015-09-15 17:56:02 +0200] warning/ApiListener: Sent delete object: {"name":"google.com2","type":"Host","version":1.0}
Context:
    (0) Deactivating object 'google.com2' of type 'Host'

[2015-09-15 17:56:02 +0200] notice/ApiListener: Relaying 'config::DeleteObject' message
[2015-09-15 17:56:02 +0200] information/ConfigObject: Dumping program state to file 'icinga2b/lib/icinga2/icinga2.state'
[2015-09-15 17:56:02 +0200] notice/ApiListener: Sending message to 'icinga2a'
[2015-09-15 17:56:02 +0200] notice/JsonRpcConnection: Received 'config::DeleteObject' message from 'icinga2a'
[2015-09-15 17:56:02 +0200] warning/ApiListener: Received update for object: {"name":"google.com2","type":"Host","version":1.0}
[2015-09-15 17:56:02 +0200] information/IcingaApplication: Icinga has shut down.

Before the object gets destroyed for good, I've added a special extension similar to DbObject, which allows the cluster config sync to properly delete the object (once called by the api), but not do that on application shutdown.

icinga-migration · 2015-09-15T16:04:54Z

Updated by mfriedrich on 2015-09-15 16:04:54 +00:00

TODO

Figure out how to sync objects between zones (for now this only words in the same zone). I'm splitting this into a separate issue in order to close this one.

icinga-migration · 2015-09-17T12:03:10Z

Updated by mfriedrich on 2015-09-17 12:03:10 +00:00

Description updated

Comments/Downtimes will be handled in #9777.

Furthermore I've been looking into the zone attribute influencing the cluster config sync (part of #9100) which has now been added as well, including additional security measures only allowing the config object sync happen top-down at the moment.

icinga-migration · 2015-09-17T12:03:21Z

Updated by mfriedrich on 2015-09-17 12:03:21 +00:00

Relates set to 9777

icinga-migration · 2015-09-17T13:59:33Z

Updated by mfriedrich on 2015-09-17 13:59:33 +00:00

Status changed from Assigned to Resolved
Done % changed from 50 to 100

icinga-migration · 2015-09-21T07:40:19Z

Updated by mfriedrich on 2015-09-21 07:40:19 +00:00

Backport? changed from TBD to No

icinga-migration · 2015-09-22T10:03:09Z

Updated by mfriedrich on 2015-09-22 10:03:09 +00:00

Relates set to 10193

icinga-migration · 2015-09-30T08:16:34Z

Updated by mfriedrich on 2015-09-30 08:16:34 +00:00

Relates set to 10249

icinga-migration · 2015-09-30T08:17:43Z

Updated by mfriedrich on 2015-09-30 08:17:43 +00:00

Relates set to 10250

icinga-migration · 2015-11-11T14:48:56Z

Updated by mfriedrich on 2015-11-11 14:48:56 +00:00

Include in Changelog changed from 1 to 0

icinga-migration closed this as completed Sep 17, 2015

icinga-migration added blocker Blocks a release or needs immediate attention enhancement New feature or request area/api REST API labels Jan 17, 2017

icinga-migration added this to the 2.4.0 milestone Jan 17, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[dev.icinga.com #9927] Figure out how to sync dynamically created objects inside the cluster #3278

[dev.icinga.com #9927] Figure out how to sync dynamically created objects inside the cluster #3278

icinga-migration commented Aug 17, 2015

icinga-migration commented Aug 17, 2015

icinga-migration commented Aug 18, 2015

icinga-migration commented Aug 20, 2015

icinga-migration commented Aug 20, 2015

icinga-migration commented Sep 4, 2015

icinga-migration commented Sep 5, 2015

icinga-migration commented Sep 10, 2015

icinga-migration commented Sep 11, 2015

icinga-migration commented Sep 11, 2015

icinga-migration commented Sep 11, 2015

icinga-migration commented Sep 12, 2015

icinga-migration commented Sep 12, 2015

icinga-migration commented Sep 15, 2015

icinga-migration commented Sep 15, 2015

icinga-migration commented Sep 15, 2015

icinga-migration commented Sep 15, 2015

icinga-migration commented Sep 17, 2015

icinga-migration commented Sep 17, 2015

icinga-migration commented Sep 17, 2015

icinga-migration commented Sep 21, 2015

icinga-migration commented Sep 22, 2015

icinga-migration commented Sep 30, 2015

icinga-migration commented Sep 30, 2015

icinga-migration commented Nov 11, 2015