[dev.icinga.com #11590] notification interval = 0 not honoured in HA clusters

[icinga-migration]

This issue has been migrated from Redmine: https://dev.icinga.com/issues/11590

Created by mnardin on 2016-04-14 09:13:01 +00:00

Assignee: gbeutner
Status: Closed (closed on 2016-06-15 16:09:31 +00:00)
Target Version: 2.5.0
Last Update: 2016-08-08 11:17:30 +00:00 (in Redmine)

Icinga Version: 2.4.4
Backport?: Not yet backported
Include in Changelog: 1

---

Hi,
in our setup (2 masters, both with notification features enabled) a notification object with interval = 0 will send out more than one notification:

Object 'backeap01-p!soa-state!mail-to-unarm-prod-server' of type 'Notification':
  % declared in '/var/lib/icinga2/api/packages/director/icingam01-p-1460620904-1/zones.d/ris-global/custom-objects/notifications/mail.conf', lines 38:1-38:57
  * __name = "backeap01-p!soa-state!mail-to-unarm-prod-server"
  * command = "mail-service-notification"
    % = modified in '/var/lib/icinga2/api/packages/director/icingam01-p-1460620904-1/zones.d/ris-global/custom-objects/templates/notification-templates.conf', lines 107:3-107:39
  * command_endpoint = ""
  * host_name = "backeap01-p"
    % = modified in '/var/lib/icinga2/api/packages/director/icingam01-p-1460620904-1/zones.d/ris-global/custom-objects/notifications/mail.conf', lines 38:1-38:57
  * interval = 0
    % = modified in '/var/lib/icinga2/api/packages/director/icingam01-p-1460620904-1/zones.d/ris-global/custom-objects/templates/notification-templates.conf', lines 123:5-123:16
  * name = "mail-to-unarm-prod-server"
  * package = "director"
    % = modified in '/var/lib/icinga2/api/packages/director/icingam01-p-1460620904-1/zones.d/ris-global/custom-objects/notifications/mail.conf', lines 38:1-38:57
  * period = "24x7"
    % = modified in '/var/lib/icinga2/api/packages/director/icingam01-p-1460620904-1/zones.d/ris-global/custom-objects/templates/notification-templates.conf', lines 129:5-129:19
  * service_name = "soa-state"
    % = modified in '/var/lib/icinga2/api/packages/director/icingam01-p-1460620904-1/zones.d/ris-global/custom-objects/notifications/mail.conf', lines 38:1-38:57
  * states = [ 1, 2, 4, 8 ]
    % = modified in '/var/lib/icinga2/api/packages/director/icingam01-p-1460620904-1/zones.d/ris-global/custom-objects/templates/notification-templates.conf', lines 109:3-109:45
  * templates = [ "mail-to-unarm-prod-server", "mail-service-notification", "custom-service-notification-properties" ]
    % = modified in '/var/lib/icinga2/api/packages/director/icingam01-p-1460620904-1/zones.d/ris-global/custom-objects/notifications/mail.conf', lines 38:1-38:57
    % = modified in '/var/lib/icinga2/api/packages/director/icingam01-p-1460620904-1/zones.d/ris-global/custom-objects/templates/notification-templates.conf', lines 106:1-106:49
    % = modified in '/var/lib/icinga2/api/packages/director/icingam01-p-1460620904-1/zones.d/ris-global/custom-objects/templates/notification-templates.conf', lines 115:1-115:62
  * times = null
  * type = "Notification"
  * types = [ 32, 16, 64, 8, 128, 256, 1, 2, 4 ]
    % = modified in '/var/lib/icinga2/api/packages/director/icingam01-p-1460620904-1/zones.d/ris-global/custom-objects/templates/notification-templates.conf', lines 110:3-112:57
  * user_groups = null
  * users = [ "unarm" ]
    % = modified in '/var/lib/icinga2/api/packages/director/icingam01-p-1460620904-1/zones.d/ris-global/custom-objects/notifications/mail.conf', lines 42:3-42:21
  * vars = null
  * zone = "RZ"
    % = modified in '/var/lib/icinga2/api/packages/director/icingam01-p-1460620904-1/zones.d/ris-global/custom-objects/notifications/mail.conf', lines 38:1-38:57

debug.log on master1:

[root@icingam01-p ~]# grep 'backeap01-p!soa-state' /var/log/icinga2/debug.log|egrep -v 'add checkable check history|notice/Dependency|Updating reachability'
[2016-04-14 04:02:29 +0200] debug/DbEvents: add log entry history for 'backeap01-p!soa-state'
[2016-04-14 04:02:29 +0200] debug/DbEvents: add state change history for 'backeap01-p!soa-state'
[2016-04-14 04:02:29 +0200] notice/Checkable: State Change: Checkable backeap01-p!soa-state soft state change from OK to CRITICAL detected.
[2016-04-14 04:05:29 +0200] debug/DbEvents: add log entry history for 'backeap01-p!soa-state'
[2016-04-14 04:06:29 +0200] debug/DbEvents: add log entry history for 'backeap01-p!soa-state'
[2016-04-14 04:07:29 +0200] debug/DbEvents: add log entry history for 'backeap01-p!soa-state'
[2016-04-14 04:07:29 +0200] debug/DbEvents: add state change history for 'backeap01-p!soa-state'
[2016-04-14 04:07:29 +0200] notice/Checkable: State Change: Checkable backeap01-p!soa-state hard state change from CRITICAL to CRITICAL detected.
[2016-04-14 04:07:29 +0200] information/Checkable: Checking for configured notifications for object 'backeap01-p!soa-state'
[2016-04-14 04:07:29 +0200] debug/Checkable: Checkable 'backeap01-p!soa-state' has 1 notification(s).
[2016-04-14 04:07:29 +0200] notice/Notification: Attempting to send notifications for notification object 'backeap01-p!soa-state!mail-to-unarm-prod-server'.
[2016-04-14 04:07:29 +0200] information/Notification: Sending notification 'backeap01-p!soa-state!mail-to-unarm-prod-server' for user 'unarm'
[2016-04-14 04:07:29 +0200] debug/DbEvents: add notification history for 'backeap01-p!soa-state'
[2016-04-14 04:07:29 +0200] debug/DbEvents: add contact notification history for service 'backeap01-p!soa-state' and user 'unarm'.
[2016-04-14 04:07:29 +0200] debug/DbEvents: add log entry history for 'backeap01-p!soa-state'
[2016-04-14 04:07:29 +0200] information/Notification: Completed sending notification 'backeap01-p!soa-state!mail-to-unarm-prod-server' for checkable 'backeap01-p!soa-state'
[2016-04-14 08:57:46 +0200] debug/ServiceDbObject: service dependencies for 'backeap01-p!soa-state'
[2016-04-14 08:57:46 +0200] debug/ServiceDbObject: service contacts: backeap01-p!soa-state
[2016-04-14 08:57:46 +0200] debug/ServiceDbObject: service contactgroups: backeap01-p!soa-state
[2016-04-14 09:02:28 +0200] debug/ServiceDbObject: service dependencies for 'backeap01-p!soa-state'
[2016-04-14 09:02:28 +0200] debug/ServiceDbObject: service contacts: backeap01-p!soa-state
[2016-04-14 09:02:28 +0200] debug/ServiceDbObject: service contactgroups: backeap01-p!soa-state
[2016-04-14 09:09:33 +0200] debug/ServiceDbObject: service dependencies for 'backeap01-p!soa-state'
[2016-04-14 09:09:33 +0200] debug/ServiceDbObject: service contacts: backeap01-p!soa-state
[2016-04-14 09:09:33 +0200] debug/ServiceDbObject: service contactgroups: backeap01-p!soa-state

debug.log on master2:

[root@icingam02-p ~]# grep 'backeap01-p!soa-state' /var/log/icinga2/debug.log|egrep -v 'add checkable check history|notice/Dependency|Updating reachability'
[2016-04-14 04:02:29 +0200] debug/DbEvents: add log entry history for 'backeap01-p!soa-state'
[2016-04-14 04:02:29 +0200] debug/DbEvents: add state change history for 'backeap01-p!soa-state'
[2016-04-14 04:02:29 +0200] notice/Checkable: State Change: Checkable backeap01-p!soa-state soft state change from OK to CRITICAL detected.
[2016-04-14 04:05:29 +0200] debug/DbEvents: add log entry history for 'backeap01-p!soa-state'
[2016-04-14 04:06:29 +0200] debug/DbEvents: add log entry history for 'backeap01-p!soa-state'
[2016-04-14 04:07:29 +0200] debug/DbEvents: add log entry history for 'backeap01-p!soa-state'
[2016-04-14 04:07:29 +0200] debug/DbEvents: add state change history for 'backeap01-p!soa-state'
[2016-04-14 04:07:29 +0200] notice/Checkable: State Change: Checkable backeap01-p!soa-state hard state change from CRITICAL to CRITICAL detected.
[2016-04-14 08:57:55 +0200] debug/ServiceDbObject: service dependencies for 'backeap01-p!soa-state'
[2016-04-14 08:57:55 +0200] debug/ServiceDbObject: service contacts: backeap01-p!soa-state
[2016-04-14 08:57:55 +0200] debug/ServiceDbObject: service contactgroups: backeap01-p!soa-state
[2016-04-14 08:57:58 +0200] notice/NotificationComponent: Attempting to send reminder notification for object 'backeap01-p!soa-state'
[2016-04-14 08:57:58 +0200] notice/Notification: Attempting to send notifications for notification object 'backeap01-p!soa-state!mail-to-unarm-prod-server'.
[2016-04-14 08:57:58 +0200] information/Notification: Sending notification 'backeap01-p!soa-state!mail-to-unarm-prod-server' for user 'unarm'
[2016-04-14 08:57:58 +0200] debug/DbEvents: add notification history for 'backeap01-p!soa-state'
[2016-04-14 08:57:58 +0200] debug/DbEvents: add contact notification history for service 'backeap01-p!soa-state' and user 'unarm'.
[2016-04-14 08:57:58 +0200] debug/DbEvents: add log entry history for 'backeap01-p!soa-state'
[2016-04-14 08:57:58 +0200] information/Notification: Completed sending notification 'backeap01-p!soa-state!mail-to-unarm-prod-server' for checkable 'backeap01-p!soa-state'
[2016-04-14 09:02:36 +0200] debug/ServiceDbObject: service dependencies for 'backeap01-p!soa-state'
[2016-04-14 09:02:36 +0200] debug/ServiceDbObject: service contacts: backeap01-p!soa-state
[2016-04-14 09:02:36 +0200] debug/ServiceDbObject: service contactgroups: backeap01-p!soa-state
[2016-04-14 09:09:41 +0200] debug/ServiceDbObject: service dependencies for 'backeap01-p!soa-state'
[2016-04-14 09:09:41 +0200] debug/ServiceDbObject: service contacts: backeap01-p!soa-state
[2016-04-14 09:09:41 +0200] debug/ServiceDbObject: service contactgroups: backeap01-p!soa-state
[2016-04-14 10:02:07 +0200] debug/ServiceDbObject: service dependencies for 'backeap01-p!soa-state'
[2016-04-14 10:02:07 +0200] debug/ServiceDbObject: service contacts: backeap01-p!soa-state
[2016-04-14 10:02:07 +0200] debug/ServiceDbObject: service contactgroups: backeap01-p!soa-state

I don't understand why this happens even with interval = 0:

[2016-04-14 08:57:58 +0200] notice/NotificationComponent: Attempting to send reminder notification for object 'backeap01-p!soa-state'

Regards
Mirko

---

Relations:

• relates #11590

[icinga-migration]

Updated by mfriedrich on 2016-04-14 09:24:34 +00:00

• Relates set to 11562

[icinga-migration]

Updated by mfriedrich on 2016-04-14 09:25:18 +00:00

• Category set to Notifications

This is most likely a duplicate of #11562.

[icinga-migration]

Updated by mfriedrich on 2016-05-06 13:38:06 +00:00

• Relates deleted 11562

[icinga-migration]

Updated by mfriedrich on 2016-05-06 13:38:36 +00:00

• Parent Id set to 11311

[icinga-migration]

Updated by mfriedrich on 2016-06-07 19:14:47 +00:00

• Subject changed from notification interval = 0 not honored to notification interval = 0 not honoured in HA clusters
• Target Version set to 2.5.0

Left node feels responsible for sending the notification first, then you'll restart/stop the left node. Then the right node is responsible for the notification object. It does not get the last_problem_notification timestamp synced (#11562) and will therefore fire its first local notification. The user gets 2 notifications and thinks that interval = 0 does not work. Apparently it is the missing cluster exchange of notification messages - each node normally calculates the notification times on their own.

[icinga-migration]

Updated by mfriedrich on 2016-06-15 16:09:31 +00:00

• Status changed from New to Closed
• Assigned to set to gbeutner
• Done % changed from 0 to 100

Fixed as part of #11562 and 8a714d6

[icinga-migration]

Updated by mfriedrich on 2016-08-08 11:17:30 +00:00

• Parent Id deleted 11311

[icinga-migration]

Updated by mfriedrich on 2016-08-16 12:47:17 +00:00

• Relates set to 12402