[dev.icinga.com #13237] Icinga2 won't run in background, only foreground

[icinga-migration]

This issue has been migrated from Redmine: https://dev.icinga.com/issues/13237

Created by andyb2000 on 2016-11-18 08:53:06 +00:00

Assignee: andyb2000
Status: Feedback
Target Version: (none)
Last Update: 2016-12-07 21:22:10 +00:00 (in Redmine)

Icinga Version: r2.5.4-1
Backport?: Not yet backported
Include in Changelog: 1

---

Icinga2 running for weeks without issues (in background, using init script /etc/init.d/icinga2 start), this morning process won't start, but it will run in foreground.

No configuration changes were made between the normal running and failure.

When trying to start with the init script (/etc/init.d/icinga2 start), no startup.log written, no log files written, no crash files written. Just no process running.
If I then run with /etc/init.d/icinga2 foreground it starts and works without issue.

I suspect the start of the crash was logged in the icinga2.err:
icinga2: /usr/include/boost/smart_ptr/intrusive_ptr.hpp:174: T* boost::intrusive_ptr::operator->() const [with T = icinga::Stream]: Assertion `px != 0' failed.
Caught SIGABRT.
Current time: 2016-11-17 21:35:52 +0000

icinga2 --version
icinga2 - The Icinga 2 network monitoring daemon (version: r2.5.4-1)

Copyright © 2012-2016 Icinga Development Team (https://www.icinga.org/)
License GPLv2+: GNU GPL version 2 or later <http://gnu.org/licenses/gpl2.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Application information:
Installation root: /usr
Sysconf directory: /etc
Run directory: /run
Local state directory: /var
Package data directory: /usr/share/icinga2
State path: /var/lib/icinga2/icinga2.state
Modified attributes path: /var/lib/icinga2/modified-attributes.conf
Objects path: /var/cache/icinga2/icinga2.debug
Vars path: /var/cache/icinga2/icinga2.vars
PID path: /run/icinga2/icinga2.pid

System information:
Platform: Ubuntu
Platform version: 16.04.1 LTS (Xenial Xerus)
Kernel: Linux
Kernel version: 3.16.0-77-generic
Architecture: x86_64

Build information:
Compiler: GNU 5.4.0
Build host: lgw01-29

Disabled features: compatlog debuglog gelf graphite icingastatus influxdb opentsdb perfdata syslog
Enabled features: api checker command ido-mysql livestatus mainlog notification statusdata

information/cli: Icinga application loader (version: r2.5.4-1)
information/cli: Loading configuration file(s).
information/ConfigItem: Committing config item(s).
information/ApiListener: My API identity: myhost
information/ConfigItem: Instantiated 2 ApiUsers.
information/ConfigItem: Instantiated 1 ApiListener.
information/ConfigItem: Instantiated 2 Zones.
information/ConfigItem: Instantiated 1 FileLogger.
information/ConfigItem: Instantiated 2 Endpoints.
information/ConfigItem: Instantiated 16 NotificationCommands.
information/ConfigItem: Instantiated 21988 Notifications.
information/ConfigItem: Instantiated 222 CheckCommands.
information/ConfigItem: Instantiated 10 Downtimes.
information/ConfigItem: Instantiated 840 Hosts.
information/ConfigItem: Instantiated 1 IcingaApplication.
information/ConfigItem: Instantiated 345 HostGroups.
information/ConfigItem: Instantiated 183 Comments.
information/ConfigItem: Instantiated 26 UserGroups.
information/ConfigItem: Instantiated 27 Users.
information/ConfigItem: Instantiated 10155 Services.
information/ConfigItem: Instantiated 3 TimePeriods.
information/ConfigItem: Instantiated 1 ServiceGroup.
information/ConfigItem: Instantiated 1 NotificationComponent.
information/ConfigItem: Instantiated 1 LivestatusListener.
information/ConfigItem: Instantiated 1 CheckerComponent.
information/ConfigItem: Instantiated 1 IdoMysqlConnection.
information/ConfigItem: Instantiated 1 ExternalCommandListener.
information/ConfigItem: Instantiated 1 StatusDataWriter.
information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
information/cli: Finished validating the configuration file(s).

Please advise what else I can test/carry out to further diagnose.
I carried out the #development instructions got gdb, etc, etc, however running under gdb it works fine (again I suspect because it's running in foreground).

---

Relations:

• relates #12667

[icinga-migration]

Updated by mfriedrich on 2016-11-18 09:23:29 +00:00

• Category set to API
• Status changed from New to Feedback
• Assigned to set to andyb2000

You're probably hitting the same bug as many others with a crash in your http server connection for the API. Are you using the API for anything (Dashing, Event Streams, etc.) at this point?

[icinga-migration]

Updated by mfriedrich on 2016-11-18 09:23:57 +00:00

• Relates set to 12667

[icinga-migration]

Updated by andyb2000 on 2016-11-18 09:27:21 +00:00

Hi,
I am using API (for http API requests GET /v1/ objects, etc) however this has been the case for weeks and weeks without issue. Whilst running in foreground the API requests are still ongoing and not crashing, so not sure if this is related/similar enough?

If I stop all API request and try to run in background would that be a suitable test?

[icinga-migration]

Updated by andyb2000 on 2016-11-18 09:32:22 +00:00

Just an update, I shutdown all remote API connections (shutdown apache2, etc) so nothing was connecting to the API, and the same issue persists so I suspect this may not be related.

[icinga-migration]

Updated by mfriedrich on 2016-11-18 10:05:52 +00:00

A full crash log would be interesting, including the stack trace.

[icinga-migration]

Updated by andyb2000 on 2016-11-18 10:58:44 +00:00

thanks for looking, I'm trying to find a way to replicate and produce a crash/stack trace. I'm starting to suspect systemd is the issue not Icinga2 itself, but if I could get a few pointers?
(I'm using the package provided by 'deb http://packages.icinga.org/ubuntu icinga-xenial main')

/usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2 daemon
/etc/init.d/icinga2 foreground
gdb --args /usr/lib/x86_64-linux-gnu/icinga2/sbin/icinga2 daemon -x debug -DUseVfork=0

All work without any problems (but stay foregrounded to my console) and don't crash.
I've also tried:
/sbin/start-stop-daemon --start --pidfile /var/run/icinga2/icinga2.pid -exec /usr/sbin/icinga2 - daemon "-e /var/log/icinga2/icinga2.err"

Which also runs, which leads me to believe it's the systemd /etc/init.d/icinga2 that's causing the issue here in daemonising the package.

Is there a way I can run icinga2 in background under gdb? I can't run it and attach gdb as the process never lives long enough to start.
I've also tried to add a bit of debugging to /etc/init.d/icinga2 shell script (to determine how far into the file I get) but this doesn't output my debug echo statements.
When I call it:
/var/log/icinga2# /etc/init.d/icinga2 start
Starting icinga2 (via systemctl): icinga2.service.

If I call it from systemctl:

1. systemctl start icinga2
   /var/log/icinga2# echo $?
   0

but no process running

The best I could do was set debug on systemctl:

/var/log/icinga2# SYSTEMD_LOG_LEVEL=debug /bin/systemctl start icinga2
Calling manager for StartUnit on icinga2.service, replace
Sent message type=method_call sender=n/a destination=org.freedesktop.systemd1 object=/org/freedesktop/systemd1 interface=org.freedesktop.systemd1.Manager member=StartUnit cookie=1 reply_cookie=0 error=n/a
Got message type=method_return sender=n/a destination=n/a object=n/a interface=n/a member=n/a cookie=1 reply_cookie=1 error=n/a
Sent message type=method_call sender=n/a destination=org.freedesktop.systemd1 object=/org/freedesktop/systemd1 interface=org.freedesktop.systemd1.Manager member=GetUnit cookie=2 reply_cookie=0 error=n/a
Got message type=method_return sender=n/a destination=n/a object=n/a interface=n/a member=n/a cookie=4 reply_cookie=2 error=n/a
Sent message type=method_call sender=n/a destination=org.freedesktop.systemd1 object=/org/freedesktop/systemd1/unit/icinga2_2eservice interface=org.freedesktop.DBus.Properties member=Get cookie=3 reply_cookie=0 error=n/a
Got message type=method_return sender=n/a destination=n/a object=n/a interface=n/a member=n/a cookie=5 reply_cookie=3 error=n/a
Adding /org/freedesktop/systemd1/job/257353 to the set
Got message type=signal sender=n/a destination=n/a object=/org/freedesktop/systemd1 interface=org.freedesktop.systemd1.Manager member=JobNew cookie=2 reply_cookie=0 error=n/a
Got message type=signal sender=n/a destination=n/a object=/org/freedesktop/systemd1 interface=org.freedesktop.systemd1.Manager member=JobRemoved cookie=3 reply_cookie=0 error=n/a
Got result done/Resource temporarily unavailable for job icinga2.service

Could it somehow be trying to call an X11 interface for some reason? (This is a headless machine without X installed)

[icinga-migration]

Updated by mfriedrich on 2016-12-07 21:22:10 +00:00

Icinga 2 doesn't need X11, that's a dead end. Could you test the snapshot packages, they include fixes for the initial stream crash problem.

[dnsmichi]

I think it is related to #5148 and #4910. I'm closing here since it is an old issue, and not fully debugged. Might have resolved itself already.