Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[dev.icinga.com #8687] Crash in Dependency::Stop #2739

Closed
icinga-migration opened this issue Mar 11, 2015 · 12 comments
Closed

[dev.icinga.com #8687] Crash in Dependency::Stop #2739

icinga-migration opened this issue Mar 11, 2015 · 12 comments
Labels
bug Something isn't working
Milestone
2.3.1

Comments

@icinga-migration
Copy link

This issue has been migrated from Redmine: https://dev.icinga.com/issues/8687

Created by mcobden on 2015-03-11 12:36:00 +00:00

Assignee: mfriedrich
Status: Resolved (closed on 2015-03-12 10:50:02 +00:00)
Target Version: 2.3.1
Last Update: 2015-03-12 10:52:12 +00:00 (in Redmine)

Icinga Version: v2.3.0-8-g330c93f
Backport?: Not yet backported
Include in Changelog: 1

icinga2: /usr/include/boost/smart_ptr/intrusive_ptr.hpp:162: T* boost::intrusive_ptr::operator->() const [with T = icinga::Checkable]: Assertion `px != 0' failed.
Caught SIGABRT.

Application information:
  Application version: v2.3.0-8-g330c93f
  Installation root: /usr
  Sysconf directory: /etc
  Run directory: /run
  Local state directory: /var
  Package data directory: /usr/share/icinga2
  State path: /var/lib/icinga2/icinga2.state
  Objects path: /var/cache/icinga2/icinga2.debug
  Vars path: /var/cache/icinga2/icinga2.vars
  PID path: /run/icinga2/icinga2.pid
  Application type: icinga/IcingaApplication
Stacktrace:

    (0) libc.so.6:  (+0x36d40) [0x7f4457d8ad40]
    (1) libc.so.6: gsignal (+0x39) [0x7f4457d8acc9]
    (2) libc.so.6: abort (+0x148) [0x7f4457d8e0d8]
    (3) libc.so.6:  (+0x2fb86) [0x7f4457d83b86]
    (4) libc.so.6:  (+0x2fc32) [0x7f4457d83c32]
    (5) libicinga.so:  (+0x173723) [0x7f4451dce723]
    (6) libicinga.so: icinga::Dependency::Stop() (+0x50) [0x7f4451e2dcb4]
    (7) libbase.so: icinga::DynamicObject::Deactivate() (+0x1e6) [0x7f4457408eea]
    (8) libbase.so: icinga::DynamicObject::StopObjects() (+0x3a2) [0x7f445740b7a6]
    (9) libbase.so: icinga::Application::RunEventLoop() (+0x3a3) [0x7f4457410c2b]
    (10) libicinga.so: icinga::IcingaApplication::Main() (+0x1a6) [0x7f4451dfac42]
    (11) libbase.so: icinga::Application::Run() (+0x118) [0x7f44574131d6]
    (12) libcli.so: icinga::DaemonCommand::Run(boost::program_options::variables_map const&, std::vector > const&) const (+0xc89) [0x7f4456d2e735]
    (13) icinga2: Main() (+0x2eae) [0x4171ca]
    (14) icinga2: main (+0x87) [0x413bbb]
    (15) libc.so.6: __libc_start_main (+0xf5) [0x7f4457d75ec5]
    (16) /usr/sbin/icinga2() [0x407449]

***
* This would indicate a runtime problem or configuration error. If you believe this is a bug in Icinga 2
* please submit a bug report at https://dev.icinga.org/ and include this stack trace as well as any other
* information that might be useful in order to reproduce this problem.
***
Could not attach to process.  If your uid matches the uid of the target
process, check the setting of /proc/sys/kernel/yama/ptrace_scope, or try
again as the root user.  For more details, see /etc/sysctl.d/10-ptrace.conf
ptrace: Operation not permitted.
The program is not being run.

Attachments

Changesets

2015-03-12 10:44:55 +00:00 by (unknown) 8573636

Fix crash in Dependency::Stop()

This partially reverts the fix in #8436

fixes #8687
refs #8436

2015-03-12 10:56:20 +00:00 by (unknown) 64214bd

Fix crash in Dependency::Stop()

This partially reverts the fix in #8436

fixes #8687
refs #8436

Conflicts:
	lib/db_ido_pgsql/idopgsqlconnection.cpp

Relations:
@icinga-migration
Copy link
Author

Updated by gbeutner on 2015-03-11 13:54:00 +00:00

  • Category set to libicinga
  • Status changed from New to Feedback
  • Assigned to set to mcobden

Please disable ptrace_scope and try to reproduce the problem:

echo 0 > /proc/sys/kernel/yama/ptrace_scope

@icinga-migration
Copy link
Author

Updated by mcobden on 2015-03-11 14:34:39 +00:00

It got a bit long: http://pastebin.com/qM618Ug0

@icinga-migration
Copy link
Author

Updated by gbeutner on 2015-03-11 17:41:07 +00:00

  • Subject changed from Crash on v2.3.0-8-g330c93f to Crash in Dependency::Stop
  • Status changed from Feedback to New
  • Assigned to deleted mcobden

For some reason m_Child for that dependency object is a null pointer. This is fairly unusual because the config validation should ensure that this can never happen.

@icinga-migration
Copy link
Author

Updated by gbeutner on 2015-03-11 17:43:17 +00:00

  • File added qM618Ug0.txt

@icinga-migration
Copy link
Author

Updated by gbeutner on 2015-03-11 17:44:28 +00:00

Can you show me the output of "icinga2 daemon -C"?

@icinga-migration
Copy link
Author

Updated by mcobden on 2015-03-12 08:45:55 +00:00

information/cli: Icinga application loader (version: v2.3.0-8-g330c93f)
information/cli: Loading application type: icinga/IcingaApplication
information/Utility: Loading library 'libicinga.so'
information/ConfigCompiler: Compiling config file: /etc/icinga2/icinga2.conf
information/ConfigCompiler: Compiling config file: /etc/icinga2/constants.conf
information/ConfigCompiler: Compiling config file: /etc/icinga2/zones.conf
information/ConfigCompiler: Compiling config file: /usr/share/icinga2/include/itl
information/ConfigCompiler: Compiling config file: /usr/share/icinga2/include/plugins
information/ConfigCompiler: Compiling config file: /usr/share/icinga2/include/command.conf
information/ConfigCompiler: Compiling config file: /etc/icinga2/features-enabled/api.conf
information/ConfigCompiler: Compiling config file: /usr/share/icinga2/include/command-icinga.conf
information/ConfigCompiler: Compiling config file: /etc/icinga2/features-enabled/checker.conf
information/ConfigCompiler: Compiling config file: /etc/icinga2/features-enabled/command.conf
information/ConfigCompiler: Compiling config file: /usr/share/icinga2/include/timeperiod.conf
information/ConfigCompiler: Compiling config file: /usr/share/icinga2/include/command-plugins.conf
information/Utility: Loading library 'libchecker.so'
information/ConfigCompiler: Compiling config file: /etc/icinga2/features-enabled/graphite.conf
information/ConfigCompiler: Compiling config file: /etc/icinga2/features-enabled/ido-pgsql.conf
information/Utility: Loading library 'libmethods.so'
information/ConfigCompiler: Compiling config file: /etc/icinga2/features-enabled/livestatus.conf
information/Utility: Loading library 'libdb_ido_pgsql.so'
information/ConfigCompiler: Compiling config file: /etc/icinga2/features-enabled/mainlog.conf
information/ConfigCompiler: Compiling config file: /etc/icinga2/features-enabled/notification.conf
information/ConfigCompiler: Compiling config file: /etc/icinga2/features-enabled/perfdata.conf
information/ConfigCompiler: Compiling config file: /etc/icinga2/conf.d/apt.conf
information/ConfigCompiler: Compiling config file: /etc/icinga2/conf.d/commands.conf
information/ConfigCompiler: Compiling config file: /etc/icinga2/conf.d/downtimes.conf
information/ConfigCompiler: Compiling config file: /etc/icinga2/conf.d/groups.conf
information/Utility: Loading library 'libcompat.so'
information/ConfigCompiler: Compiling config file: /etc/icinga2/conf.d/hosts.conf
information/ConfigCompiler: Compiling config file: /etc/icinga2/conf.d/notifications.conf
information/ConfigCompiler: Compiling config file: /etc/icinga2/conf.d/satellite.conf
information/ConfigCompiler: Compiling config file: /etc/icinga2/conf.d/services.conf
information/ConfigCompiler: Compiling config file: /etc/icinga2/conf.d/templates.conf
information/ConfigCompiler: Compiling config file: /etc/icinga2/conf.d/timeperiods.conf
information/ConfigCompiler: Compiling config file: /etc/icinga2/conf.d/users.conf
information/Utility: Loading library 'liblivestatus.so'
information/Utility: Loading library 'libperfdata.so'
information/Utility: Loading library 'libperfdata.so'
information/Utility: Loading library 'libnotification.so'
information/ConfigCompiler: Compiling config file: /etc/icinga2/conf.d/../../icinga2_ttm_hosts.conf
information/ConfigItem: Committing config items
information/ApiListener: My API identity: foo.bar.baz.com
warning/ApplyRule: Apply rule 'satellite-host' (in /etc/icinga2/conf.d/satellite.conf: 29:1-29:41) for type 'Dependency' does not match anywhere!
information/ConfigItem: Checked 1 ApiListener(s).
information/ConfigItem: Checked 1 Endpoint(s).
information/ConfigItem: Checked 1 Zone(s).
information/ConfigItem: Checked 64 CheckCommand(s).
information/ConfigItem: Checked 2 NotificationCommand(s).
information/ConfigItem: Checked 3 TimePeriod(s).
information/ConfigItem: Checked 1 CheckerComponent(s).
information/ConfigItem: Checked 1 ExternalCommandListener(s).
information/ConfigItem: Checked 1 GraphiteWriter(s).
information/ConfigItem: Checked 1 IdoPgsqlConnection(s).
information/ConfigItem: Checked 1 LivestatusListener(s).
information/ConfigItem: Checked 1 FileLogger(s).
information/ConfigItem: Checked 1 NotificationComponent(s).
information/ConfigItem: Checked 1 PerfdataWriter(s).
information/ConfigItem: Checked 29 HostGroup(s).
information/ConfigItem: Checked 3 ServiceGroup(s).
information/ConfigItem: Checked 396 Host(s).
information/ConfigItem: Checked 7250 Service(s).
information/ConfigItem: Checked 1 User(s).
information/ConfigItem: Checked 51 Notification(s).
information/ConfigItem: Checked 1 UserGroup(s).
information/ConfigItem: Checked 1 IcingaApplication(s).
information/ConfigItem: Checked 4877 Dependency(s).
information/ConfigItem: Checked 1 ScheduledDowntime(s).
information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
information/cli: Finished validating the configuration file(s).

@icinga-migration
Copy link
Author

Updated by gbeutner on 2015-03-12 10:01:11 +00:00

Can you show me the output of

SELECT * FROM icinga_dbversion

for your IDO database.

@icinga-migration
Copy link
Author

Updated by gbeutner on 2015-03-12 10:04:03 +00:00

  • Relates set to 8436

@icinga-migration
Copy link
Author

Updated by mcobden on 2015-03-12 10:06:32 +00:00

 dbversion_id |   name   | version |          create_time          |          modify_time          
--------------+----------+---------+-------------------------------+-------------------------------
            1 | idoutils | 1.12.0  | 2015-01-15 12:30:16.641878+00 | 2015-03-05 14:59:02.243517+00
(1 row)

@icinga-migration
Copy link
Author

Updated by mcobden on 2015-03-12 10:14:58 +00:00

I remember something going awry with the package manager when I was upgrading icinga, something caused it to fail to update the schema.

So i've just updated the schema with the following, and now things run okay.

psql ... < /usr/share/icinga2-ido-pgsql/schema/upgrade/2.3.0.sql

@icinga-migration
Copy link
Author

Updated by Anonymous on 2015-03-12 10:50:02 +00:00

  • Status changed from New to Resolved
  • Done % changed from 0 to 100

Applied in changeset 8573636.

@icinga-migration
Copy link
Author

Updated by mfriedrich on 2015-03-12 10:52:12 +00:00

  • Category changed from libicinga to libbase
  • Assigned to set to mfriedrich
  • Target Version set to 2.3.1
  • Estimated Hours set to 2

We've analyzed the problem and it only occurs when DB IDO requests a shutdown and the (Dependency) object is not fully created yet. This is related to #8436 and the fix partially reverts this.

@icinga-migration icinga-migration added bug Something isn't working libbase labels Jan 17, 2017
@icinga-migration icinga-migration added this to the 2.3.1 milestone Jan 17, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
2.3.1
Development

No branches or pull requests
1 participant
@icinga-migration