Unable to start icinga2 with kernel-3.10.0-514.21.2 RHEL7 #5367

alexlud · 2017-06-20T06:36:19Z

General Notes

This seems to be an upstream Kernel regression in RHEL 7 only.

Please read the published advisory and our twitter channel where we keep posting updates on the matter.

https://www.icinga.com/2017/06/20/advisory-for-latest-security-updates-on-rhel-7/

Original Description

Hello,
I've applied the latest kernel update on my Icinga2 box. After booting the new kernel icinga2 is no longer able to start.
Running the previous kernel version is my current workaround.

Log:
Jun 20 08:15:05 icinga.example.com prepare-dirs[2629]: execvp: Argument list too long
Jun 20 08:15:05 icinga.example.com prepare-dirs[2629]: Could not fetch RunAsUser variable. Error ''. Exiting.
Jun 20 08:15:05 icinga.example.com systemd[1]: icinga2.service: control process exited, code=exited status=6
Jun 20 08:15:05 icinga.example.com systemd[1]: Failed to start Icinga host/service/network monitoring system.
Jun 20 08:15:05 icinga.example.com systemd[1]: Unit icinga2.service entered failed state.
Jun 20 08:15:05 icinga.example.com systemd[1]: icinga2.service failed.

Icinga2 version is 2.6.3
RHEL7.3 with all updates
kernel-3.10.0-514.21.2.el7.x86_64

pefmeister · 2017-06-20T06:50:58Z

Hi there,

got the same problem. This happend after upgrading to the newest RHEL kernel / glibc. The following (quick and dirty) fix did at least let me start Icinga again.

Change in /usr/sbin/icinga2 the last line to look like this:

exec $ICINGA2_BIN --no-stack-rlimit "$@"

pefmeister · 2017-06-20T07:39:13Z

When running strace, two systems with different patchlevel behave differently:

System with 3.10.0-514.21.1.el7.x86_64:

setrlimit(RLIMIT_NOFILE, {rlim_cur=16*1024, rlim_max=16*1024}) = 0
setrlimit(RLIMIT_NPROC, {rlim_cur=16*1024, rlim_max=16*1024}) = 0
getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM64_INFINITY}) = 0
setrlimit(RLIMIT_STACK, {rlim_cur=256*1024, rlim_max=RLIM64_INFINITY}) = 0
execve("/usr/lib64/icinga2/sbin/icinga2", ["/usr/lib64/icinga2/sbin/icinga2", "--no-stack-rlimit"], [/* 25 vars */]) = 0
brk(0)                                  = 0x243b000`

System with 3.10.0-514.21.2.el7.x86_64:

setrlimit(RLIMIT_NOFILE, {rlim_cur=16*1024, rlim_max=16*1024}) = 0
setrlimit(RLIMIT_NPROC, {rlim_cur=16*1024, rlim_max=16*1024}) = 0
getrlimit(RLIMIT_STACK, {rlim_cur=RLIM64_INFINITY, rlim_max=RLIM64_INFINITY}) = 0
setrlimit(RLIMIT_STACK, {rlim_cur=256*1024, rlim_max=RLIM64_INFINITY}) = 0
execve("/usr/lib64/icinga2/sbin/icinga2", ["/usr/lib64/icinga2/sbin/icinga2", "--no-stack-rlimit"], [/* 21 vars */]) = -1 E2BIG (Argument list too long)

There seems to be some major change in the behavior of the kernels. Any idea how to change that?

pefmeister · 2017-06-20T07:52:52Z

I suppose this is related to https://rhn.redhat.com/errata/RHSA-2017-1484.html. Is this something that must be fixed within Icinga?

gunnarbeutner · 2017-06-20T07:55:27Z

Looks like their security fix inadvertently breaks legitimate uses of setrlimit(RLIMIT_STACK, ...).

dnsmichi · 2017-06-20T08:48:37Z

Thanks for the report, we'll look into that and are therefore postponing today's v2.7 release.

dnsmichi · 2017-06-20T08:58:59Z

CVE-2017-1000364 seems fixed/applied in Debian too.

https://security-tracker.debian.org/tracker/CVE-2017-1000364

dnsmichi · 2017-06-20T09:01:50Z

Related too: https://security-tracker.debian.org/tracker/CVE-2017-1000365

mcktr · 2017-06-20T09:19:38Z

@dnsmichi I applied the patches this morning to our Debian 8 'jessie' system and Icinga 2 is still starting after a reboot.

root@[HOSTNAME]:~# uname -a
Linux [HOSTNAME] 3.16.0-4-amd64 #1 SMP Debian 3.16.43-2+deb8u1 (2017-06-18) x86_64 GNU/Linux

Is there anything else I can send you to help with this problem?

dnsmichi · 2017-06-20T09:38:47Z

@mcktr thanks a lot, it's good to know that Debian does not seem to be affected.

We're currently investigating on the RHEL kernel update, diff'ing -1 and -2 source rpms.

dnsmichi · 2017-06-20T10:07:05Z

stack_guard_gap = 256UL>>PAGE_SHIFT

expendable_stack_area()

Setting 4.5 MB stack size works, 4 MB does not.

dnsmichi · 2017-06-20T10:09:06Z

Kernel patch:

https://patchwork.kernel.org/patch/9797613/
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=1be7107fbe18eed3e319a6c3e83c78254b693acb

Affected RHEL releases:

https://bugzilla.redhat.com/show_bug.cgi?id=1461333

dnsmichi · 2017-06-20T11:22:59Z

We're lowering the stack size not to reserve too much memory for spawned threads. An older version just attempted to set ulimit -u inside the init script which failed on Debian Jessie in #1006.

Options:

remove RLIMIT_STACK and make it a systemd/init script option again
increase rlimit to a hardcoded size

dnsmichi · 2017-06-20T11:40:31Z

RHEL vulnerability patch from https://access.redhat.com/security/vulnerabilities/stackguard
Debian patched but not affected
SLES https://www.suse.com/de-de/security/cve/CVE-2017-1000364/

Crunsher · 2017-06-20T11:59:40Z

Alright, for now you can use @pefmeister workaround, we'll have a blogpost detailing the issues out in the coming days and 2.7 will come with a longterm solution.

lazyfrosch · 2017-06-20T12:25:20Z

We reported a bug to RedHat mentioning the problem:
https://bugzilla.redhat.com/show_bug.cgi?id=1463241

The bug is currently private (I guess default for kernel)

You can reproduce the problem in a more simple way:

$ ulimit -s 1024
$ /bin/true
bash: /bin/true: Argument list too long

$ ulimit -s 4096
$ /bin/true
bash: /bin/true: Argument list too long

dnsmichi · 2017-06-20T13:22:46Z

https://www.icinga.com/2017/06/20/advisory-for-latest-security-updates-on-rhel-7/

lazyfrosch · 2017-06-20T13:37:57Z

RHEL 6 seems to be fine:

[root@rhel6-test ~]# uname -a
Linux rhel6-test.localdomain 2.6.32-696.3.2.el6.x86_64 #1 SMP Wed Jun 7 11:51:39 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux
[root@rhel6-test ~]# bash -c "ulimit -s 256; /bin/true; echo 'Works.'"
Works.

dnsmichi · 2017-06-20T14:33:44Z

Issues to consider:

dnsmichi · 2017-06-20T15:43:50Z

The workaround for systemd also requires the prepare-dirs script being patched.

diff --git a/etc/initsystem/prepare-dirs b/etc/initsystem/prepare-dirs
index 6c4a08869..5677a787a 100644
--- a/etc/initsystem/prepare-dirs
+++ b/etc/initsystem/prepare-dirs
@@ -13,13 +13,13 @@ else
 fi


-ICINGA2_USER=`$DAEMON variable get --current RunAsUser`
+ICINGA2_USER=`$DAEMON variable get --current RunAsUser --no-stack-rlimit`
 if [ $? != 0 ]; then
         echo "Could not fetch RunAsUser variable. Error '$ICINGA2_USER'. Exiting."
         exit 6
 fi

-ICINGA2_GROUP=`$DAEMON variable get --current RunAsGroup`
+ICINGA2_GROUP=`$DAEMON variable get --current RunAsGroup --no-stack-rlimit`
 if [ $? != 0 ]; then
         echo "Could not fetch RunAsGroup variable. Error '$ICINGA2_GROUP'. Exiting."
         exit 6

dnsmichi · 2017-06-20T16:05:14Z

Reference: https://monitoring-portal.org/index.php?thread/41070-problems-with-rhel-7-kernel-update-kernel-3-10-0-514-21-2-and-icinga-2/

dnsmichi · 2017-06-20T17:23:31Z

CentOS 7 is currently rolling the kernel update onto the mirrors. The main mirror has it available.

[root@icinga2 ~]# uname -a ; ulimit -s 1024 && /bin/true && echo "works"
Linux icinga2 3.10.0-514.21.2.el7.x86_64 #1 SMP Tue Jun 20 12:24:47 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
-bash: /bin/true: Argument list too long

lazyfrosch · 2017-06-20T17:46:12Z

Looks like there are related problems with the Kernel Update, but also on Debian jessie here:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=865311

dnsmichi · 2017-06-20T17:52:28Z

Reported CentOS bug: https://bugs.centos.org/view.php?id=13453

dnsmichi · 2017-06-20T20:02:36Z

https://lkml.org/lkml/2017/6/19/1515

lazyfrosch · 2017-06-22T11:20:20Z

We received a test-build from RedHat that works fine in my test environment.

…mitProcesses and RLimitStack refs #5367

Make rlimits configurable by adding three variables: RLimitFiles, RLimitProcesses and RLimitStack refs #5367

dnsmichi · 2017-06-23T11:02:49Z

Our advisory is updated with everything that happened.

https://www.icinga.com/2017/06/20/advisory-for-latest-security-updates-on-rhel-7/

Please ensure to open a support case at RedHat to ask for an accelerated fix, or a test RPM. This raises awareness that they'll release it soon enough.

The configuration options have been added for v2.7. I would leave this issue open until RedHat/CentOS released a new Kernel update.

alexlud · 2017-06-29T07:42:13Z

New kernel from RH is available.
kernel-3.10.0-514.26.1

No issues so far.

pefmeister · 2017-06-29T07:57:04Z

I can confirm this, too. Seems to be working with the new kernel. Let's go 2.7!

dnsmichi · 2017-06-29T09:46:49Z

Thanks for your tests. We'll wait until everything is publicly resolved.

https://bugzilla.redhat.com/show_bug.cgi?id=1463241 is not clear about its state, CentOS still has the old Kernel version.

It is also highly likely that Debian was affected as they recently changed their patch set.
https://tracker.debian.org/media/packages/l/linux/changelog-4.9.30-2%2Bdeb9u2
https://lists.debian.org/debian-security-announce/2017/msg00160.html

There might be more patches or regressions coming in, see e.g. torvalds/linux@98da7d0

Let's wait and see when the Kernel problems will calm down, then we'll may start a release cycle for 2.7 again.

lazyfrosch · 2017-06-29T10:10:33Z

A knowledge base entry has been published, saying solution is in progress

https://access.redhat.com/solutions/3092281

lazyfrosch · 2017-06-29T11:41:28Z

Also related: https://access.redhat.com/solutions/3098341

Copis · 2017-06-30T09:52:04Z

Hi,

CentOS has released new kernel update 3.10.0-514.26.1 and i could confirm that icinga2 process starts well.

dnsmichi · 2017-07-20T12:33:23Z

Catching up after vacation - the CentOS bug tracker item (https://bugs.centos.org/view.php?id=13453) is resolved and RedHat has published multiple Kernel versions too. Tested that inside the Vagrant box, works fine.

[root@icinga2 ~]# uname -a && icinga2 daemon -C
Linux icinga2 3.10.0-514.26.2.el7.x86_64 #1 SMP Tue Jul 4 15:04:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
information/cli: Icinga application loader (version: v2.6.3-399-gc7d71b0)
information/cli: Loading configuration file(s).
information/ConfigItem: Committing config item(s).
information/ApiListener: My API identity: icinga2
warning/ApplyRule: Apply rule 'satellite-host' (in /etc/icinga2/conf.d/satellite.conf: 29:1-29:41) for type 'Dependency' does not match anywhere!
information/ConfigItem: Instantiated 4 ApiUsers.
information/ConfigItem: Instantiated 1 ApiListener.
information/ConfigItem: Instantiated 3 Zones.
information/ConfigItem: Instantiated 1 FileLogger.
information/ConfigItem: Instantiated 1 Endpoint.
information/ConfigItem: Instantiated 1 UserGroup.
information/ConfigItem: Instantiated 28 Notifications.
information/ConfigItem: Instantiated 2 NotificationCommands.
information/ConfigItem: Instantiated 177 CheckCommands.
information/ConfigItem: Instantiated 1 Downtime.
information/ConfigItem: Instantiated 4 HostGroups.
information/ConfigItem: Instantiated 1 IcingaApplication.
information/ConfigItem: Instantiated 157 Hosts.
information/ConfigItem: Instantiated 318 Comments.
information/ConfigItem: Instantiated 1 User.
information/ConfigItem: Instantiated 3 TimePeriods.
information/ConfigItem: Instantiated 161 Services.
information/ConfigItem: Instantiated 3 ServiceGroups.
information/ConfigItem: Instantiated 1 ScheduledDowntime.
information/ConfigItem: Instantiated 1 IdoMysqlConnection.
information/ConfigItem: Instantiated 1 NotificationComponent.
information/ConfigItem: Instantiated 1 GraphiteWriter.
information/ConfigItem: Instantiated 1 CheckerComponent.
information/ScriptGlobal: Dumping variables to file '/var/cache/icinga2/icinga2.vars'
information/cli: Finished validating the configuration file(s).

We'll discuss the 2.7 release once everyone involved returned from holidays, probably next week or so.

Closing here, thanks to everyone involved 👍

…mitProcesses and RLimitStack refs #5367

dnsmichi added core/crash Shouldn't happen, requires attention blocker Blocks a release or needs immediate attention labels Jun 20, 2017

dnsmichi added this to the 2.7.0 milestone Jun 20, 2017

dnsmichi assigned dnsmichi, gunnarbeutner and Crunsher Jun 20, 2017

dnsmichi added the area/cli Command line helpers label Jun 20, 2017

knerhood mentioned this issue Jun 20, 2017

Unable to render existing config with kernel 3.10.0-514.21.2 CentOS7 Icinga/icingaweb2-module-director#973

Closed

dnsmichi mentioned this issue Jun 21, 2017

WIP: Increase limit for large scale/high performance #111

Closed

lazyfrosch mentioned this issue Jun 22, 2017

Make rlimits configurable by adding three variables: RLimitFiles, RLimitProcesses and RLimitStack #5373

Merged

dnsmichi pushed a commit that referenced this issue Jun 23, 2017

Make rlimits configurable by adding three variables: RLimitFiles, RLi…

c8b4fee

…mitProcesses and RLimitStack refs #5367

dnsmichi pushed a commit that referenced this issue Jun 23, 2017

Merge pull request #5373 from Icinga/feature/rlimit-options

0e423df

Make rlimits configurable by adding three variables: RLimitFiles, RLimitProcesses and RLimitStack refs #5367

gunnarbeutner mentioned this issue Jun 28, 2017

unable to start icinga2 after reinstallation #5380

Closed

dnsmichi closed this as completed Jul 20, 2017

Crunsher pushed a commit that referenced this issue Jul 28, 2017

Make rlimits configurable by adding three variables: RLimitFiles, RLi…

74bfa33

…mitProcesses and RLimitStack refs #5367

dnsmichi added the bug Something isn't working label Aug 3, 2017

dnsmichi unassigned dnsmichi, gunnarbeutner and Crunsher Sep 17, 2019

dmitriy-terzeman mentioned this issue Apr 13, 2020

Icinga2 fails to start with kernel-3.10.0-514.21.2 RHEL7 #7973

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to start icinga2 with kernel-3.10.0-514.21.2 RHEL7 #5367

Unable to start icinga2 with kernel-3.10.0-514.21.2 RHEL7 #5367

alexlud commented Jun 20, 2017 •

edited by dnsmichi

Loading

pefmeister commented Jun 20, 2017

pefmeister commented Jun 20, 2017

pefmeister commented Jun 20, 2017

gunnarbeutner commented Jun 20, 2017

dnsmichi commented Jun 20, 2017

dnsmichi commented Jun 20, 2017

dnsmichi commented Jun 20, 2017

mcktr commented Jun 20, 2017 •

edited

Loading

dnsmichi commented Jun 20, 2017

dnsmichi commented Jun 20, 2017

dnsmichi commented Jun 20, 2017

dnsmichi commented Jun 20, 2017

dnsmichi commented Jun 20, 2017

Crunsher commented Jun 20, 2017

lazyfrosch commented Jun 20, 2017

dnsmichi commented Jun 20, 2017

lazyfrosch commented Jun 20, 2017

dnsmichi commented Jun 20, 2017

dnsmichi commented Jun 20, 2017

dnsmichi commented Jun 20, 2017

dnsmichi commented Jun 20, 2017

lazyfrosch commented Jun 20, 2017

dnsmichi commented Jun 20, 2017

dnsmichi commented Jun 20, 2017

lazyfrosch commented Jun 22, 2017

dnsmichi commented Jun 23, 2017

alexlud commented Jun 29, 2017

pefmeister commented Jun 29, 2017

dnsmichi commented Jun 29, 2017

lazyfrosch commented Jun 29, 2017

lazyfrosch commented Jun 29, 2017

Copis commented Jun 30, 2017

dnsmichi commented Jul 20, 2017

Unable to start icinga2 with kernel-3.10.0-514.21.2 RHEL7 #5367

Unable to start icinga2 with kernel-3.10.0-514.21.2 RHEL7 #5367

Comments

alexlud commented Jun 20, 2017 • edited by dnsmichi Loading

General Notes

Original Description

pefmeister commented Jun 20, 2017

pefmeister commented Jun 20, 2017

pefmeister commented Jun 20, 2017

gunnarbeutner commented Jun 20, 2017

dnsmichi commented Jun 20, 2017

dnsmichi commented Jun 20, 2017

dnsmichi commented Jun 20, 2017

mcktr commented Jun 20, 2017 • edited Loading

dnsmichi commented Jun 20, 2017

dnsmichi commented Jun 20, 2017

dnsmichi commented Jun 20, 2017

dnsmichi commented Jun 20, 2017

dnsmichi commented Jun 20, 2017

Crunsher commented Jun 20, 2017

lazyfrosch commented Jun 20, 2017

dnsmichi commented Jun 20, 2017

lazyfrosch commented Jun 20, 2017

dnsmichi commented Jun 20, 2017

dnsmichi commented Jun 20, 2017

dnsmichi commented Jun 20, 2017

dnsmichi commented Jun 20, 2017

lazyfrosch commented Jun 20, 2017

dnsmichi commented Jun 20, 2017

dnsmichi commented Jun 20, 2017

lazyfrosch commented Jun 22, 2017

dnsmichi commented Jun 23, 2017

alexlud commented Jun 29, 2017

pefmeister commented Jun 29, 2017

dnsmichi commented Jun 29, 2017

lazyfrosch commented Jun 29, 2017

lazyfrosch commented Jun 29, 2017

Copis commented Jun 30, 2017

dnsmichi commented Jul 20, 2017

alexlud commented Jun 20, 2017 •

edited by dnsmichi

Loading

mcktr commented Jun 20, 2017 •

edited

Loading