Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[201803][monit] Restart rsyslog service if rsyslogd consumes > 800 MB memory #2963

Merged
merged 1 commit into from
Jun 4, 2019
Merged

[201803][monit] Restart rsyslog service if rsyslogd consumes > 800 MB memory #2963

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions build_debian.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -268,6 +268,10 @@ check system $HOST
if memory usage > 50% for 5 times within 10 cycles then alert
if cpu usage (user) > 90% for 5 times within 10 cycles then alert
if cpu usage (system) > 90% for 5 times within 10 cycles then alert
check process rsyslog with pidfile /var/run/rsyslogd.pid
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/var/run/rsyslogd.pid [](start = 35, length = 21)

How about the rsyslog processes inside docker? Do they matter?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have not seen the rsyslogd memory leak occur on a rsyslogd process inside any Docker container. The assumption is that those rsyslogd processes have a very light load, whereas the rsyslogd process in the host image also acts as the rsyslog server for all of those processes, so it handles a much higher load of messages.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

rsyslog within container is better managed within the container, for example use superlance with supervisord or use container option to limit the whole memory consumption for the container.

start program = "/bin/systemctl start rsyslog.service"
stop program = "/bin/systemctl stop rsyslog.service"
if totalmem > 800 MB for 5 times within 10 cycles then restart
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

restart [](start = 57, length = 7)

Do we need to keep a restart counter somewhere?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good question, what is the log message for such restart. we can search the syslog for such cases.

Copy link
Contributor Author

@jleveque jleveque Jun 4, 2019 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Each cycle that monit detects the memory has exceeded the threshold it will log the following:

ERR monit[607]: 'rsyslog' total mem amount of 1.6 GB matches resource limit [total mem amount>800.0 MB]

And if it meets the criteria (5 of these within 10 cycles), it will log the following when it attempts to restart the service:

INFO monit[607]: 'rsyslog' trying to restart

EOF

## Config sysctl
Expand Down