1.5.2 fails to shutdown #3575
Labels
Comments
|
+1 We hare the same issue, Our Logstash instance is taking some load form 40-50 machines. We run the official Docker images and when we do docker-compose kill, the container kill process hangs. We need to restart the docker service, which then creates a dead container. To top it off with the latest release of docker-compose, dead containers break the usage of docker-compose (fix in process). So we have no choice but to restart the host to get ride of the defunct process... Please, try fixing this issue as it is becoming more then annoying. |
|
Thanks for the report guys. @tootedom looking at your log traces, since the lumberjack input hits the circuit breaker this indicates that the downstream output plugins are getting stalled somehow which in turn prevents the proper propagation of the shutdown event. It seems you are in fact facing the general pipeline shutdown stall condition described in #3451 Note that we are working on a fix for this problem for 1.6. We will also create a meta-issue about that to link the related issues to the shutdown stalling bigger problem and track progress. |
7 tasks
|
+1 seeing this issue as well. Logs are also being spammed with |
|
Hi there, we just hit this on another environment (where logstash fails to shutdown, and a second process is started), here is a stack trace. Interesting information on this node is that the output (lumberjack - The logstash instance is shipping logs to a Lumberjack input on another node), is that the connection to other node (the lumberjack input) is stuck in SYN_SENT (the firewall is dropping the traffic - something we need to fix our end); but perhaps this an easy way to replicate unable to shutdown (will try when I have a moment). |
|
+1 same here |
|
Here, also affected. |
1 similar comment
|
Here, also affected. |
|
+1 |
|
+1 |
|
+1 This quite nasty - in my case the (many) upstream systems have a large inrush (logstash-forwarder) of log payload so if you stop logstash it cannot start as the breakers trip. Work around is to run logstash in debug mode (--debug), which seems to throttle enough. |
|
+1 |
|
+1 |
|
I can confirm that @bluebenno workaround with |
|
Using protocol => "transport" works very well and has been extremely stable for me ( no --debug required). YMMV etc. i.e. |
|
+1 |
|
the workaround of @bluebenno for the moment works well for me too |
|
I've upgraded to |
|
@pbruna any reason not to use 1.5.4? |
|
@ph no really, it just wasn't available at that moment, Im going to give it a try. |
|
Hi, Im using version 1.5.4 ad still having this problem. {:timestamp=>"2015-09-04T16:25:11.542000+0800", :message=>["INFLIGHT_EVENTS_REPORT", "2015-09-04T16:25:11+08:00", {"input_to_filter"=>20, "filter_to_output"=>0, "outputs"=>[]}], :level=>:warn} |
|
I have also updated to 1.5.4 and am still having the issue as well. |
|
+1 |
|
+1 |
1 similar comment
|
+1 |
|
I fixed it with Also logstash has failed so frequently with following message I fixed this issue temporarily by restarting logstash every 1 hour using cron job |
|
2.x added forced shutdown and stall detection. https://www.elastic.co/guide/en/logstash/current/stalled-shutdown.html |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hi there,
We've hit the following issue with logstash failing to shutdown on our lumberjack indexing node.
Our architecture is:
logstash (shipper) file inputs to a lumberjack output ---------> logstash (indexer) lumberjack input port 555, filters, output ----> elasticsearch
When restarting the logstash (indexer) instance that is listening on the lumberjack input
The result is that the pipeline shuts down, but the process isn't ended (see the SIGTERM in the below). This is not occurring all the time, and we've only hit this issue twice:
The result is that you have client still connected to SIGTERM'd pipeline, as the process is still up, but it is doing nothing. It is not accepting data or sending any data to any output (elasticsearch in out case):
The additional issue is the the restart script (sysv init script), does not notice that the process isn't killed, and attempts to start another logstash instance, which you can see from the log (failure to bind to 5555 as the old process is still up and is bound to that port):
Apologies I do not have a thread dump or heap for you. I shall attempt to get this the next time we hit this issue (This is the 2nd time we've seen this).
The text was updated successfully, but these errors were encountered: