-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
0.14.6 seems to deadlock on file buffers #1258
Comments
Could you paste a entire log and sigdump result? |
|
Hmm... it seems this error happens inside signal context. |
I've seen this with file buffer for forward_in, elasticsearch, and s3 plugins. Running with -v, everything runs fine for a while. But, then there is no further log output or activity for 10 minutes or more (should flush at least once a minute). After another 5 minutes of nothing and hitting a ctrl-c, the above is the output. |
Here is a the end of a trace log https://gist.github.com/shawnbutts/5219e076472242aa17d643e468938f77 After the last line of the log, it just stops doing anything. |
Thanks for the detailed log. @tagomoris Could you handle this? |
I'm checking code, but couldn't find clear causes right now. |
The more threads, the longer it takes. But, yes, it happens every time. |
Thank you! I'm digging the situation further more. |
I pushed a branch with fix for output thread handling in shutdown sequence. I couldn't reproduce the exact same problem (deadlock detected) in my environment, but found some problematic log messages with fluent-plugin-s3 and Fluentd v0.14, and this branch has changes to solve it. @shawnbutts Could you try this branch for your problem? |
Sorry, I wasn't able to test this. I've moved back to 0.12.x where this does not appear to be happening. |
I think I may be having a similar issue. I upgraded to the latest release recently and now fluentd no longer seems to function. I continually receive buffer errors and nothing is output to Elasticsearch. Here is my config: fluentd-nxlog_1 | 2016-10-07 21:34:33 +0000 [info]: fluent/supervisor.rb:656:read_config: reading config file path="/fluentd/etc/fluentd-nxlog.conf" Upon startup I just constantly receive: luentd-nxlog_1 | 2016-10-07 21:28:47 +0000 [trace]: plugin/buffer.rb:175:add_metadata: adding metadata instance=47341386269800 metadata=# |
@kkniffin it's just TRACE log to trace plugin's internal behavior, not to show any troubles or errors. |
Yah, but it just constantly floods the screen and never actually sends any Also I am not sure why it even is using that as I am not using buffering I On Oct 7, 2016 11:11 PM, "tagomoris" notifications@github.com wrote:
|
@kkniffin Trace log is for debugging and identifying problems. Filter them out on your environment not to send them to elasticsearch. It's not related with this issue. |
I'm expecting that #1264 solves this problem, so let me close this. |
All threads stop working. During the restarts I see one of these for each thread.
2016-10-04 13:20:06 +0000 [warn]: plugin_helper/thread.rb:74:ensure in block in thread_create: thread doesn't exit correctly (killed or other reason) plugin=Fluent::S3Output title=:enqueue_thread thread=#<Thread:0x000000025dc9e0@/usr/local/share/ruby/gems/2.3/gems/fluentd-0.14.6/lib/fluent/plugin_helper/thread.rb:58 aborting> error=nil
seems to be like #1010
The text was updated successfully, but these errors were encountered: