Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

0.14.6 seems to deadlock on file buffers #1258

Closed
shawnbutts opened this issue Oct 4, 2016 · 16 comments
Closed

0.14.6 seems to deadlock on file buffers #1258

shawnbutts opened this issue Oct 4, 2016 · 16 comments
Assignees
Labels
bug Something isn't working v0.14

Comments

@shawnbutts
Copy link

  • fluentd 0.14.6 (ruby 2.3.0).
  • Amazon Linux

All threads stop working. During the restarts I see one of these for each thread.

2016-10-04 13:20:06 +0000 [warn]: plugin_helper/thread.rb:74:ensure in block in thread_create: thread doesn't exit correctly (killed or other reason) plugin=Fluent::S3Output title=:enqueue_thread thread=#<Thread:0x000000025dc9e0@/usr/local/share/ruby/gems/2.3/gems/fluentd-0.14.6/lib/fluent/plugin_helper/thread.rb:58 aborting> error=nil

seems to be like #1010

@repeatedly
Copy link
Member

Could you paste a entire log and sigdump result?

@shawnbutts
Copy link
Author

shawnbutts commented Oct 4, 2016

2016-10-04 13:39:36 +0000 [debug]: fluent/supervisor.rb:536:block in install_main_process_signal_handlers: fluentd main process get SIGINT
2016-10-04 13:39:36 +0000 [info]: serverengine/server.rb:51:stop: Received graceful stop
2016-10-04 13:39:37 +0000 [debug]: fluent/supervisor.rb:547:block in install_main_process_signal_handlers: fluentd main process get SIGTERM
2016-10-04 13:39:37 +0000 [debug]: fluent/supervisor.rb:550:block in install_main_process_signal_handlers: getting start to shutdown main process
2016-10-04 13:39:37 +0000 [info]: fluent/engine.rb:190:run: shutting down fluentd
2016-10-04 13:39:37 +0000 [debug]: fluent/root_agent.rb:173:block (2 levels) in shutdown: calling stop on input plugin type=:forward plugin_id="object:17600c4"
2016-10-04 13:39:37 +0000 [debug]: fluent/root_agent.rb:173:block (2 levels) in shutdown: calling stop on output plugin type=:s3 plugin_id="object:1318264"
2016-10-04 13:39:37 +0000 [info]: fluent/root_agent.rb:199:block (3 levels) in shutdown: preparing shutdown input plugin type=:forward plugin_id="object:17600c4"
2016-10-04 13:39:37 +0000 [info]: fluent/root_agent.rb:199:block (3 levels) in shutdown: preparing shutdown output plugin type=:s3 plugin_id="object:1318264"
2016-10-04 13:39:37 +0000 [info]: fluent/root_agent.rb:199:block (3 levels) in shutdown: shutting down input plugin type=:forward plugin_id="object:17600c4"
2016-10-04 13:39:37 +0000 [warn]: plugin_helper/thread.rb:74:ensure in block in thread_create: thread doesn't exit correctly (killed or other reason) plugin=Fluent::S3Output title=:flush_thread_0 thread=#<Thread:0x0000000408c088@/usr/local/share/ruby/gems/2.3/gems/fluentd-0.14.6/lib/fluent/plugin_helper/thread.rb:58 aborting> error=nil
2016-10-04 13:39:37 +0000 [warn]: plugin_helper/thread.rb:74:ensure in block in thread_create: thread doesn't exit correctly (killed or other reason) plugin=Fluent::S3Output title=:flush_thread_1 thread=#<Thread:0x0000000408bd40@/usr/local/share/ruby/gems/2.3/gems/fluentd-0.14.6/lib/fluent/plugin_helper/thread.rb:58 aborting> error=nil
2016-10-04 13:39:37 +0000 [warn]: plugin_helper/thread.rb:74:ensure in block in thread_create: thread doesn't exit correctly (killed or other reason) plugin=Fluent::S3Output title=:enqueue_thread thread=#<Thread:0x0000000408ba20@/usr/local/share/ruby/gems/2.3/gems/fluentd-0.14.6/lib/fluent/plugin_helper/thread.rb:58 aborting> error=nil
/usr/local/share/ruby/gems/2.3/gems/fluentd-0.14.6/lib/fluent/root_agent.rb:192:in `join': No live threads left. Deadlock? (fatal)
        from /usr/local/share/ruby/gems/2.3/gems/fluentd-0.14.6/lib/fluent/root_agent.rb:192:in `block (3 levels) in shutdown'
        from /usr/local/share/ruby/gems/2.3/gems/fluentd-0.14.6/lib/fluent/root_agent.rb:192:in `each'
        from /usr/local/share/ruby/gems/2.3/gems/fluentd-0.14.6/lib/fluent/root_agent.rb:192:in `block (2 levels) in shutdown'
        from /usr/local/share/ruby/gems/2.3/gems/fluentd-0.14.6/lib/fluent/root_agent.rb:131:in `block in lifecycle'
        from /usr/local/share/ruby/gems/2.3/gems/fluentd-0.14.6/lib/fluent/root_agent.rb:113:in `each'
        from /usr/local/share/ruby/gems/2.3/gems/fluentd-0.14.6/lib/fluent/root_agent.rb:113:in `lifecycle'
        from /usr/local/share/ruby/gems/2.3/gems/fluentd-0.14.6/lib/fluent/root_agent.rb:195:in `block in shutdown'
        from /usr/local/share/ruby/gems/2.3/gems/fluentd-0.14.6/lib/fluent/root_agent.rb:215:in `shutdown'
        from /usr/local/share/ruby/gems/2.3/gems/fluentd-0.14.6/lib/fluent/engine.rb:215:in `shutdown'
        from /usr/local/share/ruby/gems/2.3/gems/fluentd-0.14.6/lib/fluent/engine.rb:191:in `run'
        from /usr/local/share/ruby/gems/2.3/gems/fluentd-0.14.6/lib/fluent/supervisor.rb:677:in `run_engine'
        from /usr/local/share/ruby/gems/2.3/gems/fluentd-0.14.6/lib/fluent/supervisor.rb:436:in `block in run_worker'
        from /usr/local/share/ruby/gems/2.3/gems/fluentd-0.14.6/lib/fluent/supervisor.rb:606:in `main_process'
        from /usr/local/share/ruby/gems/2.3/gems/fluentd-0.14.6/lib/fluent/supervisor.rb:431:in `run_worker'
        from /usr/local/share/ruby/gems/2.3/gems/fluentd-0.14.6/lib/fluent/command/fluentd.rb:271:in `<top (required)>'
        from /usr/share/ruby/vendor_ruby/2.3/rubygems/core_ext/kernel_require.rb:68:in `require'
        from /usr/share/ruby/vendor_ruby/2.3/rubygems/core_ext/kernel_require.rb:68:in `require'
        from /usr/local/share/ruby/gems/2.3/gems/fluentd-0.14.6/bin/fluentd:5:in `<top (required)>'
        from /usr/local/bin/fluentd:23:in `load'
        from /usr/local/bin/fluentd:23:in `<main>'
2016-10-04 13:39:39 +0000 [info]: serverengine/multi_process_server.rb:136:alive?: Worker 0 finished with status 1

@repeatedly
Copy link
Member

Hmm... it seems this error happens inside signal context.
We will investigate which mutex/lock causes this message.

@repeatedly repeatedly added bug Something isn't working v0.14 labels Oct 4, 2016
@shawnbutts
Copy link
Author

I've seen this with file buffer for forward_in, elasticsearch, and s3 plugins. Running with -v, everything runs fine for a while. But, then there is no further log output or activity for 10 minutes or more (should flush at least once a minute). After another 5 minutes of nothing and hitting a ctrl-c, the above is the output.

@shawnbutts
Copy link
Author

Here is a the end of a trace log https://gist.github.com/shawnbutts/5219e076472242aa17d643e468938f77

After the last line of the log, it just stops doing anything.

@repeatedly
Copy link
Member

repeatedly commented Oct 4, 2016

Thanks for the detailed log.

@tagomoris Could you handle this?

@tagomoris
Copy link
Member

I'm checking code, but couldn't find clear causes right now.
@shawnbutts Does the your problem occur every time? Could you paste your configuration?

@shawnbutts
Copy link
Author

The more threads, the longer it takes. But, yes, it happens every time.
I've update the gist. The "startup.log" has the config, etc from the beginning of the log.

@tagomoris
Copy link
Member

Thank you! I'm digging the situation further more.

@tagomoris
Copy link
Member

I pushed a branch with fix for output thread handling in shutdown sequence.
https://github.com/fluent/fluentd/tree/wakeup-threads-in-shutdown-sequence-to-quit-sleep

I couldn't reproduce the exact same problem (deadlock detected) in my environment, but found some problematic log messages with fluent-plugin-s3 and Fluentd v0.14, and this branch has changes to solve it.

@shawnbutts Could you try this branch for your problem?

@shawnbutts
Copy link
Author

Sorry, I wasn't able to test this. I've moved back to 0.12.x where this does not appear to be happening.

@kkniffin
Copy link

kkniffin commented Oct 7, 2016

I think I may be having a similar issue. I upgraded to the latest release recently and now fluentd no longer seems to function. I continually receive buffer errors and nothing is output to Elasticsearch.

Here is my config:

fluentd-nxlog_1 | 2016-10-07 21:34:33 +0000 [info]: fluent/supervisor.rb:656:read_config: reading config file path="/fluentd/etc/fluentd-nxlog.conf"
fluentd-nxlog_1 | 2016-10-07 21:34:33 +0000 [info]: fluent/supervisor.rb:504:supervise: starting fluentd-0.14.7
fluentd-nxlog_1 | 2016-10-07 21:34:33 +0000 [info]: fluent/supervisor.rb:525:supervise: spawn command to main: /usr/bin/ruby -Eascii-8bit:ascii-8bit /home/fluent/.gem/ruby/2.3.0/bin/fluentd -c /fluentd/etc/fluentd-nxlog.conf -p /fluentd/plugins -vv --under-supervisor
fluentd-nxlog_1 | 2016-10-07 21:34:34 +0000 [info]: fluent/supervisor.rb:656:read_config: reading config file path="/fluentd/etc/fluentd-nxlog.conf"
fluentd-nxlog_1 | 2016-10-07 21:34:34 +0000 [info]: fluent/supervisor.rb:447:run_worker: starting fluentd-0.14.7 without supervision
fluentd-nxlog_1 | 2016-10-07 21:34:34 +0000 [info]: fluent/engine.rb:116:block in configure: gem 'fluent-plugin-elasticsearch' version '1.7.0'
fluentd-nxlog_1 | 2016-10-07 21:34:34 +0000 [info]: fluent/engine.rb:116:block in configure: gem 'fluent-plugin-forest' version '0.3.3'
fluentd-nxlog_1 | 2016-10-07 21:34:34 +0000 [info]: fluent/engine.rb:116:block in configure: gem 'fluent-plugin-grok-parser' version '2.0.0'
fluentd-nxlog_1 | 2016-10-07 21:34:34 +0000 [info]: fluent/engine.rb:116:block in configure: gem 'fluent-plugin-parser' version '0.6.1'
fluentd-nxlog_1 | 2016-10-07 21:34:34 +0000 [info]: fluent/engine.rb:116:block in configure: gem 'fluent-plugin-record-modifier' version '0.5.0'
fluentd-nxlog_1 | 2016-10-07 21:34:34 +0000 [info]: fluent/engine.rb:116:block in configure: gem 'fluent-plugin-record-reformer' version '0.8.2'
fluentd-nxlog_1 | 2016-10-07 21:34:34 +0000 [info]: fluent/engine.rb:116:block in configure: gem 'fluent-plugin-rewrite' version '0.0.13'
fluentd-nxlog_1 | 2016-10-07 21:34:34 +0000 [info]: fluent/engine.rb:116:block in configure: gem 'fluent-plugin-rewrite-tag-filter' version '1.5.5'
fluentd-nxlog_1 | 2016-10-07 21:34:34 +0000 [info]: fluent/engine.rb:116:block in configure: gem 'fluentd' version '0.14.7'
fluentd-nxlog_1 | 2016-10-07 21:34:34 +0000 [info]: fluent/engine.rb:116:block in configure: gem 'fluentd' version '0.12.29'
fluentd-nxlog_1 | 2016-10-07 21:34:34 +0000 [info]: fluent/agent.rb:144:add_filter: adding filter pattern="nxlog" type="record_modifier"
fluentd-nxlog_1 | 2016-10-07 21:34:34 +0000 [trace]: fluent/plugin.rb:140:register_impl: registered filter plugin 'record_modifier'
fluentd-nxlog_1 | 2016-10-07 21:34:34 +0000 [warn]: config/element.rb:115:[]: 'type' is deprecated parameter name. use '@type' instead.
fluentd-nxlog_1 | 2016-10-07 21:34:34 +0000 [info]: fluent/agent.rb:123:add_match: adding match pattern="nxlog." type="rewrite_tag_filter"
fluentd-nxlog_1 | 2016-10-07 21:34:34 +0000 [trace]: fluent/plugin.rb:140:register_impl: registered output plugin 'rewrite_tag_filter'
fluentd-nxlog_1 | 2016-10-07 21:34:34 +0000 [info]: plugin/out_rewrite_tag_filter.rb:55:block in configure: adding rewrite_tag_filter rule: rewriterule1 ["NXLogFileType", /^dhcp$/, "", "dhcp.log"]
fluentd-nxlog_1 | 2016-10-07 21:34:34 +0000 [info]: plugin/out_rewrite_tag_filter.rb:55:block in configure: adding rewrite_tag_filter rule: rewriterule2 ["NXLogFileType", /^nps$/, "", "nps.log"]
fluentd-nxlog_1 | 2016-10-07 21:34:34 +0000 [info]: plugin/out_rewrite_tag_filter.rb:55:block in configure: adding rewrite_tag_filter rule: rewriterule3 ["NXLogFileType", /^dns$/, "", "dns.log"]
fluentd-nxlog_1 | 2016-10-07 21:34:34 +0000 [info]: plugin/out_rewrite_tag_filter.rb:55:block in configure: adding rewrite_tag_filter rule: rewriterule4 ["NXLogFileType", /^wineventlog$/, "", "winevent.log"]
fluentd-nxlog_1 | 2016-10-07 21:34:34 +0000 [info]: plugin/out_rewrite_tag_filter.rb:55:block in configure: adding rewrite_tag_filter rule: rewriterule5 ["NXLogFileType", /^iis$/, "", "iis.log"]
fluentd-nxlog_1 | 2016-10-07 21:34:34 +0000 [info]: plugin/out_rewrite_tag_filter.rb:55:block in configure: adding rewrite_tag_filter rule: rewriterule6 ["NXLogFileType", /.+/, "", "rawbeat.nonmatched"]
fluentd-nxlog_1 | 2016-10-07 21:34:34 +0000 [info]: fluent/agent.rb:123:add_match: adding match pattern="{*beat,tacacs,dhcp,nps,dns,iis,winevent}.
" type="forest"
fluentd-nxlog_1 | 2016-10-07 21:34:34 +0000 [trace]: fluent/plugin.rb:140:register_impl: registered output plugin 'forest'
fluentd-nxlog_1 | 2016-10-07 21:34:34 +0000 [info]: fluent/root_agent.rb:230:add_source: adding source type="tcp"
fluentd-nxlog_1 | 2016-10-07 21:34:34 +0000 [trace]: fluent/plugin.rb:140:register_impl: registered input plugin 'tcp'
fluentd-nxlog_1 | 2016-10-07 21:34:34 +0000 [trace]: fluent/plugin.rb:140:register_impl: registered parser plugin 'regexp'
fluentd-nxlog_1 | 2016-10-07 21:34:34 +0000 [trace]: fluent/plugin.rb:140:register_impl: registered parser plugin 'json'
fluentd-nxlog_1 | 2016-10-07 21:34:34 +0000 [trace]: fluent/plugin.rb:140:register_impl: registered parser plugin 'tsv'
fluentd-nxlog_1 | 2016-10-07 21:34:34 +0000 [trace]: fluent/plugin.rb:140:register_impl: registered parser plugin 'ltsv'
fluentd-nxlog_1 | 2016-10-07 21:34:34 +0000 [trace]: fluent/plugin.rb:140:register_impl: registered parser plugin 'csv'
fluentd-nxlog_1 | 2016-10-07 21:34:34 +0000 [trace]: fluent/plugin.rb:140:register_impl: registered parser plugin 'none'
fluentd-nxlog_1 | 2016-10-07 21:34:34 +0000 [trace]: fluent/plugin.rb:140:register_impl: registered parser plugin 'apache2'
fluentd-nxlog_1 | 2016-10-07 21:34:34 +0000 [trace]: fluent/plugin.rb:140:register_impl: registered parser plugin 'syslog'
fluentd-nxlog_1 | 2016-10-07 21:34:34 +0000 [trace]: fluent/plugin.rb:140:register_impl: registered parser plugin 'multiline'
fluentd-nxlog_1 | 2016-10-07 21:34:34 +0000 [info]: fluent/engine.rb:123:configure: using configuration file:
fluentd-nxlog_1 |
fluentd-nxlog_1 | @type tcp
fluentd-nxlog_1 | format "json"
fluentd-nxlog_1 | port 5140
fluentd-nxlog_1 | tag "nxlog"
fluentd-nxlog_1 |
fluentd-nxlog_1 |
fluentd-nxlog_1 | @type record_modifier
fluentd-nxlog_1 |
fluentd-nxlog_1 | SyslogReceived ${Time.at(time).to_s}
fluentd-nxlog_1 |
fluentd-nxlog_1 |
fluentd-nxlog_1 | <match nxlog.>
fluentd-nxlog_1 | type rewrite_tag_filter
fluentd-nxlog_1 | rewriterule1 NXLogFileType ^dhcp$ dhcp.log
fluentd-nxlog_1 | rewriterule2 NXLogFileType ^nps$ nps.log
fluentd-nxlog_1 | rewriterule3 NXLogFileType ^dns$ dns.log
fluentd-nxlog_1 | rewriterule4 NXLogFileType ^wineventlog$ winevent.log
fluentd-nxlog_1 | rewriterule5 NXLogFileType ^iis$ iis.log
fluentd-nxlog_1 | rewriterule6 NXLogFileType .+ rawbeat.nonmatched
fluentd-nxlog_1 |

fluentd-nxlog_1 | <match {*beat,tacacs,dhcp,nps,dns,iis,winevent}.
>
fluentd-nxlog_1 | @type forest
fluentd-nxlog_1 | subtype "copy"
fluentd-nxlog_1 |
fluentd-nxlog_1 |
fluentd-nxlog_1 | @type elasticsearch
fluentd-nxlog_1 | hosts 1.1.1.1,1.1.1.2
fluentd-nxlog_1 | user elastic
fluentd-nxlog_1 | password changeme
fluentd-nxlog_1 | port 9200
fluentd-nxlog_1 | logstash_format true
fluentd-nxlog_1 | logstash_prefix ${tag_parts[0]}
fluentd-nxlog_1 | utc_index true
fluentd-nxlog_1 | flatten_hashes true
fluentd-nxlog_1 | flatten_hashes_separator _
fluentd-nxlog_1 |
fluentd-nxlog_1 |
fluentd-nxlog_1 |
fluentd-nxlog_1 |
fluentd-nxlog_1 | 2016-10-07 21:34:34 +0000 [info]: plugin/in_tcp.rb:30:listen: listening tcp socket on 0.0.0.0:5140

Upon startup I just constantly receive:

luentd-nxlog_1 | 2016-10-07 21:28:47 +0000 [trace]: plugin/buffer.rb:175:add_metadata: adding metadata instance=47341386269800 metadata=#
fluentd-nxlog_1 | 2016-10-07 21:28:47 +0000 [trace]: plugin/buffer.rb:198:write: writing events into buffer instance=47341386269800 metadata_size=1
fluentd-nxlog_1 | 2016-10-07 21:28:47 +0000 [trace]: plugin/buffer.rb:175:add_metadata: adding metadata instance=47341386269800 metadata=#
fluentd-nxlog_1 | 2016-10-07 21:28:47 +0000 [trace]: plugin/buffer.rb:198:write: writing events into buffer instance=47341386269800 metadata_size=1
fluentd-nxlog_1 | 2016-10-07 21:28:47 +0000 [trace]: plugin/buffer.rb:175:add_metadata: adding metadata instance=47341386269800 metadata=#
fluentd-nxlog_1 | 2016-10-07 21:28:47 +0000 [trace]: plugin/buffer.rb:198:write: writing events into buffer instance=47341386269800 metadata_size=1
fluentd-nxlog_1 | 2016-10-07 21:28:47 +0000 [trace]: plugin/buffer.rb:175:add_metadata: adding metadata instance=47341386269800 metadata=#
fluentd-nxlog_1 | 2016-10-07 21:28:47 +0000 [trace]: plugin/buffer.rb:198:write: writing events into buffer instance=47341386269800 metadata_size=1
fluentd-nxlog_1 | 2016-10-07 21:28:47 +0000 [trace]: plugin/buffer.rb:175:add_metadata: adding metadata instance=47341386269800 metadata=#
fluentd-nxlog_1 | 2016-10-07 21:28:47 +0000 [trace]: plugin/buffer.rb:198:write: writing events into buffer instance=47341386269800 metadata_size=1
fluentd-nxlog_1 | 2016-10-07 21:28:47 +0000 [trace]: plugin/buffer.rb:175:add_metadata: adding metadata instance=47341386269800 metadata=#
fluentd-nxlog_1 | 2016-10-07 21:28:47 +0000 [trace]: plugin/buffer.rb:198:write: writing events into buffer instance=47341386269800 metadata_size=1
fluentd-nxlog_1 | 2016-10-07 21:28:47 +0000 [trace]: plugin/buffer.rb:175:add_metadata: adding metadata instance=47341386269800 metadata=#
fluentd-nxlog_1 | 2016-10-07 21:28:47 +0000 [trace]: plugin/buffer.rb:198:write: writing events into buffer instance=47341386269800 metadata_size=1
fluentd-nxlog_1 | 2016-10-07 21:28:47 +0000 [trace]: plugin/buffer.rb:175:add_metadata: adding metadata instance=47341386269800 metadata=#
fluentd-nxlog_1 | 2016-10-07 21:28:47 +0000 [trace]: plugin/buffer.rb:198:write: writing events into buffer instance=47341386269800 metadata_size=1
fluentd-nxlog_1 | 2016-10-07 21:28:47 +0000 [trace]: plugin/buffer.rb:175:add_metadata: adding metadata instance=47341386269800 metadata=#
fluentd-nxlog_1 | 2016-10-07 21:28:47 +0000 [trace]: plugin/buffer.rb:198:write: writing events into buffer instance=47341386269800 metadata_size=1
fluentd-nxlog_1 | 2016-10-07 21:28:47 +0000 [trace]: plugin/buffer.rb:175:add_metadata: adding metadata instance=47341386269800 metadata=#
fluentd-nxlog_1 | 2016-10-07 21:28:47 +0000 [trace]: plugin/buffer.rb:198:write: writing events into buffer instance=47341386269800 metadata_size=1
fluentd-nxlog_1 | 2016-10-07 21:28:47 +0000 [trace]: plugin/buffer.rb:175:add_metadata: adding metadata instance=47341386269800 metadata=#
fluentd-nxlog_1 | 2016-10-07 21:28:47 +0000 [trace]: plugin/buffer.rb:198:write: writing events into buffer instance=47341386269800 metadata_size=1
fluentd-nxlog_1 | 2016-10-07 21:28:47 +0000 [trace]: plugin/buffer.rb:175:add_metadata: adding metadata instance=47341386269800 metadata=#
fluentd-nxlog_1 | 2016-10-07 21:28:47 +0000 [trace]: plugin/buffer.rb:198:write: writing events into buffer instance

@tagomoris
Copy link
Member

@kkniffin it's just TRACE log to trace plugin's internal behavior, not to show any troubles or errors.

@kkniffin
Copy link

kkniffin commented Oct 8, 2016

Yah, but it just constantly floods the screen and never actually sends any
information to elasticsearch. Rolling back the code produces none of those
trace errors and elasticsearch functionality works again.

Also I am not sure why it even is using that as I am not using buffering I
didn't think.

On Oct 7, 2016 11:11 PM, "tagomoris" notifications@github.com wrote:

@kkniffin https://github.com/kkniffin it's just TRACE log to trace
plugin's internal behavior, not to show any troubles or errors.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#1258 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/APGfcVK9FkjyRmhS0F9tAXIqjtLEsZXSks5qxwnygaJpZM4KNrAS
.

@tagomoris
Copy link
Member

@kkniffin Trace log is for debugging and identifying problems. Filter them out on your environment not to send them to elasticsearch. It's not related with this issue.

@tagomoris
Copy link
Member

I'm expecting that #1264 solves this problem, so let me close this.
Please reopen this or create another one for same/similar problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working v0.14
Projects
None yet
Development

No branches or pull requests

4 participants