plugin throws errors if not receiving messages and crashes on exit

[jsvd]

On logstash 1.5.0 on macosx:

input {
  zeromq {
    topology => "pushpull"
      address => ["tcp://127.0.0.1:2100"]
      mode => "client"
      sockopt => ["ZMQ::RCVTIMEO", 5000]
  }
}
output { stdout { codec => rubydebug } }

Logstash startup completed
ZeroMQ error while in recv_string {:error_code=>-1, :level=>:error}
ZeroMQ error while in recv_string {:error_code=>-1, :level=>:error}
ZeroMQ error while in recv_string {:error_code=>-1, :level=>:error}
ZeroMQ error while in recv_string {:error_code=>-1, :level=>:error}
^CSIGINT received. Shutting down the pipeline. {:level=>:warn}
Assertion failed: ok (mailbox.cpp:82)
[1]    5760 abort (core dumped)  bin/logstash -f zmq-client.conf

migrated from elastic/logstash#1320

[felipegs]

We have this error here too

[felipegs]

Hi guys, any expectations to fix this issue?

[jsvd]

Not sure how to go about fixing this, the library this plugin uses, ffi-rzmq, is up to date, so this needs to be fixed upstream, I think. Maybe we can experiment with a different library, wrapping https://github.com/zeromq/jeromq?

[achimnol]

I have a similar issue.
I'm getting an infinite stream of

{:timestamp=>"2015-08-24T12:38:18.221000+0000", :message=>"ZeroMQ error while in recv_string", :error_code=>-1, :level=>:error}
{:timestamp=>"2015-08-24T12:38:18.222000+0000", :message=>"ZeroMQ error while in recv_string", :error_code=>-1, :level=>:error}
{:timestamp=>"2015-08-24T12:38:18.222000+0000", :message=>"ZeroMQ error while in recv_string", :error_code=>-1, :level=>:error}
{:timestamp=>"2015-08-24T12:38:18.223000+0000", :message=>"ZeroMQ error while in recv_string", :error_code=>-1, :level=>:error}
{:timestamp=>"2015-08-24T12:38:18.224000+0000", :message=>"ZeroMQ error while in recv_string", :error_code=>-1, :level=>:error}
{:timestamp=>"2015-08-24T12:38:18.224000+0000", :message=>"ZeroMQ error while in recv_string", :error_code=>-1, :level=>:error}
{:timestamp=>"2015-08-24T12:38:18.225000+0000", :message=>"ZeroMQ error while in recv_string", :error_code=>-1, :level=>:error}
...

where my configuration is

input {
  zeromq {
    topology => "pushpull"
    address => ["tcp://*:2120"]
    sockopt => ["ZMQ::RCVTIMEO", 1000, "ZMQ::RCVHWM", 50]
  }
}

with no pushers connected.

I'm using logstash 1.5.4 inside a docker container and apt-get-installed libzmq3-dev (ZeroMQ 4.0.5) on the vanilla image which bases on Debian Jessie. The above output stream is retrieved using docker logs -f <container-name>.

I'd like to check out the errno set by the underlying libzmq using ZMQ::Util.errno, but do not know how to modify the gem source codes on a running system because I am new to Ruby/JRuby. I tried to change /opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-input-zeromq-1.0.0/lib/logstash/util/zeromq.rb and restart the container, but it seems to have no effect. :(
I want to know if this is a configuration issue or an internal bug that could be fixed soon.

[achimnol]

Oh, I found the solution for my case.
It was actually due to misconfigured output plugin.

• I'm using a single output plugin: s3.
• I gave it a set of IAM access key and secret to access my s3 bucket, and allowed "PutObject" permission only. (I didn't know exactly what permissions it requires, so tried the minimal configuration. I see no mention on the documentation--it says only "write" permission--and there was no warning/errors of logstash on the default setup.)
• After some debugging with --debug option, I noticed that the s3 output plugin tries to put and delete a temporary object (e.g., logstash-programmatic-access-test-object-1440423290) to test its configuration during logstash startup.
• It failed to delete the temporary object, and logstash starts with no output plugin.
• Then logstash somehow _ignores_ the receive timeout of the zeromq input plugin. (!)

After adding "DeleteObject" permission to the s3 bucket, my logstash began to work as expected.

[achimnol]

So, my suggested fix/improvements are:

• Add an explicit permission list to the s3 plugin documentation.
• Let the zeromq plugin to explicitly report the errno when it encounters errors and before it retries the loop execution.
• Fix logstash to respect the timeout of the zeromq plugin when it has no output plugin (or failed output plugin), or to explicitly warn or raise an error to the user when it has configuration errors.