Reliable queue delivery: Redis, Kafka, ... ?

[mostolog]

Hi

In our ELK deployment we're looking for a 0.00% event loss.
Actually we are running a rsyslog using RELP to send logs to a Redis queue/list, readed from Logstash which inserts into Elasticsearch.

Are there any plans to implement Redis RPOPLPUSH & LREM for reliable queue delivery?

Are the efforts regarding reliability placed on kafka plugin (as a reliable platform) instead?

As seems elastic was (is?) working for a reliable delivery elastic/logstash#3693, should we consider other approach?

Regards

[jordansissel]

Can you narrow the focus of this issue to just redis since it is filed in
the redis input plugin?

On Wednesday, September 7, 2016, mostolog notifications@github.com wrote:

Hi

In our ELK deployment we're looking for a 0.00% event loss.
Actually we are running a rsyslog using RELP to send logs to a Redis
queue/list, readed from Logstash which inserts into Elasticsearch.

Are there any plans to implement Redis RPOPLPUSH
http://redis.io/commands/rpoplpush & LREM
http://redis.io/commands/lrem for reliable queue delivery?

Are the efforts regarding reliability placed on kafka plugin (as a
reliable platform) instead?

As seems elastic was (is?) working for a reliable delivery
elastic/logstash#3693 elastic/logstash#3693,
should we consider other approach?

Regards

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#51, or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAIC6k1gQZAuyY32uHjCuPj8VWMS_WrXks5qnrligaJpZM4J26DZ
.

[mostolog]

I was dealing with Redis, RPOPLPUSH & LREM are Redis commands, so I supposed this was the best place.

As you suggest, reliability isn't a Redis issue only, so probably I should reference this at elastic/logstash#5636 and elastic/logstash#5283

Anyway, this request it's to implement those commands, which will provide REDIS to LOGSTASH reliability. For the sake of truth, LREM could be a https://github.com/logstash-plugins/logstash-output-redis issue.