Feature Request: Add zlib compression support

[zackwine]

Currently we use fluentd to send compressed logs from on-prem to kinesis. The compression provides quite bit of cost savings in network traffic.

Are there any plans to add zlib compression support?

Reference to the fluentd support we are currently attempting to replace:
https://github.com/awslabs/aws-fluent-plugin-kinesis#compression

[hossain-rayhan]

Hi @zackwine Thanks for creating the issue. To me, it seems like fluent-bit also supports compression using gzip (right now only for out_http plugin). I will update after further study.

[bradfordlittooysonos]

+1 To this request. I'm evaluating fluentbit at the moment for shipping logs to kinesis and compression is a must for us due to size of our logs and cost implications of sending uncompressed data to kinesis. Thanks for maintaining this project!

[jonathanviber]

The advantage of using the AWS KPL format is that it is pretty efficient since it uses protobuf and it can easily be de-aggregated on the other side using the AWS KCL library. This is exactly why my company is not using Fluentbit. Without aggregation and compression (or at least a space efficient format), we will end up needing much more kinesis shards which will be too expensive.
Is there much to be gained from compressing protobuf?

[zackwine]

@jonathanviber We found zlib compression would further reduce bandwidth required to ship logs. This is mainly a concern for a specific use case where we have fairly large on-prem deployments shipping logs to AWS/Kinesis. In these cases the bandwidth usage to AWS is either limited or costly. I think your milage would vary depending on the types of logs/records shipped, so I don't have metrics on the percentage of bandwidth saved.

NOTE: We do not enable zlib when shipping logs from AWS resources to Kinesis given the added cost to compress/decompress the logs.

[jonathanviber]

Ok. I understand. Our primary use-case is shipping logs from EC2 and ECS to Kinesis. We tend to max out the 1000 write p/sec limit so we need to make sure that each write contains as much data as possible. This is our main reason for wanting to use the KPL aggregation format.

[PettitWesley]

@jonathanviber I agree KPL is a high priority... we have a lot to work with Fluent Bit, but I'm hopeful that we'll get to it sometime this year. I'm less certain about zlib.

[timesking]

I have a question when using compression, Do you decompress message in lambda before sending to firehose? especially when using kinesis_streams_aggregated, the standard KCL and firehose doesn't support zlib compressed message. so when do you usually decompress them?

[hossain-rayhan]

Hi @timesking , Kinesis Data Stream stores data as raw bytes. So, if you compress your data using some kind of compression tool (e.g. zlib), you need to write decompression logic yourself. Data won't be decompressed automatically. Now, where you want to write the decompression logic depends on specific use case.

[PettitWesley]

released