Increase Fluentd Buffer Queue Size #1877
Conversation
This increases the number of chunks that can be queued to be sent to S3. The [documentation][1] claims that this number is unlimited when not set, but the default was in fact [recently set][2] to `1`, which causes a backlog of chunks to build up when there is a larger number of log files. [1]: https://docs.fluentd.org/v1.0/articles/buffer-section#buffering-parameters [2]: fluent/fluentd#2173 Signed-off-by: Christian Berg <berg.christian@gmail.com>
3c59f2c to
cd297c1
Compare
|
👍 |
1 similar comment
|
👍 |
|
How do we know that 100 is a good number? |
|
I admit that 100 is somewhat arbitrary. After looking into the Fluentd source code and running a few tests (which I'm documenting in more detail and will link here when done), I believe it is safe to set this relatively high. Both the buffered chunks and the queued chunks are stored on disk, and the only difference between the two (that I can see), is that the former have a This means that no additional resources are consumed by putting more chunks into the queue. But chunks are only moved into the queue at regular intervals (~ every 5 seconds with our time-slicing settings, if I followed the calculations correctly), and if the queue is too small (i.e. the default length of 1), the S3 upload thread is starved and a backlog builds up. 100 was sufficient in my tests to not starve the thread, but a lower number would probably also be fine. When the log throughput gets high enough for the queue to fill up with 100 chunks, other factors are actually more limiting (specifically CPU consumption of Fluentd) and would need to be addressed first. 100,000 events/minute can be handled with the current settings. |
|
👍 |
This increases the number of chunks that can be queued to be sent to
S3. The documentation claims that this number is unlimited when
not set, but the default was in fact recently set to
1, whichcauses a backlog of chunks to build up when there is a larger number
of log files.