Support fluent-bit also in addition/replacement for fluentd

[rverma-nikiai]

Is your feature request related to a problem? Please describe.
Fluentd is still a resource hoggy service, also multiple elk/splunk etc consumers prefers fluent-bit.

Describe the solution you'd like
We should support fluent-bit also, there is some initial work as well https://github.com/cosmo0920/fluent-bit-go-loki
Additionally, fluent-bit native support for kubernetes allows gathering all labels/annotations by default.

Can we manage this officially at https://github.com/fluent/fluent-bit/tree/master/plugins

[cyriltovena]

A big yes !

[cyriltovena]

@cosmo0920 would you be interested into merging this into our repository ?

[cosmo0920]

Yeah. I'm interested in merging my fluent-bit go loki plugin here.

But, I have questions to start to work:

• How do I manage fluent-bit-go-loki plugin's library dependencies?
• I want to continue to improve fluent-bit-go-loki plugin (most of my hobby work, not Fluentd/fluent-bit contributor work). How should I support to improvement here?
• Need to rectify loki's main Makefile for this work?

[cyriltovena]

I think the ultimate goal will be to have an official plugin in the fluentbit repository. But let's fist incubate a version here. I'll get in touch with the fluentbit team see how that would work. I'm not sure if the a go version can become official. Looks like most officials one are written in C.

How do I manage fluent-bit-go-loki plugin's library dependencies?

Right now we use dep. You can create a new cmd/fluent-bit folder.

How should I support to improvement here?

You would still be able to contribute here, though you would need review first by us. However I'm working on a governance document and you could become a member of this Repository. You would still need review that how we manage this repo. Basically decisions will be taken together as a team.

Need to rectify loki's main Makefile for this work?

Yes and we're here to help you for this. I think you should not worry too much about this for now, let's start step by step. I think it would be easier to contribute one small PR at the time:

• Fluentbit plugin code
• documentation
• Makefile & release.

[cosmo0920]

I'll get in touch with the fluentbit team see how that would work. I'm not sure if the a go version can become official.

AFAIK, it must be written in C to become official fluent-bit plugin.
There is no exception for this.

[cyriltovena]

Fair enough.

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.

[cyriltovena]

Fixed by #847