-
Notifications
You must be signed in to change notification settings - Fork 256
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support file system as storage layer in service section of fluenbit #825
Support file system as storage layer in service section of fluenbit #825
Conversation
@karan56625 Thanks for the pr. git commit -s --amend
git push -f |
@@ -25,6 +25,8 @@ spec: | |||
skipLongLines: {{ .Values.fluentbit.input.tail.skipLongLines }} | |||
db: /fluent-bit/tail/pos.db | |||
dbSync: Normal | |||
storageType: {{ .Values.fluentbit.input.tail.storageType }} | |||
pauseOnChunksOverlimit: {{ .Values.fluentbit.input.tail.pauseOnChunksOverlimit }} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
pauseOnChunksOverlimit
field is only functional when the storage type is fileSystem
right? If that's the case then maybe add a check for this field to see what the storage type is
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Although it will by default off and also no effect if it is enabled with fileSystem
as storage. Still I have added check on this. Its always better to have one check on higher level.
2712a73
to
0cb2240
Compare
fixed |
This is the case for statefulset only? Does it output logs in pvc? |
This is to configure the storage type of buffer that is getting used by fluent-bit pipeline. By default, buffer storage is memory. In this PR, we allow customer to customise that in fluent-bit operator to use file-system as buffer storage type instead of memory only. see: https://docs.fluentbit.io/manual/administration/buffering-and-storage |
@@ -80,6 +101,8 @@ type Service struct { | |||
LogLevel string `json:"logLevel,omitempty"` | |||
// Optional 'parsers' config file (can be multiple) | |||
ParsersFile string `json:"parsersFile,omitempty"` | |||
// Configure a global environment for the storage layer in Service |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@karan56625 I think we should add a note here to tell users it's better to set up volume and volumemount separately for this storage, it should be hostpath type for fluentbit daemonset. we shouldn't write chuncks directly to containers. And the volume & volumemount can be added like https://github.com/fluent/fluent-operator/blob/master/charts/fluent-operator/values.yaml#L129
We'd better add file buffer config to https://github.com/fluent/fluent-operator/blob/master/charts/fluent-operator/values.yaml#L129 too
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@benjaminhuo , I have added a note in that comment. Also one example is put in the values.yaml. Let me know if any change is required in those.
charts/fluent-operator/values.yaml
Outdated
# Uncomment the code if you intend to create the volume for buffer storage in case the storage type "filesystem" is being used in the configuration of the fluentbit service. | ||
# - name: hostBuffer | ||
# hostPath: | ||
# path: /tmp/ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
maybe like below?
# path: /tmp/ | |
# path: /tmp/fluent-bit-buffer |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed.
Signed-off-by: karan k <karan.k@oracle.com>
95a000b
to
88ab30b
Compare
@karan56625 Thank you! |
…em_as_storage Support file system as storage layer in service section of fluenbit
…em_as_storage (#14) Support file system as storage layer in service section of fluenbit Co-authored-by: Benjamin Huo <benjamin@kubesphere.io>
@karan56625 Have add you as the fluent operator maintainer, thanks for your contribution! @wanjunlei @wenchajun @patrick-stephens |
What this PR does / why we need it:
Which issue(s) this PR fixes:
Fixes #826
Does this PR introduced a user-facing change?
Additional documentation, usage docs, etc.: