-
Notifications
You must be signed in to change notification settings - Fork 3.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Loki: Change live tailing to only allow mutating the log line not the number of streams. #6063
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…e with the pipeline to allow mutating the content of the line but don't split it into new streams. Signed-off-by: Edward Welch <edward.welch@grafana.com>
Signed-off-by: Edward Welch <edward.welch@grafana.com>
owen-d
reviewed
May 1, 2022
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ooh, nice find. How'd you catch this?
Signed-off-by: Edward Welch <edward.welch@grafana.com>
owen-d
approved these changes
May 2, 2022
Signed-off-by: Edward Welch <edward.welch@grafana.com>
Signed-off-by: Edward Welch <edward.welch@grafana.com>
sandeepsukhani
approved these changes
May 2, 2022
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice one! LGTM!
This was referenced Jun 28, 2022
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What this PR does / why we need it:
Stops live tailing from splitting a stream into more streams when using parsers while still maintaining the ability to mutate the log line using parsers and
line_format
The current implementation will allow creating many new streams from a tailed stream when using parsers, but this multiplicative affect can make it difficult to then tail the stream because it creates a lot more overhead in the amount of data pushed down the websocket.
For the purposes of live tailing which is meant primarily for human consumption, returning the parsed and extracted labels and manipulating the streams isn't helpful for the end user visualization and instead makes the process less efficient and more likely to struggle to keep up with higher volume streams.
Update: Also included in this PR is a change to no longer reuse the same pipeline for parsing every push request, the pipeline has a label cache built into it which, when reused, would lead to unbounded memory growth when the log lines being processed had extremely high cardinality data being parsed.
Which issue(s) this PR fixes:
Fixes #
Special notes for your reviewer:
Checklist
CHANGELOG.md
.docs/sources/upgrading/_index.md