Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[pkg/stanza] improve performance #16027

Merged
merged 5 commits into from
Nov 3, 2022

Conversation

newly12
Copy link
Contributor

@newly12 newly12 commented Nov 2, 2022

Description:

The idea is to avoid making []byte every time to improve performance. In fact we had previously done benchmark against v0.52.0 and recently v0.60.0 and noticed the performance regression, give it one CPU and 40 files to be ingested, emitted logs/s reduced from ~254k/s to ~140k/s. most of cpu time is used by decoding and underlying runtime.makeslice, the new implementation is similar to v0.52.0.

Link to tracking Issue:
#16028

Testing:

Documentation:

$ benchstat old.txt new.txt
name                        old time/op  new time/op  delta
FileInput/Single-10         1.30µs ±13%  1.05µs ±10%  -19.45%  (p=0.008 n=5+5)
FileInput/Glob-10           6.32µs ±12%  4.25µs ± 8%  -32.73%  (p=0.008 n=5+5)
FileInput/MultiGlob-10      5.98µs ± 3%  4.17µs ± 9%  -30.29%  (p=0.008 n=5+5)
FileInput/MaxConcurrent-10  5.42µs ± 4%  4.26µs ±15%  -21.54%  (p=0.008 n=5+5)
FileInput/FngrPrntLarge-10  1.21µs ± 1%  1.06µs ±12%  -12.52%  (p=0.008 n=5+5)
FileInput/FngrPrntSmall-10  1.34µs ± 7%  1.00µs ± 6%  -25.32%  (p=0.008 n=5+5)

image

@newly12 newly12 requested a review from a team November 2, 2022 06:04
@newly12 newly12 requested a review from djaglowski as a code owner November 2, 2022 06:04
Copy link
Member

@djaglowski djaglowski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great find, @newly12. Thanks for the improvement.

pkg/stanza/operator/helper/encoding.go Outdated Show resolved Hide resolved
Copy link
Member

@djaglowski djaglowski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @newly12

@djaglowski djaglowski merged commit b8ed96c into open-telemetry:main Nov 3, 2022
@newly12 newly12 deleted the filelog_file_perf branch November 4, 2022 03:07
dineshg13 pushed a commit to DataDog/opentelemetry-collector-contrib that referenced this pull request Nov 21, 2022
shalper2 pushed a commit to shalper2/opentelemetry-collector-contrib that referenced this pull request Dec 6, 2022
@plantfansam plantfansam mentioned this pull request Jul 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants