Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Try reverting "compress on upload" #34536

Closed
adeebshihadeh opened this issue Feb 6, 2025 · 2 comments · Fixed by #34549
Closed

Try reverting "compress on upload" #34536

adeebshihadeh opened this issue Feb 6, 2025 · 2 comments · Fixed by #34549

Comments

@adeebshihadeh
Copy link
Contributor

#34528 broke compressed logs. It wasn't caught in CI since we use what's on disk in CI, which isn't what gets uploaded. We can add tests, but I think we should also investigate moving the compression back to loggerd.

IIRC we did that for two reason:

  1. CPU usage
  2. Getting as much as possible in the case of sudden power loss

#1 should be much better with zstd, and #2 is better defined now.

@adeebshihadeh
Copy link
Contributor Author

@deanlee want to try this out?

@deanlee
Copy link
Contributor

deanlee commented Feb 10, 2025

#34528 broke compressed logs

@adeebshihadeh The issue occurred because the original LogReader was using zstd.decompress(data) and was unable to handle streaming zstd-compressed logs. This has been fixed in in PR #34549.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants