S3 input: try to detect GZIPped objects #18764
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Pinging @elastic/integrations-platforms (Team:Platforms) |
💚 Build Succeeded
Expand to view the summary
Build stats
Test stats 🧪
|
jsoriano
reviewed
May 28, 2020
jsoriano
approved these changes
May 28, 2020
This was referenced Jun 3, 2020
ycombinator
added a commit
that referenced
this pull request
Jun 3, 2020
ycombinator
added a commit
that referenced
this pull request
Aug 4, 2020
* S3 input: try to detect GZIPped objects (#18764) * Try to detect GZIP based on content encoding header * Check GZIP contents * Log error before returning it * Fixing typo * Add comment * Adding comment * Adding CHANGELOG entry * Add test case for empty contents * Cleaning up CHANGELOG
melchiormoulin
pushed a commit
to melchiormoulin/beats
that referenced
this pull request
Oct 14, 2020
* Try to detect GZIP based on content encoding header * Check GZIP contents * Log error before returning it * Fixing typo * Add comment * Adding comment * Adding CHANGELOG entry * Add test case for empty contents
leweafan
pushed a commit
to leweafan/beats
that referenced
this pull request
Apr 28, 2023
…c#18944) * S3 input: try to detect GZIPped objects (elastic#18764) * Try to detect GZIP based on content encoding header * Check GZIP contents * Log error before returning it * Fixing typo * Add comment * Adding comment * Adding CHANGELOG entry * Add test case for empty contents * Cleaning up CHANGELOG
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What does this PR do?
This PR enhances the S3 input to try harder to automatically detect GZIPped objects.
Why is it important?
Sometimes objects returned by the S3 API are gzipped but don't come with proper headers indicating their gzipped nature. So we try different ways to auto-detect if the object is gzipped.
Checklist
I have made corresponding changes to the documentationI have made corresponding change to the default configuration filesCHANGELOG.next.asciidocorCHANGELOG-developer.next.asciidoc.How to test this PR locally
Follow the instructions in https://www.elastic.co/blog/getting-aws-logs-from-s3-using-filebeat-and-the-elastic-stack to setup an S3 bucket and SQS queue and configure Filebeat. When configuring Filebeat use the
consoleoutput for easy debugging.Create a sample log file.
Gzip the sample log file while keeping the original around.
Make a copy of the gzipped file, but remove the gzip extension to try and "fool" our S3 input.
Start Filebeat
One by one, upload the 3 files to your S3 bucket. After each upload, check the Filebeat log to make sure there are no errors. Also make sure that the S3 input successfully generates the expected events.
Related issues