Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add S3 Support in Filebeat #12582

Closed
1 of 2 tasks
kaiyan-sheng opened this issue Jun 17, 2019 · 1 comment
Closed
1 of 2 tasks

Add S3 Support in Filebeat #12582

kaiyan-sheng opened this issue Jun 17, 2019 · 1 comment
Assignees
Labels
enhancement Filebeat Filebeat Team:Integrations Label for the Integrations team v7.4.0

Comments

@kaiyan-sheng
Copy link
Contributor

kaiyan-sheng commented Jun 17, 2019

This is an issue to track the design process for adding s3 Filebeat input. Different logs from different services can be stored in S3. For example:

  • S3 server access logs: provides detailed records for the requests that are made to a bucket.

  • VPC flow logs: records for all of the monitored network interfaces are published to a series of log file objects that are stored in the bucket.

  • ELB access logs: capture detailed information about requests sent to the load balancer. Each log contains information such as the time the request was received, the client's IP address, latencies, request paths, and server responses.

  • Cloudwatch logs: users can choose to export all data from an amazon cloudwatch log group to a specific s3 bucket.

With all the different logs in S3 from different services, it will be good to have a dedicated Filebeat input to retrieve raw lines from S3 objects. Then under AWS module in Filebeat, we can add supports for different services with proper grok patterns in ingest pipelines.

Please see more information in mini-design documentation:
https://docs.google.com/document/d/1lbma_7nBLRwVy2ioLs-bbEzQV2Nz0fRUB9ot535NDzA/edit

@kaiyan-sheng
Copy link
Contributor Author

Closing this issue since s3 filebeat input and s3access fileset is merged for 7.4.0. Will keep #13128 open till all tests are properly added.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Filebeat Filebeat Team:Integrations Label for the Integrations team v7.4.0
Projects
None yet
Development

No branches or pull requests

2 participants