You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This is an issue to track the design process for adding s3 Filebeat input. Different logs from different services can be stored in S3. For example:
S3 server access logs: provides detailed records for the requests that are made to a bucket.
VPC flow logs: records for all of the monitored network interfaces are published to a series of log file objects that are stored in the bucket.
ELB access logs: capture detailed information about requests sent to the load balancer. Each log contains information such as the time the request was received, the client's IP address, latencies, request paths, and server responses.
Cloudwatch logs: users can choose to export all data from an amazon cloudwatch log group to a specific s3 bucket.
With all the different logs in S3 from different services, it will be good to have a dedicated Filebeat input to retrieve raw lines from S3 objects. Then under AWS module in Filebeat, we can add supports for different services with proper grok patterns in ingest pipelines.
This is an issue to track the design process for adding s3 Filebeat input. Different logs from different services can be stored in S3. For example:
S3 server access logs: provides detailed records for the requests that are made to a bucket.
VPC flow logs: records for all of the monitored network interfaces are published to a series of log file objects that are stored in the bucket.
ELB access logs: capture detailed information about requests sent to the load balancer. Each log contains information such as the time the request was received, the client's IP address, latencies, request paths, and server responses.
Cloudwatch logs: users can choose to export all data from an amazon cloudwatch log group to a specific s3 bucket.
With all the different logs in S3 from different services, it will be good to have a dedicated Filebeat input to retrieve raw lines from S3 objects. Then under AWS module in Filebeat, we can add supports for different services with proper grok patterns in ingest pipelines.
Please see more information in mini-design documentation:
https://docs.google.com/document/d/1lbma_7nBLRwVy2ioLs-bbEzQV2Nz0fRUB9ot535NDzA/edit
Initial PR to add s3 input: Add S3 input to retrieve logs from AWS S3 buckets #12640
Add more tests [Filebeat] Add more tests for s3 input #13128
The text was updated successfully, but these errors were encountered: