Add multiline support to awss3 input #25710

leehinman · 2021-05-13T19:55:42Z

What does this PR do?

Adds multiline and encoding reader support to aws-s3 input. This does
not change the processing of JSON logs by aws-s3 input.

Why is it important?

This is needed so you can read logs that have embedded new lines. For
example XML Windows events.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Need S3 bucket & SQS queue, then upload multiline logs.

Related issues

Closes [Filebeat] Support multiline options in aws s3 input #25249

elasticmachine · 2021-05-13T19:56:33Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

mergify · 2021-05-13T19:59:19Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 25249_awss3_input_multiline2 upstream/25249_awss3_input_multiline2
git merge upstream/master
git push upstream 25249_awss3_input_multiline2

elasticmachine · 2021-05-13T20:12:35Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #25710 updated
Start Time: 2021-05-17T15:13:31.892+0000
Duration: 115 min 39 sec
Commit: 8a3b408

Test stats 🧪

Test	Results
Failed	0
Passed	7031
Skipped	1193
Total	8224

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	7031
Skipped	1193
Total	8224

andrewkroh

Awesome 😄 . I took a very quick pass at it. Plan to look a little deeper and try it out tomorrow.

x-pack/filebeat/input/awss3/collector.go

andrewkroh · 2021-05-13T20:37:13Z

x-pack/filebeat/input/awss3/collector.go

-		offset += int64(len(log))
-		event := createEvent(log, offset, info, objectHash, s3Ctx)
+		event := createEvent(string(message.Content), offset, info, objectHash, s3Ctx)
+		offset += int64(message.Bytes)
 		err = c.forwardEvent(event)
 		if err != nil {


Can you combine this into a one-liner, e.g. if err = c.forwardEvent(event); err != nil {.

x-pack/filebeat/input/awss3/collector.go

andrewkroh · 2021-05-13T20:39:21Z

x-pack/filebeat/input/awss3/config.go

+	VisibilityTimeout        time.Duration           `config:"visibility_timeout"`
+	AwsConfig                awscommon.ConfigAWS     `config:",inline"`
+	MaxBytes                 int                     `config:"max_bytes" validate:"min=0,nonzero"`
+	Multiline                *multiline.Config       `config:"multiline"`


Do you think these options should be available in the FileSelectorCfg?

Hmmmm we could...

The use case would be a single S3 bucket that has a mix of multiline and non-multiline log files.

I'll add it, and we can see if we like it.

The parser options could be combined into a struct that is embedded in both config and FileSelectorCfg to avoid having to duplicate the same config struct tags.

I'm thinking of changing this so you always have a file_selector, the default is just to match any filename. That should make the config and code paths cleaner. I know we want to get a build out with multiline "soon". You OK with merging as is, with a new PR this week to clean this up?

x-pack/filebeat/docs/inputs/input-aws-s3.asciidoc

marc-gr

Do you think could be worth adding a new test with the multiline config in s3_integration_test.go?

mergify · 2021-05-14T15:22:06Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 25249_awss3_input_multiline2 upstream/25249_awss3_input_multiline2
git merge upstream/master
git push upstream 25249_awss3_input_multiline2

leehinman · 2021-05-14T20:15:27Z

Do you think could be worth adding a new test with the multiline config in s3_integration_test.go?

added it. running it is a little weird you have to setup S3 bucket & SQS ahead of time, then upload the 2 sample files, then run go test --tags=integration,aws

Long term I'd like to improve this.

- only applies to non JSON logstash Closes elastic#25249

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

marc-gr · 2021-05-17T12:12:27Z

I think some files need to be formatted

marc-gr

LGTM once the formatting is fixed

* Add multiline support to awss3 input - only applies to non JSON logs Closes #25249 Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co> (cherry picked from commit 5f242e3)

kaiyan-sheng

LGTM! Thank you for adding this feature!! Are you planning to add this into the AWS package later? I'm adding the label just to make sure we don't forget about it :)

* Add multiline support to awss3 input - only applies to non JSON logs Closes #25249 Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co> (cherry picked from commit 5f242e3)

* Add multiline support to awss3 input - only applies to non JSON logs Closes #25249 Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co> (cherry picked from commit 5f242e3) Co-authored-by: Lee Hinman <57081003+leehinman@users.noreply.github.com>

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label May 13, 2021

leehinman added backport-v7.14.0 Automated backport with mergify enhancement Team:Security-External Integrations labels May 13, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label May 13, 2021

leehinman requested review from kaiyan-sheng and marc-gr May 13, 2021 19:56

leehinman force-pushed the 25249_awss3_input_multiline2 branch from 5b1a550 to 356cb8c Compare May 13, 2021 19:59

andrewkroh reviewed May 13, 2021

View reviewed changes

marc-gr reviewed May 14, 2021

View reviewed changes

leehinman closed this May 14, 2021

leehinman deleted the 25249_awss3_input_multiline2 branch May 14, 2021 14:48

leehinman reopened this May 14, 2021

leehinman restored the 25249_awss3_input_multiline2 branch May 14, 2021 14:51

leehinman and others added 6 commits May 14, 2021 15:16

Add multiline support to awss3 input

e2cf26d

- only applies to non JSON logstash Closes elastic#25249

Update x-pack/filebeat/input/awss3/collector.go

d90b4b7

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

Update x-pack/filebeat/input/awss3/collector.go

8558f49

Co-authored-by: Andrew Kroh <andrew.kroh@elastic.co>

Add feedback

423cf7d

allow multiline to be turned on/off per file selector

85c7985

Add multiline test

da10563

leehinman force-pushed the 25249_awss3_input_multiline2 branch from 52120e1 to da10563 Compare May 14, 2021 20:16

marc-gr approved these changes May 17, 2021

View reviewed changes

update docs and fmt code

8a3b408

andrewkroh approved these changes May 17, 2021

View reviewed changes

leehinman merged commit 5f242e3 into elastic:master May 17, 2021

mergify bot mentioned this pull request May 17, 2021

[7.x](backport #25710) Add multiline support to awss3 input #25740

Merged

kaiyan-sheng approved these changes May 17, 2021

View reviewed changes

kaiyan-sheng added the needs_integration_sync Changes in this PR need synced to elastic/integrations. label May 17, 2021

andrewkroh mentioned this pull request May 18, 2021

[Filebeat] Fix aws-s3 config initialization plus cleanup #25763

Merged

4 tasks

andrewkroh mentioned this pull request Jun 23, 2021

[Filebeat] Support multiline options in aws s3 input #25249

Closed

leehinman deleted the 25249_awss3_input_multiline2 branch August 17, 2021 21:14

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add multiline support to awss3 input #25710

Add multiline support to awss3 input #25710

leehinman commented May 13, 2021 •

edited

Loading

elasticmachine commented May 13, 2021

mergify bot commented May 13, 2021

elasticmachine commented May 13, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

Trends 🧪

Test stats 🧪

andrewkroh left a comment

andrewkroh May 13, 2021

andrewkroh May 13, 2021

leehinman May 13, 2021

andrewkroh May 17, 2021 •

edited

Loading

leehinman May 17, 2021

marc-gr left a comment

mergify bot commented May 14, 2021

leehinman commented May 14, 2021

marc-gr commented May 17, 2021

marc-gr left a comment

kaiyan-sheng left a comment •

edited

Loading

Add multiline support to awss3 input #25710

Add multiline support to awss3 input #25710

Conversation

leehinman commented May 13, 2021 • edited Loading

What does this PR do?

Why is it important?

Checklist

How to test this PR locally

Related issues

elasticmachine commented May 13, 2021

mergify bot commented May 13, 2021

elasticmachine commented May 13, 2021 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

andrewkroh left a comment

Choose a reason for hiding this comment

andrewkroh May 13, 2021

Choose a reason for hiding this comment

andrewkroh May 13, 2021

Choose a reason for hiding this comment

leehinman May 13, 2021

Choose a reason for hiding this comment

andrewkroh May 17, 2021 • edited Loading

Choose a reason for hiding this comment

leehinman May 17, 2021

Choose a reason for hiding this comment

marc-gr left a comment

Choose a reason for hiding this comment

mergify bot commented May 14, 2021

leehinman commented May 14, 2021

marc-gr commented May 17, 2021

marc-gr left a comment

Choose a reason for hiding this comment

kaiyan-sheng left a comment • edited Loading

Choose a reason for hiding this comment

leehinman commented May 13, 2021 •

edited

Loading

elasticmachine commented May 13, 2021 •

edited by jenkins-beats-ci bot

Loading

andrewkroh May 17, 2021 •

edited

Loading

kaiyan-sheng left a comment •

edited

Loading