[libbeat] New decode xml wineventlog processor #25115

marc-gr · 2021-04-15T16:06:38Z

What does this PR do?

Draft of the changes required to move wineventlog decoding to a new processor

From the discussion at #25109 to see which approach we prefer.

Why is it important?

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

elasticmachine · 2021-04-15T18:03:59Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #25115 updated
Start Time: 2021-04-20T07:26:49.899+0000
Duration: 60 min 33 sec
Commit: cfcb4f1

Test stats 🧪

Test	Results
Failed	0
Passed	47145
Skipped	5134
Total	52279

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	47145
Skipped	5134
Total	52279

leehinman

still testing but found 2 things

libbeat/processors/decode_xml_wineventlog/processor.go

elasticmachine · 2021-04-19T10:33:04Z

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

leehinman

LGTM. Thank You.

mergify · 2021-04-19T22:40:35Z

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b new_decode_xml_wineventlog_processor upstream/new_decode_xml_wineventlog_processor
git merge upstream/master
git push upstream new_decode_xml_wineventlog_processor

andrewkroh · 2021-04-19T23:12:56Z

libbeat/processors/decode_xml_wineventlog/docs/decode_xml_wineventlog.asciidoc

+exist in the event are overwritten by keys from the decoded XML object. The
+default value is `true`.
+
+`map_ecs_fields`:: (Optional) A boolean that specifies whether to map additional ECS


I think it would be more clear to explicitly state that this writes keys outside of target_field.

andrewkroh · 2021-04-19T23:15:26Z

libbeat/processors/decode_xml_wineventlog/docs/decode_xml_wineventlog.asciidoc

+`map_ecs_fields`:: (Optional) A boolean that specifies whether to map additional ECS
+fields when possible. The default value is `true`.
+
+`document_id`:: (Optional) XML key to use as the document ID. If configured, the


I don't think this will be useful for event log XML data. I can't think of a field that has enough uniqueness on its own. I'd would probably use a fingerprint processor to combine a few fields like record_id, channel, computer_name, timestamp if I wanted a unique _id.

…-github-pr-comment-template * upstream/master: [Ingest Manager] Keep http and logging config during enroll (elastic#25132) Refactor kubernetes autodiscover to avoid skipping short-living pods (elastic#24742) [libbeat] New decode xml wineventlog processor (elastic#25115) Add svc to agent k8s clusterRole (elastic#25146) Add awsfargate module to collect container logs from Amazon ECS on Fargate (elastic#25041) [Filebeat][Cisco ASA] log enhancement and performance (elastic#24744) Watch kubernetes namespaces for autodiscover metadata for pods (elastic#25117) Cyberark Privileged Access Security module (elastic#24803) [Elastic Agent] Log the container command output with LOGS_PATH (elastic#25150) Fix for tests after `device...` field has been removed (elastic#25141) [Ingest Manager] Restart process on output change (elastic#24907) Set --insecure in container when FLEET_SERVER_ENABLE and FLEET_INSECURE set. (elastic#25137) [filebeat] Update documentation / changelog / beta warnings for the syslog input (elastic#25047) Add support for ignore_inactive in filestream input (elastic#25036) Fix bug with annotations dedot config on k8s not used (elastic#25111)

* Move enrich raw functionality to common package * Enrich Raw fields when possible in decode_xml * Add ECS mappings when decoding wineventlog xml * Add decode_xml_wineventlog processor * Add missing fields to config checks * Change event.code type * Fix PR number in changelog * Fix test * Remove document_id and make docs more clear (cherry picked from commit 8cf8f51)

* Move enrich raw functionality to common package * Enrich Raw fields when possible in decode_xml * Add ECS mappings when decoding wineventlog xml * Add decode_xml_wineventlog processor * Add missing fields to config checks * Change event.code type * Fix PR number in changelog * Fix test * Remove document_id and make docs more clear (cherry picked from commit 8cf8f51) Co-authored-by: Marc Guasch <marc-gr@users.noreply.github.com>

…ng-versions-stack * upstream/master: (28 commits) Add support for parsers in filestream input (elastic#24763) Skip flaky test TestFilestreamTruncate (elastic#25218) backport: Add 7.13 branch (elastic#25189) Update decode_json_fields.asciidoc (elastic#25056) [Elastic Agent] Fix status and inspect command to work inside running container (elastic#25204) Check native environment before starting (elastic#25186) Change event.code and winlog.event_id type (elastic#25176) [Ingest Manager] Proxy processes/elastic-agent to stats (elastic#25193) Update mergify backporting to 7.x and 7.13 (elastic#25196) [Heartbeat]: ensure synthetics version co* [Heartbeat]: ensure synthetics version compatability for suites * address review and fix notice * fix lowercase struct * fix version conflict and rebase * update go.* stuff to master * fix notice.txt * move validate inside sourcempatability for suites (elastic#24777) [Filebeat] Ensure Kibana audit `event.category` and `event.type` are still processed as strings. (elastic#25101) Update replace.asciidoc (elastic#25055) Fix nil panic when overwriting metadata (elastic#24741) [Filebeat] Add Malware Bazaar to Threat Intel Module (elastic#24570) Fix k8s svc selectors mapping (elastic#25169) [Ingest Manager] Make agent retry values for bootstraping configurable (elastic#25163) [Metricbeat] Remove elasticsearc.index.created from the SM code (elastic#25113) [Ingest Manager] Keep http and logging config during enroll (elastic#25132) Refactor kubernetes autodiscover to avoid skipping short-living pods (elastic#24742) [libbeat] New decode xml wineventlog processor (elastic#25115) ...

marc-gr added enhancement Team:Security-External Integrations labels Apr 15, 2021

marc-gr requested a review from leehinman April 15, 2021 16:06

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Apr 15, 2021

leehinman requested changes Apr 15, 2021

View reviewed changes

libbeat/processors/decode_xml_wineventlog/processor.go Show resolved Hide resolved

libbeat/processors/decode_xml_wineventlog/processor.go Outdated Show resolved Hide resolved

marc-gr marked this pull request as ready for review April 19, 2021 10:33

marc-gr force-pushed the new_decode_xml_wineventlog_processor branch from c739801 to 3f8b423 Compare April 19, 2021 15:56

marc-gr mentioned this pull request Apr 19, 2021

[libbeat] Decode xml enhance wineventlog #25109

Closed

5 tasks

marc-gr requested a review from leehinman April 19, 2021 15:58

leehinman approved these changes Apr 19, 2021

View reviewed changes

marc-gr added the backport-v7.13.0 Automated backport with mergify label Apr 19, 2021

andrewkroh approved these changes Apr 19, 2021

View reviewed changes

marc-gr force-pushed the new_decode_xml_wineventlog_processor branch from ab53093 to 7eafa15 Compare April 20, 2021 06:13

marc-gr added 9 commits April 20, 2021 09:25

Move enrich raw functionality to common package

ea2a84f

Enrich Raw fields when possible in decode_xml

9987efe

Add ECS mappings when decoding wineventlog xml

16155f3

Add decode_xml_wineventlog processor

0c2fca8

Add missing fields to config checks

61aac68

Change event.code type

38c6a32

Fix PR number in changelog

d6ec2fd

Fix test

50209e1

Remove document_id and make docs more clear

cfcb4f1

marc-gr force-pushed the new_decode_xml_wineventlog_processor branch from 3fefb41 to cfcb4f1 Compare April 20, 2021 07:25

adriansr approved these changes Apr 20, 2021

View reviewed changes

marc-gr merged commit 8cf8f51 into elastic:master Apr 20, 2021

marc-gr deleted the new_decode_xml_wineventlog_processor branch April 20, 2021 08:31

mergify bot mentioned this pull request Apr 20, 2021

[libbeat] New decode xml wineventlog processor (backport #25115) #25165

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[libbeat] New decode xml wineventlog processor #25115

[libbeat] New decode xml wineventlog processor #25115

marc-gr commented Apr 15, 2021

elasticmachine commented Apr 15, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

Trends 🧪

Test stats 🧪

leehinman left a comment

elasticmachine commented Apr 19, 2021

leehinman left a comment

mergify bot commented Apr 19, 2021

andrewkroh Apr 19, 2021

andrewkroh Apr 19, 2021

[libbeat] New decode xml wineventlog processor #25115

[libbeat] New decode xml wineventlog processor #25115

Conversation

marc-gr commented Apr 15, 2021

What does this PR do?

Why is it important?

Checklist

elasticmachine commented Apr 15, 2021 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

Trends 🧪

💚 Flaky test report

Test stats 🧪

leehinman left a comment

Choose a reason for hiding this comment

elasticmachine commented Apr 19, 2021

leehinman left a comment

Choose a reason for hiding this comment

mergify bot commented Apr 19, 2021

andrewkroh Apr 19, 2021

Choose a reason for hiding this comment

andrewkroh Apr 19, 2021

Choose a reason for hiding this comment

elasticmachine commented Apr 15, 2021 •

edited by jenkins-beats-ci bot

Loading