[Filebeat] ASA Ingest Pipeline fixes

[legoguy1000]

What does this PR do?

Modifies the cisco ASA FTD Ingest pipeline

• Adds community_id processor
• Fixes 805001 dissect to properly set network.transport
• Update 304001 grok pattern to account for source.user.* fields
• Update 106023 grok pattern for when network.iana_number is present instead of network.transport
• Update 602304 dissect to remove incorrect network.inner usage

Why is it important?

Grok/dissect patterns aren't correct.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have made corresponding change to the default configuration files
• I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

• [ ]

How to test this PR locally

cd beats/x-pack/filebeat
TESTING_FILEBEAT_MODULES=cisco mage -v pythonIntegTest

Related issues

• Closes Cisco ASA Ingress Fixes #26869
• Closes filebeat-<agent.version>-cisco-asa-asa-ftd-pipeline creates fields network.inner as string  #27137

Use cases

Screenshots

Logs

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2021-09-14T11:50:34.984+0000

• Duration: 103 min 9 sec

• Commit: 5f4c04b

Test stats 🧪

Test	Results
Failed	0
Passed	15314
Skipped	2314
Total	17628

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	15314
Skipped	2314
Total	17628

[marc-gr]

/test

[elasticmachine]

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

[trexx]

There are also a few message-ids where the various *.user.name fields include the surrounding brackets.
Message-id 302014 and some others are examples.

Some like 302013, 302020 don't include the brackets.

[legoguy1000]

There are also a few message-ids where the various *.user.name fields include the surrounding brackets.
Message-id 302014 and some others are examples.

Some like 302013, 302020 don't include the brackets.

There aren't any sample logs with those message IDs that include the optional username. Can you provide some so we can test that?

[trexx]

106023:
Built inbound TCP connection 195207391 for OUTSIDE:85.0.0.1/12312 (62.0.0.1/34534)(LOCAL\USER001) to OUTSIDE:81.0.0.1/443 (81.0.0.1/443) (USER001)
Built inbound TCP connection 195207391 for OUTSIDE:85.0.0.1/12312 (62.0.0.1/34534)(LOCAL\user@domain.tld) to OUTSIDE:81.0.0.1/443 (81.0.0.1/443) (user@domain.tld)

302020:
Built inbound ICMP connection for faddr 85.0.0.1/0(LOCAL\USER001) gaddr 81.0.0.1/0 laddr 81.0.0.1/0 (USER001) type 3 code 3
Built inbound ICMP connection for faddr 85.0.0.1/0(LOCAL\user@domain.tld) gaddr 81.0.0.1/0 laddr 81.0.0.1/0 (user@domain.tld) type 3 code 3

[trexx]

https://www.cisco.com/c/en/us/td/docs/security/asa/syslog/b_syslog/syslog-messages-302003-to-342008.html#con_4770598
https://www.cisco.com/c/en/us/td/docs/security/asa/syslog/b_syslog/syslogs3.html#con_4771041

Message-id 305012 is not included in the ingress pipeline, however I have noticed that a similar message-id 302012 is present.
However when I check the official documentation , it looks like 302012 is for a completely different message.

I can see 305012 message ids in the test logs but its not in the ingress pipeline.
https://github.com/elastic/beats/blob/v7.13.3/x-pack/filebeat/module/cisco/ftd/test/asa.log#L71

I think the pattern for 302012 in the ingress pipeline should be for 305012.

[legoguy1000]

https://www.cisco.com/c/en/us/td/docs/security/asa/syslog/b_syslog/syslog-messages-302003-to-342008.html#con_4770598
https://www.cisco.com/c/en/us/td/docs/security/asa/syslog/b_syslog/syslogs3.html#con_4771041

Message-id 305012 is not included in the ingress pipeline, however I have noticed that a similar message-id 302012 is present.
However when I check the official documentation , it looks like 302012 is for a completely different message.

I can see 305012 message ids in the test logs but its not in the ingress pipeline.
https://github.com/elastic/beats/blob/v7.13.3/x-pack/filebeat/module/cisco/ftd/test/asa.log#L71

I think the pattern for 302012 in the ingress pipeline should be for 305012.

I think you're right. Fixed.

[trexx]

message-id 302020 gets incorrectly tagged with event.action: flow-expiration when the message referrers to building a icmp connection.
302021 is the teardown message id and seems to be properly tagged.

609001 is also another message id that gets tagged with flow-expiration but refers to building a connection.
609002 is the tear down message id.

message ids that refer to building connections (such as 302013) get tagged as event.action: "firewall-rule". This is applied because event.id is not blank. It maybe more informational if the event.action refers to the built / teardown actions in the log messages. Or perhaps these were meant to be tagged with connection-started?

[trexx]

https://www.cisco.com/c/en/us/td/docs/security/asa/syslog/b_syslog/syslog-messages-302003-to-342008.html#con_4770603
https://github.com/elastic/beats/blob/v7.13.3/x-pack/filebeat/module/cisco/shared/ingest/asa-ftd-pipeline.yml#L332
message-id: 302013

Example:
Built inbound TCP connection 197220814 for OUTSIDE:81.0.0.1/60991 (18.0.0.1/60991)(LOCAL\USER001) to OUTSIDE:82.0.0.1/80 (82.0.0.1/80) (USER001)

This message id has the AAA user at the end and no destination user. This gets picked up by the ingress pipeline as the destination username when in fact the destination has no user.

I'd expect the destination user, similar to the source user, to be right after the destination port with no space in between. I think this gets picked up because theres a space before the bracket in the destination user pattern.

Quite the Goliathan task parsing these Cisco Syslog messages.

[legoguy1000]

106023:
Built inbound TCP connection 195207391 for OUTSIDE:85.0.0.1/12312 (62.0.0.1/34534)(LOCAL\USER001) to OUTSIDE:81.0.0.1/443 (81.0.0.1/443) (USER001)
Built inbound TCP connection 195207391 for OUTSIDE:85.0.0.1/12312 (62.0.0.1/34534)(LOCAL[user@domain.tld](mailto:user@domain.tld)) to OUTSIDE:81.0.0.1/443 (81.0.0.1/443) (user@domain.tld)

302020:
Built inbound ICMP connection for faddr 85.0.0.1/0(LOCAL\USER001) gaddr 81.0.0.1/0 laddr 81.0.0.1/0 (USER001) type 3 code 3
Built inbound ICMP connection for faddr 85.0.0.1/0(LOCAL[user@domain.tld](mailto:user@domain.tld)) gaddr 81.0.0.1/0 laddr 81.0.0.1/0 (user@domain.tld) type 3 code 3

Does it make sense to keep the LOCAL ever??

[trexx]

Does it make sense to keep the LOCAL ever??

I believe it maybe worth keeping. Its possible other deployments out there are authenticating differently and it'll show the correct domain. I'm certain its down to how we have deployed and configured Cisco Identity Services Engine and/or Cisco Context Directory Agent.

https://www.cisco.com/c/en/us/td/docs/security/ibf/cda_10/Install_Config_guide/cda10/cda_wrkng.html

For users who are authenticated through ISE against , the domain that ISE is joined to is used as the domain name. For users who are authenticated through ISE but not against , do not have a domain and “LOCAL” is used as the domain name.

[legoguy1000]

Does it make sense to keep the LOCAL ever??

I believe it maybe worth keeping. Its possible other deployments out there are authenticating differently and it'll show the correct domain. I'm certain its down to how we have deployed and configured Cisco Identity Services Engine and/or Cisco Context Directory Agent.

https://www.cisco.com/c/en/us/td/docs/security/ibf/cda_10/Install_Config_guide/cda10/cda_wrkng.html

For users who are authenticated through ISE against , the domain that ISE is joined to is used as the domain name. For users who are authenticated through ISE but not against , do not have a domain and “LOCAL” is used as the domain name.

Copy. I'll have to go back and take some time to make sure its all parsed properly.

[trexx]

For message id 733100
burst.object has some leading whitespace in the field which changes depending on its contents, weirdly. Might need to have a trim processor clean it up.

Examples:

[ Scanning] drop rate-2 exceeded. Current burst rate is 0 per second, max configured rate is 8; Current average rate is 5 per second, max configured rate is 4; Cumulative total count is 19269
[   192.168.0.1] drop rate-1 exceeded. Current burst rate is 0 per second, max configured rate is 10; Current average rate is 5 per second, max configured rate is 5; Cumulative total count is 6018
[     Port-5432 5432] drop rate-1 exceeded. Current burst rate is 8 per second, max configured rate is 10; Current average rate is 20 per second, max configured rate is 5; Cumulative total count is 12466
[           RDP 3389] drop rate-1 exceeded. Current burst rate is 63 per second, max configured rate is 10; Current average rate is 5 per second, max configured rate is 5; Cumulative total count is 3054

[legoguy1000]

@trexx My latest commit should address everything ( I think). Please take a look and let me know. If there is more sample data that you think we should add to identifies these pattern issues, please post them.

[trexx]

@trexx My latest commit should address everything ( I think). Please take a look and let me know. If there is more sample data that you think we should add to identifies these pattern issues, please post them.

@legoguy1000 All good. No more grok errors. *.users look good too.

[trexx]

Oh looks like there's a mapping error in my logs.

Failed to parse field [source.port] of type [long]
Preview of field "57019(LOCAL\USER001)"

[legoguy1000]

Oh looks like there's a mapping error in my logs.

Failed to parse field [source.port] of type [long]
Preview of field "57019(LOCAL\USER001)"

Which log I'd was that and can u post the original log?

[trexx]

The dissect for message-id 305012 is missing patterns to detect user name, it was renamed from message id 302012. This was causing "mapper_parsing_exception"s

<174>%ASA-6-305012: Teardown dynamic TCP translation from OUTSIDE:192.168.0.1/59677(LOCAL\USER001) to OUTSIDE:75.0.0.1/18449 duration 0:00:00

Its structure is similar to that of message id 305011 which is the "Built" event and is matching user names.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b 26869-asa-fixes upstream/26869-asa-fixes
git merge upstream/master
git push upstream 26869-asa-fixes

[trexx]

@trexx do all the changes in this branch solve all the issues u identified?

Yes. All issues have been solved with these changes.

[P1llus]

@legoguy1000 Now that the issues has been resolved, is this now ready to be reviewed?

[legoguy1000]

@legoguy1000 Now that the issues has been resolved, is this now ready to be reviewed?

Yes I think so.

[P1llus]

jenkins run tests

[legoguy1000]

@P1llus This will need to be synced with the integrations