Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Filebeat] Add Dashboards to Threat Intel Module #24488

Merged
merged 2 commits into from
Mar 17, 2021
Merged

[Filebeat] Add Dashboards to Threat Intel Module #24488

merged 2 commits into from
Mar 17, 2021

Conversation

peasead
Copy link
Contributor

@peasead peasead commented Mar 10, 2021

What does this PR do?

This PR adds a saved search, visualizations, and dashboards for the Threat Intel Filebeat Module.

Why is it important?

The overview dashboard provides valuable information on the health of the module, what type of data is being ingested, how much, and allows you to analyze module metrics.

The additional five dashboards provide a view into the different feeds and allow an analyst to hunt and find connections and associations that may not have previously been exposed.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

  • Validated that ./filebeat setup imports the saved search, visualization, and dashboard
  • Validated that data provided by the Threat Intel Filebeat module populates the dashboards

How to test this PR locally

This can be tested by:

cd beats/x-pack/filebeat
mage update && mage build
./filebeat modules enable threatintel
# if using Elastic Cloud, update filebeat.yml to cloud.id and cloud.auth
./filebeat setup -E setup.dashboards.directory=./build/kibana
./filebeat

Go to KIbana -> dashboards -> apply the threat intel tag

Related issues

Resolves #24487

Use cases

Threat hunting, security operations, and intelligence analysis.

Screenshots

Threat Intel Overview
filebeat-threatintel-overview

Threat Intel Abuse Malware
filebeat-threatintel-abuse-malware

Threat Intel Abuse URL
filebeat-threatintel-abuse-url

Threat Intel AlienVault OTX
filebeat-threatintel-alienvault-otx

Threat Intel Anomali Limo
filebeat-threatintel-anomali-limo

Threat Intel MISP
filebeat-threatintel-misp

Logs

@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Mar 10, 2021
@elasticmachine
Copy link
Collaborator

elasticmachine commented Mar 10, 2021

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Pull request #24488 updated

  • Start Time: 2021-03-15T15:21:07.344+0000

  • Duration: 74 min 34 sec

  • Commit: 223a4c5

Test stats 🧪

Test Results
Failed 0
Passed 13160
Skipped 2243
Total 15403

Trends 🧪

Image of Build Times

Image of Tests

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test Results
Failed 0
Passed 13160
Skipped 2243
Total 15403

@elasticmachine
Copy link
Collaborator

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Mar 11, 2021
Copy link
Contributor

@marc-gr marc-gr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@marc-gr marc-gr merged commit 141273b into elastic:master Mar 17, 2021
marc-gr pushed a commit to marc-gr/beats that referenced this pull request Mar 17, 2021
* added dashboards & docs

* ran mage fmt update

(cherry picked from commit 141273b)
marc-gr pushed a commit to marc-gr/beats that referenced this pull request Mar 17, 2021
* added dashboards & docs

* ran mage fmt update

(cherry picked from commit 141273b)
v1v added a commit to v1v/beats that referenced this pull request Mar 17, 2021
* upstream/master:
  [CI] bump gvm version and use the binary (elastic#24571)
  [Filebeat] Add Dashboards to Threat Intel Module (elastic#24488)
  Fix default scope in add_nomad_metadata (elastic#24559)
  [Heartbeat] Produce error rather than panic on missing source (elastic#24404)
  [Auditbeat] btmp offset check (elastic#24515)
  Clarify that the Tomcat module is for ingesting access logs (elastic#24543)
  [Ingest Manager] Move logging defaults to agent (elastic#24535)
marc-gr added a commit that referenced this pull request Mar 17, 2021
* added dashboards & docs

* ran mage fmt update

(cherry picked from commit 141273b)

Co-authored-by: Andrew Pease <7442091+peasead@users.noreply.github.com>
marc-gr added a commit that referenced this pull request Mar 17, 2021
* added dashboards & docs

* ran mage fmt update

(cherry picked from commit 141273b)

Co-authored-by: Andrew Pease <7442091+peasead@users.noreply.github.com>
v1v added a commit to v1v/beats that referenced this pull request Mar 17, 2021
…rchive-for-functionbeat

* upstream/master:
  [CI] bump gvm version and use the binary (elastic#24571)
  [Filebeat] Add Dashboards to Threat Intel Module (elastic#24488)
  Fix default scope in add_nomad_metadata (elastic#24559)
v1v added a commit to v1v/beats that referenced this pull request Mar 17, 2021
…reporting-changes

* upstream/master:
  [CI] bump gvm version and use the binary (elastic#24571)
  [Filebeat] Add Dashboards to Threat Intel Module (elastic#24488)
  Fix default scope in add_nomad_metadata (elastic#24559)
  [Heartbeat] Produce error rather than panic on missing source (elastic#24404)
narph pushed a commit to narph/beats that referenced this pull request Mar 17, 2021
* added dashboards & docs

* ran mage fmt update
narph added a commit that referenced this pull request Mar 18, 2021
* mofidy doc

* work on fix

* update changelog

* update notice

* Update notice

* [Ingest Manager] Sync on rename on windows (#24504)

* Add tests for encoding settings of filestream input (#24426)

* [Elastic Agent] Add the ability to provide custom CA's inside of docker. (#24486)

* Add the ability to provide custom CA's for Elastic Agent docker.

* Add changelog.

* Update Golang to 1.15.9 (#24442)

* Add syntax for multiple selector logging (#24207) (#24497)

* Add syntax for multiple selector logging

* Update libbeat/docs/loggingconfig.asciidoc

Co-authored-by: EamonnTP <Eamonn.Smith@elastic.co>

Co-authored-by: EamonnTP <Eamonn.Smith@elastic.co>

Co-authored-by: AndyHunt66 <andrew.hunt@elastic.co>

* chore(ci): use beat_version instead of PR version (#24446)

* Add test for close.reader.after_interval to filestream input (#24423)

* Refactor use of system.hostfs to fix cgroup metrics (#24334)

* refactor use of system.hostfs to fix cgroup metrics

* add changelog

* remove comment

* add cfgwarn

* move changelog

* shift around CLI config location and rep warning

* add comment about system.hostfs usage

* update docs

* capitalization

* fix grammar, add conditional

* change docs phrasing

* [Elastic Agent] Add verification check when updating communication to Kibana. (#24489)

* Add verification check when updating communication to Kibana.

* Add changelog.

* Add const.

* Fix typo in mqtt input docs (#24509)

* Update input-http-endpoint.asciidoc (#24490)

* [Ingest Manager] Move logging defaults to agent (#24535)

[Ingest Manager] Move logging defaults to agent (#24535)

* Clarify that the Tomcat module is for ingesting access logs (#24543)

The Tomcat module is for ingesting access logs, not Catalina or localhost logs.

* [Auditbeat] btmp offset check (#24515)

* auditbeat btmp offset check

Add check that saved offset is not larger than the current file size
to prevent seeking past the end of file

* [Heartbeat] Produce error rather than panic on missing source (#24404)

Fixes #24403.

With the changes to the heartbeat config syntax in 7.12 the `source`
field is now required. Our config validation code didn't actually check
for this field's presence, which caused an NPE.

This PR adds a validation checking for that config's presence. It also
adds tests for the validation code for config sub-fields. There were no
defects found in the validations for source.inline, or source.browser,
but a few tests were missing.

Instead of the panic seen in #24403 users will now get the error seen
below.

```
2021-03-05T15:41:40.146-0600	ERROR	instance/beat.go:952	Exiting: could not create monitor: job err could not parse suite config: config 'source' must be specified for this monitor, if upgrading from a previous experimental version please see our new config docs accessing 'heartbeat.monitors.0' (source:'sample-synthetics-config/heartbeat.yml')
Exiting: could not create monitor: job err could not parse suite config: config 'source' must be specified for this monitor, if upgrading from a previous experimental version please see our new config docs accessing 'heartbeat.monitors.0' (source:'sample-synthetics-config/heartbeat.yml')
```

* Fix default scope in add_nomad_metadata (#24559)

Fix default scope in add_nomad_metadata. It is set to local, but
it should be node. Fix also error message that showed that local
is a valid value.

* [Filebeat] Add Dashboards to Threat Intel Module (#24488)

* added dashboards & docs

* ran mage fmt update

* [CI] bump gvm version and use the binary (#24571)

* [CI] Add resilience when installing required tooling (#24542)

* [CI] enable new flaky detector (#24464)

* chore: do not pass beat version (#24586)

We will be delegating the version calculation to the e2e tests, using
target branch values as defaults

Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>
Co-authored-by: Michal Pristas <michal.pristas@gmail.com>
Co-authored-by: Noémi Ványi <kvch@users.noreply.github.com>
Co-authored-by: Blake Rouse <blake.rouse@elastic.co>
Co-authored-by: EamonnTP <Eamonn.Smith@elastic.co>
Co-authored-by: AndyHunt66 <andrew.hunt@elastic.co>
Co-authored-by: Manuel de la Peña <mdelapenya@gmail.com>
Co-authored-by: Alex K <8418476+fearful-symmetry@users.noreply.github.com>
Co-authored-by: DeDe Morton <dede.morton@elastic.co>
Co-authored-by: Lee Hinman <57081003+leehinman@users.noreply.github.com>
Co-authored-by: Andrew Cholakian <andrew@andrewvc.com>
Co-authored-by: Andrew Pease <7442091+peasead@users.noreply.github.com>
Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>
narph added a commit to narph/beats that referenced this pull request Mar 18, 2021
* mofidy doc

* work on fix

* update changelog

* update notice

* Update notice

* [Ingest Manager] Sync on rename on windows (elastic#24504)

* Add tests for encoding settings of filestream input (elastic#24426)

* [Elastic Agent] Add the ability to provide custom CA's inside of docker. (elastic#24486)

* Add the ability to provide custom CA's for Elastic Agent docker.

* Add changelog.

* Update Golang to 1.15.9 (elastic#24442)

* Add syntax for multiple selector logging (elastic#24207) (elastic#24497)

* Add syntax for multiple selector logging

* Update libbeat/docs/loggingconfig.asciidoc

Co-authored-by: EamonnTP <Eamonn.Smith@elastic.co>

Co-authored-by: EamonnTP <Eamonn.Smith@elastic.co>

Co-authored-by: AndyHunt66 <andrew.hunt@elastic.co>

* chore(ci): use beat_version instead of PR version (elastic#24446)

* Add test for close.reader.after_interval to filestream input (elastic#24423)

* Refactor use of system.hostfs to fix cgroup metrics (elastic#24334)

* refactor use of system.hostfs to fix cgroup metrics

* add changelog

* remove comment

* add cfgwarn

* move changelog

* shift around CLI config location and rep warning

* add comment about system.hostfs usage

* update docs

* capitalization

* fix grammar, add conditional

* change docs phrasing

* [Elastic Agent] Add verification check when updating communication to Kibana. (elastic#24489)

* Add verification check when updating communication to Kibana.

* Add changelog.

* Add const.

* Fix typo in mqtt input docs (elastic#24509)

* Update input-http-endpoint.asciidoc (elastic#24490)

* [Ingest Manager] Move logging defaults to agent (elastic#24535)

[Ingest Manager] Move logging defaults to agent (elastic#24535)

* Clarify that the Tomcat module is for ingesting access logs (elastic#24543)

The Tomcat module is for ingesting access logs, not Catalina or localhost logs.

* [Auditbeat] btmp offset check (elastic#24515)

* auditbeat btmp offset check

Add check that saved offset is not larger than the current file size
to prevent seeking past the end of file

* [Heartbeat] Produce error rather than panic on missing source (elastic#24404)

Fixes elastic#24403.

With the changes to the heartbeat config syntax in 7.12 the `source`
field is now required. Our config validation code didn't actually check
for this field's presence, which caused an NPE.

This PR adds a validation checking for that config's presence. It also
adds tests for the validation code for config sub-fields. There were no
defects found in the validations for source.inline, or source.browser,
but a few tests were missing.

Instead of the panic seen in elastic#24403 users will now get the error seen
below.

```
2021-03-05T15:41:40.146-0600	ERROR	instance/beat.go:952	Exiting: could not create monitor: job err could not parse suite config: config 'source' must be specified for this monitor, if upgrading from a previous experimental version please see our new config docs accessing 'heartbeat.monitors.0' (source:'sample-synthetics-config/heartbeat.yml')
Exiting: could not create monitor: job err could not parse suite config: config 'source' must be specified for this monitor, if upgrading from a previous experimental version please see our new config docs accessing 'heartbeat.monitors.0' (source:'sample-synthetics-config/heartbeat.yml')
```

* Fix default scope in add_nomad_metadata (elastic#24559)

Fix default scope in add_nomad_metadata. It is set to local, but
it should be node. Fix also error message that showed that local
is a valid value.

* [Filebeat] Add Dashboards to Threat Intel Module (elastic#24488)

* added dashboards & docs

* ran mage fmt update

* [CI] bump gvm version and use the binary (elastic#24571)

* [CI] Add resilience when installing required tooling (elastic#24542)

* [CI] enable new flaky detector (elastic#24464)

* chore: do not pass beat version (elastic#24586)

We will be delegating the version calculation to the e2e tests, using
target branch values as defaults

Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>
Co-authored-by: Michal Pristas <michal.pristas@gmail.com>
Co-authored-by: Noémi Ványi <kvch@users.noreply.github.com>
Co-authored-by: Blake Rouse <blake.rouse@elastic.co>
Co-authored-by: EamonnTP <Eamonn.Smith@elastic.co>
Co-authored-by: AndyHunt66 <andrew.hunt@elastic.co>
Co-authored-by: Manuel de la Peña <mdelapenya@gmail.com>
Co-authored-by: Alex K <8418476+fearful-symmetry@users.noreply.github.com>
Co-authored-by: DeDe Morton <dede.morton@elastic.co>
Co-authored-by: Lee Hinman <57081003+leehinman@users.noreply.github.com>
Co-authored-by: Andrew Cholakian <andrew@andrewvc.com>
Co-authored-by: Andrew Pease <7442091+peasead@users.noreply.github.com>
Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>
(cherry picked from commit 5c6f1b6)
narph added a commit to narph/beats that referenced this pull request Mar 18, 2021
* mofidy doc

* work on fix

* update changelog

* update notice

* Update notice

* [Ingest Manager] Sync on rename on windows (elastic#24504)

* Add tests for encoding settings of filestream input (elastic#24426)

* [Elastic Agent] Add the ability to provide custom CA's inside of docker. (elastic#24486)

* Add the ability to provide custom CA's for Elastic Agent docker.

* Add changelog.

* Update Golang to 1.15.9 (elastic#24442)

* Add syntax for multiple selector logging (elastic#24207) (elastic#24497)

* Add syntax for multiple selector logging

* Update libbeat/docs/loggingconfig.asciidoc

Co-authored-by: EamonnTP <Eamonn.Smith@elastic.co>

Co-authored-by: EamonnTP <Eamonn.Smith@elastic.co>

Co-authored-by: AndyHunt66 <andrew.hunt@elastic.co>

* chore(ci): use beat_version instead of PR version (elastic#24446)

* Add test for close.reader.after_interval to filestream input (elastic#24423)

* Refactor use of system.hostfs to fix cgroup metrics (elastic#24334)

* refactor use of system.hostfs to fix cgroup metrics

* add changelog

* remove comment

* add cfgwarn

* move changelog

* shift around CLI config location and rep warning

* add comment about system.hostfs usage

* update docs

* capitalization

* fix grammar, add conditional

* change docs phrasing

* [Elastic Agent] Add verification check when updating communication to Kibana. (elastic#24489)

* Add verification check when updating communication to Kibana.

* Add changelog.

* Add const.

* Fix typo in mqtt input docs (elastic#24509)

* Update input-http-endpoint.asciidoc (elastic#24490)

* [Ingest Manager] Move logging defaults to agent (elastic#24535)

[Ingest Manager] Move logging defaults to agent (elastic#24535)

* Clarify that the Tomcat module is for ingesting access logs (elastic#24543)

The Tomcat module is for ingesting access logs, not Catalina or localhost logs.

* [Auditbeat] btmp offset check (elastic#24515)

* auditbeat btmp offset check

Add check that saved offset is not larger than the current file size
to prevent seeking past the end of file

* [Heartbeat] Produce error rather than panic on missing source (elastic#24404)

Fixes elastic#24403.

With the changes to the heartbeat config syntax in 7.12 the `source`
field is now required. Our config validation code didn't actually check
for this field's presence, which caused an NPE.

This PR adds a validation checking for that config's presence. It also
adds tests for the validation code for config sub-fields. There were no
defects found in the validations for source.inline, or source.browser,
but a few tests were missing.

Instead of the panic seen in elastic#24403 users will now get the error seen
below.

```
2021-03-05T15:41:40.146-0600	ERROR	instance/beat.go:952	Exiting: could not create monitor: job err could not parse suite config: config 'source' must be specified for this monitor, if upgrading from a previous experimental version please see our new config docs accessing 'heartbeat.monitors.0' (source:'sample-synthetics-config/heartbeat.yml')
Exiting: could not create monitor: job err could not parse suite config: config 'source' must be specified for this monitor, if upgrading from a previous experimental version please see our new config docs accessing 'heartbeat.monitors.0' (source:'sample-synthetics-config/heartbeat.yml')
```

* Fix default scope in add_nomad_metadata (elastic#24559)

Fix default scope in add_nomad_metadata. It is set to local, but
it should be node. Fix also error message that showed that local
is a valid value.

* [Filebeat] Add Dashboards to Threat Intel Module (elastic#24488)

* added dashboards & docs

* ran mage fmt update

* [CI] bump gvm version and use the binary (elastic#24571)

* [CI] Add resilience when installing required tooling (elastic#24542)

* [CI] enable new flaky detector (elastic#24464)

* chore: do not pass beat version (elastic#24586)

We will be delegating the version calculation to the e2e tests, using
target branch values as defaults

Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>
Co-authored-by: Michal Pristas <michal.pristas@gmail.com>
Co-authored-by: Noémi Ványi <kvch@users.noreply.github.com>
Co-authored-by: Blake Rouse <blake.rouse@elastic.co>
Co-authored-by: EamonnTP <Eamonn.Smith@elastic.co>
Co-authored-by: AndyHunt66 <andrew.hunt@elastic.co>
Co-authored-by: Manuel de la Peña <mdelapenya@gmail.com>
Co-authored-by: Alex K <8418476+fearful-symmetry@users.noreply.github.com>
Co-authored-by: DeDe Morton <dede.morton@elastic.co>
Co-authored-by: Lee Hinman <57081003+leehinman@users.noreply.github.com>
Co-authored-by: Andrew Cholakian <andrew@andrewvc.com>
Co-authored-by: Andrew Pease <7442091+peasead@users.noreply.github.com>
Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>
(cherry picked from commit 5c6f1b6)
@peasead peasead deleted the filebeat-ti-dashboards branch March 23, 2021 20:35
narph added a commit that referenced this pull request Mar 26, 2021
* mofidy doc

* work on fix

* update changelog

* update notice

* Update notice

* [Ingest Manager] Sync on rename on windows (#24504)

* Add tests for encoding settings of filestream input (#24426)

* [Elastic Agent] Add the ability to provide custom CA's inside of docker. (#24486)

* Add the ability to provide custom CA's for Elastic Agent docker.

* Add changelog.

* Update Golang to 1.15.9 (#24442)

* Add syntax for multiple selector logging (#24207) (#24497)

* Add syntax for multiple selector logging

* Update libbeat/docs/loggingconfig.asciidoc

Co-authored-by: EamonnTP <Eamonn.Smith@elastic.co>

Co-authored-by: EamonnTP <Eamonn.Smith@elastic.co>

Co-authored-by: AndyHunt66 <andrew.hunt@elastic.co>

* chore(ci): use beat_version instead of PR version (#24446)

* Add test for close.reader.after_interval to filestream input (#24423)

* Refactor use of system.hostfs to fix cgroup metrics (#24334)

* refactor use of system.hostfs to fix cgroup metrics

* add changelog

* remove comment

* add cfgwarn

* move changelog

* shift around CLI config location and rep warning

* add comment about system.hostfs usage

* update docs

* capitalization

* fix grammar, add conditional

* change docs phrasing

* [Elastic Agent] Add verification check when updating communication to Kibana. (#24489)

* Add verification check when updating communication to Kibana.

* Add changelog.

* Add const.

* Fix typo in mqtt input docs (#24509)

* Update input-http-endpoint.asciidoc (#24490)

* [Ingest Manager] Move logging defaults to agent (#24535)

[Ingest Manager] Move logging defaults to agent (#24535)

* Clarify that the Tomcat module is for ingesting access logs (#24543)

The Tomcat module is for ingesting access logs, not Catalina or localhost logs.

* [Auditbeat] btmp offset check (#24515)

* auditbeat btmp offset check

Add check that saved offset is not larger than the current file size
to prevent seeking past the end of file

* [Heartbeat] Produce error rather than panic on missing source (#24404)

Fixes #24403.

With the changes to the heartbeat config syntax in 7.12 the `source`
field is now required. Our config validation code didn't actually check
for this field's presence, which caused an NPE.

This PR adds a validation checking for that config's presence. It also
adds tests for the validation code for config sub-fields. There were no
defects found in the validations for source.inline, or source.browser,
but a few tests were missing.

Instead of the panic seen in #24403 users will now get the error seen
below.

```
2021-03-05T15:41:40.146-0600	ERROR	instance/beat.go:952	Exiting: could not create monitor: job err could not parse suite config: config 'source' must be specified for this monitor, if upgrading from a previous experimental version please see our new config docs accessing 'heartbeat.monitors.0' (source:'sample-synthetics-config/heartbeat.yml')
Exiting: could not create monitor: job err could not parse suite config: config 'source' must be specified for this monitor, if upgrading from a previous experimental version please see our new config docs accessing 'heartbeat.monitors.0' (source:'sample-synthetics-config/heartbeat.yml')
```

* Fix default scope in add_nomad_metadata (#24559)

Fix default scope in add_nomad_metadata. It is set to local, but
it should be node. Fix also error message that showed that local
is a valid value.

* [Filebeat] Add Dashboards to Threat Intel Module (#24488)

* added dashboards & docs

* ran mage fmt update

* [CI] bump gvm version and use the binary (#24571)

* [CI] Add resilience when installing required tooling (#24542)

* [CI] enable new flaky detector (#24464)

* chore: do not pass beat version (#24586)

We will be delegating the version calculation to the e2e tests, using
target branch values as defaults

Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>
Co-authored-by: Michal Pristas <michal.pristas@gmail.com>
Co-authored-by: Noémi Ványi <kvch@users.noreply.github.com>
Co-authored-by: Blake Rouse <blake.rouse@elastic.co>
Co-authored-by: EamonnTP <Eamonn.Smith@elastic.co>
Co-authored-by: AndyHunt66 <andrew.hunt@elastic.co>
Co-authored-by: Manuel de la Peña <mdelapenya@gmail.com>
Co-authored-by: Alex K <8418476+fearful-symmetry@users.noreply.github.com>
Co-authored-by: DeDe Morton <dede.morton@elastic.co>
Co-authored-by: Lee Hinman <57081003+leehinman@users.noreply.github.com>
Co-authored-by: Andrew Cholakian <andrew@andrewvc.com>
Co-authored-by: Andrew Pease <7442091+peasead@users.noreply.github.com>
Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>
(cherry picked from commit 5c6f1b6)
narph added a commit that referenced this pull request Mar 26, 2021
* Update gosigar package after fix (#24502)

* mofidy doc

* work on fix

* update changelog

* update notice

* Update notice

* [Ingest Manager] Sync on rename on windows (#24504)

* Add tests for encoding settings of filestream input (#24426)

* [Elastic Agent] Add the ability to provide custom CA's inside of docker. (#24486)

* Add the ability to provide custom CA's for Elastic Agent docker.

* Add changelog.

* Update Golang to 1.15.9 (#24442)

* Add syntax for multiple selector logging (#24207) (#24497)

* Add syntax for multiple selector logging

* Update libbeat/docs/loggingconfig.asciidoc

Co-authored-by: EamonnTP <Eamonn.Smith@elastic.co>

Co-authored-by: EamonnTP <Eamonn.Smith@elastic.co>

Co-authored-by: AndyHunt66 <andrew.hunt@elastic.co>

* chore(ci): use beat_version instead of PR version (#24446)

* Add test for close.reader.after_interval to filestream input (#24423)

* Refactor use of system.hostfs to fix cgroup metrics (#24334)

* refactor use of system.hostfs to fix cgroup metrics

* add changelog

* remove comment

* add cfgwarn

* move changelog

* shift around CLI config location and rep warning

* add comment about system.hostfs usage

* update docs

* capitalization

* fix grammar, add conditional

* change docs phrasing

* [Elastic Agent] Add verification check when updating communication to Kibana. (#24489)

* Add verification check when updating communication to Kibana.

* Add changelog.

* Add const.

* Fix typo in mqtt input docs (#24509)

* Update input-http-endpoint.asciidoc (#24490)

* [Ingest Manager] Move logging defaults to agent (#24535)

[Ingest Manager] Move logging defaults to agent (#24535)

* Clarify that the Tomcat module is for ingesting access logs (#24543)

The Tomcat module is for ingesting access logs, not Catalina or localhost logs.

* [Auditbeat] btmp offset check (#24515)

* auditbeat btmp offset check

Add check that saved offset is not larger than the current file size
to prevent seeking past the end of file

* [Heartbeat] Produce error rather than panic on missing source (#24404)

Fixes #24403.

With the changes to the heartbeat config syntax in 7.12 the `source`
field is now required. Our config validation code didn't actually check
for this field's presence, which caused an NPE.

This PR adds a validation checking for that config's presence. It also
adds tests for the validation code for config sub-fields. There were no
defects found in the validations for source.inline, or source.browser,
but a few tests were missing.

Instead of the panic seen in #24403 users will now get the error seen
below.

```
2021-03-05T15:41:40.146-0600	ERROR	instance/beat.go:952	Exiting: could not create monitor: job err could not parse suite config: config 'source' must be specified for this monitor, if upgrading from a previous experimental version please see our new config docs accessing 'heartbeat.monitors.0' (source:'sample-synthetics-config/heartbeat.yml')
Exiting: could not create monitor: job err could not parse suite config: config 'source' must be specified for this monitor, if upgrading from a previous experimental version please see our new config docs accessing 'heartbeat.monitors.0' (source:'sample-synthetics-config/heartbeat.yml')
```

* Fix default scope in add_nomad_metadata (#24559)

Fix default scope in add_nomad_metadata. It is set to local, but
it should be node. Fix also error message that showed that local
is a valid value.

* [Filebeat] Add Dashboards to Threat Intel Module (#24488)

* added dashboards & docs

* ran mage fmt update

* [CI] bump gvm version and use the binary (#24571)

* [CI] Add resilience when installing required tooling (#24542)

* [CI] enable new flaky detector (#24464)

* chore: do not pass beat version (#24586)

We will be delegating the version calculation to the e2e tests, using
target branch values as defaults

Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>
Co-authored-by: Michal Pristas <michal.pristas@gmail.com>
Co-authored-by: Noémi Ványi <kvch@users.noreply.github.com>
Co-authored-by: Blake Rouse <blake.rouse@elastic.co>
Co-authored-by: EamonnTP <Eamonn.Smith@elastic.co>
Co-authored-by: AndyHunt66 <andrew.hunt@elastic.co>
Co-authored-by: Manuel de la Peña <mdelapenya@gmail.com>
Co-authored-by: Alex K <8418476+fearful-symmetry@users.noreply.github.com>
Co-authored-by: DeDe Morton <dede.morton@elastic.co>
Co-authored-by: Lee Hinman <57081003+leehinman@users.noreply.github.com>
Co-authored-by: Andrew Cholakian <andrew@andrewvc.com>
Co-authored-by: Andrew Pease <7442091+peasead@users.noreply.github.com>
Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>
(cherry picked from commit 5c6f1b6)

* upadte changelog
leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023
…stic#24578)

* added dashboards & docs

* ran mage fmt update

(cherry picked from commit 1e65129)

Co-authored-by: Andrew Pease <7442091+peasead@users.noreply.github.com>
leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023
…lastic#24627)

* Update gosigar package after fix (elastic#24502)

* mofidy doc

* work on fix

* update changelog

* update notice

* Update notice

* [Ingest Manager] Sync on rename on windows (elastic#24504)

* Add tests for encoding settings of filestream input (elastic#24426)

* [Elastic Agent] Add the ability to provide custom CA's inside of docker. (elastic#24486)

* Add the ability to provide custom CA's for Elastic Agent docker.

* Add changelog.

* Update Golang to 1.15.9 (elastic#24442)

* Add syntax for multiple selector logging (elastic#24207) (elastic#24497)

* Add syntax for multiple selector logging

* Update libbeat/docs/loggingconfig.asciidoc

Co-authored-by: EamonnTP <Eamonn.Smith@elastic.co>

Co-authored-by: EamonnTP <Eamonn.Smith@elastic.co>

Co-authored-by: AndyHunt66 <andrew.hunt@elastic.co>

* chore(ci): use beat_version instead of PR version (elastic#24446)

* Add test for close.reader.after_interval to filestream input (elastic#24423)

* Refactor use of system.hostfs to fix cgroup metrics (elastic#24334)

* refactor use of system.hostfs to fix cgroup metrics

* add changelog

* remove comment

* add cfgwarn

* move changelog

* shift around CLI config location and rep warning

* add comment about system.hostfs usage

* update docs

* capitalization

* fix grammar, add conditional

* change docs phrasing

* [Elastic Agent] Add verification check when updating communication to Kibana. (elastic#24489)

* Add verification check when updating communication to Kibana.

* Add changelog.

* Add const.

* Fix typo in mqtt input docs (elastic#24509)

* Update input-http-endpoint.asciidoc (elastic#24490)

* [Ingest Manager] Move logging defaults to agent (elastic#24535)

[Ingest Manager] Move logging defaults to agent (elastic#24535)

* Clarify that the Tomcat module is for ingesting access logs (elastic#24543)

The Tomcat module is for ingesting access logs, not Catalina or localhost logs.

* [Auditbeat] btmp offset check (elastic#24515)

* auditbeat btmp offset check

Add check that saved offset is not larger than the current file size
to prevent seeking past the end of file

* [Heartbeat] Produce error rather than panic on missing source (elastic#24404)

Fixes elastic#24403.

With the changes to the heartbeat config syntax in 7.12 the `source`
field is now required. Our config validation code didn't actually check
for this field's presence, which caused an NPE.

This PR adds a validation checking for that config's presence. It also
adds tests for the validation code for config sub-fields. There were no
defects found in the validations for source.inline, or source.browser,
but a few tests were missing.

Instead of the panic seen in elastic#24403 users will now get the error seen
below.

```
2021-03-05T15:41:40.146-0600	ERROR	instance/beat.go:952	Exiting: could not create monitor: job err could not parse suite config: config 'source' must be specified for this monitor, if upgrading from a previous experimental version please see our new config docs accessing 'heartbeat.monitors.0' (source:'sample-synthetics-config/heartbeat.yml')
Exiting: could not create monitor: job err could not parse suite config: config 'source' must be specified for this monitor, if upgrading from a previous experimental version please see our new config docs accessing 'heartbeat.monitors.0' (source:'sample-synthetics-config/heartbeat.yml')
```

* Fix default scope in add_nomad_metadata (elastic#24559)

Fix default scope in add_nomad_metadata. It is set to local, but
it should be node. Fix also error message that showed that local
is a valid value.

* [Filebeat] Add Dashboards to Threat Intel Module (elastic#24488)

* added dashboards & docs

* ran mage fmt update

* [CI] bump gvm version and use the binary (elastic#24571)

* [CI] Add resilience when installing required tooling (elastic#24542)

* [CI] enable new flaky detector (elastic#24464)

* chore: do not pass beat version (elastic#24586)

We will be delegating the version calculation to the e2e tests, using
target branch values as defaults

Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co>
Co-authored-by: Michal Pristas <michal.pristas@gmail.com>
Co-authored-by: Noémi Ványi <kvch@users.noreply.github.com>
Co-authored-by: Blake Rouse <blake.rouse@elastic.co>
Co-authored-by: EamonnTP <Eamonn.Smith@elastic.co>
Co-authored-by: AndyHunt66 <andrew.hunt@elastic.co>
Co-authored-by: Manuel de la Peña <mdelapenya@gmail.com>
Co-authored-by: Alex K <8418476+fearful-symmetry@users.noreply.github.com>
Co-authored-by: DeDe Morton <dede.morton@elastic.co>
Co-authored-by: Lee Hinman <57081003+leehinman@users.noreply.github.com>
Co-authored-by: Andrew Cholakian <andrew@andrewvc.com>
Co-authored-by: Andrew Pease <7442091+peasead@users.noreply.github.com>
Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>
(cherry picked from commit b5b66f5)

* upadte changelog
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Filebeat] Dashboards for Threat Intel Module
4 participants