Add test for close.reader.after_interval of filestream input #24423

kvch · 2021-03-08T17:11:07Z

What does this PR do?

This PR adds test for close.reader.after_interval option, aka close_timeout in log input.

Why is it important?

Increase test coverage.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
~~- [ ] I have made corresponding changes to the documentation~~
~~- [ ] I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
~~- [ ] I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.~~

elasticmachine · 2021-03-08T18:10:06Z

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Build Cause: Pull request #24423 updated
Start Time: 2021-03-11T14:19:54.012+0000
Duration: 81 min 21 sec
Commit: 63c8034

Test stats 🧪

Test	Results
Failed	1
Passed	13159
Skipped	2243
Total	15403

Trends 🧪

Test errors

Expand to view the tests failures

`Build&Test / x-pack/filebeat-build / test_fileset_file_155_cisco – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest`

Expand to view the error details

 AssertionError: The following expected object doesn't match:    Diff:   {'values_changed': {"root['cisco.amp.detection_id']": {'new_value': '6159251516445163617', 'old_value': '6159251516445163618'}, "root['log.offset']": {'new_value': 79352, 'old_value': 78028}}}, full object:    {'log.offset': 79352, 'fileset.name': 'amp', 'tags': ['cisco-amp', 'forwarded'], 'input.type': 'log', '@timestamp': '2021-01-15T10:17:39.000Z', 'file.path': '\\\\?\\C:\\Users\\Administrator\\AppData\\Roaming\\rjtsbks.exe', 'file.name': 'rjtsbks.exe', 'file.hash.sha1': 'e654d39cd13414b5151e8cf0d8f5b166dddd45cb', 'file.hash.sha256': '3372c1edab46837f1e973164fa2d726c5c5e17bcb888828ccd7c4dfcc234a370', 'file.hash.md5': '209a288c68207d57e0ce6e60ebf60729', 'related.hosts': ['Demo_TeslaCrypt'], 'related.ip': ['8.8.8.8', '10.10.10.10'], 'related.hash': ['3372c1edab46837f1e973164fa2d726c5c5e17bcb888828ccd7c4dfcc234a370', '209a288c68207d57e0ce6e60ebf60729', 'e654d39cd13414b5151e8cf0d8f5b166dddd45cb'], 'service.type': 'cisco', 'host.hostname': 'Demo_TeslaCrypt', 'host.os.family': 'windows', 'host.os.platform': 'windows', 'host.name': 'Demo_TeslaCrypt', 'event.severity': 2, 'event.kind': 'alert', 'event.module': 'cisco', 'event.action': 'Threat Detected', 'event.id': 6159251516445164000, 'event.category': ['file', 'malware'], 'event.dataset': 'cisco.amp', 'cisco.amp.detection': 'W32.DFC.MalParent', 'cisco.amp.group_guids': ['test_group_guid'], 'cisco.amp.event_type_id': 1090519054, 'cisco.amp.timestamp_nanoseconds': 988000000, 'cisco.amp.computer.network_addresses': [{'ip': '10.10.10.10', 'mac': '90:61:b5:c9:13:79'}], 'cisco.amp.computer.connector_guid': 'test_connector_guid', 'cisco.amp.computer.active': True, 'cisco.amp.computer.external_ip': '8.8.8.8', 'cisco.amp.file.disposition': 'Malicious', 'cisco.amp.connector_guid': 'test_connector_guid', 'cisco.amp.related.mac': ['90:61:b5:c9:13:79'], 'cisco.amp.detection_id': '6159251516445163617'} assert 1 == 0  +  where 1 = len({'values_changed': {"root['cisco.amp.detection_id']": {'new_value': '6159251516445163617', 'old_value': '6159251516445163618'}, "root['log.offset']": {'new_value': 79352, 'old_value': 78028}}})

Expand to view the stacktrace

 a = (<test_xpack_modules.XPackTest testMethod=test_fileset_file_155_cisco>,)

    @wraps(func)
    def standalone_func(*a):
>       return func(*(a + p.args), **p.kwargs)

../../build/ve/docker/lib/python3.7/site-packages/parameterized/parameterized.py:518: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../filebeat/tests/system/test_modules.py:99: in test_fileset_file
    cfgfile=cfgfile)
../../filebeat/tests/system/test_modules.py:185: in run_on_file
    self._test_expected_events(test_file, objects)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <test_xpack_modules.XPackTest testMethod=test_fileset_file_155_cisco>
test_file = '/go/src/github.com/elastic/beats/x-pack/filebeat/module/cisco/amp/test/cisco_amp2.ndjson.log'
objects = [{'@timestamp': '2021-01-15T11:59:52.000Z', 'agent': {'ephemeral_id': '52ff9103-c14c-4bea-a78e-20e2fda70f75', 'id': '8...}, 'connector_guid': 'test_connector_guid', 'event_type_id': 1107296274, ...}}, 'ecs': {'version': '1.7.0'}, ...}, ...]

    def _test_expected_events(self, test_file, objects):
    
        # Generate expected files if GENERATE env variable is set
        if os.getenv("GENERATE"):
            with open(test_file + "-expected.json", 'w') as f:
                # Flatten an cleanup objects
                # This makes sure when generated on different machines / version the expected.json stays the same.
                for k, obj in enumerate(objects):
                    objects[k] = self.flatten_object(obj, {}, "")
                    clean_keys(objects[k])
    
                json.dump(objects, f, indent=4, separators=(',', ': '), sort_keys=True)
    
        with open(test_file + "-expected.json", "r") as f:
            expected = json.load(f)
    
        assert len(expected) == len(objects), "expected {} events to compare but got {}".format(
            len(expected), len(objects))
    
        for idx in range(len(expected)):
            ev = expected[idx]
            obj = objects[idx]
    
            # Flatten objects for easier comparing
            obj = self.flatten_object(obj, {}, "")
            clean_keys(obj)
            clean_keys(ev)
    
            d = DeepDiff(ev, obj, ignore_order=True)
    
>           assert len(d) == 0, "The following expected object doesn't match:\n Diff:\n{}, full object: \n{}".format(d, obj)
E           AssertionError: The following expected object doesn't match:
E              Diff:
E             {'values_changed': {"root['cisco.amp.detection_id']": {'new_value': '6159251516445163617', 'old_value': '6159251516445163618'}, "root['log.offset']": {'new_value': 79352, 'old_value': 78028}}}, full object: 
E             {'log.offset': 79352, 'fileset.name': 'amp', 'tags': ['cisco-amp', 'forwarded'], 'input.type': 'log', '@timestamp': '2021-01-15T10:17:39.000Z', 'file.path': '\\\\?\\C:\\Users\\Administrator\\AppData\\Roaming\\rjtsbks.exe', 'file.name': 'rjtsbks.exe', 'file.hash.sha1': 'e654d39cd13414b5151e8cf0d8f5b166dddd45cb', 'file.hash.sha256': '3372c1edab46837f1e973164fa2d726c5c5e17bcb888828ccd7c4dfcc234a370', 'file.hash.md5': '209a288c68207d57e0ce6e60ebf60729', 'related.hosts': ['Demo_TeslaCrypt'], 'related.ip': ['8.8.8.8', '10.10.10.10'], 'related.hash': ['3372c1edab46837f1e973164fa2d726c5c5e17bcb888828ccd7c4dfcc234a370', '209a288c68207d57e0ce6e60ebf60729', 'e654d39cd13414b5151e8cf0d8f5b166dddd45cb'], 'service.type': 'cisco', 'host.hostname': 'Demo_TeslaCrypt', 'host.os.family': 'windows', 'host.os.platform': 'windows', 'host.name': 'Demo_TeslaCrypt', 'event.severity': 2, 'event.kind': 'alert', 'event.module': 'cisco', 'event.action': 'Threat Detected', 'event.id': 6159251516445164000, 'event.category': ['file', 'malware'], 'event.dataset': 'cisco.amp', 'cisco.amp.detection': 'W32.DFC.MalParent', 'cisco.amp.group_guids': ['test_group_guid'], 'cisco.amp.event_type_id': 1090519054, 'cisco.amp.timestamp_nanoseconds': 988000000, 'cisco.amp.computer.network_addresses': [{'ip': '10.10.10.10', 'mac': '90:61:b5:c9:13:79'}], 'cisco.amp.computer.connector_guid': 'test_connector_guid', 'cisco.amp.computer.active': True, 'cisco.amp.computer.external_ip': '8.8.8.8', 'cisco.amp.file.disposition': 'Malicious', 'cisco.amp.connector_guid': 'test_connector_guid', 'cisco.amp.related.mac': ['90:61:b5:c9:13:79'], 'cisco.amp.detection_id': '6159251516445163617'}
E           assert 1 == 0
E            +  where 1 = len({'values_changed': {"root['cisco.amp.detection_id']": {'new_value': '6159251516445163617', 'old_value': '6159251516445163618'}, "root['log.offset']": {'new_value': 79352, 'old_value': 78028}}})

../../filebeat/tests/system/test_modules.py:217: AssertionError

Steps errors

Expand to view the steps failures

`x-pack/filebeat-build - mage build test`

Took 32 min 41 sec . View more details on here
Description: mage build test

`Error signal`

Took 0 min 0 sec . View more details on here
Description: Error 'hudson.AbortException: script returned exit code 1'

Log output

Expand to view the last 100 lines of log output

[2021-03-11T15:40:34.219Z] 056125ce7a35: Waiting
[2021-03-11T15:40:34.219Z] 94dc982a8d06: Waiting
[2021-03-11T15:40:34.219Z] e993f077b676: Waiting
[2021-03-11T15:40:34.219Z] a79979c1dc53: Waiting
[2021-03-11T15:40:34.219Z] 5a2c8a1119e6: Waiting
[2021-03-11T15:40:34.657Z] fc07c1b59178: Layer already exists
[2021-03-11T15:40:34.657Z] 07e72b49afee: Layer already exists
[2021-03-11T15:40:34.657Z] 70bf974c0674: Layer already exists
[2021-03-11T15:40:34.657Z] d8788b1c29f4: Layer already exists
[2021-03-11T15:40:34.657Z] c092672313d2: Layer already exists
[2021-03-11T15:40:34.657Z] 2e6bd77da548: Layer already exists
[2021-03-11T15:40:35.034Z] 5a2c8a1119e6: Layer already exists
[2021-03-11T15:40:35.034Z] 056125ce7a35: Layer already exists
[2021-03-11T15:40:35.034Z] 94dc982a8d06: Layer already exists
[2021-03-11T15:40:35.034Z] e993f077b676: Layer already exists
[2021-03-11T15:40:35.034Z] a79979c1dc53: Layer already exists
[2021-03-11T15:40:35.034Z] e48e4f4d88e8: Layer already exists
[2021-03-11T15:40:36.313Z] 63c8034c05ba8e801c7cea1f3435ed6e5445a179-arm64: digest: sha256:48b81fbf934071e1e9eb3acdd7a60ebffa5caf8ba4b751cdc690126825320578 size: 2824
[2021-03-11T15:40:37.232Z] Cleaning up /var/lib/jenkins/workspace/PR-24423-3-219ae753-2e1a-413b-bb58-2ae26d69f4c1
[2021-03-11T15:40:37.232Z] Client: Docker Engine - Community
[2021-03-11T15:40:37.232Z]  Version:           19.03.8
[2021-03-11T15:40:37.232Z]  API version:       1.40
[2021-03-11T15:40:37.232Z]  Go version:        go1.12.17
[2021-03-11T15:40:37.232Z]  Git commit:        afacb8b
[2021-03-11T15:40:37.232Z]  Built:             Wed Mar 11 01:26:27 2020
[2021-03-11T15:40:37.232Z]  OS/Arch:           linux/arm64
[2021-03-11T15:40:37.232Z]  Experimental:      false
[2021-03-11T15:40:37.232Z] 
[2021-03-11T15:40:37.232Z] Server: Docker Engine - Community
[2021-03-11T15:40:37.232Z]  Engine:
[2021-03-11T15:40:37.232Z]   Version:          19.03.8
[2021-03-11T15:40:37.232Z]   API version:      1.40 (minimum version 1.12)
[2021-03-11T15:40:37.232Z]   Go version:       go1.12.17
[2021-03-11T15:40:37.232Z]   Git commit:       afacb8b
[2021-03-11T15:40:37.232Z]   Built:            Wed Mar 11 01:24:59 2020
[2021-03-11T15:40:37.232Z]   OS/Arch:          linux/arm64
[2021-03-11T15:40:37.232Z]   Experimental:     false
[2021-03-11T15:40:37.232Z]  containerd:
[2021-03-11T15:40:37.232Z]   Version:          1.2.13
[2021-03-11T15:40:37.232Z]   GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
[2021-03-11T15:40:37.232Z]  runc:
[2021-03-11T15:40:37.232Z]   Version:          1.0.0-rc10
[2021-03-11T15:40:37.232Z]   GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
[2021-03-11T15:40:37.233Z]  docker-init:
[2021-03-11T15:40:37.233Z]   Version:          0.18.0
[2021-03-11T15:40:37.233Z]   GitCommit:        fec3683
[2021-03-11T15:40:37.233Z] Change ownership of all files inside the specific folder from root/root to current user/group
[2021-03-11T15:40:38.531Z] Change permissions with write access of all files inside the specific folder
[2021-03-11T15:40:40.293Z] + python .ci/scripts/pre_archive_test.py
[2021-03-11T15:40:41.581Z] Copy ./x-pack/filebeat/build into build/x-pack/filebeat/build
[2021-03-11T15:40:41.581Z] Copy ./x-pack/filebeat/build/package/filebeat/filebeat-linux-arm64.docker/docker-build into build/x-pack/filebeat/build/package/filebeat/filebeat-linux-arm64.docker/docker-build
[2021-03-11T15:40:41.581Z] Copy ./x-pack/filebeat/build/package/filebeat-ubi8/filebeat-linux-arm64.docker/docker-build into build/x-pack/filebeat/build/package/filebeat-ubi8/filebeat-linux-arm64.docker/docker-build
[2021-03-11T15:40:41.581Z] Copy ./x-pack/filebeat/build/golang-crossbuild into build/x-pack/filebeat/build/golang-crossbuild
[2021-03-11T15:40:41.772Z] Running in /var/lib/jenkins/workspace/PR-24423-3-219ae753-2e1a-413b-bb58-2ae26d69f4c1/src/github.com/elastic/beats/build
[2021-03-11T15:40:42.594Z] + rm -rf ve
[2021-03-11T15:40:42.594Z] + find . -type d -name vendor -exec rm -r {} ;
[2021-03-11T15:40:42.728Z] Recording test results
[2021-03-11T15:40:44.362Z] None of the test reports contained any result
[2021-03-11T15:40:44.370Z] [Checks API] No suitable checks publisher found.
[2021-03-11T15:40:45.002Z] + go clean -modcache
[2021-03-11T15:40:48.285Z] Cleaning up /var/lib/jenkins/workspace/PR-24423-3-219ae753-2e1a-413b-bb58-2ae26d69f4c1
[2021-03-11T15:40:48.286Z] Client: Docker Engine - Community
[2021-03-11T15:40:48.286Z]  Version:           19.03.8
[2021-03-11T15:40:48.286Z]  API version:       1.40
[2021-03-11T15:40:48.286Z]  Go version:        go1.12.17
[2021-03-11T15:40:48.286Z]  Git commit:        afacb8b
[2021-03-11T15:40:48.286Z]  Built:             Wed Mar 11 01:26:27 2020
[2021-03-11T15:40:48.286Z]  OS/Arch:           linux/arm64
[2021-03-11T15:40:48.286Z]  Experimental:      false
[2021-03-11T15:40:48.286Z] 
[2021-03-11T15:40:48.286Z] Server: Docker Engine - Community
[2021-03-11T15:40:48.286Z]  Engine:
[2021-03-11T15:40:48.286Z]   Version:          19.03.8
[2021-03-11T15:40:48.286Z]   API version:      1.40 (minimum version 1.12)
[2021-03-11T15:40:48.286Z]   Go version:       go1.12.17
[2021-03-11T15:40:48.286Z]   Git commit:       afacb8b
[2021-03-11T15:40:48.286Z]   Built:            Wed Mar 11 01:24:59 2020
[2021-03-11T15:40:48.286Z]   OS/Arch:          linux/arm64
[2021-03-11T15:40:48.286Z]   Experimental:     false
[2021-03-11T15:40:48.286Z]  containerd:
[2021-03-11T15:40:48.286Z]   Version:          1.2.13
[2021-03-11T15:40:48.286Z]   GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
[2021-03-11T15:40:48.286Z]  runc:
[2021-03-11T15:40:48.286Z]   Version:          1.0.0-rc10
[2021-03-11T15:40:48.286Z]   GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
[2021-03-11T15:40:48.286Z]  docker-init:
[2021-03-11T15:40:48.286Z]   Version:          0.18.0
[2021-03-11T15:40:48.286Z]   GitCommit:        fec3683
[2021-03-11T15:40:48.286Z] Change ownership of all files inside the specific folder from root/root to current user/group
[2021-03-11T15:40:53.829Z] Change permissions with write access of all files inside the specific folder
[2021-03-11T15:40:54.007Z] Running in /var/lib/jenkins/workspace/PR-24423-3-219ae753-2e1a-413b-bb58-2ae26d69f4c1
[2021-03-11T15:40:58.852Z] Stage "Packaging" skipped due to earlier failure(s)
[2021-03-11T15:40:58.982Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-24423/src/github.com/elastic/beats
[2021-03-11T15:41:14.422Z] Running on beats-ci-immutable-ubuntu-2004-1615476419122630475 in /var/lib/jenkins/workspace/Beats_beats_PR-24423
[2021-03-11T15:41:14.479Z] [INFO] getVaultSecret: Getting secrets
[2021-03-11T15:41:14.586Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-03-11T15:41:15.426Z] + chmod 755 generate-build-data.sh
[2021-03-11T15:41:15.426Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-24423/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-24423/runs/3 FAILURE 4881115
[2021-03-11T15:41:15.426Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-24423/runs/3/steps/?limit=10000 -o steps-info.json
[2021-03-11T15:41:16.819Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-24423/runs/3/tests/?status=FAILED -o tests-errors.json

🐛 Flaky test report

❕ There are test failures but not known flaky tests.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	1
Passed	13159
Skipped	2243
Total	15403

Genuine test errors

💔 There are test failures but not known flaky tests, most likely a genuine test failure.

Name: Build&Test / x-pack/filebeat-build / test_fileset_file_155_cisco – x-pack.filebeat.tests.system.test_xpack_modules.XPackTest

elasticmachine · 2021-03-08T21:07:20Z

Pinging @elastic/agent (Team:Agent)

filebeat/input/filestream/input_integration_test.go

michalpristas

seems like failure in test is not related and this test works

…#24423)

…reporting-changes * upstream/master: (23 commits) [Auditbeat] btmp offset check (elastic#24515) Clarify that the Tomcat module is for ingesting access logs (elastic#24543) [Ingest Manager] Move logging defaults to agent (elastic#24535) Update input-http-endpoint.asciidoc (elastic#24490) Fix typo in mqtt input docs (elastic#24509) [Elastic Agent] Add verification check when updating communication to Kibana. (elastic#24489) Refactor use of system.hostfs to fix cgroup metrics (elastic#24334) Add test for close.reader.after_interval to filestream input (elastic#24423) chore(ci): use beat_version instead of PR version (elastic#24446) Add syntax for multiple selector logging (elastic#24207) (elastic#24497) Update Golang to 1.15.9 (elastic#24442) [Elastic Agent] Add the ability to provide custom CA's inside of docker. (elastic#24486) Add tests for encoding settings of filestream input (elastic#24426) [Ingest Manager] Sync on rename on windows (elastic#24504) Port four Harvester tests of log input to filestream in Golang (elastic#24250) [DOCS] Restructure content for SSL settings (elastic#24342) Move example to the correct location in reference docs (elastic#24455) Add unit tests for harvester.go of input-logfile package (elastic#24107) Fix type for uwsgi.status.worker.rss field (elastic#24468) [Ingest Manager] Logging to file disabled on enroll (elastic#24466) ...

* mofidy doc * work on fix * update changelog * update notice * Update notice * [Ingest Manager] Sync on rename on windows (#24504) * Add tests for encoding settings of filestream input (#24426) * [Elastic Agent] Add the ability to provide custom CA's inside of docker. (#24486) * Add the ability to provide custom CA's for Elastic Agent docker. * Add changelog. * Update Golang to 1.15.9 (#24442) * Add syntax for multiple selector logging (#24207) (#24497) * Add syntax for multiple selector logging * Update libbeat/docs/loggingconfig.asciidoc Co-authored-by: EamonnTP <Eamonn.Smith@elastic.co> Co-authored-by: EamonnTP <Eamonn.Smith@elastic.co> Co-authored-by: AndyHunt66 <andrew.hunt@elastic.co> * chore(ci): use beat_version instead of PR version (#24446) * Add test for close.reader.after_interval to filestream input (#24423) * Refactor use of system.hostfs to fix cgroup metrics (#24334) * refactor use of system.hostfs to fix cgroup metrics * add changelog * remove comment * add cfgwarn * move changelog * shift around CLI config location and rep warning * add comment about system.hostfs usage * update docs * capitalization * fix grammar, add conditional * change docs phrasing * [Elastic Agent] Add verification check when updating communication to Kibana. (#24489) * Add verification check when updating communication to Kibana. * Add changelog. * Add const. * Fix typo in mqtt input docs (#24509) * Update input-http-endpoint.asciidoc (#24490) * [Ingest Manager] Move logging defaults to agent (#24535) [Ingest Manager] Move logging defaults to agent (#24535) * Clarify that the Tomcat module is for ingesting access logs (#24543) The Tomcat module is for ingesting access logs, not Catalina or localhost logs. * [Auditbeat] btmp offset check (#24515) * auditbeat btmp offset check Add check that saved offset is not larger than the current file size to prevent seeking past the end of file * [Heartbeat] Produce error rather than panic on missing source (#24404) Fixes #24403. With the changes to the heartbeat config syntax in 7.12 the `source` field is now required. Our config validation code didn't actually check for this field's presence, which caused an NPE. This PR adds a validation checking for that config's presence. It also adds tests for the validation code for config sub-fields. There were no defects found in the validations for source.inline, or source.browser, but a few tests were missing. Instead of the panic seen in #24403 users will now get the error seen below. ``` 2021-03-05T15:41:40.146-0600 ERROR instance/beat.go:952 Exiting: could not create monitor: job err could not parse suite config: config 'source' must be specified for this monitor, if upgrading from a previous experimental version please see our new config docs accessing 'heartbeat.monitors.0' (source:'sample-synthetics-config/heartbeat.yml') Exiting: could not create monitor: job err could not parse suite config: config 'source' must be specified for this monitor, if upgrading from a previous experimental version please see our new config docs accessing 'heartbeat.monitors.0' (source:'sample-synthetics-config/heartbeat.yml') ``` * Fix default scope in add_nomad_metadata (#24559) Fix default scope in add_nomad_metadata. It is set to local, but it should be node. Fix also error message that showed that local is a valid value. * [Filebeat] Add Dashboards to Threat Intel Module (#24488) * added dashboards & docs * ran mage fmt update * [CI] bump gvm version and use the binary (#24571) * [CI] Add resilience when installing required tooling (#24542) * [CI] enable new flaky detector (#24464) * chore: do not pass beat version (#24586) We will be delegating the version calculation to the e2e tests, using target branch values as defaults Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co> Co-authored-by: Michal Pristas <michal.pristas@gmail.com> Co-authored-by: Noémi Ványi <kvch@users.noreply.github.com> Co-authored-by: Blake Rouse <blake.rouse@elastic.co> Co-authored-by: EamonnTP <Eamonn.Smith@elastic.co> Co-authored-by: AndyHunt66 <andrew.hunt@elastic.co> Co-authored-by: Manuel de la Peña <mdelapenya@gmail.com> Co-authored-by: Alex K <8418476+fearful-symmetry@users.noreply.github.com> Co-authored-by: DeDe Morton <dede.morton@elastic.co> Co-authored-by: Lee Hinman <57081003+leehinman@users.noreply.github.com> Co-authored-by: Andrew Cholakian <andrew@andrewvc.com> Co-authored-by: Andrew Pease <7442091+peasead@users.noreply.github.com> Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com>

* mofidy doc * work on fix * update changelog * update notice * Update notice * [Ingest Manager] Sync on rename on windows (elastic#24504) * Add tests for encoding settings of filestream input (elastic#24426) * [Elastic Agent] Add the ability to provide custom CA's inside of docker. (elastic#24486) * Add the ability to provide custom CA's for Elastic Agent docker. * Add changelog. * Update Golang to 1.15.9 (elastic#24442) * Add syntax for multiple selector logging (elastic#24207) (elastic#24497) * Add syntax for multiple selector logging * Update libbeat/docs/loggingconfig.asciidoc Co-authored-by: EamonnTP <Eamonn.Smith@elastic.co> Co-authored-by: EamonnTP <Eamonn.Smith@elastic.co> Co-authored-by: AndyHunt66 <andrew.hunt@elastic.co> * chore(ci): use beat_version instead of PR version (elastic#24446) * Add test for close.reader.after_interval to filestream input (elastic#24423) * Refactor use of system.hostfs to fix cgroup metrics (elastic#24334) * refactor use of system.hostfs to fix cgroup metrics * add changelog * remove comment * add cfgwarn * move changelog * shift around CLI config location and rep warning * add comment about system.hostfs usage * update docs * capitalization * fix grammar, add conditional * change docs phrasing * [Elastic Agent] Add verification check when updating communication to Kibana. (elastic#24489) * Add verification check when updating communication to Kibana. * Add changelog. * Add const. * Fix typo in mqtt input docs (elastic#24509) * Update input-http-endpoint.asciidoc (elastic#24490) * [Ingest Manager] Move logging defaults to agent (elastic#24535) [Ingest Manager] Move logging defaults to agent (elastic#24535) * Clarify that the Tomcat module is for ingesting access logs (elastic#24543) The Tomcat module is for ingesting access logs, not Catalina or localhost logs. * [Auditbeat] btmp offset check (elastic#24515) * auditbeat btmp offset check Add check that saved offset is not larger than the current file size to prevent seeking past the end of file * [Heartbeat] Produce error rather than panic on missing source (elastic#24404) Fixes elastic#24403. With the changes to the heartbeat config syntax in 7.12 the `source` field is now required. Our config validation code didn't actually check for this field's presence, which caused an NPE. This PR adds a validation checking for that config's presence. It also adds tests for the validation code for config sub-fields. There were no defects found in the validations for source.inline, or source.browser, but a few tests were missing. Instead of the panic seen in elastic#24403 users will now get the error seen below. ``` 2021-03-05T15:41:40.146-0600 ERROR instance/beat.go:952 Exiting: could not create monitor: job err could not parse suite config: config 'source' must be specified for this monitor, if upgrading from a previous experimental version please see our new config docs accessing 'heartbeat.monitors.0' (source:'sample-synthetics-config/heartbeat.yml') Exiting: could not create monitor: job err could not parse suite config: config 'source' must be specified for this monitor, if upgrading from a previous experimental version please see our new config docs accessing 'heartbeat.monitors.0' (source:'sample-synthetics-config/heartbeat.yml') ``` * Fix default scope in add_nomad_metadata (elastic#24559) Fix default scope in add_nomad_metadata. It is set to local, but it should be node. Fix also error message that showed that local is a valid value. * [Filebeat] Add Dashboards to Threat Intel Module (elastic#24488) * added dashboards & docs * ran mage fmt update * [CI] bump gvm version and use the binary (elastic#24571) * [CI] Add resilience when installing required tooling (elastic#24542) * [CI] enable new flaky detector (elastic#24464) * chore: do not pass beat version (elastic#24586) We will be delegating the version calculation to the e2e tests, using target branch values as defaults Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co> Co-authored-by: Michal Pristas <michal.pristas@gmail.com> Co-authored-by: Noémi Ványi <kvch@users.noreply.github.com> Co-authored-by: Blake Rouse <blake.rouse@elastic.co> Co-authored-by: EamonnTP <Eamonn.Smith@elastic.co> Co-authored-by: AndyHunt66 <andrew.hunt@elastic.co> Co-authored-by: Manuel de la Peña <mdelapenya@gmail.com> Co-authored-by: Alex K <8418476+fearful-symmetry@users.noreply.github.com> Co-authored-by: DeDe Morton <dede.morton@elastic.co> Co-authored-by: Lee Hinman <57081003+leehinman@users.noreply.github.com> Co-authored-by: Andrew Cholakian <andrew@andrewvc.com> Co-authored-by: Andrew Pease <7442091+peasead@users.noreply.github.com> Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com> (cherry picked from commit 5c6f1b6)

* mofidy doc * work on fix * update changelog * update notice * Update notice * [Ingest Manager] Sync on rename on windows (#24504) * Add tests for encoding settings of filestream input (#24426) * [Elastic Agent] Add the ability to provide custom CA's inside of docker. (#24486) * Add the ability to provide custom CA's for Elastic Agent docker. * Add changelog. * Update Golang to 1.15.9 (#24442) * Add syntax for multiple selector logging (#24207) (#24497) * Add syntax for multiple selector logging * Update libbeat/docs/loggingconfig.asciidoc Co-authored-by: EamonnTP <Eamonn.Smith@elastic.co> Co-authored-by: EamonnTP <Eamonn.Smith@elastic.co> Co-authored-by: AndyHunt66 <andrew.hunt@elastic.co> * chore(ci): use beat_version instead of PR version (#24446) * Add test for close.reader.after_interval to filestream input (#24423) * Refactor use of system.hostfs to fix cgroup metrics (#24334) * refactor use of system.hostfs to fix cgroup metrics * add changelog * remove comment * add cfgwarn * move changelog * shift around CLI config location and rep warning * add comment about system.hostfs usage * update docs * capitalization * fix grammar, add conditional * change docs phrasing * [Elastic Agent] Add verification check when updating communication to Kibana. (#24489) * Add verification check when updating communication to Kibana. * Add changelog. * Add const. * Fix typo in mqtt input docs (#24509) * Update input-http-endpoint.asciidoc (#24490) * [Ingest Manager] Move logging defaults to agent (#24535) [Ingest Manager] Move logging defaults to agent (#24535) * Clarify that the Tomcat module is for ingesting access logs (#24543) The Tomcat module is for ingesting access logs, not Catalina or localhost logs. * [Auditbeat] btmp offset check (#24515) * auditbeat btmp offset check Add check that saved offset is not larger than the current file size to prevent seeking past the end of file * [Heartbeat] Produce error rather than panic on missing source (#24404) Fixes #24403. With the changes to the heartbeat config syntax in 7.12 the `source` field is now required. Our config validation code didn't actually check for this field's presence, which caused an NPE. This PR adds a validation checking for that config's presence. It also adds tests for the validation code for config sub-fields. There were no defects found in the validations for source.inline, or source.browser, but a few tests were missing. Instead of the panic seen in #24403 users will now get the error seen below. ``` 2021-03-05T15:41:40.146-0600 ERROR instance/beat.go:952 Exiting: could not create monitor: job err could not parse suite config: config 'source' must be specified for this monitor, if upgrading from a previous experimental version please see our new config docs accessing 'heartbeat.monitors.0' (source:'sample-synthetics-config/heartbeat.yml') Exiting: could not create monitor: job err could not parse suite config: config 'source' must be specified for this monitor, if upgrading from a previous experimental version please see our new config docs accessing 'heartbeat.monitors.0' (source:'sample-synthetics-config/heartbeat.yml') ``` * Fix default scope in add_nomad_metadata (#24559) Fix default scope in add_nomad_metadata. It is set to local, but it should be node. Fix also error message that showed that local is a valid value. * [Filebeat] Add Dashboards to Threat Intel Module (#24488) * added dashboards & docs * ran mage fmt update * [CI] bump gvm version and use the binary (#24571) * [CI] Add resilience when installing required tooling (#24542) * [CI] enable new flaky detector (#24464) * chore: do not pass beat version (#24586) We will be delegating the version calculation to the e2e tests, using target branch values as defaults Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co> Co-authored-by: Michal Pristas <michal.pristas@gmail.com> Co-authored-by: Noémi Ványi <kvch@users.noreply.github.com> Co-authored-by: Blake Rouse <blake.rouse@elastic.co> Co-authored-by: EamonnTP <Eamonn.Smith@elastic.co> Co-authored-by: AndyHunt66 <andrew.hunt@elastic.co> Co-authored-by: Manuel de la Peña <mdelapenya@gmail.com> Co-authored-by: Alex K <8418476+fearful-symmetry@users.noreply.github.com> Co-authored-by: DeDe Morton <dede.morton@elastic.co> Co-authored-by: Lee Hinman <57081003+leehinman@users.noreply.github.com> Co-authored-by: Andrew Cholakian <andrew@andrewvc.com> Co-authored-by: Andrew Pease <7442091+peasead@users.noreply.github.com> Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com> (cherry picked from commit 5c6f1b6)

* Update gosigar package after fix (#24502) * mofidy doc * work on fix * update changelog * update notice * Update notice * [Ingest Manager] Sync on rename on windows (#24504) * Add tests for encoding settings of filestream input (#24426) * [Elastic Agent] Add the ability to provide custom CA's inside of docker. (#24486) * Add the ability to provide custom CA's for Elastic Agent docker. * Add changelog. * Update Golang to 1.15.9 (#24442) * Add syntax for multiple selector logging (#24207) (#24497) * Add syntax for multiple selector logging * Update libbeat/docs/loggingconfig.asciidoc Co-authored-by: EamonnTP <Eamonn.Smith@elastic.co> Co-authored-by: EamonnTP <Eamonn.Smith@elastic.co> Co-authored-by: AndyHunt66 <andrew.hunt@elastic.co> * chore(ci): use beat_version instead of PR version (#24446) * Add test for close.reader.after_interval to filestream input (#24423) * Refactor use of system.hostfs to fix cgroup metrics (#24334) * refactor use of system.hostfs to fix cgroup metrics * add changelog * remove comment * add cfgwarn * move changelog * shift around CLI config location and rep warning * add comment about system.hostfs usage * update docs * capitalization * fix grammar, add conditional * change docs phrasing * [Elastic Agent] Add verification check when updating communication to Kibana. (#24489) * Add verification check when updating communication to Kibana. * Add changelog. * Add const. * Fix typo in mqtt input docs (#24509) * Update input-http-endpoint.asciidoc (#24490) * [Ingest Manager] Move logging defaults to agent (#24535) [Ingest Manager] Move logging defaults to agent (#24535) * Clarify that the Tomcat module is for ingesting access logs (#24543) The Tomcat module is for ingesting access logs, not Catalina or localhost logs. * [Auditbeat] btmp offset check (#24515) * auditbeat btmp offset check Add check that saved offset is not larger than the current file size to prevent seeking past the end of file * [Heartbeat] Produce error rather than panic on missing source (#24404) Fixes #24403. With the changes to the heartbeat config syntax in 7.12 the `source` field is now required. Our config validation code didn't actually check for this field's presence, which caused an NPE. This PR adds a validation checking for that config's presence. It also adds tests for the validation code for config sub-fields. There were no defects found in the validations for source.inline, or source.browser, but a few tests were missing. Instead of the panic seen in #24403 users will now get the error seen below. ``` 2021-03-05T15:41:40.146-0600 ERROR instance/beat.go:952 Exiting: could not create monitor: job err could not parse suite config: config 'source' must be specified for this monitor, if upgrading from a previous experimental version please see our new config docs accessing 'heartbeat.monitors.0' (source:'sample-synthetics-config/heartbeat.yml') Exiting: could not create monitor: job err could not parse suite config: config 'source' must be specified for this monitor, if upgrading from a previous experimental version please see our new config docs accessing 'heartbeat.monitors.0' (source:'sample-synthetics-config/heartbeat.yml') ``` * Fix default scope in add_nomad_metadata (#24559) Fix default scope in add_nomad_metadata. It is set to local, but it should be node. Fix also error message that showed that local is a valid value. * [Filebeat] Add Dashboards to Threat Intel Module (#24488) * added dashboards & docs * ran mage fmt update * [CI] bump gvm version and use the binary (#24571) * [CI] Add resilience when installing required tooling (#24542) * [CI] enable new flaky detector (#24464) * chore: do not pass beat version (#24586) We will be delegating the version calculation to the e2e tests, using target branch values as defaults Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co> Co-authored-by: Michal Pristas <michal.pristas@gmail.com> Co-authored-by: Noémi Ványi <kvch@users.noreply.github.com> Co-authored-by: Blake Rouse <blake.rouse@elastic.co> Co-authored-by: EamonnTP <Eamonn.Smith@elastic.co> Co-authored-by: AndyHunt66 <andrew.hunt@elastic.co> Co-authored-by: Manuel de la Peña <mdelapenya@gmail.com> Co-authored-by: Alex K <8418476+fearful-symmetry@users.noreply.github.com> Co-authored-by: DeDe Morton <dede.morton@elastic.co> Co-authored-by: Lee Hinman <57081003+leehinman@users.noreply.github.com> Co-authored-by: Andrew Cholakian <andrew@andrewvc.com> Co-authored-by: Andrew Pease <7442091+peasead@users.noreply.github.com> Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com> (cherry picked from commit 5c6f1b6) * upadte changelog

…lastic#24627) * Update gosigar package after fix (elastic#24502) * mofidy doc * work on fix * update changelog * update notice * Update notice * [Ingest Manager] Sync on rename on windows (elastic#24504) * Add tests for encoding settings of filestream input (elastic#24426) * [Elastic Agent] Add the ability to provide custom CA's inside of docker. (elastic#24486) * Add the ability to provide custom CA's for Elastic Agent docker. * Add changelog. * Update Golang to 1.15.9 (elastic#24442) * Add syntax for multiple selector logging (elastic#24207) (elastic#24497) * Add syntax for multiple selector logging * Update libbeat/docs/loggingconfig.asciidoc Co-authored-by: EamonnTP <Eamonn.Smith@elastic.co> Co-authored-by: EamonnTP <Eamonn.Smith@elastic.co> Co-authored-by: AndyHunt66 <andrew.hunt@elastic.co> * chore(ci): use beat_version instead of PR version (elastic#24446) * Add test for close.reader.after_interval to filestream input (elastic#24423) * Refactor use of system.hostfs to fix cgroup metrics (elastic#24334) * refactor use of system.hostfs to fix cgroup metrics * add changelog * remove comment * add cfgwarn * move changelog * shift around CLI config location and rep warning * add comment about system.hostfs usage * update docs * capitalization * fix grammar, add conditional * change docs phrasing * [Elastic Agent] Add verification check when updating communication to Kibana. (elastic#24489) * Add verification check when updating communication to Kibana. * Add changelog. * Add const. * Fix typo in mqtt input docs (elastic#24509) * Update input-http-endpoint.asciidoc (elastic#24490) * [Ingest Manager] Move logging defaults to agent (elastic#24535) [Ingest Manager] Move logging defaults to agent (elastic#24535) * Clarify that the Tomcat module is for ingesting access logs (elastic#24543) The Tomcat module is for ingesting access logs, not Catalina or localhost logs. * [Auditbeat] btmp offset check (elastic#24515) * auditbeat btmp offset check Add check that saved offset is not larger than the current file size to prevent seeking past the end of file * [Heartbeat] Produce error rather than panic on missing source (elastic#24404) Fixes elastic#24403. With the changes to the heartbeat config syntax in 7.12 the `source` field is now required. Our config validation code didn't actually check for this field's presence, which caused an NPE. This PR adds a validation checking for that config's presence. It also adds tests for the validation code for config sub-fields. There were no defects found in the validations for source.inline, or source.browser, but a few tests were missing. Instead of the panic seen in elastic#24403 users will now get the error seen below. ``` 2021-03-05T15:41:40.146-0600 ERROR instance/beat.go:952 Exiting: could not create monitor: job err could not parse suite config: config 'source' must be specified for this monitor, if upgrading from a previous experimental version please see our new config docs accessing 'heartbeat.monitors.0' (source:'sample-synthetics-config/heartbeat.yml') Exiting: could not create monitor: job err could not parse suite config: config 'source' must be specified for this monitor, if upgrading from a previous experimental version please see our new config docs accessing 'heartbeat.monitors.0' (source:'sample-synthetics-config/heartbeat.yml') ``` * Fix default scope in add_nomad_metadata (elastic#24559) Fix default scope in add_nomad_metadata. It is set to local, but it should be node. Fix also error message that showed that local is a valid value. * [Filebeat] Add Dashboards to Threat Intel Module (elastic#24488) * added dashboards & docs * ran mage fmt update * [CI] bump gvm version and use the binary (elastic#24571) * [CI] Add resilience when installing required tooling (elastic#24542) * [CI] enable new flaky detector (elastic#24464) * chore: do not pass beat version (elastic#24586) We will be delegating the version calculation to the e2e tests, using target branch values as defaults Co-authored-by: Jaime Soriano Pastor <jaime.soriano@elastic.co> Co-authored-by: Michal Pristas <michal.pristas@gmail.com> Co-authored-by: Noémi Ványi <kvch@users.noreply.github.com> Co-authored-by: Blake Rouse <blake.rouse@elastic.co> Co-authored-by: EamonnTP <Eamonn.Smith@elastic.co> Co-authored-by: AndyHunt66 <andrew.hunt@elastic.co> Co-authored-by: Manuel de la Peña <mdelapenya@gmail.com> Co-authored-by: Alex K <8418476+fearful-symmetry@users.noreply.github.com> Co-authored-by: DeDe Morton <dede.morton@elastic.co> Co-authored-by: Lee Hinman <57081003+leehinman@users.noreply.github.com> Co-authored-by: Andrew Cholakian <andrew@andrewvc.com> Co-authored-by: Andrew Pease <7442091+peasead@users.noreply.github.com> Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com> (cherry picked from commit b5b66f5) * upadte changelog

kvch requested a review from urso March 8, 2021 17:11

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Mar 8, 2021

This was referenced Mar 8, 2021

Add tests and support for truncated files in filestream input #24424

Merged

Finish porting all tests from Python to Golang from test_harvester.py #24397

Closed

andresrc added the Team:Elastic-Agent Label for the Agent team label Mar 8, 2021

botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Mar 8, 2021

urso reviewed Mar 9, 2021

View reviewed changes