[7.14](backport #27646) [7.x](backport #27638) Filebeat auditd: Fix Top Exec Commands dashboard visualization #27649

mergify · 2021-08-30T17:08:04Z

This is an automatic backport of pull request #27646 done by Mergify.
Cherry-pick of 9b574ef has failed:

On branch mergify/bp/7.14/pr-27646
Your branch is up to date with 'origin/7.14'.

You are currently cherry-picking commit 9b574ef77.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	modified:   CHANGELOG.next.asciidoc

Unmerged paths:
  (use "git add/rm <file>..." as appropriate to mark resolution)
	deleted by us:   filebeat/module/auditd/_meta/kibana/7/dashboard/Filebeat-auditd.ndjson

To fix up this pull request, you can check it out locally. See documentation: https://docs.github.com/en/github/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally

Mergify commands and options

More conditions and actions can be found in the documentation.

You can also trigger Mergify actions by commenting on this pull request:

@Mergifyio refresh will re-evaluate the rules
@Mergifyio rebase will rebase this PR on its base branch
@Mergifyio update will merge the base branch into this PR
@Mergifyio backport <destination> will backport this PR on <destination> branch

Additionally, on Mergify dashboard you can:

look at your merge queues
generate the Mergify configuration with the config editor.

Finally, you can contact us on https://mergify.io/

botelastic · 2021-08-30T17:08:09Z

This pull request doesn't have a Team:<team> label.

elasticmachine · 2021-08-30T18:06:33Z

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-08-30T17:08:19.125+0000
Duration: 58 min 55 sec
Commit: 1b398b1

Test stats 🧪

Test	Results
Failed	1
Passed	5734
Skipped	661
Total	6396

Trends 🧪

Test errors

Expand to view the tests failures

`Build&Test / filebeat-windows-windows-2019 / test_file_no_permission – filebeat.tests.system.test_crawler.Test`

Expand to view the error details

 PermissionError: [WinError 5] Access is denied: 'C:\\Users\\jenkins\\workspace\\PR-27649-1-e99f9ee1-a135-41eb-a1d3-777019a3b5cc\\src\\github.com\\elastic\\beats\\filebeat\\build\\system-tests\\run\\test_crawler.Test.test_file_no_permission\\log\\test.log'

Expand to view the stacktrace

 ..\libbeat\tests\system\beat\beat.py:331: in setUp
    shutil.rmtree(self.working_dir)
C:\Python38\lib\shutil.py:737: in rmtree
    return _rmtree_unsafe(path, onerror)
C:\Python38\lib\shutil.py:610: in _rmtree_unsafe
    _rmtree_unsafe(fullname, onerror)
C:\Python38\lib\shutil.py:615: in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

path = 'C:\\Users\\jenkins\\workspace\\PR-27649-1-e99f9ee1-a135-41eb-a1d3-777019a3b5cc\\src\\github.com\\elastic\\beats\\filebeat\\build\\system-tests\\run\\test_crawler.Test.test_file_no_permission\\log'
onerror = <function rmtree.<locals>.onerror at 0x0000020BB23C3C10>

    def _rmtree_unsafe(path, onerror):
        try:
            with os.scandir(path) as scandir_it:
                entries = list(scandir_it)
        except OSError:
            onerror(os.scandir, path, sys.exc_info())
            entries = []
        for entry in entries:
            fullname = entry.path
            if _rmtree_isdir(entry):
                try:
                    if entry.is_symlink():
                        # This can only happen if someone replaces
                        # a directory with a symlink after the call to
                        # os.scandir or entry.is_dir above.
                        raise OSError("Cannot call rmtree on a symbolic link")
                except OSError:
                    onerror(os.path.islink, fullname, sys.exc_info())
                    continue
                _rmtree_unsafe(fullname, onerror)
            else:
                try:
>                   os.unlink(fullname)
E                   PermissionError: [WinError 5] Access is denied: 'C:\\Users\\jenkins\\workspace\\PR-27649-1-e99f9ee1-a135-41eb-a1d3-777019a3b5cc\\src\\github.com\\elastic\\beats\\filebeat\\build\\system-tests\\run\\test_crawler.Test.test_file_no_permission\\log\\test.log'

C:\Python38\lib\shutil.py:613: PermissionError

Steps errors

Expand to view the steps failures

`filebeat-windows-windows-2019 - mage build unitTest`

Took 6 min 59 sec . View more details on here
Description: mage build unitTest

`filebeat-windows-windows-2019 - mage build unitTest`

Took 4 min 52 sec . View more details on here
Description: mage build unitTest

`filebeat-windows-windows-2019 - mage build unitTest`

Took 4 min 50 sec . View more details on here
Description: mage build unitTest

`Error signal`

Took 0 min 0 sec . View more details on here
Description: Error 'hudson.AbortException: script returned exit code 1'

Log output

Expand to view the last 100 lines of log output

[2021-08-30T17:47:13.923Z] 
[2021-08-30T17:47:13.923Z] C:\Users\jenkins\workspace\PR-27649-1-e99f9ee1-a135-41eb-a1d3-777019a3b5cc\src\github.com\elastic\beats>gcloud auth activate-service-account --key-file **** 
[2021-08-30T17:47:14.862Z] Activated service account credentials for: [beats-ci-gcs-plugin@elastic-ci-prod.iam.gserviceaccount.com]
[2021-08-30T17:47:19.080Z] 
[2021-08-30T17:47:19.080Z] C:\Users\jenkins\workspace\PR-27649-1-e99f9ee1-a135-41eb-a1d3-777019a3b5cc\src\github.com\elastic\beats>go clean -modcache 
[2021-08-30T17:47:24.577Z] ........................................................................ [ 47%]
[2021-08-30T17:48:16.983Z] Failed in branch filebeat-windows-windows-2019
[2021-08-30T17:54:13.111Z] ........................................................................ [ 64%]
[2021-08-30T17:59:27.255Z] ........................................................................ [ 81%]
[2021-08-30T18:04:44.483Z] ........................................................................ [ 97%]
[2021-08-30T18:05:37.216Z] .........                                                                [100%]
[2021-08-30T18:05:37.216Z] 
[2021-08-30T18:05:37.216Z] =============================== warnings summary ===============================
[2021-08-30T18:05:37.216Z] x-pack/filebeat/tests/system/test_xpack_modules.py: 406 warnings
[2021-08-30T18:05:37.216Z]   /go/src/github.com/elastic/beats/build/ve/docker/lib/python3.7/site-packages/elasticsearch/connection/base.py:177: ElasticsearchDeprecationWarning: [script.max_compilations_rate] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2021-08-30T18:05:37.216Z]     warnings.warn(message, category=ElasticsearchDeprecationWarning)
[2021-08-30T18:05:37.216Z] 
[2021-08-30T18:05:37.216Z] x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_000_rabbitmq
[2021-08-30T18:05:37.216Z]   /go/src/github.com/elastic/beats/build/ve/docker/lib/python3.7/site-packages/elasticsearch/connection/base.py:177: ElasticsearchDeprecationWarning: [script.cache.max_size] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2021-08-30T18:05:37.216Z]     warnings.warn(message, category=ElasticsearchDeprecationWarning)
[2021-08-30T18:05:37.216Z] 
[2021-08-30T18:05:37.216Z] -- Docs: https://docs.pytest.org/en/stable/warnings.html
[2021-08-30T18:05:37.216Z] - generated xml file: /go/src/github.com/elastic/beats/x-pack/filebeat/build/TEST-python-integration.xml -
[2021-08-30T18:05:37.216Z] ============================= slowest 20 durations =============================
[2021-08-30T18:05:37.216Z] 38.79s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_257_checkpoint
[2021-08-30T18:05:37.216Z] 26.14s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_000_rabbitmq
[2021-08-30T18:05:37.216Z] 18.10s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_356_o365
[2021-08-30T18:05:37.216Z] 16.94s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_188_snort
[2021-08-30T18:05:37.216Z] 15.55s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_350_o365
[2021-08-30T18:05:37.216Z] 15.19s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_024_threatintel
[2021-08-30T18:05:37.216Z] 11.50s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_023_threatintel
[2021-08-30T18:05:37.216Z] 11.09s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_215_panw
[2021-08-30T18:05:37.216Z] 11.00s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_404_ibmmq
[2021-08-30T18:05:37.216Z] 10.87s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_175_cisco
[2021-08-30T18:05:37.216Z] 10.79s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_183_cisco
[2021-08-30T18:05:37.216Z] 10.77s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_217_panw
[2021-08-30T18:05:37.216Z] 10.73s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_216_panw
[2021-08-30T18:05:37.216Z] 10.73s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_212_panw
[2021-08-30T18:05:37.216Z] 10.42s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_405_ibmmq
[2021-08-30T18:05:37.216Z] 10.37s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_030_threatintel
[2021-08-30T18:05:37.216Z] 10.23s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_159_cisco
[2021-08-30T18:05:37.216Z] 10.08s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_263_gcp
[2021-08-30T18:05:37.216Z] 10.04s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_197_oracle
[2021-08-30T18:05:37.216Z] 9.98s call     x-pack/filebeat/tests/system/test_xpack_modules.py::XPackTest::test_fileset_file_248_juniper
[2021-08-30T18:05:37.216Z] ================ 430 passed, 407 warnings in 1938.78s (0:32:18) ================
[2021-08-30T18:05:37.216Z] >> python test: Integration Testing Complete
[2021-08-30T18:05:40.113Z] Timeout set to expire in 5 min 0 sec
[2021-08-30T18:05:40.489Z] Cleaning up /var/lib/jenkins/workspace/PR-27649-1-cd6e70b2-a49b-4efb-889a-65d553e48f9b
[2021-08-30T18:05:40.489Z] Change ownership of all files inside the specific folder from root/root to current user/group
[2021-08-30T18:05:40.489Z] ++ id -u
[2021-08-30T18:05:40.489Z] ++ id -g
[2021-08-30T18:05:40.489Z] + docker run -v /var/lib/jenkins/workspace/PR-27649-1-cd6e70b2-a49b-4efb-889a-65d553e48f9b:/beat alpine:3.4 sh -c 'find /beat -user 0 -exec chown -h 1170:1171 {} \;'
[2021-08-30T18:05:40.489Z] Unable to find image 'alpine:3.4' locally
[2021-08-30T18:05:41.060Z] 3.4: Pulling from library/alpine
[2021-08-30T18:05:41.320Z] c1e54eec4b57: Pulling fs layer
[2021-08-30T18:05:41.582Z] c1e54eec4b57: Verifying Checksum
[2021-08-30T18:05:41.582Z] c1e54eec4b57: Download complete
[2021-08-30T18:05:41.848Z] c1e54eec4b57: Pull complete
[2021-08-30T18:05:41.848Z] Digest: sha256:b733d4a32c4da6a00a84df2ca32791bb03df95400243648d8c539e7b4cce329c
[2021-08-30T18:05:41.848Z] Status: Downloaded newer image for alpine:3.4
[2021-08-30T18:05:43.761Z] + set -e
[2021-08-30T18:05:43.761Z] + echo 'Change permissions with write access of all files inside the specific folder'
[2021-08-30T18:05:43.761Z] Change permissions with write access of all files inside the specific folder
[2021-08-30T18:05:43.761Z] + chmod -R +w /var/lib/jenkins/workspace/PR-27649-1-cd6e70b2-a49b-4efb-889a-65d553e48f9b
[2021-08-30T18:05:44.445Z] Running in /var/lib/jenkins/workspace/PR-27649-1-cd6e70b2-a49b-4efb-889a-65d553e48f9b/src/github.com/elastic/beats/build
[2021-08-30T18:05:44.787Z] + rm -rf ve
[2021-08-30T18:05:44.787Z] + find . -type d -name vendor -exec rm -r {} ;
[2021-08-30T18:05:45.198Z] + python .ci/scripts/pre_archive_test.py
[2021-08-30T18:05:47.749Z] Copy ./x-pack/filebeat/build into build/x-pack/filebeat/build
[2021-08-30T18:05:47.786Z] Running in /var/lib/jenkins/workspace/PR-27649-1-cd6e70b2-a49b-4efb-889a-65d553e48f9b/src/github.com/elastic/beats/build
[2021-08-30T18:05:47.842Z] Recording test results
[2021-08-30T18:05:49.640Z] [Checks API] No suitable checks publisher found.
[2021-08-30T18:05:50.148Z] + go clean -modcache
[2021-08-30T18:05:52.754Z] Timeout set to expire in 5 min 0 sec
[2021-08-30T18:05:53.099Z] Cleaning up /var/lib/jenkins/workspace/PR-27649-1-cd6e70b2-a49b-4efb-889a-65d553e48f9b
[2021-08-30T18:05:53.099Z] Change ownership of all files inside the specific folder from root/root to current user/group
[2021-08-30T18:05:53.099Z] ++ id -u
[2021-08-30T18:05:53.099Z] ++ id -g
[2021-08-30T18:05:53.099Z] + docker run -v /var/lib/jenkins/workspace/PR-27649-1-cd6e70b2-a49b-4efb-889a-65d553e48f9b:/beat alpine:3.4 sh -c 'find /beat -user 0 -exec chown -h 1170:1171 {} \;'
[2021-08-30T18:06:02.955Z] + set -e
[2021-08-30T18:06:02.955Z] + echo 'Change permissions with write access of all files inside the specific folder'
[2021-08-30T18:06:02.955Z] Change permissions with write access of all files inside the specific folder
[2021-08-30T18:06:02.955Z] + chmod -R +w /var/lib/jenkins/workspace/PR-27649-1-cd6e70b2-a49b-4efb-889a-65d553e48f9b
[2021-08-30T18:06:03.063Z] Running in /var/lib/jenkins/workspace/PR-27649-1-cd6e70b2-a49b-4efb-889a-65d553e48f9b
[2021-08-30T18:06:07.524Z] + gsutil --version
[2021-08-30T18:06:08.951Z] Masking supported pattern matches of $FILE_CREDENTIAL
[2021-08-30T18:06:09.308Z] + gcloud auth activate-service-account --key-file ****
[2021-08-30T18:06:09.877Z] Activated service account credentials for: [beats-ci-gcs-plugin@elastic-ci-prod.iam.gserviceaccount.com]
[2021-08-30T18:06:10.271Z] + gsutil -m -q cp eC1wYWNrL2ZpbGViZWF0LXB5dGhvbkludGVnVGVzdDFiMzk4YjFiNjNkMWI5MzYyZWIwYmRhZDQ3ODRmNzY2NGM3MTNhZWY gs://beats-ci-temp/ci/cache/
[2021-08-30T18:06:12.149Z] Stage "Extended" skipped due to earlier failure(s)
[2021-08-30T18:06:12.253Z] Stage "Packaging" skipped due to earlier failure(s)
[2021-08-30T18:06:12.361Z] Stage "Packaging-Pipeline" skipped due to earlier failure(s)
[2021-08-30T18:06:12.491Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-27649/src/github.com/elastic/beats
[2021-08-30T18:06:13.193Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-27649
[2021-08-30T18:06:13.299Z] [INFO] getVaultSecret: Getting secrets
[2021-08-30T18:06:13.353Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2021-08-30T18:06:14.391Z] + chmod 755 generate-build-data.sh
[2021-08-30T18:06:14.392Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-27649/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-27649/runs/1 FAILURE 3474986
[2021-08-30T18:06:14.392Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-27649/runs/1/steps/?limit=10000 -o steps-info.json
[2021-08-30T18:06:15.735Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-27649/runs/1/tests/?status=FAILED -o tests-errors.json

🐛 Flaky test report

❕ There are test failures but not known flaky tests.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	1
Passed	5734
Skipped	661
Total	6396

Genuine test errors

💔 There are test failures but not known flaky tests, most likely a genuine test failure.

Name: Build&Test / filebeat-windows-windows-2019 / test_file_no_permission – filebeat.tests.system.test_crawler.Test

… (#27646) This visualization was expecting an uppercase EXECVE value in event.action while the ingest pipeline was lowercasing this value. (cherry picked from commit 9b574ef)

mergify bot added backport conflicts There is a conflict in the backported pull request labels Aug 30, 2021

mergify bot assigned adriansr Aug 30, 2021

botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Aug 30, 2021

Filebeat auditd: Fix Top Exec Commands dashboard visualization (#27638)…

e3c9257

… (#27646) This visualization was expecting an uppercase EXECVE value in event.action while the ingest pipeline was lowercasing this value. (cherry picked from commit 9b574ef)

adriansr force-pushed the mergify/bp/7.14/pr-27646 branch from 1b398b1 to e3c9257 Compare August 31, 2021 19:36

adriansr merged commit f230b31 into 7.14 Aug 31, 2021

mergify bot deleted the mergify/bp/7.14/pr-27646 branch August 31, 2021 19:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[7.14](backport #27646) [7.x](backport #27638) Filebeat auditd: Fix Top Exec Commands dashboard visualization #27649

[7.14](backport #27646) [7.x](backport #27638) Filebeat auditd: Fix Top Exec Commands dashboard visualization #27649

mergify bot commented Aug 30, 2021 •

edited by zube bot

Loading

botelastic bot commented Aug 30, 2021

elasticmachine commented Aug 30, 2021

Build stats

Test stats 🧪

Trends 🧪

`Build&Test / filebeat-windows-windows-2019 / test_file_no_permission – filebeat.tests.system.test_crawler.Test`

`filebeat-windows-windows-2019 - mage build unitTest`

`filebeat-windows-windows-2019 - mage build unitTest`

`filebeat-windows-windows-2019 - mage build unitTest`

`Error signal`

Test stats 🧪

Genuine test errors

[7.14](backport #27646) [7.x](backport #27638) Filebeat auditd: Fix Top Exec Commands dashboard visualization #27649

[7.14](backport #27646) [7.x](backport #27638) Filebeat auditd: Fix Top Exec Commands dashboard visualization #27649

Conversation

mergify bot commented Aug 30, 2021 • edited by zube bot Loading

botelastic bot commented Aug 30, 2021

elasticmachine commented Aug 30, 2021

💔 Tests Failed

Build stats

Test stats 🧪

Trends 🧪

Test errors

Build&Test / filebeat-windows-windows-2019 / test_file_no_permission – filebeat.tests.system.test_crawler.Test

Steps errors

filebeat-windows-windows-2019 - mage build unitTest

filebeat-windows-windows-2019 - mage build unitTest

filebeat-windows-windows-2019 - mage build unitTest

Error signal

Log output

🐛 Flaky test report

Test stats 🧪

Genuine test errors

mergify bot commented Aug 30, 2021 •

edited by zube bot

Loading

`Build&Test / filebeat-windows-windows-2019 / test_file_no_permission – filebeat.tests.system.test_crawler.Test`

`filebeat-windows-windows-2019 - mage build unitTest`

`filebeat-windows-windows-2019 - mage build unitTest`

`filebeat-windows-windows-2019 - mage build unitTest`

`Error signal`