Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cherry-pick #20150 to 7.10: Log debug message if the Kibana dashboard can not be imported #22761

Closed
wants to merge 1 commit into from

Conversation

kvch
Copy link
Contributor

@kvch kvch commented Nov 25, 2020

Cherry-pick of PR #20150 to 7.10 branch. Original message:

This PR is the updated verion of a community contribution which was abandoned in #13387. The original PR description is the following:

As described in #12211, depending on the value of setup.dashboards.beat there is a required directory structure for successful import of the Kibana dashboards from the archive file.

This PR adds additional path in case when no Kibana dashboards can be imported from the archive file, which logs debug message as shown below:

2019-08-29T22:40:00.122+0200    DEBUG   [dashboards]    dashboards/kibana_loader.go:146 Unzip archive /tmp/tmp263803916
2019-08-29T22:40:00.126+0200    DEBUG   [dashboards]    dashboards/kibana_loader.go:146 Importing Kibana from /tmp/tmp263803916/kibana/filebeat
.. 
// this is added 
2019-08-29T22:40:00.126+0200    DEBUG   [dashboards]    dashboards/kibana_loader.go:146 Skipping import of /tmp/tmp263803916/kibana/filebeat directory. Beat name: metricbeat, base dir name: filebeat.
// end

2019-08-29T22:40:00.126+0200    INFO    instance/beat.go:776    Kibana dashboards successfully loaded.

Implementation of point 2 from #12211 proposal.

There is a table in #12211 that contains test cases for this issue, but I currently don't know how to mock all necessary dependencies and implement unit tests for this without too much changes in libbeat/dashboards/importer.go.

I see that there are some integration tests for loading dashboards in libbeat/tests/system/test_dashboard.py so I'll check those out.

Fixes #12211
Closes #13387

…ed from the archive (elastic#12211) (elastic#20150)

There is a predefined combination of directory structure and setup.dashboards.beat property that has to be defined to successfully import dashboards to the Kibana.

Co-authored-by: dplavcic <dalibor.plavcic@gmail.com>
(cherry picked from commit c2bb3f0)
@kvch kvch added [zube]: In Review backport Team:Services (Deprecated) Label for the former Integrations-Services team labels Nov 25, 2020
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Nov 25, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations-services (Team:Services)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Nov 25, 2020
@elasticmachine
Copy link
Collaborator

💔 Tests Failed

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: Pull request #22761 opened

  • Start Time: 2020-11-25T12:58:09.557+0000

  • Duration: 58 min 1 sec

Test stats 🧪

Test Results
Failed 1
Passed 16523
Skipped 1382
Total 17906

Test errors 1

Expand to view the tests failures

Build&Test / metricbeat-goIntegTest / TestFetch – status

    Expand to view the error details

     Failed 
    

    Expand to view the stacktrace

         stats_integration_test.go:59: kibana/stats event: {
              "elasticsearch": {
                "cluster": {
                  "id": "Gp2aJkgAQ4uAufaDUsfZ7Q"
                }
              },
              "kibana": {
                "stats": {
                  "concurrent_connections": 0,
                  "host": {
                    "name": "0"
                  },
                  "index": "kibana",
                  "name": "kibana",
                  "process": {
                    "event_loop_delay": {
                      "ms": 7.705736999982037
                    },
                    "memory": {
                      "heap": {
                        "size_limit": {
                          "bytes": 1526909922
                        },
                        "total": {
                          "bytes": 635740160
                        },
                        "used": {
                          "bytes": 456435400
                        }
                      }
                    },
                    "uptime": {
                      "ms": 82658
                    }
                  },
                  "request": {
                    "disconnects": 0,
                    "total": 1
                  },
                  "response_time": {
                    "avg": {
                      "ms": 22
                    },
                    "max": {
                      "ms": 22
                    }
                  },
                  "snapshot": false,
                  "status": "green"
                }
              },
              "process": {
                "pid": 6
              },
              "service": {
                "address": "0:5601",
                "id": "4d71f669-7213-4b8e-8b83-75ffe79e5b55",
                "name": "kibana",
                "type": "kibana",
                "version": "7.9.2"
              }
            }
        status_integration_test.go:38:  
    

Steps errors 3

Expand to view the steps failures

metricbeat-goIntegTest - mage goIntegTest

  • Took 24 min 54 sec . View more details on here
  • Description: mage goIntegTest

Error signal

  • Took 0 min 0 sec . View more details on here
  • Description: Error 'hudson.AbortException: script returned exit code 1'

Terraform Apply on x-pack/metricbeat/module/aws

  • Took 0 min 15 sec . View more details on here

Log output

Expand to view the last 100 lines of log output

[2020-11-25T13:55:43.495Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/x-pack-dockerlogbeat-Lint-1606310027122
[2020-11-25T13:55:43.602Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/x-pack-dockerlogbeat-build-1606310037130
[2020-11-25T13:55:43.711Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/journalbeat-unitTest-1606310041377
[2020-11-25T13:55:43.828Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/filebeat-Lint-1606310046883
[2020-11-25T13:55:43.932Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/x-pack-auditbeat-Lint-1606310051811
[2020-11-25T13:55:44.039Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/packetbeat-Lint-1606310065124
[2020-11-25T13:55:44.158Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/journalbeat-Lint-1606310072431
[2020-11-25T13:55:44.261Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/x-pack-winlogbeat-Lint-immutable----ubuntu-18-1606310074505
[2020-11-25T13:55:44.367Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/x-pack-metricbeat-Lint-1606310080174
[2020-11-25T13:55:44.473Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/auditbeat-crosscompile-1606310087144
[2020-11-25T13:55:44.573Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/winlogbeat-crosscompile-1606310088864
[2020-11-25T13:55:44.684Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/x-pack-filebeat-Lint-1606310106850
[2020-11-25T13:55:44.796Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/auditbeat-Lint-1606310115167
[2020-11-25T13:55:44.903Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/libbeat-stress-tests-1606310119512
[2020-11-25T13:55:45.013Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/libbeat-crosscompile-1606310143196
[2020-11-25T13:55:45.123Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/x-pack-functionbeat-build-1606310146097
[2020-11-25T13:55:45.234Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/heartbeat-Lint-1606310156820
[2020-11-25T13:55:45.357Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/x-pack-functionbeat-Lint-1606310173311
[2020-11-25T13:55:45.478Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/x-pack-elastic-agent-build-1606310203678
[2020-11-25T13:55:45.584Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/libbeat-Lint-1606310226002
[2020-11-25T13:55:45.702Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/metricbeat-Lint-1606310238585
[2020-11-25T13:55:45.808Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/x-pack-libbeat-Lint-1606310245141
[2020-11-25T13:55:45.915Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/metricbeat-crosscompile-1606310261432
[2020-11-25T13:55:46.018Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/metricbeat-unitTest-1606310284626
[2020-11-25T13:55:46.136Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/packetbeat-build-1606310351764
[2020-11-25T13:55:46.253Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/generator-metricbeat-test-1606310365329
[2020-11-25T13:55:46.380Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/x-pack-auditbeat-windows-windows-2019-1606310396778
[2020-11-25T13:55:46.504Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/auditbeat-build-1606310399954
[2020-11-25T13:55:46.622Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/x-pack-elastic-agent-windows-windows-2019-1606310410481
[2020-11-25T13:55:46.727Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/heartbeat-windows-windows-2019-1606310415883
[2020-11-25T13:55:46.839Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/winlogbeat-windows-windows-2008-r2-1606310440256
[2020-11-25T13:55:46.959Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/x-pack-libbeat-build-1606310455405
[2020-11-25T13:55:47.075Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/x-pack-functionbeat-windows-windows-2019-1606310468690
[2020-11-25T13:55:47.180Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/winlogbeat-windows-windows-2019-1606310468833
[2020-11-25T13:55:47.293Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/heartbeat-build-1606310478180
[2020-11-25T13:55:47.404Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/auditbeat-windows-windows-2019-1606310482722
[2020-11-25T13:55:47.521Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/x-pack-filebeat-windows-windows-2019-1606310489645
[2020-11-25T13:55:47.632Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/x-pack-winlogbeat-build-windows-2019-1606310494049
[2020-11-25T13:55:47.772Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/x-pack-auditbeat-build-1606310534000
[2020-11-25T13:55:47.890Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/generator-beat-test-1606310558123
[2020-11-25T13:55:47.996Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/filebeat-windows-windows-2019-1606310653118
[2020-11-25T13:55:48.112Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/packetbeat-windows-windows-2019-1606310695327
[2020-11-25T13:55:48.224Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/metricbeat-windows-windows-2019-1606310718855
[2020-11-25T13:55:48.343Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/x-pack-metricbeat-windows-windows-2019-1606310732300
[2020-11-25T13:55:48.453Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/libbeat-build-1606310997361
[2020-11-25T13:55:48.571Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/filebeat-build-1606311073398
[2020-11-25T13:55:48.688Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/metricbeat-goIntegTest-1606311443027
[2020-11-25T13:55:48.804Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/x-pack-filebeat-build-1606311724964
[2020-11-25T13:55:48.912Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/metricbeat-pythonIntegTest-1606311772412
[2020-11-25T13:55:49.033Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/x-pack-metricbeat-build-1606312193009
[2020-11-25T13:55:49.479Z] + cat
[2020-11-25T13:55:49.479Z] + /usr/local/bin/runbld ./runbld-test-reports --job-name elastic+beats+pull-request
[2020-11-25T13:55:49.479Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-11-25T13:55:54.761Z] runbld>>> runbld started
[2020-11-25T13:55:54.761Z] runbld>>> 1.6.12/f45d832f2ba0aa2722ab4ec1fda8ad140f027f8b
[2020-11-25T13:55:56.671Z] runbld>>> The following profiles matched the job 'elastic+beats+pull-request' in order of occurrence in the config (last value wins).
[2020-11-25T13:55:56.671Z] runbld>>> Matches in the system config:
[2020-11-25T13:55:56.671Z] runbld>>> - Matched ^elastic\+beats
[2020-11-25T13:55:56.671Z] runbld>>> - Matched ^elastic\+beats\+pull-request
[2020-11-25T13:55:57.611Z] runbld>>> Debug logging enabled.
[2020-11-25T13:55:57.611Z] runbld>>> Storing result
[2020-11-25T13:55:57.870Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-11-25T13:55:57.870Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1597739501209/t/20201125135557-D0C44D3F
[2020-11-25T13:55:57.870Z] runbld>>> Adding system facts.
[2020-11-25T13:55:58.811Z] runbld>>> Adding vcs info for the latest commit:  f6eee7243c90238df3f89e4bd61f430efef72481
[2020-11-25T13:55:58.811Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-11-25T13:55:58.811Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-11-25T13:55:58.811Z] Processing JUnit reports with runbld...
[2020-11-25T13:55:58.811Z] + echo 'Processing JUnit reports with runbld...'
[2020-11-25T13:55:59.071Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-11-25T13:55:59.071Z] runbld>>> DURATION: 13ms
[2020-11-25T13:55:59.071Z] runbld>>> STDOUT: 40 bytes
[2020-11-25T13:55:59.071Z] runbld>>> STDERR: 49 bytes
[2020-11-25T13:55:59.071Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-11-25T13:55:59.071Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats_PR-22761
[2020-11-25T13:56:00.011Z] runbld>>> Storing build metadata: 
[2020-11-25T13:56:00.011Z] runbld>>> Adding test report.
[2020-11-25T13:56:00.011Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats
[2020-11-25T13:56:00.951Z] runbld>>> Found 125 test output files
[2020-11-25T13:56:00.951Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/metricbeat-goIntegTest-1606311443027/metricbeat/build/TEST-go-integration-graphite.xml
[2020-11-25T13:56:00.951Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/metricbeat-goIntegTest-1606311443027/metricbeat/build/TEST-go-integration-windows.xml
[2020-11-25T13:56:02.860Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/x-pack-metricbeat-build-1606312193009/x-pack/metricbeat/build/TEST-go-integration-activemq.xml
[2020-11-25T13:56:02.860Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/x-pack-metricbeat-build-1606312193009/x-pack/metricbeat/build/TEST-go-integration-tomcat.xml
[2020-11-25T13:56:02.860Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/x-pack-metricbeat-build-1606312193009/x-pack/metricbeat/build/TEST-go-integration-openmetrics.xml
[2020-11-25T13:56:02.860Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/x-pack-metricbeat-build-1606312193009/x-pack/metricbeat/build/TEST-go-integration-iis.xml
[2020-11-25T13:56:02.860Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats/x-pack-metricbeat-build-1606312193009/x-pack/metricbeat/build/TEST-go-integration-istio.xml
[2020-11-25T13:56:03.434Z] runbld>>> Test output logs contained: Errors: 0 Failures: 1 Tests: 17906 Skipped: 1091
[2020-11-25T13:56:03.702Z] runbld>>> Storing result
[2020-11-25T13:56:03.702Z] runbld>>> FAILURES: 1
[2020-11-25T13:56:03.961Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-11-25T13:56:03.961Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1597739501209/t/20201125135557-D0C44D3F
[2020-11-25T13:56:04.221Z] runbld>>> Email notification disabled by environment variable.
[2020-11-25T13:56:04.221Z] runbld>>> Slack notification disabled by environment variable.
[2020-11-25T13:56:09.575Z] Running in /var/lib/jenkins/workspace/Beats_beats_PR-22761/src/github.com/elastic/beats
[2020-11-25T13:56:09.780Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats_PR-22761
[2020-11-25T13:56:09.892Z] [INFO] getVaultSecret: Getting secrets
[2020-11-25T13:56:09.967Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-11-25T13:56:10.825Z] + chmod 755 generate-build-data.sh
[2020-11-25T13:56:10.825Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-22761/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-22761/runs/1 FAILURE 3480997
[2020-11-25T13:56:10.825Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats/PR-22761/runs/1/steps/?limit=10000 -o steps-info.json

🐛 Flaky test report

❕ There are test failures but not known flaky tests.

Expand to view the summary

Test stats 🧪

Test Results
Failed 1
Passed 16523
Skipped 1382
Total 17906

Genuine test errors 1

💔 There are test failures but not known flaky tests, most likely a genuine test failure.

  • Name: Build&Test / metricbeat-goIntegTest / TestFetch – status

@kvch kvch closed this Nov 25, 2020
@zube zube bot removed the [zube]: Done label Feb 24, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
backport Team:Services (Deprecated) Label for the former Integrations-Services team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants