Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cherry-pick #19398 to 7.x: Add support for named ports in autodiscover (hints & templates) #19694

Merged
merged 1 commit into from
Jul 7, 2020
Merged

Cherry-pick #19398 to 7.x: Add support for named ports in autodiscover (hints & templates) #19694

merged 1 commit into from
Jul 7, 2020

Conversation

ChrsMark
Copy link
Member

@ChrsMark ChrsMark commented Jul 7, 2020 •
edited by zube bot
Loading

Cherry-pick of PR #19398 to 7.x branch. Original message:

What does this PR do?

Adds support for named ports in hints based autodiscover.
When for instance we have a target container like:

spec:
  containers:
    - name: natsqueue
      image: nats
      command: ["/nats-server"]
      args: ["-m", "8222"]
      ports:
        - name: web
          containerPort: 8222
          protocol: TCP

then we can reference the name of the port with co.elastic.metrics/hosts: '${data.host}:${data.ports.web}' in the annotations.

Why is it important?

To cover cases like #15796.

Manual Testing

Hints based Autodiscover

  1. Deploy Metricbeat on k8s with hints autodiscover enabled:
metricbeat.autodiscover:
  providers:
    - type: kubernetes
      node: ${NODE_NAME}
      hints.enabled: true
  1. Deploy a target Pod using the following spec:
apiVersion: v1
kind: Service
metadata:
  name: nats
  labels:
    app: nats
spec:
  clusterIP: None
  ports:
  - name: web
    port: 8222
    protocol: TCP
  selector:
    app: nats
  type: ClusterIP
---
apiVersion: v1
kind: Pod
metadata:
  name: nats
  labels:
    app: name
  annotations:
    co.elastic.metrics/module: nats
    co.elastic.metrics/metricsets: stats
    co.elastic.metrics/hosts: '${data.host}:${data.ports.web}'
spec:
  containers:
    - name: natsqueue
      image: nats
      command: ["/nats-server"]
      args: ["-m", "8222"]
      ports:
        - name: web
          containerPort: 8222
          protocol: TCP
  1. Make sure that Nats metrics are being collected
  2. Verify that old behaviour is not broken by deploying target with the following 2 different annotations:
    co.elastic.metrics/hosts: '${data.host}:8222' and co.elastic.metrics/hosts: '${data.host}:${data.port}

Template based Autodiscover

  1. Deploy Metricbeat on k8s with hints autodiscover enabled:
metricbeat.autodiscover:
  providers:
    - type: kubernetes
      host: "${NODE_NAME}"
      templates:
        - condition:
            contains:
              kubernetes.annotations.nats.io/scrape: "true"
          config:
            - module: nats
              period: 10s
              hosts: ["${data.host}:${data.ports.web}"]
  1. Deploy a target Pod using the following spec:
apiVersion: v1
kind: Service
metadata:
  name: nats
  labels:
    app: nats
spec:
  clusterIP: None
  ports:
  - name: web
    port: 8222
    protocol: TCP
  selector:
    app: nats
  type: ClusterIP
---
apiVersion: v1
kind: Pod
metadata:
  name: nats
  labels:
    app: name
  annotations:
    nats.io/scrape: "true"
spec:
  containers:
    - name: natsqueue
      image: nats
      command: ["/nats-server"]
      args: ["-m", "8222"]
      ports:
        - name: web
          containerPort: 8222
          protocol: TCP
  1. Make sure that Nats metrics are being collected
  2. Verify that old behaviour is not broken by deploying target with the following 2 different annotations:
    co.elastic.metrics/hosts: '${data.host}:8222' and co.elastic.metrics/hosts: '${data.host}:${data.port}

Closes #15796

@ChrsMark ChrsMark added [zube]: In Review backport Team:Platforms Label for the Integrations - Platforms team labels Jul 7, 2020
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jul 7, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/integrations-platforms (Team:Platforms)

@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jul 7, 2020
@ChrsMark ChrsMark self-assigned this Jul 7, 2020
@elasticmachine
Copy link
Collaborator

💔 Build Failed

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #19694 opened]

  • Start Time: 2020-07-07T08:55:08.679+0000

  • Duration: 123 min 28 sec

Test stats 🧪

Test Results
Failed 0
Passed 8918
Skipped 1575
Total 10493

Steps errors

Expand to view the steps failures

  • Name: Make -C libbeat testsuite

    • Description: make -C libbeat testsuite

    • Duration: 36 min 51 sec

    • Start Time: 2020-07-07T09:19:05.518+0000

    • log

Log output

Expand to view the last 100 lines of log output 

[2020-07-07T10:58:03.217Z] + [ -f packetbeat/build/coverage/full.cov ]
[2020-07-07T10:58:03.217Z] + FILE=winlogbeat/build/coverage/full.cov
[2020-07-07T10:58:03.217Z] + [ -f winlogbeat/build/coverage/full.cov ]
[2020-07-07T10:58:03.217Z] + FILE=journalbeat/build/coverage/full.cov
[2020-07-07T10:58:03.217Z] + [ -f journalbeat/build/coverage/full.cov ]
[2020-07-07T10:58:04.655Z] Failed in branch Filebeat oss
[2020-07-07T10:58:06.390Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19694/src/github.com/elastic/beats
[2020-07-07T10:58:06.733Z] + find . -type f -name TEST*.xml -path */build/* -delete
[2020-07-07T10:58:06.755Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19694/src/github.com/elastic/beats/Lint
[2020-07-07T10:58:06.902Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19694/src/github.com/elastic/beats/Elastic-Agent-Mac-OS-X
[2020-07-07T10:58:07.052Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19694/src/github.com/elastic/beats/Elastic-Agent-x-pack
[2020-07-07T10:58:07.200Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19694/src/github.com/elastic/beats/Auditbeat-crosscompile
[2020-07-07T10:58:07.353Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19694/src/github.com/elastic/beats/Winlogbeat-oss
[2020-07-07T10:58:07.505Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19694/src/github.com/elastic/beats/Filebeat-x-pack-Mac-OS-X
[2020-07-07T10:58:07.659Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19694/src/github.com/elastic/beats/Dockerlogbeat
[2020-07-07T10:58:07.806Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19694/src/github.com/elastic/beats/Journalbeat-oss
[2020-07-07T10:58:07.956Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19694/src/github.com/elastic/beats/Generators-Metricbeat-Linux
[2020-07-07T10:58:08.103Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19694/src/github.com/elastic/beats/Auditbeat-oss-Mac-OS-X
[2020-07-07T10:58:08.255Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19694/src/github.com/elastic/beats/Functionbeat-x-pack
[2020-07-07T10:58:08.403Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19694/src/github.com/elastic/beats/Elastic-Agent-x-pack-Windows
[2020-07-07T10:58:08.549Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19694/src/github.com/elastic/beats/Filebeat-Mac-OS-X
[2020-07-07T10:58:08.695Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19694/src/github.com/elastic/beats/Auditbeat-x-pack-Mac-OS-X
[2020-07-07T10:58:08.842Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19694/src/github.com/elastic/beats/Metricbeat-OSS-Unit-tests
[2020-07-07T10:58:08.989Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19694/src/github.com/elastic/beats/Auditbeat-oss-Windows
[2020-07-07T10:58:09.138Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19694/src/github.com/elastic/beats/Metricbeat-Mac-OS-X
[2020-07-07T10:58:09.289Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19694/src/github.com/elastic/beats/Auditbeat-x-pack
[2020-07-07T10:58:09.442Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19694/src/github.com/elastic/beats/Winlogbeat-Windows-x-pack
[2020-07-07T10:58:09.591Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19694/src/github.com/elastic/beats/Filebeat-x-pack-Windows
[2020-07-07T10:58:09.739Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19694/src/github.com/elastic/beats/Heartbeat-oss
[2020-07-07T10:58:09.891Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19694/src/github.com/elastic/beats/Auditbeat-x-pack-Windows
[2020-07-07T10:58:10.041Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19694/src/github.com/elastic/beats/Metricbeat-crosscompile
[2020-07-07T10:58:10.191Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19694/src/github.com/elastic/beats/Filebeat-Windows
[2020-07-07T10:58:10.343Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19694/src/github.com/elastic/beats/Libbeat-x-pack
[2020-07-07T10:58:10.495Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19694/src/github.com/elastic/beats/Auditbeat-oss-Linux
[2020-07-07T10:58:10.643Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19694/src/github.com/elastic/beats/Metricbeat-Windows
[2020-07-07T10:58:10.792Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19694/src/github.com/elastic/beats/Metricbeat-x-pack-Windows
[2020-07-07T10:58:10.945Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19694/src/github.com/elastic/beats/Winlogbeat-Windows
[2020-07-07T10:58:11.096Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19694/src/github.com/elastic/beats/Heartbeat-Mac-OS-X
[2020-07-07T10:58:11.248Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19694/src/github.com/elastic/beats/Packetbeat-oss
[2020-07-07T10:58:11.401Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19694/src/github.com/elastic/beats/Metricbeat-x-pack-Mac-OS-X
[2020-07-07T10:58:11.552Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19694/src/github.com/elastic/beats/Generators-Beat-Linux
[2020-07-07T10:58:11.775Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19694/src/github.com/elastic/beats/Heartbeat-Windows
[2020-07-07T10:58:11.926Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19694/src/github.com/elastic/beats/Metricbeat-Python-integration-tests
[2020-07-07T10:58:12.077Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19694/src/github.com/elastic/beats/Libbeat-oss
[2020-07-07T10:58:12.229Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19694/src/github.com/elastic/beats/Functionbeat-Mac-OS-X-x-pack
[2020-07-07T10:58:12.383Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19694/src/github.com/elastic/beats/Generators-Metricbeat-Mac-OS-X
[2020-07-07T10:58:12.532Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19694/src/github.com/elastic/beats/Generators-Beat-Mac-OS-X
[2020-07-07T10:58:12.679Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19694/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests
[2020-07-07T10:58:12.828Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19694/src/github.com/elastic/beats/Metricbeat-x-pack
[2020-07-07T10:58:12.978Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19694/src/github.com/elastic/beats/Functionbeat-Windows
[2020-07-07T10:58:13.128Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19694/src/github.com/elastic/beats/Filebeat-x-pack
[2020-07-07T10:58:13.280Z] Running in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19694/src/github.com/elastic/beats/Filebeat-oss
[2020-07-07T10:58:13.722Z] + cat
[2020-07-07T10:58:13.722Z] + /usr/local/bin/runbld ./runbld-script
[2020-07-07T10:58:13.722Z] Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding=UTF8
[2020-07-07T10:58:20.323Z] runbld>>> runbld started
[2020-07-07T10:58:20.323Z] runbld>>> 1.6.12/f45d832f2ba0aa2722ab4ec1fda8ad140f027f8b
[2020-07-07T10:58:22.241Z] runbld>>> The following profiles matched the job 'Beats/beats-beats-mbp/PR-19694' in order of occurrence in the config (last value wins).
[2020-07-07T10:58:23.626Z] runbld>>> Debug logging enabled.
[2020-07-07T10:58:23.626Z] runbld>>> Storing result
[2020-07-07T10:58:23.887Z] runbld>>> Store result: created {:total 2, :successful 2, :failed 0} 1
[2020-07-07T10:58:23.887Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200707105823-D93DFC86
[2020-07-07T10:58:23.887Z] runbld>>> Adding system facts.
[2020-07-07T10:58:24.831Z] runbld>>> Adding vcs info for the latest commit:  095ad9fe9e788e7b7f53a223ba3bf852f8496ba1
[2020-07-07T10:58:24.831Z] runbld>>> >>>>>>>>>>>> SCRIPT EXECUTION BEGIN >>>>>>>>>>>>
[2020-07-07T10:58:24.831Z] runbld>>> Adding /usr/lib/jvm/java-8-openjdk-amd64/bin to the path.
[2020-07-07T10:58:24.831Z] Processing JUnit reports with runbld...
[2020-07-07T10:58:24.831Z] + echo 'Processing JUnit reports with runbld...'
[2020-07-07T10:58:25.093Z] runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<
[2020-07-07T10:58:25.093Z] runbld>>> DURATION: 31ms
[2020-07-07T10:58:25.093Z] runbld>>> STDOUT: 40 bytes
[2020-07-07T10:58:25.093Z] runbld>>> STDERR: 49 bytes
[2020-07-07T10:58:25.093Z] runbld>>> WRAPPED PROCESS: SUCCESS (0)
[2020-07-07T10:58:25.093Z] runbld>>> Searching for build metadata in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19694/src/github.com/elastic/beats
[2020-07-07T10:58:26.487Z] runbld>>> Storing build metadata: 
[2020-07-07T10:58:26.487Z] runbld>>> Adding test report.
[2020-07-07T10:58:26.487Z] runbld>>> Searching for junit test output files with the pattern: TEST-.*\.xml$ in: /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19694/src/github.com/elastic/beats
[2020-07-07T10:58:27.060Z] runbld>>> Found 112 test output files
[2020-07-07T10:58:27.633Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19694/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-istio.xml
[2020-07-07T10:58:27.634Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19694/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-tomcat.xml
[2020-07-07T10:58:27.634Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19694/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-iis.xml
[2020-07-07T10:58:27.634Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19694/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-openmetrics.xml
[2020-07-07T10:58:27.634Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19694/src/github.com/elastic/beats/Metricbeat-x-pack/x-pack/metricbeat/build/TEST-go-integration-activemq.xml
[2020-07-07T10:58:29.023Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19694/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests/metricbeat/build/TEST-go-integration-graphite.xml
[2020-07-07T10:58:29.023Z] runbld>>> No testsuite node found in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19694/src/github.com/elastic/beats/Metricbeat-OSS-Integration-tests/metricbeat/build/TEST-go-integration-windows.xml
[2020-07-07T10:58:29.285Z] runbld>>> Test output logs contained: Errors: 0 Failures: 0 Tests: 10343 Skipped: 1338
[2020-07-07T10:58:29.547Z] runbld>>> Storing result
[2020-07-07T10:58:29.547Z] runbld>>> FAILURES: 0
[2020-07-07T10:58:29.808Z] runbld>>> Store result: updated {:total 2, :successful 2, :failed 0} 2
[2020-07-07T10:58:29.808Z] runbld>>> BUILD: https://c150076387b5421f9154dfbf536e5c60.us-west1.gcp.cloud.es.io:9243/build-1587637540455/t/20200707105823-D93DFC86
[2020-07-07T10:58:29.808Z] runbld>>> Email notification disabled by environment variable.
[2020-07-07T10:58:29.808Z] runbld>>> Slack notification disabled by environment variable.
[2020-07-07T10:58:35.440Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_beats-beats-mbp_PR-19694
[2020-07-07T10:58:35.606Z] [INFO] getVaultSecret: Getting secrets
[2020-07-07T10:58:35.692Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-07-07T10:58:36.836Z] + chmod 755 generate-build-data.sh
[2020-07-07T10:58:36.836Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19694/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19694/runs/1 FAILURE 7407887
[2020-07-07T10:58:36.836Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19694/runs/1/steps/?limit=10000 -o steps-info.json
[2020-07-07T10:58:38.697Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19694/runs/1/tests/?status=FAILED -o tests-errors.json
[2020-07-07T10:58:38.948Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/beats-beats-mbp/PR-19694/runs/1/log/ -o pipeline-log.txt

@ChrsMark ChrsMark merged commit 5db114f into elastic:7.x Jul 7, 2020
@zube zube bot removed the [zube]: Done label Oct 13, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jsoriano jsoriano jsoriano approved these changes

Assignees

@ChrsMark ChrsMark

Labels
backport Team:Platforms Label for the Integrations - Platforms team
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@ChrsMark @elasticmachine @jsoriano