Metricbeat fails to collect docker metadata if configured with Docker API

[aviratna]

Metricbeat version: 7.1.1

We have configured Metricbeat with remote Docker API and able to get the docker stats.

However to get additional details related to container (e.g docker label), we updated the metricbeat.yml "add_docker_metadata" configuration to point to Docker API instead of unix socket to fetch additional details.

Issue: Metricbeat service will fail with panic error.

Error Message:
FATAL [metricbeat] instance/beat.go:155 Failed due to panic. {"panic": "runtime error: invalid memory address or nil pointer dereference"

Expected: Metricbeat should be able to read additional details which are available in remote docker API when docker metadata is configured instead of unix socket. Remote Docker API has additional details available like Docker Label, IP Address, Port etc.

Can you please let me know if I am missing some configuration or if its bug.

If add docker metadata is not supported currently with docker API, can you please help how I can get the docker label for each container in cpu, memory metricset as per document.

https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-metricset-docker-cpu.html

I have also raised a question on forum ,waiting for reply.

https://discuss.elastic.co/t/metricbeat-docker-module-container-labels-are-missing-for-docker-api/185123

@malikkal

[fearful-symmetry]

@aviratna Can you share the metricbeat config file you used?

[aviratna]

@fearful-symmetry Thanks for response.
Please find the details below.

Metricbeat Configuration:

Docker Metadata:

Note: We are using VMWare Integrated Containers. There is no docker daemon or Docker Engine running on Host. It provides Virtual Container Host(VCH) which exposes the Remote Docker API Endpoint. Its similar to AWS Firecracker project.

I did some test for VCH API (Remote Docker API):

1. Get List Container API:
   https://VCH Docker API:2376/containers/json
   It returns the list of all containers running however other details like Docker Lable, networking details etc are missing. I have logged issue, they are going to fix it.

2. Docker Inspect Container API:
   https://VCH Docker API:2376/containers/"container id"/json
   It returns the details related to container id, docker label, network details etc.

I wanted to check which API is used by Metricbeat to fetch the container details and update the Metricset ?

If Metricbeat is using the Get List Container API, I hope once VCH API is fixed to return the docker label as well, Metricset will get updated.

And for Docker Metadata error:
As we are using remote docker API from Virtual Container Host (Host), there is no local docker engine running.
We would like to check if Docker Metadata configuration supports the Remote Docker API or needs Unix Socket and docker engine running locally ?

@malikkal

[fearful-symmetry]

networking details etc are missing

@aviratna This is the fundamental cause of the metadata issue. The metadata handler expects the networking data to be there, and when it's not, it panics. It looks like the VIC issue is being tracked here: vmware/vic#8564

[fearful-symmetry]

@aviratna Just to be clear, are you also running into issues with other docker metricsets? It looks like a few are using the /containers/json API.

[aviratna]

@fearful-symmetry Thanks for the confirmation on API.
Yes, issue is for other Metricset as well. Docker Label and networking data is missing. Once vmware/vic#8564 I think it will solve this issue.

[fearful-symmetry]

We have a fix in now to fix the panic, which I believe is only happening in docker_add_metadata. This missing data will have to wait until we get an upstream fix.

[fearful-symmetry]

Closing this for now as we have changes in place to prevent the panics, and upstream will have to implement fixes to actually add the data.