docker-driver: not adding the container_name as documented

[rgl]

Describe the bug

the docker-driver is not adding the container_name as documented.

To Reproduce

Steps to reproduce the behavior:

1. Start Loki 2.2.1
2. Install the docker-driver 2.2.1 docker plugin
3. Set the following docker 20.10.7 daemon configuration:

   {
     "debug": true,
     "log-driver": "loki",
     "log-opts": {
       "loki-url": "http://10.10.0.2:3100/loki/api/v1/push",
       "loki-external-labels": "job=docker"
     }
   }

4. Query: logcli query --tail '{job="docker"}'

Got:

2021-06-07T07:37:57+01:00 {filename="/var/log/docker/ce6574ee7e2385634b16bc6b277dd4530e0d4826e37bf6eb9402df2cffc1f664/json.log", host="linuxkit-525400226029", job="docker", source="stdout"} hello docker Mon Jun 7 06:37:57 UTC 2021

Expected behavior

I expected the container_name label.

Workaround

"loki-external-labels": "job=docker,container_name={{.Name}}"

[kavirajk]

https://github.com/grafana/loki/blob/main/clients/cmd/docker-driver/config.go#L215-L218

After quick search, I see container_name is added only if loki-external-labels is empty! ( as default external labels).

If that's expected behaviour, docs should be updated!

[rgl]

Oh indeed, the default value of loki-external-labels is actually documented as container_name={{.Name}}.

But loki-external-labels only affects the container_name label, the other labels "defaults" do not come from this setting, which somehow, drove me off.

Maybe making the source of the labels explicit in the Labels section of the docs would help (at least I think it would have helped me).

[kavirajk]

@rgl Definitely document could have been more clear. you think this makes it clear?
#3814

[rgl]

I do.

But maybe, the documentation should also mention how can we modify each label. That is, whether we have to use loki-relabel-config or loki-pipeline-stages.

FWIW, this is what I'm going to comment in my loki log-driver configuration file:

    # NB loki-relabel-config is executed once per container.
    # NB loki-pipeline-stages is executed once per container log line.
    # NB the source label is set per container log line.

[rgl]

Thank you!