Added container_labels parameter

[noxer]

Fixes #1263

[sparrc]

please remove the mention of the prometheus output, this is too specific for an option that is in fact general.

[sparrc]

you should have the container_labels array support glob matching using https://github.com/gobwas/glob.

If the user specifies multiple patterns in container_labels you can construct a list of matches similar to how we do filtering here: https://github.com/influxdata/telegraf/blob/master/internal/models/filter.go#L87

you should only compile the filter once.

[noxer]

How would you implement inserting placeholders with those filters?

[sparrc]

@noxer what do you mean by that?

[noxer]

@sparrc I'm inserting placeholders ("-") for labels that are in d.ContainerLabels but not in container.Labels: tags[l] = "-". Using glob would make that impossible (I can't insert something for "*xyz"). I would suggest inserting such filters as a separate parameter.

[sparrc]

hmm I see what you mean. I'd actually rather not merge this, those placeholder labels will appear in every output plugin but don't mean anything except in the prometheus case. I think that you need to come up with a solution that is specific to the prometheus output.

[noxer]

@sparrc This seems to be a problem specific to docker input + prometheus output, prometheus needs a fixed set of tags (per input plugin) and docker generates a variable set of tags (due to the container labels). At least in our setup docker is the only input plugin that does that.

[sparrc]

this is true for your particular use-case, but in reality it could be any of the more than 70 input plugins that generate variable numbers of tags.

[lightblu]

These placeholders will only appear for nonexistent labels which where specified in container_labels, if you do not specify container_labels you get completely the current behavior and no placeholders at all.

I am still not sure if it is generally ok for a plugin to provide inconsistent and arbitrary labels for a plugin/metric, because no other input plugin I used so far does this (mongo, rabbitmq, and all the default ones like mem, cpu, disk etc.).

I just found that what you propose may be possible with Telegrafs configuration (taginclude) alone, e.g. specifying taginclude = ['network', and all other ones that are not extracted from the container labels, because I need those] on the docker plugin, still it feels wrong for me that you have to specify this to not break an output plugin and to allow for inconsistent and arbitrary labels. Will try this now.

[noxer]

Turns out taginclude does exactly what we need, making this PR obsolete.

[sparrc]

@lightblu the labels are not "arbitrary", you are specifying them as docker labels 😑

as I said before, this is only a problem for prometheus. It doesn't make sense and it's very bad programming design to have 100 little workarounds in every single input plugin just for the sake of a single output plugin.

[lightblu]

I agree its fine if this is the general approach within Telegraf/Influx, but to me it looked like bad design to have inconsistent label sets on the same metric (biased by prometheus) and the docker plugin to be the only input plugin which shows this behaviour (although I am wrong with this it seems), if this is the general approach I'm sorry.
But the labels are arbitrary in the sense that you usually do not build up all your containers from scratch but use prebuilt ones from registries, where you are bound to the arbitrary labels the creators have already added (see examples in the issue), you cannot remove these when running containers but just add/overwrite.