Add support for %%id%% and %%pid%% template variables

[sophaskins]

What does this PR do?

This PR adds two new template variables to the service-discovery check configs:

• pid the host PID namespace PID of a container processes' PID 0
• id the docker container id hash

Motivation

We have a few custom integrations that collect information from /proc. When we were running only one instance of a process per-host (pre-containers), it was easy to determine the correct pid to interrogate. Now that we're using Kubernetes and Docker, there is often many such processes per machine. These variables help us find the process we want.

Notes

This PR Is actually authored by @jnewland - I'm submitting it upstream because I've been working on our dd-agent packaging setup.

[hkaj]

Hi @sophaskins and @jnewland
Thanks for the contrib! I think I understand the use case for pid, what do you need the container ID for though? I'm a bit wary about adding it as a template variable as if it's used as a tag it could be very spammy for us.

[sophaskins]

I took a look at how we were using the container id, and it's for something that we can definitely just use the pid for (in our case, we were using the container id to identify if processes were part of the container, rather than doing something simpler like "get all the children of %%pid%%)

Shall I remove the id part of the PR?

[xvello]

I'm ok with keeping id, as it could be used in custom checks, but I'd rename it cid for disambiguation.

Indeed, someone using either pid or cid as a tag value would give our backend a hard time, but I think we should find a way to forbid using them (and even host btw) in the tag array. What do you think @hkaj?

[hkaj]

@sophaskins great, thanks for the details! Then yes, please remove the id part. @xvello I agree we should move in this direction, but we don't have this in place yet and this id variable is too tempting to use, it's bound to happen if we keep it. We can always add it back later once a tag filtering system is in place.

[sophaskins]

Fixed!