RFD 197 - Prometheus metrics guidelines

[hugoShaka]

Following the addition of the internal metric registry in Teleport, here's a mini RFD about how to add metrics in teleport.

Rendered version.

[hugoShaka]

Do we want this to be true for every executable? Should tbot metrics be prefixed by tbot?

[rosstimothy]

Are there any gotchas in converting existing metrics from the global registry to a local registry?

[rosstimothy]

Do we have any strategies for migrating legacy metrics which do not have a namespace? Should we register the same metric with and without the namespace?

[hugoShaka]

Currently I did not suggest a migration strategy because this is a pretty disruptive change. I don't like double-registering metrics very much because it increases cardinality.

I guess we could pull a metric breaking change in a major version.

[rosstimothy]

Would it be terrible to double register the same metric in major version N and announce that the non-namespaced variants will be removed in version N+2? That would give people ~8 months of notice to adjust to use the correct metrics.

[hugoShaka]

I think that's acceptable, maybe we can gate the behaviour behind a TELEPORT_UNSTABLE_ env var or something, so if someone has an issue with this they can disable the duplication and use only the new or old metrics.