Skip to content

Commit

Permalink
Docs: Hint about potential high cardinality metrics (#5603)
Browse files Browse the repository at this point in the history
* Hint about potential high cardinality metrics

Add warning about potential high cardinality metrics due to the file path being included in some of the metrics. 
Depending on the setup, such config can result in a very large amount of label values, which is not recommended by Prometheus.
This can result in increased storage, slow down of queries, extra costs and so on.

Address what has been discussed in #5553.

* Reword observability.md

Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>

Co-authored-by: Karen Miller <84039272+KMiller-Grafana@users.noreply.github.com>
  • Loading branch information
m1keil and KMiller-Grafana authored May 20, 2022
1 parent 0075085 commit 5206e3a
Showing 1 changed file with 7 additions and 0 deletions.
7 changes: 7 additions & 0 deletions docs/sources/operations/observability.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,13 @@ exposed by Promtail at its `/metrics` endpoint. See Promtail's documentation on
An example Grafana dashboard was built by the community and is available as
dashboard [10004](https://grafana.com/dashboards/10004).

## Metrics cardinality

Some of the Loki observability metrics are emitted per tracked file (active), with the file path included in labels.
This increases the quantity of label values across the environment, thereby increasing cardinality. Best practices with Prometheus [labels](https://prometheus.io/docs/practices/naming/#labels) discourage increasing cardinality in this way.
Review your emitted metrics before scraping with Prometheus, and configure the scraping to avoid this issue.


## Mixins

The Loki repository has a [mixin](https://github.com/grafana/loki/blob/master/production/loki-mixin) that includes a
Expand Down

0 comments on commit 5206e3a

Please sign in to comment.