Metrics: Counter expiring too soon

[divanikus]

I'm using a simple stage which increments counter on specific log lines (WARN, ERROR, INFO, etc). Recently I've noticed that some counters just dissappear after a while. That could be a problem for rare lines. For example ERROR lines are kind of rare, maybe one in several hours, so If I try to look at them in Grafana, I see just a couple of points during day, not even connected in a line.

    pipeline_stages:
      - regex:
          expression: '^(?P<time>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}) (?P<level>\w+) '
      - labels:
          level:
      - metrics:
          lines_total:
            type: Counter
            prefix: mc_log_
            source: level
            config:
              action: inc
      - match:
          selector: '{level="DEBUG"}'
          action: drop

I'm scraping promtail metrics from prometheus-server. But if I curl the /metrics endpoint of the promtail itself, I also do not see those counters after a while.

Did I miss any expiry option here? I believe it shouldn't expire so fast.

[cyriltovena]

I think this is mostly how explore will show metrics, Have you tried using this query into a dashboard ? you can even connect zero in the options if you look around.

[divanikus]

I mostly surprised why it is expiring so fast? As you might see, it expires within less than 30 minutes. I actually thought of the counter as a something long living, I don't think that it would eat too much resources. Could it be a tunable at least?

[cyriltovena]

Sorry I didn't realize this was a counter.

[cyriltovena]

I realize this is not well documented.

# Label values on metrics are dynamic which can cause exported metrics
# to go stale (for example when a stream stops receiving logs).
# To prevent unbounded growth of the /metrics endpoint any metrics which
# have not been updated within this time will be removed.
# Must be greater than or equal to '1s', if undefined default is '5m'
[max_idle_duration: <string>]

see https://github.com/grafana/loki/blob/master/docs/clients/promtail/stages/metrics.md

It applies to all metrics.

[cyriltovena]

You should try 6h may be here ? depend how variable you stream is ? from what I can see in the labels, even 30d would work.

[divanikus]

Yeah, it helped. Thanks