Ruler: Recording Rules

[dannykopping]

What this PR does / why we need it:
This PR adds Prometheus-style recording rules to Loki.

Recording rules are evaluated in the Ruler component. All recording rules are metric queries evaluated against the configured store, and the resulting samples are sent to any system that implements a Prometheus remote-write receiver; such systems include:

• Prometheus (>=v2.25.0 with --enable-feature=remote-write-receiver)
• Cortex
• Thanos

Which issue(s) this PR fixes:
Fixes #3765

Special notes for your reviewer:
All rule groups are associated to a tenant. To ensure isolation, each rule group gets assigned its own remote-write appender, and each appender keeps its own FIFO queue of samples to write to the remote-write target. If the target cannot receive samples (e.g. service is down), the samples need to be kept in memory until the target is available again.

I've created an EvictingQueue to this end; ultimately we can't store an infinite number of samples, so this queue will evict the oldest samples when it reaches its defined capacity (globally/per-tenant configurable).

I've opted for this approach rather than a full WAL implementation for now due to its simplicity; we can always deprecate this in favour of a WAL should the need arise in the future.

Metrics added:

• recording_rules_samples_evicted_total
• recording_rules_samples_queued_total
• recording_rules_samples_queued_current
• recording_rules_samples_queue_capacity
• recording_rules_remote_write_errors

This is obviously biasing towards newer rather than older samples; should a user want to disable this functionality (an argument could be made biasing towards older samples) and rather alert when a remote-write queue is full, that could be implemented fairly easily.

Checklist

• Documentation added
• Tests updated

[cyriltovena]

LGTM

[sandeepsukhani]

LGTM

[owen-d]

The remote write config itself isn't specifiable via flags, but I think that is OK. It's not our package (thus would be hard to guarantee we always have flags for every field) and doesn't implement this function already. We've been increasingly favoring file configuration to flags as well.

[owen-d]

I would love to see these instantiated rather than globally defined (example: https://github.com/grafana/loki/blob/main/pkg/ingester/metrics.go)

[owen-d]

This is a per tenant limit, but the metric is partitioned by group. Can we make this only partitioned by tenant?

[dannykopping]

Great point, will do.

[owen-d]

Suggested change

	// PrepareRequest takes the given queue and serialized it into a compressed
	// PrepareRequest takes the given queue and serializes it into a compressed

[owen-d]

nit: What you're doing here looks good, but we could be a bit safer in the case the entries somehow become significantly larger than the capacity (doesn't look possible in the current code).

This change would make Append work as expected regardless of the prior length

Suggested change

	q.evictOldest()
	q.onEvict()
	start := (len(q.entries)-q.Capacity()) + 1
	q.entries = append(q.entries[:0], q.entries[start:]...)

[owen-d]

It looks like this is unused

[owen-d]

I'm a bit hesitant to add this to the overrides because the overrides are expected to be updated during runtime. This would mean that all internal queue capacities would need to be adjusted at some regular interval (for instance if the queue capacity configuration was increased or decreased). I'm not opposed to either approach:

1. Removing this as a runtime configuration and making it a static config across all tenants in the ruler config block
2. Exposing it as a runtime config per tenant and re-adjusting queue capacities on the fly

(2) Would entail some more work, though.

Edit:
After thinking about this a bit more, I'd like to expose it as a per tenant config, primarily because
(A) While unlikely, some tenants may wish to evaluate rules significantly faster than others.
(2) It's a pain to move from a global configuration in a component block to a per tenant configuration in the limits because there are then 3 places where a config can be set: (the component, the base limits, per tenant overrides)

[owen-d]

All in all this looks great. I've left a few nits and one issue about runtime configurations not reloading which I'd like us to solve.

[owen-d]

Since this is ultimately called on every evaluation cycle in the Ruler, we can probably add a method WithCapacity which can mutate an appenders capacity and thus will be kept up to date with the overrides (would need a test)

Suggested change

	return appender
	return appender.WithCapacity(capacity)

[owen-d]

Nice work 🎉

[owen-d]

I don't think we need to make these private, but I did link you to ingesterMetrics as an example to follow which are private though 😆. In the long term, I think we should expose them consistently and I'd favor public Metric structs where possible.