Support metric_relabel_configs in distributor #3329

roidelapluie · 2020-10-12T23:08:19Z

Signed-off-by: Julien Pivotto roidelapluie@inuits.eu

What this PR does:
Support metric_relabel_configs in distributor

Which issue(s) this PR fixes:
Fixes #1507

Checklist

Tests updated
Documentation added
CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

csmarchbanks

🥳, I am excited for this!

Do you think it is worthwhile to allow this to be specified per user rather than just globally? I imagine a common use case will be to stop very high cardinality or high churn data from a specific tenant without disrupting data for all tenants.

drequena · 2020-10-13T03:35:57Z

CHANGELOG.md

@@ -49,6 +49,7 @@
 * [CHANGE] Increased default `-<prefix>.redis.timeout` from `100ms` to `500ms`. #3301
 * [FEATURE] Added support for shuffle-sharding queriers in the query-frontend. When configured (`-frontend.max-queriers-per-tenant` globally, or using per-tenant limit `max_queriers_per_tenant`), each tenants's requests will be handled by different set of queriers. #3113 #3257
 * [FEATURE] Query-frontend: added `compression` config to support results cache with compression. #3217
+* [FEATURE] Support `metric_relabel_confis` in distributor. #3329


configs instead of confis (just a simples typo) ;)

pstibrany

Can we move the configuration into Overrides? Then it can be per-user, and reloaded when changed on disk, without restarting Cortex. WDYT?

pstibrany · 2020-10-13T08:47:52Z

pkg/distributor/distributor.go

@@ -170,6 +171,9 @@ type Config struct {
 	// when true the distributor does not validate the label name, Cortex doesn't directly use
 	// this (and should never use it) but this feature is used by other projects built on top of it
 	SkipLabelNameValidation bool `yaml:"-"`
+
+	// Metrics relabeling
+	MetricRelabelConfigs []*relabel.Config `yaml:"metric_relabel_configs,omitempty" doc:"nocli|description=List of metric relabel configurations. Note that in most situations, it is more effective to use metrics relabeling directly in the Prometheus server, e.g. remote_write.write_relabel_configs."`


We should note that this is global, for ALL tenants.

roidelapluie · 2020-10-13T09:36:20Z

I was already thinking about #3318 (comment) when I set this up. WDYT? Should we have both?

pstibrany · 2020-10-13T09:48:47Z

I was already thinking about #3318 (comment) when I set this up. WDYT? Should we have both?

If by "both" you refer to global and per-tenant config, then Overrides / Limits already support that, and yes, I think it makes sense to have both. For example in cloud offering, we wouldn't use global one, but could use per-tenant config in specific cases. On the other hand, some people running Cortex internally may only need global one.

bboreham · 2020-10-13T11:27:01Z

Thanks! Any idea what the CPU/memory overhead is?

roidelapluie · 2020-10-13T11:30:37Z

That depends on the relabeling rules..

roidelapluie · 2020-10-13T15:56:59Z

I have moved it to limits

jtlisi

LGTM!

jtlisi · 2020-10-13T17:43:04Z

pkg/distributor/distributor.go

@@ -455,6 +456,11 @@ func (d *Distributor) Push(ctx context.Context, req *client.WriteRequest) (*clie
 			removeLabel(labelName, &ts.Labels)
 		}

+		if mrc := d.limits.MetricRelabelConfigs(userID); len(mrc) > 0 {


question: Is there any instrumentation or debug logging that could be added that would be helpful for operators/users when working with this feature?

IMHO the metrics distributor_received_samples_total and distributor_samples_in_total cover that already.

jtlisi · 2020-10-13T17:47:59Z

CHANGELOG.md

@@ -49,6 +49,7 @@
 * [CHANGE] Increased default `-<prefix>.redis.timeout` from `100ms` to `500ms`. #3301
 * [FEATURE] Added support for shuffle-sharding queriers in the query-frontend. When configured (`-frontend.max-queriers-per-tenant` globally, or using per-tenant limit `max_queriers_per_tenant`), each tenants's requests will be handled by different set of queriers. #3113 #3257
 * [FEATURE] Query-frontend: added `compression` config to support results cache with compression. #3217
+* [FEATURE] Support `metric_relabel_configs` in distributor. #3329


Suggested change

* [FEATURE] Support `metric_relabel_configs` in distributor. #3329

* [FEATURE] Added the support for applying a Prometheus metrics relabel config on series received by the distributor. The `metric_relabel_configs` field was added to the limits config. #3329

Thanks, I updated my comment

pstibrany · 2020-10-20T08:14:39Z

Thank you, this looks very good! Would it be possible to add small test showing parsing of relabeling YAML rules into per-user limits?

roidelapluie · 2020-10-20T12:03:55Z

Thank you, this looks very good! Would it be possible to add small test showing parsing of relabeling YAML rules into per-user limits?

Do we have tests like this already?

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>

pstibrany · 2020-10-20T13:19:17Z

Thank you, this looks very good! Would it be possible to add small test showing parsing of relabeling YAML rules into per-user limits?

Do we have tests like this already?

Until now structures we've put into overrides YAML file were pretty basic. I can only see TestLimitsLoadingFromYaml. Something similar in size but with relabel config to make sure the wiring is all correct would be good to have. WDYT?

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>

roidelapluie · 2020-10-20T21:36:13Z

Done :)

pracucci

Could we mark it experimental in docs/configuration/v1-guarantees.md, please?

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>

roidelapluie · 2020-10-26T17:32:13Z

done

pracucci

Thanks for working on this and sorry for the super late review (I would like days of 48h 😢 ). Looks flawless to me. I just left a nit, definitely not a blocker.

docs/configuration/config-file-reference.template

pstibrany

Thanks a lot Julien!

Co-authored-by: Marco Pracucci <marco@pracucci.com> Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

bboreham · 2021-03-15T18:11:05Z

I just used this feature in production to remove a high-cardinality metric; worked super well!

Even better if we had a metric to track how many samples got dropped this way - right now they disappear from cortex_distributor_received_samples_total and don't appear in cortex_discarded_samples_total.

pull-request-size bot added the size/M label Oct 12, 2020

roidelapluie force-pushed the rlcdistributor branch from 8f8ee23 to b2310ad Compare October 12, 2020 23:09

pull-request-size bot added size/L and removed size/M labels Oct 12, 2020

csmarchbanks reviewed Oct 13, 2020

View reviewed changes

drequena reviewed Oct 13, 2020

View reviewed changes

pstibrany reviewed Oct 13, 2020

View reviewed changes

jtlisi approved these changes Oct 13, 2020

View reviewed changes

jtlisi reviewed Oct 13, 2020

View reviewed changes

Support metric_relabel_configs in distributor

c0ccea9

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>

roidelapluie force-pushed the rlcdistributor branch from 96095ee to c0ccea9 Compare October 20, 2020 12:09

Add unmarshal test

8b657af

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>

pracucci reviewed Oct 26, 2020

View reviewed changes

Mark as experimental

f8e05bd

Signed-off-by: Julien Pivotto <roidelapluie@inuits.eu>

Merge branch 'master' into rlcdistributor

78a2a26

pracucci approved these changes Oct 26, 2020

View reviewed changes

docs/configuration/config-file-reference.template Outdated Show resolved Hide resolved

pracucci requested a review from pstibrany October 26, 2020 18:14

pstibrany approved these changes Oct 26, 2020

View reviewed changes

Update docs/configuration/config-file-reference.template

199e07f

Co-authored-by: Marco Pracucci <marco@pracucci.com> Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

pstibrany force-pushed the rlcdistributor branch from 7939d3b to 199e07f Compare October 26, 2020 21:08

Fix docs.

8a1e3f8

Signed-off-by: Peter Štibraný <peter.stibrany@grafana.com>

pstibrany merged commit 2abac33 into cortexproject:master Oct 26, 2020

bboreham mentioned this pull request Mar 15, 2021

Add a metric to track samples dropped in distributor by metric_relabel #3954

Closed

ghost mentioned this pull request Jun 14, 2021

metric_relabel_configs in cortex-helm-chart cortexproject/cortex-helm-chart#163

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support metric_relabel_configs in distributor #3329

Support metric_relabel_configs in distributor #3329

roidelapluie commented Oct 12, 2020 •

edited

Loading

csmarchbanks left a comment

drequena Oct 13, 2020

pstibrany left a comment

pstibrany Oct 13, 2020

roidelapluie commented Oct 13, 2020

pstibrany commented Oct 13, 2020

bboreham commented Oct 13, 2020

roidelapluie commented Oct 13, 2020

roidelapluie commented Oct 13, 2020

jtlisi left a comment

jtlisi Oct 13, 2020

roidelapluie Oct 13, 2020 •

edited

Loading

jtlisi Oct 13, 2020

roidelapluie Oct 14, 2020

pstibrany commented Oct 20, 2020

roidelapluie commented Oct 20, 2020

pstibrany commented Oct 20, 2020

roidelapluie commented Oct 20, 2020

pracucci left a comment

roidelapluie commented Oct 26, 2020

pracucci left a comment

pstibrany left a comment

bboreham commented Mar 15, 2021

	* [FEATURE] Support `metric_relabel_configs` in distributor. #3329
	* [FEATURE] Added the support for applying a Prometheus metrics relabel config on series received by the distributor. The `metric_relabel_configs` field was added to the limits config. #3329

Support metric_relabel_configs in distributor #3329

Support metric_relabel_configs in distributor #3329

Conversation

roidelapluie commented Oct 12, 2020 • edited Loading

csmarchbanks left a comment

Choose a reason for hiding this comment

drequena Oct 13, 2020

Choose a reason for hiding this comment

pstibrany left a comment

Choose a reason for hiding this comment

pstibrany Oct 13, 2020

Choose a reason for hiding this comment

roidelapluie commented Oct 13, 2020

pstibrany commented Oct 13, 2020

bboreham commented Oct 13, 2020

roidelapluie commented Oct 13, 2020

roidelapluie commented Oct 13, 2020

jtlisi left a comment

Choose a reason for hiding this comment

jtlisi Oct 13, 2020

Choose a reason for hiding this comment

roidelapluie Oct 13, 2020 • edited Loading

Choose a reason for hiding this comment

jtlisi Oct 13, 2020

Choose a reason for hiding this comment

roidelapluie Oct 14, 2020

Choose a reason for hiding this comment

pstibrany commented Oct 20, 2020

roidelapluie commented Oct 20, 2020

pstibrany commented Oct 20, 2020

roidelapluie commented Oct 20, 2020

pracucci left a comment

Choose a reason for hiding this comment

roidelapluie commented Oct 26, 2020

pracucci left a comment

Choose a reason for hiding this comment

pstibrany left a comment

Choose a reason for hiding this comment

bboreham commented Mar 15, 2021

roidelapluie commented Oct 12, 2020 •

edited

Loading

roidelapluie Oct 13, 2020 •

edited

Loading