Fix Prometheus metrics #773

rofafor · 2024-11-26T14:24:29Z

Issue

Ratelimit Prometheus metrics are kind of mess with the latest Envoy Gateway. The included patch tries to solve these issues:

~~Avoid repeating (optional) value if it's identical to key. This simplifies Prometheus labels by removing unnecessary duplication.~~
Statsd mapping configuration defaults to glob rules where dots are reserved characters. One needs to replace these and my approach was to hotfix only IPv4 addresses used in CIDR matches.
Simplify default statsd mappings configuration to use more generic (but slower) regex filters for fallbacks.

Ratelimit configuration

- name: LOG_LEVEL
  value: debug
- name: USE_STATSD
  value: "false"
- name: USE_PROMETHEUS
  value: "true"

Envoy configuration

"traffic": {
  "rateLimit": {
    "global": {
      "rules": [
        {
          "headerMatches": [],
          "cidrMatch": {
            "cidr": "0.0.0.0/0",
            "ip": "0.0.0.0",
            "maskLen": 0,
            "isIPv6": false,
            "distinct": true
          },
          "limit": {
            "requests": 1000,
            "unit": "Hour"
          }
        }
      ]
    }
  },
  "timeout": {
    "http": {
      "requestTimeout": "15s"
    }
  }
},

Envoy Ratelimit logs

envoy-ratelimit time="2024-11-26T14:03:45Z" level=debug msg="loading domain: envoy-gateway-system/internal/https"
envoy-ratelimit time="2024-11-26T14:03:45Z" level=debug msg="loading descriptor: key=envoy-gateway-system/internal/https.httproute/envoydemo/envoydemo/rule/0/match/0/envoydemo_domain_com_httproute/envoydemo/envoydemo/rule/0/match/0/envoydemo_domain_com"
envoy-ratelimit time="2024-11-26T14:03:45Z" level=debug msg="loading descriptor: key=envoy-gateway-system/internal/https.httproute/envoydemo/envoydemo/rule/0/match/0/envoydemo_domain_com_httproute/envoydemo/envoydemo/rule/0/match/0/envoydemo_domain_com.masked_remote_address_0.0.0.0/0"
envoy-ratelimit time="2024-11-26T14:03:45Z" level=debug msg="Creating stats for key: 'envoy-gateway-system/internal/https.httproute/envoydemo/envoydemo/rule/0/match/0/envoydemo_domain_com_httproute/envoydemo/envoydemo/rule/0/match/0/envoydemo_domain_com.masked_remote_address_0.0.0.0/0.remote_address'"
envoy-ratelimit time="2024-11-26T14:03:45Z" level=debug msg="loading descriptor: key=envoy-gateway-system/internal/https.httproute/envoydemo/envoydemo/rule/0/match/0/envoydemo_domain_com_httproute/envoydemo/envoydemo/rule/0/match/0/envoydemo_domain_com.masked_remote_address_0.0.0.0/0.remote_address ratelimit={requests_per_unit=1000, unit=HOUR, unlimited=false, shadow_mode=false}"

Ratelimit /metrics in current master

# HELP ratelimit_service_rate_limit_envoy_gateway_system_internal_https_httproute_envoydemo_envoydemo_rule_0_match_0_envoydemo_domain_com_httproute_envoydemo_envoydemo_rule_0_match_0_envoydemo_domain_com_masked_remote_address_0_0_0_0_0_remote_address_total_hits Metric autogenerated by statsd_exporter.
# TYPE ratelimit_service_rate_limit_envoy_gateway_system_internal_https_httproute_envoydemo_envoydemo_rule_0_match_0_envoydemo_domain_com_httproute_envoydemo_envoydemo_rule_0_match_0_envoydemo_domain_com_masked_remote_address_0_0_0_0_0_remote_address_total_hits counter
ratelimit_service_rate_limit_envoy_gateway_system_internal_https_httproute_envoydemo_envoydemo_rule_0_match_0_envoydemo_domain_com_httproute_envoydemo_envoydemo_rule_0_match_0_envoydemo_domain_com_masked_remote_address_0_0_0_0_0_remote_address_total_hits 1
# HELP ratelimit_service_rate_limit_envoy_gateway_system_internal_https_httproute_envoydemo_envoydemo_rule_0_match_0_envoydemo_domain_com_httproute_envoydemo_envoydemo_rule_0_match_0_envoydemo_domain_com_masked_remote_address_0_0_0_0_0_remote_address_within_limit Metric autogenerated by statsd_exporter.
# TYPE ratelimit_service_rate_limit_envoy_gateway_system_internal_https_httproute_envoydemo_envoydemo_rule_0_match_0_envoydemo_domain_com_httproute_envoydemo_envoydemo_rule_0_match_0_envoydemo_domain_com_masked_remote_address_0_0_0_0_0_remote_address_within_limit counter
ratelimit_service_rate_limit_envoy_gateway_system_internal_https_httproute_envoydemo_envoydemo_rule_0_match_0_envoydemo_domain_com_httproute_envoydemo_envoydemo_rule_0_match_0_envoydemo_domain_com_masked_remote_address_0_0_0_0_0_remote_address_within_limit 1

Ratelimit /metrics with this patch

# HELP ratelimit_service_rate_limit_total_hits Metric autogenerated by statsd_exporter.
# TYPE ratelimit_service_rate_limit_total_hits counter
ratelimit_service_rate_limit_total_hits{domain="envoy-gateway-system/internal/https",key1="httproute/envoydemo/envoydemo/rule/0/match/0/envoydemo_domain_com",key2="masked_remote_address_0_0_0_0/0"} 3
# HELP ratelimit_service_rate_limit_within_limit Metric autogenerated by statsd_exporter.
# TYPE ratelimit_service_rate_limit_within_limit counter
ratelimit_service_rate_limit_within_limit{domain="envoy-gateway-system/internal/https",key1="httproute/envoydemo/envoydemo/rule/0/match/0/envoydemo_domain_com",key2="masked_remote_address_0_0_0_0/0"} 3
# HELP ratelimit_service_response_time_seconds Metric autogenerated by statsd_exporter.

arkodg · 2024-11-26T14:49:33Z

cc @zirain

arkodg · 2024-11-26T17:30:07Z

dup of #716 ?

rofafor · 2024-11-26T18:28:19Z

dup of #716 ?

Yes, I missed that one. The chore: replace dots in ipv4 addresses with slashes is pretty much the same as the mentioned PR - just a bit more generic regexp and without tests.

zirain · 2024-11-27T00:22:45Z

I'd like to close my PR as this one is better.

src/config/config_impl.go

rofafor · 2024-12-05T07:01:14Z

Squashed the fixup commit for merging

arkodg · 2024-12-09T16:37:18Z

hey @rofafor can we split up the fix into 2 PRs, and use this PR for fixing the the SanitizeStatName ?
trying to figure out why EG generates an identical key, value pair downstream

rofafor · 2024-12-09T16:52:26Z

hey @rofafor can we split up the fix into 2 PRs, and use this PR for fixing the the SanitizeStatName ? trying to figure out why EG generates an identical key, value pair downstream

Done by dropping the first commit. Do you want me to create another PR for the dropped identical-key-value commit or do you track down the actual issue with a proper fix?

arkodg · 2024-12-09T17:03:36Z

hey @rofafor can we split up the fix into 2 PRs, and use this PR for fixing the the SanitizeStatName ? trying to figure out why EG generates an identical key, value pair downstream

Done by dropping the first commit. Do you want me to create another PR for the dropped identical-key-value commit or do you track down the actual issue with a proper fix?

would be great if you can create a GH issue in https://github.com/envoyproxy/gateway/issues for it

arkodg

LGTM thanks !

arkodg · 2024-12-09T17:56:19Z

cc @collin-lee @psbrar99

arkodg · 2025-01-04T00:59:03Z

@rofafor looks like a test is failing with the changes

rofafor · 2025-01-09T17:46:02Z

@rofafor looks like a test is failing with the changes

@arkodg Thanks for pointing it out. The error was due to fact that 4th capturing groups weren't optional. I can't figure out why I didn't spot this in December, but anyways now it works within my development setup (make docker_tests) and in my k8s setup with a self-built image.

zirain · 2025-01-09T22:47:10Z

src/utils/utilities.go

@@ -65,5 +66,9 @@ func MaskCredentialsInUrl(url string) string {
 // Remove invalid characters from the stat name.
 func SanitizeStatName(s string) string {
 	r := strings.NewReplacer(":", "_", "|", "_")
-	return r.Replace(s)
+	ipv4Pattern := `\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}`


would it be better to make this as const?

optimized it a bit by moving the static regex compilation out of the function

zirain

small nit

Signed-off-by: Rolf Ahrenberg <rolf.ahrenberg@saunalahti.fi>

zirain

@psbrar99 @collin-lee PTAL

zirain reviewed Nov 28, 2024

View reviewed changes

src/config/config_impl.go Outdated Show resolved Hide resolved

zirain approved these changes Dec 5, 2024

View reviewed changes

rofafor force-pushed the chore/prom branch from b3fecd4 to 0daf10d Compare December 5, 2024 06:58

rofafor force-pushed the chore/prom branch from 0daf10d to 5992d5c Compare December 9, 2024 16:48

arkodg approved these changes Dec 9, 2024

View reviewed changes

rofafor mentioned this pull request Dec 9, 2024

Simplify identical key-value pairs in descriptors #785

Closed

collin-lee previously approved these changes Dec 10, 2024

View reviewed changes

zirain mentioned this pull request Dec 23, 2024

remote address ratelimit metrics cannot be converted to prometheus format envoyproxy/gateway#4281

Closed

rofafor dismissed collin-lee’s stale review via 6717759 January 9, 2025 17:32

rofafor force-pushed the chore/prom branch from 5992d5c to 6717759 Compare January 9, 2025 17:32

zirain reviewed Jan 9, 2025

View reviewed changes

zirain approved these changes Jan 9, 2025

View reviewed changes

rofafor added 2 commits January 10, 2025 10:22

chore: replace dots in ipv4 addresses with slashes

36bec9f

Signed-off-by: Rolf Ahrenberg <rolf.ahrenberg@saunalahti.fi>

chore: tweak default mapper configuration for Prometheus

bfb6594

Signed-off-by: Rolf Ahrenberg <rolf.ahrenberg@saunalahti.fi>

rofafor force-pushed the chore/prom branch from 6717759 to bfb6594 Compare January 10, 2025 08:37

zirain approved these changes Jan 14, 2025

View reviewed changes

psbrar99 self-requested a review January 15, 2025 01:07

psbrar99 approved these changes Jan 15, 2025

View reviewed changes

psbrar99 merged commit 60d8e81 into envoyproxy:main Jan 15, 2025
6 checks passed

rofafor deleted the chore/prom branch January 15, 2025 07:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix Prometheus metrics #773

Fix Prometheus metrics #773

rofafor commented Nov 26, 2024 •

edited

Loading

arkodg commented Nov 26, 2024

arkodg commented Nov 26, 2024

rofafor commented Nov 26, 2024 •

edited

Loading

zirain commented Nov 27, 2024

rofafor commented Dec 5, 2024

arkodg commented Dec 9, 2024

rofafor commented Dec 9, 2024

arkodg commented Dec 9, 2024

arkodg left a comment

arkodg commented Dec 9, 2024

arkodg commented Jan 4, 2025

rofafor commented Jan 9, 2025

zirain Jan 9, 2025

rofafor Jan 10, 2025

zirain left a comment

zirain left a comment

Fix Prometheus metrics #773

Fix Prometheus metrics #773

Conversation

rofafor commented Nov 26, 2024 • edited Loading

Issue

Ratelimit configuration

Envoy configuration

Envoy Ratelimit logs

Ratelimit /metrics in current master

Ratelimit /metrics with this patch

arkodg commented Nov 26, 2024

arkodg commented Nov 26, 2024

rofafor commented Nov 26, 2024 • edited Loading

zirain commented Nov 27, 2024

rofafor commented Dec 5, 2024

arkodg commented Dec 9, 2024

rofafor commented Dec 9, 2024

arkodg commented Dec 9, 2024

arkodg left a comment

Choose a reason for hiding this comment

arkodg commented Dec 9, 2024

arkodg commented Jan 4, 2025

rofafor commented Jan 9, 2025

zirain Jan 9, 2025

Choose a reason for hiding this comment

rofafor Jan 10, 2025

Choose a reason for hiding this comment

zirain left a comment

Choose a reason for hiding this comment

zirain left a comment

Choose a reason for hiding this comment

rofafor commented Nov 26, 2024 •

edited

Loading

rofafor commented Nov 26, 2024 •

edited

Loading