Prometheus middleware generates new metrics for every 404

[brietaylor]

Every time a user requests a unique URL that returns 404, the Prometheus middleware generates new metrics. This can easily generate millions of timeseries, overloading a Prometheus server or triggering other limits.

Minimal reproducer: https://gist.github.com/brietaylor/c5aa76df6a2f3fa3a8e14e4f8250cab1

Debugging:

I was able to trace this back through the code somewhat. The code that generates these URLs ultimately traces back to router.go:Find() in Echo. That code initially sets Context.path to the request path, but on all success paths, it overrides Context.path to the route path. I couldn't find any way for middlewares to detect which was done, though, so this might need to be an upstream fix.

https://github.com/labstack/echo/blob/master/router.go#L527

[lammel]

We have been using our own metrics with a helper to discover the actual used route (the one configured in echo, not the actual path, so the list is not unbound) for a few years now.

Your proposed changes have been merged already with labstack/echo#2385
So this will be resolved with the next minor release.

[aldas]

Should be fixed. Please update Echo to latest version and this library to latest.

[therealak12]

Hi.

Can you confirm the issue is resolved? @brietaylor

I'm using echo v4.14.4 and echo-contrib v0.15.0. I still see new metrics for each new path with a 404 status code.
The solution is to make this assignment conditional.

I propose #111 as a fix.