Persist prometheus storage

[ltitanb]

All the prometheus metrics are currently wiped when restarting commit boost. We should have the option to persist those to disk

[aimxhaisse]

Retention of metrics is usually handled by prometheus itself and commit-boost should probably start with no context/be stateless.

The usual way is to write prometheus queries around this to handle restarts, for example: if you want total queries done to relayers in the last 3 days and you only have a counter that is initialized at 0 at startup and commit-boost restarts 5 times, it's totally fine, you can do a sum(metric...) in prom ; in general better be stupid around metrics and work around them using promql.

This approach is handy especially if you work at scale: chances are you'll have N replicas of commit-boost that can all receive queries (to have HA, be able to upgrade commit-boost without missing blocks, etc), so each commit-boost is independent and has its own metrics and via PromQL you can still craft queries to have the global view.

[ltitanb]

closed by #49