Grafana integration

[expenses]

See #814. This PR adds a HTTP server that (in a currently very bare-bones way) conforms to the grafana-json-datasource API.

This PR probably has a fair bit to go before it can be merged. Some notes:

• Currently, all metrics are stored as f32s, which is probably fine (although a bit awkward) as we're serializing into json anyway.
• The server is currently turned on by default, but it might be a good idea to put it behind a flag instead.
• I haven't yet implemented the table query response type, but I'm not sure if this is absolutely needed.
• I also haven't bothered with the annotations API, not sure about that one either.
• At the moment, the metrics grow indefinitely. This is definitely something that needs changing, but I'm not sure when a cleanup should occur.
• Ideally, we want to switch to a small, embedded timeseries database.

[expenses]

Ok, I think I've gotten things to a good-enough place now. Ideally, we'd use a database like Akumuli in the same way that we use RocksDB, but there are no rust bindings for that (I'm looking into it, no promises) and it's probably overkill.

Grafana doesn't seem to complain that the tables mode doesn't return a table, so that's probably a non-issue.

I've added a future that removed metrics that are over a week old that is ran every day, which seems reasonable for the time being.

[gavofyork]

@expenses please update the lock file. in the future i'd recommend you work in branches off this repo so that stuff like this can be done by anyone else without having to open a new PR.

[expenses]

@gavofyork done :^)

[tomaka]

i'd recommend you work in branches off this repo so that stuff like this can be done by anyone else without having to open a new PR.

For what it's worth, GitHub allows contributors to edit branches corresponding to PRs, even if they are outside of the repo.

[expenses]

Also, I'm not a part of the parity github organization yet 😉😉

[gavofyork]

[gavofyork]

Full test suite passed on build server.

[gavofyork]

i'd recommend you work in branches off this repo so that stuff like this can be done by anyone else without having to open a new PR.

For what it's worth, GitHub allows contributors to edit branches corresponding to PRs, even if they are outside of the repo.

that has never worked for me. can you explain how?

[tomaka]

that has never worked for me. can you explain how?

Hmmm
The option is enabled by default but people can disable it, that's maybe why

[bkchr]

For me it also never worked.

"You can set commit permissions when you first create a pull request from a fork. For more information, see "Creating a pull request from a fork." Additionally, you can modify an existing pull request to let repository maintainers make commits to your branch."

Most people probably have disabled it.

[expenses]

Oh, I was assuming it was enabled on this PR (as I thought that was the default) but it's not. Sorry 😅

[expenses]

[Hyung-bharvest]

@expenses @gavofyork

Hi. We are implementing a Prometheus exporter runtime in substrate and connecting into grafana server. I am afraid it is a duplicated job with almost same objective.

I am curious why you don't use already well known open source program "prometheus", but implemented subset of its feature by hand?

Our repository is here
https://github.com/nodebreaker-carlsagan/substrate/tree/prometheus_v0.1/client/prometheus

Thank you.

[expenses]

@dlguddus I decided not to use Prometheus because it's overkill if you just want to record metrics. For most use-cases, the advantages of Prometheus such as persistent metrics and being able to store metrics from multiple nodes are not needed. I believe that we should be aiming for good defaults in Substrate, and using a simple in-memory database is a better default than needing Prometheus.

That said, it would be great if we could support both. With #4187 we could do this, or if you setup your Prometheus observer to use the existing record_metrics macro, that would be great!

[Hyung-bharvest]

As a validator, we are running about 30 nodes. We expect it would be more than 100 next year. Definitely we need generalized data import from our monitoring center. This is our common usecase, as a validator because we are not only validating polkadot.

As you can see in our implementation, prometheus does not store any data in local PC, but just realtime passing the data to monitoring(grafana) server. So I expect it is as light as your implementation which stores and clear data in local memory.

Do I understand the specs correctly?

[Hyung-bharvest]

And, I think #4187 is definitely a must go work because it can minimize complexity of codebase for extracting data for various data exporter such as telemetry, prometheus, grafana-api, rpc, lcd, and more.

We will communicate there to contribute and discuss. Thanks for inviting!