Skip to content
This repository has been archived by the owner on Aug 27, 2024. It is now read-only.

docs/tutorials/visualizing-node-metrics: Update to Prometheus endpoint #461

Merged
merged 2 commits into from
Feb 19, 2020
Merged

docs/tutorials/visualizing-node-metrics: Update to Prometheus endpoint #461

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f5bfaeb
docs/tutorials/visualizing-node-metrics: Update to Prometheus endpoint
mxinden Feb 18, 2020
4ff1285
docs/tutorial/visualizing-node-metrics: Remove commit ref
mxinden Feb 19, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
148 changes: 95 additions & 53 deletions docs/tutorials/visualizing-node-metrics.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,94 +2,136 @@
title: "Visualizing Node Metrics"
---

Recent versions of Substrate record metrics, such as how many peers your node
is connected to, how much memory your node is using, etc. To visualize these
metrics, you can use a tool called [Grafana](https://grafana.com/).
Recent versions of Substrate expose metrics, such as how many peers your node is
connected to, how much memory your node is using, etc. To visualize these
metrics, you can use tools like [Prometheus](https://prometheus.io/) and
[Grafana](https://grafana.com/).

## Step 1: Install and run Grafana
> Note: In the past Substrate exposed a Grafana JSON endpoint directly. This has
> been replaced with a Prometheus metric endpoint.

If you're on
macOS, the easiest way to do that is via [Homebrew](https://brew.sh/):
A possible architecture could look like:

```bash
brew install grafana
```
+-----------+ +-------------+ +---------+
| Substrate | | Prometheus | | Grafana |
+-----------+ +-------------+ +---------+
| -----------------\ | |
| | Every 1 minute |-| |
| |----------------| | |
| | |
| GET current metric values | |
|<---------------------------------| |
| | |
| `substrate_peers_count 5` | |
|--------------------------------->| |
| | --------------------------------------------------------------------\ |
| |-| Save metric value with corresponding time stamp in local database | |
| | |-------------------------------------------------------------------| |
| | -------------------------------\ |
| | | Every time user opens graphs |-|
| | |------------------------------| |
| | |
| | GET values of metric `substrate_peers_count` from time-X to time-Y |
| |<-------------------------------------------------------------------------|
| | |
| | `substrate_peers_count (1582023828, 5), (1582023847, 4) [...]` |
| |------------------------------------------------------------------------->|
| | |
Comment on lines +16 to +40
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe some of the designers can help us out with this? @goldsteinsveta

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is ultimately beautiful, but sure, I'll draw a scheme by the end of Friday

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While I do agree that my diagram has a lot of room for improvement, I also want to emphasize the fact that one has to view it in markdown mode, otherwise tabs will not align properly.

This is ultimately beautiful

❤️

I'll draw a scheme by the end of Friday

🎉 @goldsteinsveta let me know if you need more details. I would expect the content to change a bit in the future, thus in case it is hard to do this in a way that is not easy to adapt, it might make sense to hold off a bit longer?


Grafana runs via a server, which you can run via `brew` with:

```bash
brew services start grafana
```

Downloads for other platforms are available [here](https://grafana.com/grafana/download).
<details>
<summary>Reproduce diagram</summary>

## Step 2: Install the Grafana JSON DataSource plugin:
Go to: https://textart.io/sequence

We use a simple JSON interface to serve metrics. The
[Grafana JSON DataSource](https://github.com/simPod/grafana-json-datasource) plugin can be
installed with:
```
object Substrate Prometheus Grafana
note left of Prometheus: Every 1 minute
Prometheus->Substrate: GET current metric values
Substrate->Prometheus: `substrate_peers_count 5`
note right of Prometheus: Save metric value with corresponding time stamp in local database
note left of Grafana: Every time user opens graphs
Grafana->Prometheus: GET values of metric `substrate_peers_count` from time-X to time-Y
Prometheus->Grafana: `substrate_peers_count (1582023828, 5), (1582023847, 4) [...]`
```

```bash
grafana-cli plugins install simpod-json-datasource
```
</details>

## Step 3: Run your node

As long as you're running a version of Substrate at or after commit
[`d9ca975`](https://github.com/paritytech/substrate/commit/d9ca9750dba018463d59459a3ee1c03b71ea2d46),
a server serving metrics for Grafana will start on port `9955`. You can specify
this port with `--grafana-port <PORT>` and enable it to be accessed over a
network with `--grafana-external`.

## Step 4: Set Up Grafana
## Step 1: Run your node

We will not cover setting up and running Grafana in great detail - there's the
[Getting Started guide](https://grafana.com/docs/guides/getting_started/) for
that. Here are a few pointers though:
Substrate exposes an endpoint which serves metrics in the [Prometheus exposition
format](https://prometheus.io/docs/concepts/data_model/) available on port
`9615`. You can change the port with `--prometheus-port <PORT>` and enable it to
be accessed over an interface other than local host with
`--prometheus-external`.

1. On the 'Add data source' screen, select the JSON data source in the
'Others' section.
2. Set the url of the running servers (e.g.
`http://localhost:9955`)
> NOTE: just `locahost::<PORT>` won't work.
3. Click `Save & Test`.
```bash
./substrate
```

Grafana should ping the server and show that the data source is working:
## Step 2: Retrieve the metrics

![Data Source Config](/docs/assets/tutorials/grafana/datasource-config.png)
In a second terminal run:

Creating queries is a lot simpler than in other data sources. Simply click the
drop-down and select the metric you want to visualize:
```bash
curl localhost:9615/metrics
```

![Creating a query](/docs/assets/tutorials/grafana/metric-selection.png)
Which should return a similar output to:

## Step 5: Create Your Dashboard
```
# HELP substrate_block_height_number Height of the chain
# TYPE substrate_block_height_number gauge
substrate_block_height_number{status="best"} 12591
substrate_block_height_number{status="finalized"} 11776
substrate_block_height_number{status="sync_target"} 1236089
# HELP substrate_cpu_usage_percentage Node CPU usage
# TYPE substrate_cpu_usage_percentage gauge
substrate_cpu_usage_percentage 98.90908813476563
# HELP substrate_memory_usage_bytes Node memory usage
# TYPE substrate_memory_usage_bytes gauge
substrate_memory_usage_bytes 195504
# HELP substrate_network_per_sec_bytes Networking bytes per second
# TYPE substrate_network_per_sec_bytes gauge
substrate_network_per_sec_bytes{direction="download"} 4117
substrate_network_per_sec_bytes{direction="upload"} 437
# HELP substrate_peers_count Number of network gossip peers
# TYPE substrate_peers_count gauge
substrate_peers_count 3
# HELP substrate_ready_transactions_number Number of transactions in the ready queue
# TYPE substrate_ready_transactions_number gauge
substrate_ready_transactions_number 0
```

Once you've done all that, you should be able to make a pretty neat node
dashboard!
## Next Steps

![Node Dashboard](/docs/assets/tutorials/grafana/dashboard.png)
### Configure Prometheus to scrape your Substrate node

There's a lot of work still to be done on the metrics system. Hopefully, in the
future you'll be able to log metrics from anywhere in the node runtime to a
variety of databases, such as [Prometheus](https://prometheus.io/).
https://prometheus.io/docs/guides/node-exporter/#configuring-your-prometheus-instances

If you have any suggestions, feel free to file an issue on
[the substrate repository](https://github.com/paritytech/substrate).

## Next Steps
### Visualizing Prometheus metrics with Grafana

https://prometheus.io/docs/visualization/grafana/


### Learn More

- Learn how to [set up a private Substrate network](start-a-private-network).

### Examples

- Take a look at the Grafana dashboard configuration for the [Polkadot network](https://github.com/w3f/polkadot-dashboard).
- Take a look at the Grafana dashboard configuration for the [Polkadot
network](https://github.com/w3f/polkadot-dashboard).

### References

<!-- TODO: Update this to RUSTDOC link-->

- Visit the source code for [grafana-data-source](https://github.com/paritytech/substrate/tree/master/client/grafana-data-source).
- Visit the source code for
[grafana-data-source](https://github.com/paritytech/substrate/tree/master/client/grafana-data-source).