Skip to content

Commit

Permalink
chore: Apply review feedback
Browse files Browse the repository at this point in the history
  • Loading branch information
@nodebreaker0-0
nodebreaker0-0 committed Jan 20, 2020
1 parent 53c95de commit 23cb72e
Show file tree
Hide file tree
Showing 2 changed files with 7 additions and 78 deletions.
79 changes: 4 additions & 75 deletions utils/prometheus/README.md
View file
Original file line number Diff line number Diff line change
@@ -1,70 +1,28 @@
# Substrate Prometheus Node Exporter
# Substrate Prometheus Exporter
![grants](./photo_2019-12-13_16-32-53.jpg)
## Introduction

Prometheus is one of the most widely used monitoring tool for managing highly available services supported by [Cloud Native Computing Foundation](https://www.cncf.io/). By providing Prometheus metrics in Substrate, node operators can easily adopt widely used display/alert tools such as Grafana and Alertmanager without setting-up/operating external Prometheus push gateways (which is an antipattern in the first place) through RPC connections. Easy access to such monitoring tools will benefit parachain developers/operators and validators to have much higher availability of their services.

## Table of Contents

Hack Prometheus in Substrate
- Prometheus primer
- CLI Config
- Metrics Add

Metrics
- List of available metrics

Start Prometheus
- Install prometheus
- Edit Prometheus config file
- Start Prometheus

Start Grafana
- Install Grafana

## Metrics

substrate can report and serve the Prometheus metrics, which in their turn can be consumed by Prometheus collector(s).

This functionality is disabled by default.

To enable the Prometheus metrics, set in your cli command (--prometheus-addr,--prometheus-port ).
Metrics will be served under /metrics on 33333 port by default.

### List of available metrics


Consensus metrics, namespace: `substrate`

| **Name** | **Type** | **Tags** | **Description** |
| -------------------------------------- | --------- | -------- | --------------------------------------------------------------- |
| consensus_finality_block_height_number | IntGauge | | finality Height of the chain |
| consensus_best_block_height_number | IntGauge | | best Height of the chain |
| consensus_target_syn_number | IntGauge | | syning Height target number |
| consensus_num_txs | Gauge | | Number of transactions |
| consensus_node_memory | IntGauge | | Node's primary memory |
| consensus_node_cpu | IntGauge | | Node's cpu load |
| consensus_state_cache_size | IntGauge | | used state cache size |
| p2p_peers_number | IntGauge | | Number of peers node's connected to |
| p2p_peer_receive_bytes_per_sec | IntGauge | | number of bytes received from a given peer |
| p2p_peer_send_bytes_per_sec | IntGauge | | number of bytes sent to a given peer |
| Resource_receive_bytes_per_sec(Future) | IntGauge | | Operating System of bytes received |
| Resource_send_bytes_per_sec(Future) | IntGauge | | Operating System of bytes sent |
| Resource_cpu_use(Future) | IntGauge | | Operating System cpu load |
| Resource_disk_use(Future) | IntGauge | | Operating System disk use |
| validator_sign_prevote(Future) | IntGauge | validator addr | validator sign vote list |
| validator_sign_precommit(Future) | IntGauge | validator addr | validator sign commit list |
To enable the Prometheus metrics, set in your cli command (--prometheus_external, --prometheus-port ).
Metrics will be served under /metrics on 9955 port by default.


## Start Prometheus
### Install prometheus

https://prometheus.io/download/
```bash
wget <download URL>
tar -zxvf <prometheus tar file>
```

### Edit Prometheus config file

You can visit [prometheus.yml](https://github.com/prometheus/prometheus/blob/master/documentation/examples/prometheus.yml) to download default `prometheus.yml`.
Expand All @@ -74,7 +32,7 @@ Then edit `prometheus.yml` and add `jobs` :
```yaml
- job_name: kusama
static_configs:
- targets: ['localhost:33333']
- targets: ['localhost:9955']
labels:
instance: local-validator
```
Expand All @@ -83,35 +41,6 @@ Then edit `prometheus.yml` and add `jobs` :
### Start Prometheus
```bash
cd <prometheus file>
./prometheus
```

> The above example, you can save `prometheus.yml` at `~/volumes/prometheus` on your host machine

You can visit `http://localhost:9090` to see prometheus data.



## Start Grafana
### Install Grafana
https://grafana.com/docs/installation/debian/

```bash
apt-get install -y software-properties-common
sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
sudo apt-get update
sudo apt-get install grafana
sudo service grafana-server start
./prometheus
```

You can visit `http://localhost:3000/` to open grafana and create your own dashboard.

> Tips: The default username and password are both admin. We strongly recommend immediately changing your username & password after login
### Seting Grafana

Default ID:PW is admin.
6 changes: 3 additions & 3 deletions utils/prometheus/src/lib.rs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ pub async fn init_prometheus(mut prometheus_addr: SocketAddr) -> Result<(), Err
Err(err) => match err.kind() {
io::ErrorKind::AddrInUse | io::ErrorKind::PermissionDenied if prometheus_addr.port() != 0 => {
log::warn!(
"Prometheus server to already {} port.", prometheus_addr.port()
"Prometheus metric port {} already in use.", prometheus_addr.port()
);
prometheus_addr.set_port(0);
continue;
Expand All @@ -112,12 +112,12 @@ pub async fn init_prometheus(mut prometheus_addr: SocketAddr) -> Result<(), Err
}
});

let _server = Server::builder(Incoming(listener.incoming()))
let server = Server::builder(Incoming(listener.incoming()))
.executor(Executor)
.serve(service)
.boxed();

let result = _server.await.map_err(Into::into);
let result = server.await.map_err(Into::into);

result
}
Expand Down

0 comments on commit 23cb72e

Please sign in to comment.