Add per peer prometheus metrics and a TUI to view them. #1477

jgiszczak · 2023-08-02T22:34:20Z

This pull request adds Prometheus statistics for each net_plugin peer and a text-based user interface similar to top or ntop to render them in a terminal. Note: All Prometheus statistics have been renamed with a nodeos_ prefix. Labels within samples remain unprefixed.

The new metrics are labels under nodeos_p2p_connections and are as follows:

accepting_blocks
last_received_block
first_available_block
last_available_block
unique_first_block_count
latency
bytes_received
bytes_sent
connection_start_time

Latency is in nanoseconds. The accepting_blocks metric is 0 for False, 1 for True.

In addition an informational block about the node has been added. It contains static information.

The new node info metrics are available in a new sample named nodeos_info and are as follows:

server_version
chain_id
server_version_string
server_full_version_string
earliest_available_block_num

The text-based user interface is net-util.py. The following screenshot is on a Mainnet node:

The minimum number of terminal columns required for reasonable rendering is 106.

net-util is a work in progress. Its known deficiencies include:

inability to scroll all columns in the peer list
line wrap behavior which throws all row renders off if the terminal has insufficient columns
~~Hardcoded to read from localhost:8888~~

In addition, due to limitations in the current P2P protocol, some statistics aren't updated unless a peer disconnects and reconnects. These include:

first_available_block
last_available_block

Other statistics may not update at all:

latency

~~Finally, net-util will terminate if it encounters a network error.~~

Dependencies

net-util.py uses the Debian-packaged versions of prometheus_client and urwid.

sudo apt install python3-prometheus-client python3-urwid

Resolves #1292 and #1325

Add custom logging level TRACE via the mechanism provided rather than reaching into the library's internal data structures. Actually use the logging level command line argument.

spoonincode · 2023-08-03T03:10:57Z

plugins/net_plugin/net_plugin.cpp

+            per_connection.latencies.push_back((*it)->get_peer_ping_time_ns());
+         }
+         else
+            fc_wlog(logger, "socket remote endpoint is not IPv4");


Doesn't net_plugin support ipv6 now? Why would this be a wlog?

Yes, net_plugin supports ipv6. The Prometheus integration also needs to support ipv6.

There are still comments in net_plugin saying ipv6 support is needed.

Please remove/update those comments.

spoonincode · 2023-08-03T03:18:01Z

programs/net-util/CMakeLists.txt

@@ -0,0 +1 @@
+configure_file(net-util.py net-util.py COPYONLY)


What is the level of support we'll be providing for this new tool: Is it more like a demo, or a fully supported application? Should it be installed in the .deb package? If so, would the python deps be required or optional? Should there be any documentation anywhere? Should there be some tests for the tool? (maybe there could be a -n 1 kind of mode like top, to make an easy way to validate the output?)

@stephenpdeos ^^ My understanding was this was a proof of concept.

I agree this is more of a PoC

Since a PoC, should it be put somewhere besides programs/net-util. Maybe tools/?

heifner

Unable to test net-util.py as it is returning an error for me.
Also all my endpoints are being reported as non-ipv4 even though they are ipv4.

plugins/prometheus_plugin/metrics.hpp

heifner · 2023-08-03T12:52:37Z

plugins/prometheus_plugin/metrics.hpp

@@ -70,36 +84,38 @@ struct catalog_type {


   catalog_type()
-       : http_request_counts(family<Counter>("http_requests_total", "number of HTTP requests"))
-       , p2p_connections(family<Gauge>("p2p_connections", "current number of connected p2p connections"))
+       : info(family<prometheus::Info>("nodeos", "static information about the server"))


I think this should be configurable via a command line option. Users run multiple nodeos and will likely want to provide unique identifiers. Not sure if that should also be prefixed on all the items like you are with "nodeos_".

Prometheus supports "relabeling". So the the Prometheus data collector will add the extra information required to distinguish between multiple nodes. "instance" (server + port) where the metrics are collected from is auto-added.

plugins/net_plugin/include/eosio/net_plugin/net_plugin.hpp

plugins/net_plugin/net_plugin.cpp

programs/net-util/net-util.py

Update endpoints in both constructors and when a connection is established.

Remove redundant logging level setup. Fix typo.

Remove stray debug log statement.

programs/net-util/net-util.py

Add error overlay when there's a problem connecting. Also log errors to the log file.

Remove obsolete comment regarding IPv6.

Add support for node info overlay showing all identifying information. Add missing reversed attribute for focus in some peer list columns.

plugins/net_plugin/include/eosio/net_plugin/net_plugin.hpp

plugins/net_plugin/net_plugin.cpp

plugins/prometheus_plugin/metrics.hpp

heifner · 2023-08-10T18:52:18Z

programs/net-util/CMakeLists.txt

@@ -0,0 +1 @@
+configure_file(net-util.py net-util.py COPYONLY)


Since a PoC, should it be put somewhere besides programs/net-util. Maybe tools/?

plugins/prometheus_plugin/metrics.hpp

Collect Prometheus metrics for each peer only if the update method exists. Rename a method for consistency. Rename a variable for cosmetic reasons.

…token.

Remove some unused and commented code. WIP multicolumned list in net-util.

plugins/net_plugin/net_plugin.cpp

tools/net-util.py

For traceability when Prometheus data is missing the address.

dimas1185 · 2023-08-15T20:10:20Z

tools/net-util.py

+        self.fields.update({k:v for k, v in zip(self.rightFieldLabels[1:], [labelToAttrName(e) for e in self.rightFieldLabels[1:]])})
+        self.fields.update({k:v for k, v in zip(self.infoFieldLabels, [labelToAttrName(e) for e in self.infoFieldLabels])})
+
+        parser = argparse.ArgumentParser(description='Terminal UI for monitoring and managing nodeos P2P connections',


it says for managing but I don't see any option other then monitoring. normally management means to alter in some specific way. So if it doesn't change nodeos in any way I suggest to remove that word it is just misleading

The intention was to allow the tool to use the net_api_plugin to disconnect peers. As that is not yet implemented, I have changed the wording.

dimas1185

see my comment in a python net util. minor one but still worth to change I guess

jgiszczak added 5 commits August 2, 2023 17:03

Add per peer prometheus metrics and a TUI to view them.

79c9817

Merge branch 'main' into prometheus-peer-statistics

d13dfa5

Remove debug log statement.

727a0c5

Add net-util command line support for host and port.

fbb0cec

Add custom logging level TRACE via the mechanism provided rather than reaching into the library's internal data structures. Actually use the logging level command line argument.

Update prometheus metric names in plugin_http_api_test.

86c6ced

spoonincode reviewed Aug 3, 2023

View reviewed changes

heifner added the OCI Work exclusive to OCI team label Aug 3, 2023

heifner requested changes Aug 3, 2023

View reviewed changes

jgiszczak added 9 commits August 3, 2023 14:44

Make p2p_per_connection_metrics constructor explicit.

ed9eee6

Count unique received blocks in process_signed_block

a30d7c7

Update endpoints in both constructors and when a connection is established.

Rename construction_time to connection_time and update every connect

51c5f15

Make prometheus endpoint URL a variable at the top of the script.

d511095

Remove redundant logging level setup. Fix typo.

Reserve enough space only for current connection count.

ef80f69

Rename late_initialize to update_prometheus_info.

507efb7

Remove stray debug log statement.

Change default host from localhost to 127.0.0.1.

1e14aa0

Check for non-200 status codes rather than only 404.

850a336

Add reported p2p_address to Prometheus metrics per peer.

7420c72

heifner reviewed Aug 3, 2023

View reviewed changes

programs/net-util/net-util.py Outdated Show resolved Hide resolved

jgiszczak added 5 commits August 4, 2023 17:27

Add refresh interval argument and default to the max possible.

2774dea

Add error overlay when there's a problem connecting. Also log errors to the log file.

Add support for last data received and sent timestamps on peers.

df3554d

Remove obsolete comment regarding IPv6.

Support IPv6 addresses.

7c8ebe4

Add support for node info overlay showing all identifying information. Add missing reversed attribute for focus in some peer list columns.

Convert peer metrics from struct of vectors to vector of structs.

fb00ebc

Merge branch 'main' into prometheus-peer-statistics

e1dacb9

heifner requested changes Aug 10, 2023

View reviewed changes

heifner reviewed Aug 10, 2023

View reviewed changes

plugins/prometheus_plugin/metrics.hpp Outdated Show resolved Hide resolved

jgiszczak added 4 commits August 10, 2023 14:56

Default initialization and const correctness.

98e1e39

Thread safety and collect connection time for inbound connections

690d12e

Merge branch 'main' into prometheus-peer-statistics

9ec5410

Retain chrono nanoseconds until the last possible moment.

ae87427

Collect Prometheus metrics for each peer only if the update method exists. Rename a method for consistency. Rename a variable for cosmetic reasons.

jgiszczak added 3 commits August 10, 2023 17:30

Relocate net-util to tools.

639faf2

Move accounting of bytes written to peer to async_write's completion …

9ff857c

…token.

Use connection ID as per peer Prometheus metric key.

6abba1a

Remove some unused and commented code. WIP multicolumned list in net-util.

heifner requested changes Aug 14, 2023

View reviewed changes

plugins/net_plugin/net_plugin.cpp Outdated Show resolved Hide resolved

tools/net-util.py Show resolved Hide resolved

jgiszczak added 7 commits August 14, 2023 14:24

Remove unnecessary retention of pointers to Prometheus gauges.

2754f02

Add logging when remote endpoint can't be retrieved.

ef44bfb

For traceability when Prometheus data is missing the address.

Use fc::unique_lock on an fc::mutex.

6b6ea38

Change over bandwidth field tracking to connection ID from IP.

09ea0cc

Merge branch 'main' into prometheus-peer-statistics

af76bb0

Adjust peer list column weightings.

e4aa07e

Merge branch 'main' into prometheus-peer-statistics

8914850

BenjaminGormanPMP requested review from dimas1185 and spoonincode and removed request for spoonincode August 14, 2023 21:03

heifner linked an issue Aug 15, 2023 that may be closed by this pull request

Terminal-based UI tool for managing and monitoring P2P #1325

Closed

Merge branch 'main' into prometheus-peer-statistics

e0deb07

dimas1185 reviewed Aug 15, 2023

View reviewed changes

dimas1185 approved these changes Aug 15, 2023

View reviewed changes

jgiszczak added 2 commits August 15, 2023 15:59

Update help preamble to better describe net-util's current capabilities.

1f1ea00

Merge branch 'main' into prometheus-peer-statistics

a0b3985

heifner approved these changes Aug 16, 2023

View reviewed changes

jgiszczak merged commit c3c59ed into main Aug 16, 2023

jgiszczak deleted the prometheus-peer-statistics branch August 16, 2023 17:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add per peer prometheus metrics and a TUI to view them. #1477

Add per peer prometheus metrics and a TUI to view them. #1477

jgiszczak commented Aug 2, 2023 •

edited

Loading

spoonincode Aug 3, 2023

heifner Aug 3, 2023 •

edited

Loading

jgiszczak Aug 3, 2023

heifner Aug 3, 2023

jgiszczak Aug 5, 2023

spoonincode Aug 3, 2023

heifner Aug 3, 2023

stephenpdeos Aug 3, 2023

heifner Aug 10, 2023

jgiszczak Aug 14, 2023

heifner left a comment

heifner Aug 3, 2023

matthewdarwin Aug 3, 2023

heifner Aug 10, 2023

dimas1185 Aug 15, 2023

jgiszczak Aug 15, 2023

dimas1185 left a comment

		@@ -0,0 +1 @@
		configure_file(net-util.py net-util.py COPYONLY)

Add per peer prometheus metrics and a TUI to view them. #1477

Add per peer prometheus metrics and a TUI to view them. #1477

Conversation

jgiszczak commented Aug 2, 2023 • edited Loading

Dependencies

Choose a reason for hiding this comment

heifner Aug 3, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

heifner left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dimas1185 left a comment

Choose a reason for hiding this comment

jgiszczak commented Aug 2, 2023 •

edited

Loading

heifner Aug 3, 2023 •

edited

Loading