feat: add pending connections count to metrics

[2color]

Title

Description

Notes & open questions

Change checklist

• I have performed a self-review of my own code
• I have made corresponding changes to the documentation if necessary (this includes comments as well)
• I have added tests that prove my fix is effective or that my feature works

[achingbrain]

Given that this value can change rapidly between scrapes, do you think this might be better off as a counter rather than a gauge?

It should probably also have the _total suffix - see Metric and Label naming.

[2color]

Counters can't go down. So we can't use them here since pending requests can go both up and down.

https://prometheus.io/docs/tutorials/understanding_metric_types/#counter

[achingbrain]

Counters can't go down

This is true - but when graphing them, we can use a rate() function to show their change over time. This way we don't miss spikes between scrapes.

In Graphana you add a Range Function and select Rate:

[2color]

To pick between counter and gauge, there is a simple rule of thumb: if the value can go down, it is a gauge.
Counters can only go up (and reset, such as when a process restarts). They are useful for accumulating the number of events, or the amount of something at each event. For example, the total number of HTTP requests, or the total number of bytes sent in HTTP requests. Raw counters are rarely useful. Use the rate() function to get the per-second rate at which they are increasing. https://prometheus.io/docs/practices/instrumentation/#counter-vs-gauge-summary-vs-histogram

If at t(0) we have 100 pending connections and at t(1) we have 80 pending connections, how can capture that with a counter?

This way we don't miss spikes between scrapes.

I think catching spikes is a matter of the scrape interval. We could add a counter to track the total incoming connections (prior to upgrade unlike libp2p_connection_manager_connections which only counts upgraded connections), but that would still not necessarily help catch spikes between scrapes.

[wemeetagain]

Are pending connections only inbound?

We could use a histogram or summary for the # of pending connections observed when a new inbound connection is triggered.

We can also have a histogram/summary to track time from pending to fully connected.

[achingbrain]

How about adding the pending connections count to the existing inbound/outbound connections metric as a label?

Then it'd graph with those automatically which might make things easier to observe? This is similar to the "inbound unnegotiated" and "outbound unnegotiated" stream metrics.

[wemeetagain]

How about adding the pending connections count to the existing inbound/outbound connections metric as a label?

Sounds good to me from a consistency perspective.

[2color]

LGTM with the latest changes