feat(node | das | libs/header/sync): add total uptime node metrics + totalSampled das metrics + totalSynced sync metrics

[derrandz]

Overview

This PR introduces node uptime metrics + das total sampled headers metrics to support calculating the uptime index proposed by mustafa on the monitoring side.

This PR introduces a new module named Telemetry to support node related telemetry. This module can also host all general telemetry and observability that does not interest specific modules.

Changes

• Introduced uptime metrics for node under nodebuilder/node/uptime.go
• Introduced persistent uptime metrics using datastore to persist node start time
• Testing for uptime metrics persistence using the store
• Unit testing for uptime metrics
• Integration testing for uptime metrics
• e2e testing for uptime metrics

Checklist

• New and updated code has appropriate documentation
• New and updated code has new and/or updated testing
• Required CI checks are passing
• Visual proof for any user facing features like CLI or documentation updates
• Linked issues closed with keywords

Blocked By

PR: #1537

[codecov-commenter]

Codecov Report

Merging #1638 (e2795d7) into main (b51509d) will increase coverage by 1.12%.
The diff coverage is 71.42%.

@@            Coverage Diff             @@
##             main    #1638      +/-   ##
==========================================
+ Coverage   55.11%   56.24%   +1.12%     
==========================================
  Files         230      232       +2     
  Lines       15029    15151     +122     
==========================================
+ Hits         8283     8521     +238     
+ Misses       5862     5725     -137     
- Partials      884      905      +21     

Impacted Files	Coverage Δ
das/options.go	60.00% <ø> (ø)
libs/header/sync/sync.go	68.15% <0.00%> (-1.33%)	⬇️
share/availability/mocks/availability.go	59.25% <ø> (ø)
share/mocks/getter.go	45.45% <ø> (ø)
das/metrics.go	47.86% <48.57%> (+34.38%)	⬆️
das/checkpoint.go	90.00% <50.00%> (-10.00%)	⬇️
header/metrics.go	66.66% <50.00%> (+66.66%)	⬆️
libs/header/sync/metrics.go	66.66% <66.66%> (ø)
nodebuilder/node/metrics.go	79.31% <79.31%> (ø)
das/coordinator.go	88.88% <100.00%> (+0.65%)	⬆️
... and 12 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[derrandz]

We're successfully retrieving metrics at the moment, however we need to think about calculating the uptime index (from mustafa's proposal) somewhere either in node or in mojtaba's API, since Grafana's min function returns a min over a vector and not a minimum of two values.

Here's a screen short of the two values, green being totalSampled/network_head, and yellow being (time-nodeStartTs)/total_uptime_in_s

[renaynay]

just define the const in uptime.go

[derrandz]

@mojtaba-esk and I discussed the following comment and concluded to not store any metrics on store at the node side, and defer this requirement to the parties that require it (i.e: NodeLogger and Leaderboard backend)

[derrandz]

Here's a screenshot of the last changes that shows:

1. In green: the time factor of uptime: (total node runtime in seconds) / (time() - node start time ts)
2. In yellow: the work factor of uptime (total headers sampled) / network head

[vgonkivs]

Small nits regarding type conversion but other than that lgtm