Thunderdome: Grafana Integration

[iand]

What Is It?

Send metrics and traces from Thunderdome experiments to Grafana cloud

Why Are We Doing It?

Integrating with Grafana gives us visibility of the performance of the gateways under test and gives us tools to analyse and compare results. Kubo exports dozens of metrics that report the health and operation of its subsystems such as bitswap and blockstore. We also need to see the results of the experiment from the users point of view such a the number of failed requests, the time to first byte and the overall response time. We chose Grafana cloud since we use it for most of our other infrastructure monitoring but other users could send to private Grafana instances.

Notes

Appropriate config for grafana agent to run as a sidecar alongside the gateway process. Pushing metrics, logs and tracing data to hosted Grafana with a dashboard to show the result of the experiments. Use dealgood’s vantage point to send clientside metrics and traces to Grafana

Project overview is on Notion

Tasks

• deploy grafana-agent sidecar for metrics gathering in ECS probe-lab/thunderdome#13
• Deploy grafana-agent sidecar for dealgood during ECS experiments probe-lab/thunderdome#21
• Enable trace gathering via grafana-agent in ECS experiments probe-lab/thunderdome#16
• Add OTEL tracing to dealgood probe-lab/thunderdome#18
• Support propagation of trace context headers in http gateway ipfs/kubo#9168
• Add prom metrics to dealgood probe-lab/thunderdome#17
• Dash in Grafana for ECS experiments  probe-lab/thunderdome#15
• Deploy headless dealgood during ECS experiments probe-lab/thunderdome#24
• Create and share demo of this phase

[JesseXie]

@iand please link the Create and share demo of this phase to #23

[iand]

Propogation of tracing headers in Kubo is waiting for stewards to review