OpenTelemetry with .NET

The tiniest example I could think of. Good as template.

Run

docker compose up
dotnet run --project Tel.Web

Grafana will be available (after a short while) on localhost:40300. Use admin as username and password. You can use JetBrains HTTP Client to simulate traffic by running the call defined in Tel.Web.http.

Features

• Microsoft.Extensions.Logging based logging without clutter on stdout
• Application startup logging. Before initializing OpenTelemetry
• OTEL_DIAGNOSTICS.json file included
• ASP .NET Instrumentation plus custom explicit tracing
• Grafana LGTM included as observability platform
• OpenTelemetry Collector Contrib. with configuration and self-monitoring included
• ASP .NET minimal APIs
• Source code commentary with links to reading resources
• TODO Exemplars
• TODO Samplers

Pictures

Human readable logs on stdout

Application traces

Application logs

Application metrics

OpenTelemetry Collector metrics

Side notes

⚠️ Do not use plain HTTP in production, it is too slow. Use gRPC when possible

⚠️ There are two OpenTelemetry Collectors: one baked in Grafana LGTM container and one configured explicity. This setup is whack! Good for demonstration purposes only! In production, use one OpenTelemetry Collector

Seemingly double metrics

You will see a job named opentelemetry-collector in Prometheus metrics. This exists because:

By default, the Collector generates basic metrics about itself and exposes them using the OpenTelemetry Go Prometheus exporter for scraping at http://127.0.0.1:8888/metrics. See here for mor information on default metrics

The Grafana LGTM distribution used in this example sets up scraping on localhost:8888 by default. See here the Grafana LGTM OpenTelemetry Collector configuration. The metrics for the OpenTelemetry Collector used within this example are under job name otelcol-contrib.

OpenTelemetry Configuration Schema

To better understand how OpenTelemetry service stanza can be configured browse the Go source code type definitions here.

OTEL_DIAGNOSTICS.json

Configures self-diagnostic for OpenTelemetry .NET SDK, the file is supposed to be full of NUL because it is pre-allocated. See here for further details

How do I change the labels that Loki assigns to my OTLP logs ?

See the official Loki documentation

Do I have the same concept for labels in Tempo, like Loki ?

No.