[BUG] LiveMetrics showing incorrect values

[TimothyMothra]

Library name and version

Azure.Monitor.OpenTelemetry.AspNetCore

Describe the bug

LiveMetrics is showing incorrect values.
Specifically investigating Requests per second.

Expected behavior

N/A

Actual behavior

N/A

Reproduction Steps

Here I've written a console app that starts a web application.
The default endpoint is invoked in a loop, and every second we print out the number of requests.

    internal class Program
    {
        public static async Task Main(string[] args)
        {
            var builder = WebApplication.CreateBuilder(args);
            builder.Logging.ClearProviders();
            string connectionString = "...";
            builder.Services.AddOpenTelemetry().UseAzureMonitor(x => x.ConnectionString = connectionString);
            var app = builder.Build();

            app.MapGet("/", () => "Hello World!");

            var hostTask = app.RunAsync();

            using var httpClient = new HttpClient();
            var stopwatch = Stopwatch.StartNew();
            int callCount = 0;

            while (true)
            {
                var response = await httpClient.GetStringAsync("http://localhost:5000");
                callCount++;

                if (stopwatch.ElapsedMilliseconds >= 1000)
                {
                    Console.WriteLine($"{DateTime.Now:HH:mm:ss}: {callCount} calls");
                    callCount = 0;
                    stopwatch.Restart();
                }
            }
        }
    }

Console output shows between 30-50 requests per second.

The Metrics blade is showing an average of 45 requests per second.

LiveMetrics dashboard is showing 3 requests per second

A fiddler trace is also showing that the SDK is sending 3 requests per second

Environment

No response

[TimothyMothra]

Seems like our internal buffer had a hard cap of 20. Meaning once the buffer was full, we would drop telemetry (from LiveMetrics).

azure-sdk-for-net/sdk/monitor/Azure.Monitor.OpenTelemetry.AspNetCore/src/LiveMetrics/DataCollection/DocumentBuffer.cs

Line 15 in 43fc021

private readonly int _capacity = 20;

When i changed this to an arbitrary high value, Live Metrics shows the correct value

[TimothyMothra]

FYI; The bug is related to the capacity limit on our internal buffer. The capacity was set too low.
We're discussing either increasing the capacity or removing it altogether.
We need to take some more time to review the performance implications of either change.

[vRune4]

Live metrics:

Meanwhile, what was running during this time was a load test that averaged almost 1500 requests per second.

This is a .net 9 asp.net project that benchmarks a single query running against the northwind database.

builder.Services.AddOpenTelemetry().UseAzureMonitor(opt => { opt.Credential = credential; opt.ConnectionString = appInsightsConnectionString; });

Plus I'm worried that the CPU display is also displaying a too optimistic value. At some point, in an actual service of ours, if I really hit it hard the CPU chart will rarely go above 25-40% before all reporting simply halts.