[BUG] Flooded with logs "The limit of 1000 metricsets has been reached, no new metricsets will be created"

[tomap]

APM Agent version

The version of the Elastic.Apm 1.11

Environment

Operating system and version: both windows & docker linux

.NET Framework/Core name and version : .Net 5.0.8

Application Target Framework(s) (e.g. net461, netcoreapp3.1): ASP.Net.Sdk.Web

Describe the bug

Since the last update, we re flooded with log message "The limit of 1000 metricsets has been reached, no new metricsets will be created"

comming from

apm-agent-dotnet/src/Elastic.Apm/Metrics/MetricsProvider/BreakdownMetricsProvider.cs

Line 90 in 07a4484

_logger.Warning()

I don't understand the issue... how to fix it, ...

Here is our configuration:

configuration[ConfigConsts.KeyNames.LogLevel] = "Debug";
                configuration[ConfigConsts.KeyNames.Environment] = app.ApplicationServices.GetRequiredService<IHostEnvironment>().EnvironmentName;
                configuration[ConfigConsts.KeyNames.CentralConfig] = "false";
                configuration[ConfigConsts.KeyNames.CloudProvider] = ConfigConsts.SupportedValues.CloudProviderNone;
                configuration[ConfigConsts.KeyNames.GlobalLabels] = $"product={ApiTitle}";
                configuration[ConfigConsts.KeyNames.ServiceVersion] = VersionLazy.Value;
                configuration[ConfigConsts.KeyNames.ServiceName] = ApiTitle;
                configuration[ConfigConsts.KeyNames.SanitizeFieldNames] = "password,passwd,pwd,secret,*key,*token*,*session*,*credit*,*card*,set-cookie,*auth*,*Newrelic*";
                configuration[ConfigConsts.KeyNames.StackTraceLimit] = "0";
                configuration[ConfigConsts.KeyNames.DisableMetrics] = "*";
                configuration[ConfigConsts.KeyNames.UseElasticTraceparentHeader] = "false";
                configuration[ConfigConsts.KeyNames.TransactionSampleRate] = "1"; // No sampling

                app.UseElasticApm(configuration,
                    new HttpDiagnosticsSubscriber(),
                    new SqlClientDiagnosticSubscriber(),
                    new EfCoreDiagnosticsSubscriber(),
                    new AspNetCoreErrorDiagnosticsSubscriber(),
                    new GrpcClientDiagnosticSubscriber());
            }

[marcusjsford]

I too have this problem. Is there a way to Disable the BreakdownMetricsProvider?

It looks like a new BreakdownMetricsProvider is started if the Agent is enabled and regardless of whether the Metrics are disabled.

apm-agent-dotnet/src/Elastic.Apm/AgentComponents.cs

Line 60 in d6f210b

breakdownMetricsProvider ??= new BreakdownMetricsProvider(Logger);

Even if you disable the MetricsProvider in configuration
DisableMetrics: "*",
MetricsInterval: 0

It will still allow a call into CaptureTransaction.

apm-agent-dotnet/src/Elastic.Apm/Model/Transaction.cs

Line 517 in d6f210b

_breakdownMetricsProvider?.CaptureTransaction(this);

The BreakdownMetricsProvider has an IsEnabled property but this is only checked when adding it to the collection of MetricProviders for the purpose of collecting metrics.

apm-agent-dotnet/src/Elastic.Apm/Metrics/MetricsProvider/BreakdownMetricsProvider.cs

Line 39 in d6f210b

public bool IsEnabled(IReadOnlyList<WildcardMatcher> matchers) => !WildcardMatcher.IsAnyMatch(matchers, SpanSelfTime);

So Its possible to fill the private readonly List<MetricSet> _itemsToSend quite quickly which will start to log warnings.

apm-agent-dotnet/src/Elastic.Apm/Metrics/MetricsProvider/BreakdownMetricsProvider.cs

Lines 62 to 72 in d6f210b

	if (_itemsToSend.Count < 1000)
	_itemsToSend.Add(metricSet);
	else
	{
	if (loggedWarning) continue;
	_logger.Warning()
	?.Log(
	"The limit of 1000 metricsets has been reached, no new metricsets will be created.");
	loggedWarning = true;
	}

The MetricsCollector makes the call into GetSamples If the MetricsProvider is added to the list of MetricsProviders.
It won't be added if you specify DisableMetrics : "*" in the config and so the _itemsToSend collection will never be cleared.

apm-agent-dotnet/src/Elastic.Apm/Metrics/MetricsProvider/BreakdownMetricsProvider.cs

Line 98 in d6f210b

public IEnumerable<MetricSet> GetSamples()

I've considered these options in the absence of a code change.

1. Set the ElasticApm:LogLevel to Error.
2. Increase the the MetricsCollection frequency by setting MetricsInterval to 1.

Is it possible to gate the call to CaptureTransaction with a check of the IsEnabled property or similar?

apm-agent-dotnet/src/Elastic.Apm/Model/Transaction.cs

Line 517 in d6f210b

_breakdownMetricsProvider?.CaptureTransaction(this);

[gregkalapos]

Thanks for reporting this @tomap and @marcusjsford.

I opened #1367 which will solve this problem. We'll still print a log, but it'll only happen in every metric collection round (once per 30sec by default) if you hit the limit.

I too have this problem. Is there a way to Disable the BreakdownMetricsProvider?

Yes, as an immediate workaround you can disable it by setting the DisableMetrics to span.self_time (or add it to the list if you already disabled some other metrics). With that you'll loose data form the Time spent by span type graph in Kibana under APM, but you'll get rid of the log spam. Once the fix is out you can enable it again.

Sorry about the trouble on this.

[gregkalapos]

Oh and another workaround that may help people: You could also lower the MetricsInterval. The issue here is that we only collect 1K metrics per collection round and the collection round is 30sec by default. If you have let's say 1K transaction with 100spans each within 30sec, then it'll be 1K*100 = 100K metric per collection, so you hit the limit. But you can lower the interval of the metrics collection, and with that you may come to an interval whee you don't hit the limit.

Obviously on the downside the agent will send data mer frequently to the APM Server.

[gregkalapos]

Fixed in #1367