Is there a way to synchronously flush any remaining logs that have not been exported over OTLP?

[JakeDern]

Question

Is there a way to attempt to synchronously flush any remaining logs that have not be exported over OTLP?

Context

We're running into a scenario where we have a very short lived .NET app (sometimes less than 1 second) that sets up logging with an OTLP exporter, logs a couple of statements while working, and exits. We've noticed that most the time the logs are not exported over OTLP, but are present in the console. We want a reliable way to make sure that the logs have at least been attempted to be exported before the process exits.

Attempted solutions

• Setting otlpOptions.ExportProcessorType = ExportProcessorType.Simple did not work
• Adding a sleep statement for 30 seconds before the process exits in addition to the above does result in the logs being exported, but is not reliable in general or ideal
• We've noticed there is a ForceFlush functionality described in the OTel documentation and also in the dotnet sdk here, but couldn't figure out exactly how to call it. It seems to show up in this "Unshipped Public Api" doc, so perhaps that's why.

Would appreciate any pointers you could provide, thanks!

[cijothomas]

Disposing the loggerfactory should have taken care of this.
https://github.com/open-telemetry/opentelemetry-dotnet/blob/main/docs/logs/getting-started-console/Program.cs#L39-L41

Since you mentioned even SimpleExportProcessor did not work, I am not sure if this is even related to lack of flush!. Could you share a minimal repro of the code you are using?

[BasJ93]

I'm running in to a similar problem. A short lived .net app constructed using the Host.CreateDefaultBuilder() call and adding OTLP exporting.

• Setting otlpOptions.ExportProcessorType = ExportProcessorType.Simple did not work: I tried this too, but the docs also point out this is only applicable for traces. Not for logs.

Any pointers would be appreciated.

[cijothomas]

@BasJ93 See my reply above. Need to see a minimal code setup to give any other suggestions/fixes.

[vishweshbankwar]

otlpOptions.ExportProcessorType = ExportProcessorType.Simple

This ^ is not the correct way to configure processor options for otlp log exporter. Setting that option has no effect on otlp log export processor and it will default to Batch

See https://github.com/open-telemetry/opentelemetry-dotnet/tree/main/src/OpenTelemetry.Exporter.OpenTelemetryProtocol#logrecordexportprocessoroptions

OR

opentelemetry-dotnet/examples/Console/TestLogs.cs

Line 91 in 9211cee

opt.AddOtlpExporter((exporterOptions, processorOptions) =>

[JakeDern]

@cijothomas Did not realize that the force flush was tied to logger factory disposal! After playing around with that we were able to get something working, appreciate the pointer.

@vishweshbankwar I see this is documented, but it's a bit confusing given that it sits right next to all the other options like otlpOptions.Endpoint which does affect logs. Not to mention it's settable in the scope of LoggerFactory.Create, so we wouldn't have any reason to think it doesn't apply. Do you have any insight into why it's set up this way? I want to be thinking about this in the right way going forward to avoid similar config hiccups.

[vishweshbankwar]

@JakeDern I understand the confusion and we plan to fix that in near future. See #4527

[JakeDern]

@vishweshbankwar Makes sense, thanks!

[cijothomas]

@cijothomas Did not realize that the force flush was tied to logger factory disposal! After playing around with that we were able to get something working, appreciate the pointer.

ForceFlush and disposal are slightly different things, but both achieve the goal of flushing out in-memory things!
ForceFlush() can be repeatedly called, but Dispose() (just like shutdown), is one-time only.

[JakeDern]

@cijothomas Thanks for clarifying that, just for my education is there an example of calling ForceFlush() somewhere? This is a feature that may come in handy for us later.

[cijothomas]

@cijothomas Thanks for clarifying that, just for my education is there an example of calling ForceFlush() somewhere? This is a feature that may come in handy for us later.

Almost all unit-tests in the repo (using inmemoryexporters) use ForceFlush() to avoid waiting for metrics to be exported.
Outside of that I can imagine scenario where you emitted some important telemetry (say, an exception telemetry), and want to ensure it gets exported right away, forceflush() is a good option to ensure that. Shutdown() cannot be used in that case, as that'll shutdown the entire pipeline for good!

[BasJ93]

@cijothomas I see you are talking about metrics again. I can't seem to find a public api entry that allows for calling ForceFlush() on the logging pipeline. Could you clarify?

[cijothomas]

@cijothomas I see you are talking about metrics again. I can't seem to find a public api entry that allows for calling ForceFlush() on the logging pipeline. Could you clarify?

I used metrics merely as an example only! The concept is same for traces and logs.
In you use case, are you trying to ensure logs are exporter before the app is exiting? In that case, simply disposing the LoggerFactory is sufficient (If using hosted applications like asp.net core, this should be done automatically at host shutdown.)

[CodeBlanch]

FYI if you are using a prerelease SDK version with the LoggerProvider experimental API you can call ForceFlush directly. Like this:

var appBuilder = WebApplication.CreateBuilder(args);

appBuilder.Logging.AddOpenTelemetry(builder => builder.AddOtlpExporter());

var app = appBuilder.Build();

var loggerProvider = app.Services.GetRequiredService<LoggerProvider>();

app.Run();

loggerProvider.ForceFlush();

That being said, it should not be necessary. Just make sure you dispose your host!

var appBuilder = WebApplication.CreateBuilder(args);

appBuilder.Logging.AddOpenTelemetry(builder => builder.AddOtlpExporter());

using var app = appBuilder.Build(); // <- Added "using" to this

app.Run();

Disposing the host should dispose the LoggerFactory which should cause a flush.

[BasJ93]

That helped! Just not when the System.CommandLine package as it disposes the host before running the command that was invoked.

The loggers still work, but OpenTelemetry is already disposed and therefore does not export anything.

Anyway, turns out this is not an OpenTelemetry bug for me!