[Advanced] How to enable self diagnostics for ApplicationInsights.Kubernetes

[DEPRECATED] PLEASE REFER TO: Self Diagnostics for updated document.

This is a draft page for how to enable logging for ApplicationInsights.Kubernetes in ASP.NET Core application.

Enable self-diagnostics

We understand ApplicationInsights.Kubernetes is far from perfect. Enabling self-diagnostics will help find issues easier. Please feel free to open issues and we will be glad to look into them.

To enable self-diagnostics, configure the console logging to the level of Debug while setting up the WebHost like this:

Add the configuration to your appsettings.json - "Microsoft.ApplicationInsights.Kubernetes": "Trace"

{
  "Logging": {
    "LogLevel": {
      "Default": "Warning",
      "Microsoft.ApplicationInsights.Kubernetes": "Trace"
    }
  }
}

When self-diagnostics is enabled, you will see logging being written into the pod logs. You may watch it in the Kubernetes Dashboard like it below:

Or you can fetch the log by calling kubectl logs <PodName> [ContainerName]. For example:

kubectl logs x-u-service-20450933-b61w2 x-webapi

A similar log shall be output to the terminal.

What happened when self-diagnostics is enabled

Gets the ILoggerFactory

Firstly, WebHostBuilder.ConfigureLogging(f => f.AddConsole(LogLevel.Trace)) is called, the Console output will be added as providers to ILoggerFactory. Secondly, services.EnableKubernetes() is called to initialize the ApplicationInsights.Kubernetes.

In that second step, ApplicationInsights.Kubernetes will fetch the existing ILoggerFactory:

        public static IServiceCollection EnableKubernetes(this IServiceCollection services, TimeSpan? timeout = null)
        {
            // ...
            // Fetch the logger factory
            ILoggerFactory loggerFactory = (ILoggerFactory)services.FirstOrDefault(
                s => s.ServiceType == typeof(ILoggerFactory))?.ImplementationInstance;
        }

Create ILogger using the ILoggerFactory

Now that ApplicaitonInsights.Kubernetes has a handle on the loggerFactory, it is easy to infer that we use loggerFactory to create ILogger objects:

ILogger logger = loggerFactory?.CreateLogger("K8sEnvInitializer");

or

ILogger logger = loggerFactory?.CreateLogger<K8sEnvironment>();

Consume ILogger:

ILogger provides methods to log messages in different levels:

logger?.LogDebug(Invariant($"Getting container status of container-id: {settings.ContainerId}"));

or

logger?.LogError(Invariant($"Kubernetes info is not available within given time of {timeout.TotalMilliseconds} ms."));

As you can pretty much tell now, there's no magic happening. These are how the self-diagnostics is enabled for ApplicationInsights.Kubernetes.

Caveats

During the process of building up the self-diagnostics, there are several caveats we encountered. We put it down here, hoping it will help for troubleshooting.

• ILoggerFactory provider needs to be configured early

There are 2 common places to configure providers for ILoggerFactory: In Main() method or in Configure() method of the Startup class. To have complete logs for self-diagnostics, configure of the providers need to happen in Main() method. It would be too later otherwise.

• Handle ILoggerFactory or ILogger is null within ApplicationInsights.Kubernetes

To write code for ApplicationInsights.Kubernetes, remember there is no guarantee that ILoggerFactory will be provided and thus both ILoggerFactory and ILogger objects could be none. They are, in fact, supposed to be null most of the time when users are not interested in debugging into ApplicationInsights.Kubernetes.