Unable to run promitor pod on AKS, fails with CrashLoopBackOff

[Takshika]

I am trying to run promitor on a private AKS cluster, I downloaded promitor image from docker and pushed it to ACR. Further, I downloaded promitor helm chart repo and doing helm install from this folder.

➜ helm upgrade --install promitor-agent-scraper --values=metric-declaration.yaml .

However, the container is not getting started. Please find the logs below. can someone please help me troubleshoot this or if I am missing anything?

NAME READY STATUS RESTARTS AGE
promitor-agent-scraper-54d556d868-4t6hm 0/1 CrashLoopBackOff 6 9m18s

[06:52:34 INF] Booting up Promitor v2.0.0 - Thank you for using Promitor!
[06:52:34 INF] Using configuration folder '/config/'
[06:52:34 INF] Metrics declaration is using spec version v1
[06:52:34 FTL] Promitor Scraper Agent has encountered an unexpected error. Please open an issue at https://github.com/tomkerkhove/promitor/issues to let us know about it.
System.ArgumentException: [applicationId] cannot be Null, empty or white-space. (Parameter 'applicationId')
   at GuardNet.Guard.For[TException](Func`1 predicate, TException exception)
   at GuardNet.Guard.NotNullOrWhitespace[TException](String param, TException exception)
   at GuardNet.Guard.NotNullOrWhitespace(String param, String paramName, String message)
   at GuardNet.Guard.NotNullOrWhitespace(String param, String paramName)
   at Promitor.Integrations.AzureMonitor.AzureMonitorClient..ctor(AzureEnvironment azureCloud, String tenantId, String subscriptionId, String applicationId, String applicationSecret, IOptions`1 azureMonitorLoggingConfiguration, MetricSinkWriter metricSinkWriter, IRuntimeMetricsCollector metricsCollector, ILoggerFactory loggerFactory) in /src/Promitor.Integrations.AzureMonitor/AzureMonitorClient.cs:line 47
   at Promitor.Agents.Scraper.AzureMonitorClientFactory.CreateNewAzureMonitorClient(AzureEnvironment cloud, String tenantId, String subscriptionId, MetricSinkWriter metricSinkWriter, IRuntimeMetricsCollector metricsCollector, IConfiguration configuration, IOptions`1 azureMonitorLoggingConfiguration, ILoggerFactory loggerFactory) in /src/Promitor.Agents.Scraper/AzureMonitorClientFactory.cs:line 46
   at Promitor.Agents.Scraper.AzureMonitorClientFactory.CreateIfNotExists(AzureEnvironment cloud, String tenantId, String subscriptionId, MetricSinkWriter metricSinkWriter, IRuntimeMetricsCollector metricsCollector, IConfiguration configuration, IOptions`1 azureMonitorLoggingConfiguration, ILoggerFactory loggerFactory) in /src/Promitor.Agents.Scraper/AzureMonitorClientFactory.cs:line 37
   at Microsoft.Extensions.DependencyInjection.SchedulingExtensions.ScheduleResourceScraping(IAzureResourceDefinition resource, AzureMetadata azureMetadata, MetricDefinition metric, AzureMonitorClientFactory azureMonitorClientFactory, MetricSinkWriter metricSinkWriter, IRuntimeMetricsCollector runtimeMetricCollector, IConfiguration configuration, IOptions`1 azureMonitorLoggingConfiguration, ILoggerFactory loggerFactory, ILogger`1 logger, IServiceCollection services) in /src/Promitor.Agents.Scraper/SchedulingExtensions.cs:line 68
   at Microsoft.Extensions.DependencyInjection.SchedulingExtensions.ScheduleMetricScraping(IServiceCollection services) in /src/Promitor.Agents.Scraper/SchedulingExtensions.cs:line 34
   at Promitor.Agents.Scraper.Startup.ConfigureServices(IServiceCollection services) in /src/Promitor.Agents.Scraper/Startup.cs:line 55
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor, Boolean wrapExceptions)
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.InvokeCore(Object instance, IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.<>c__DisplayClass9_0.<Invoke>g__Startup|0(IServiceCollection serviceCollection)
   at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.Invoke(Object instance, IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.ConfigureServicesBuilder.<>c__DisplayClass8_0.<Build>b__0(IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.UseStartup(Type startupType, HostBuilderContext context, IServiceCollection services)
   at Microsoft.AspNetCore.Hosting.GenericWebHostBuilder.<>c__DisplayClass12_0.<UseStartup>b__0(HostBuilderContext context, IServiceCollection services)
   at Microsoft.Extensions.Hosting.HostBuilder.CreateServiceProvider()
   at Microsoft.Extensions.Hosting.HostBuilder.Build()
   at Promitor.Agents.Scraper.Program.Main(String[] args) in /src/Promitor.Agents.Scraper/Program.cs:line 34

[tomkerkhove]

Can you share your runtime configuration please? It looks like the application ID is not configured

[Takshika]

Thanks for the update. I will get this check.

[Takshika]

I changed authorization to azureAuthentication.identity.id and azureAuthentication.identity.key. In public subnet running pod with "containers.promitor.io/tomkerkhove/promitor-agent-scraper:2.5.0" works very well. However, if I use the same image in a private subnet, it won't work as we have disabled its internet acess. ( i cannot open ports).
Events:
Normal Pulling 7s (x3 over 42s) kubelet Pulling image "containers.promitor.io/tomkerkhove/promitor-agent-scraper:2.5.0" Warning Failed 7s (x3 over 41s) kubelet Failed to pull image "containers.promitor.io/tomkerkhove/promitor-agent-scraper:2.5.0": rpc error: code = Unknown desc = failed to pull and unpack image "containers.promitor.io/tomkerkhove/promitor-agent-scraper:2.5.0": failed to resolve reference "containers.promitor.io/tomkerkhove/promitor-agent-scraper:2.5.0": failed to do request: Head https://containers.promitor.io/v2/tomkerkhove/promitor-agent-scraper/manifests/2.5.0: EOF

How and where can I access https://containers.promitor.io/v2/tomkerkhove/promitor-agent-scraper/manifests/2.5.0 ? if AKS is private, what way I can refer to image?
One workaround which I thought was to download docker image (https://hub.docker.com/r/tomkerkhove/promitor-agent-scraper), and push it to acr and refer to image from acr, however that fails with empty application id as posted above.

runtime parameters-
azureAuthentication: identity: id: "<hidden>" key: "<hidden>" azureMetadata: tenantId: '<hidden>' subscriptionId: '<hidden>' resourceGroupName: promitor metricDefaults: aggregation: interval: 00:05:00 scraping: schedule: "* * * * *"

[tomkerkhove]

Please note that images from Docker Hub are no longer maintained and you should either use the URL above or GitHub Container Registry, learn more on #1444.

I have no experience with private AKS but if you cannot use internet access then you'll need to re-tag Promitor images and host them on Azure Container Registry indeed. If you do have internet access, then it should just work.

You can refer to the image to pull through via Helm parameter: https://github.com/promitor/charts/blob/main/promitor-agent-scraper/values.yaml#L10

PS: Are you willing to become a listed Promitor end-user on http://promitor.io/? If so, feel free to reach out to kerkhove.tom@gmail.com

[Takshika]

Thank you so much for your input. Using the correct image I was able to deploy promitor on the private cluster.

[tomkerkhove]

Great to hear!