[QUERY] DefaultAzureCredential authentication failed

[maldago]

Query/Question
I am not sure if this is a bug or not or even belongs here? The reason I am posting here is the stack trace from the kudu console. We are using key vault, secrets with config builders to update our web.config during pre-application start initialisation.

Environment:

• Name: Version used in current Azure AppService, KeyVault
• Azure AppService, targeting net472

We have followed a couple of how-tos :
https://docs.microsoft.com/en-us/azure/key-vault/vs-secure-secret-appsettings
https://docs.microsoft.com/en-us/azure/key-vault/managed-identity
https://docs.microsoft.com/en-us/azure/key-vault/overview-security#identity-and-access-management

We have set up the system assigned managed identity, and have given the managed identity RBAC role read to the keyvault, and applied the access policy to secrets (Get, List).

When looking at Azure Diagnostics on the key vault we see the Authentication event is successfult but then receive the httpStatusCode_d 401 and the following stack trace. Unfortunately, the we cannot get any more information that this.

If this is the wrong place, I hope someone can point us in the correct direction. We do have a support ticket open with Azure support, but because we cannot move forward with our test cycles, I am hoping to find some insight.

Here is the following stack trace:

The pre-application start initialization method Start on type EnvSettings.SettingsProcessor threw an exception with the following error message: An error occurred creating the configuration section handler for connectionStrings: Error in Configuration Builder 'KeyVault'::GetAllValues() (D:\home\site\wwwroot\web.config line 149). at 
System.Web.Compilation.BuildManager.InvokePreStartInitMethodsCore(ICollection`1 methods, Func`1 setHostingEnvironmentCultures) at 
System.Web.Compilation.BuildManager.InvokePreStartInitMethods(ICollection`1 methods) at 
System.Web.Compilation.BuildManager.CallPreStartInitMethods(String preStartInitListPath, Boolean& isRefAssemblyLoaded) at 
System.Web.Compilation.BuildManager.ExecutePreAppStart() at 
System.Web.Hosting.HostingEnvironment.Initialize(ApplicationManager appManager, IApplicationHost appHost, IConfigMapPathFactory configMapPathFactory, HostingEnvironmentParameters hostingParameters, PolicyLevel policyLevel, Exception appDomainCreationException) An error occurred creating the configuration section handler for connectionStrings: Error in Configuration Builder 'KeyVault'::GetAllValues() (D:\home\site\wwwroot\web.config line 149) at 
System.Configuration.BaseConfigurationRecord.EvaluateOne(String[] keys, SectionInput input, Boolean isTrusted, FactoryRecord factoryRecord, SectionRecord sectionRecord, Object parentResult) at 
System.Configuration.BaseConfigurationRecord.Evaluate(FactoryRecord factoryRecord, SectionRecord sectionRecord, Object parentResult, Boolean getLkg, Boolean getRuntimeObject, Object& result, Object& resultRuntimeObject) at 
System.Configuration.BaseConfigurationRecord.GetSectionRecursive(String configKey, Boolean getLkg, Boolean checkPermission, Boolean getRuntimeObject, Boolean requestIsHere, Object& result, Object& resultRuntimeObject) at 
System.Configuration.BaseConfigurationRecord.GetSection(String configKey) at 
System.Configuration.ConfigurationManager.GetSection(String sectionName) at 
System.Configuration.ConfigurationManager.get_ConnectionStrings() at 
EnvSettings.SettingsUtils.SetConnectionString(String name, String connString, String providerName) at 
EnvSettings.SettingsUtils.ApplyEnvironments() at 
EnvSettings.SettingsProcessor.Start() Error in Configuration Builder 'KeyVault'::GetAllValues() at 
Microsoft.Configuration.ConfigurationBuilders.KeyValueConfigBuilder.EnsureGreedyInitialized() at 
Microsoft.Configuration.ConfigurationBuilders.KeyValueConfigBuilder.ProcessConfigurationSection(ConfigurationSection configSection) at 
System.Configuration.ConfigurationBuilderChain.ProcessConfigurationSection(ConfigurationSection configSection) at 
System.Configuration.RuntimeConfigurationRecord.RuntimeConfigurationFactory.CreateSectionImpl(RuntimeConfigurationRecord configRecord, FactoryRecord factoryRecord, SectionRecord sectionRecord, SectionInput sectionInput, Object parentConfig, ConfigXmlReader reader) at 
System.Configuration.RuntimeConfigurationRecord.RuntimeConfigurationFactory.CreateSectionWithRestrictedPermissions(RuntimeConfigurationRecord configRecord, FactoryRecord factoryRecord, SectionRecord sectionRecord, SectionInput sectionInput, Object parentConfig, ConfigXmlReader reader) at 
System.Configuration.RuntimeConfigurationRecord.CreateSection(Boolean inputIsTrusted, FactoryRecord factoryRecord, SectionRecord sectionRecord, SectionInput sectionInput, Object parentConfig, ConfigXmlReader reader) at 
System.Configuration.BaseConfigurationRecord.CallCreateSection(Boolean inputIsTrusted, FactoryRecord factoryRecord, SectionRecord sectionRecord, SectionInput sectionInput, Object parentConfig, ConfigXmlReader reader) **DefaultAzureCredential authentication failed.** at 
Azure.Identity.DefaultAzureCredential.<GetTokenAsync>d__10.MoveNext() --- End of stack trace from previous location where exception was thrown --- at 
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at 
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at 
Azure.Identity.DefaultAzureCredential.GetToken(TokenRequestContext requestContext, CancellationToken cancellationToken) at 
Azure.Security.KeyVault.ChallengeBasedAuthenticationPolicy.<AuthenticateRequestAsync>d__9.MoveNext() --- End of stack trace from previous location where exception was thrown --- at 
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at 
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at 
Azure.Security.KeyVault.ChallengeBasedAuthenticationPolicy.<ProcessCoreAsync>d__8.MoveNext() --- End of stack trace from previous location where exception was thrown --- at 
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at 
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at 
Azure.Security.KeyVault.ChallengeBasedAuthenticationPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline) at 
Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory`1 pipeline) at 
Azure.Core.Pipeline.RetryPolicy.<ProcessAsync>d__11.MoveNext() --- End of stack trace from previous location where exception was thrown --- at 
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at 
Azure.Core.Pipeline.RetryPolicy.<ProcessAsync>d__11.MoveNext() --- End of stack trace from previous location where exception was thrown --- at 
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at 
System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at 
Azure.Core.Pipeline.RetryPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline) at 
Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory`1 pipeline) at 
Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline) at 
Azure.Core.Pipeline.HttpPipelinePolicy.ProcessNext(HttpMessage message, ReadOnlyMemory`1 pipeline) at 
Azure.Core.Pipeline.HttpPipelineSynchronousPolicy.Process(HttpMessage message, ReadOnlyMemory`1 pipeline) at 
Azure.Core.Pipeline.HttpPipeline.Send(HttpMessage message, CancellationToken cancellationToken) at 
Azure.Core.Pipeline.HttpPipeline.SendRequest(Request request, CancellationToken cancellationToken) at 
Azure.Security.KeyVault.KeyVaultPipeline.SendRequest(Request request, CancellationToken cancellationToken) at 
Azure.Security.KeyVault.KeyVaultPipeline.GetPage[T](Uri firstPageUri, String nextLink, Func`1 itemFactory, String operationName, CancellationToken cancellationToken) at 
Azure.Security.KeyVault.Secrets.SecretClient.<>c__DisplayClass13_0.<GetPropertiesOfSecrets>b__0(String nextLink) at 
Azure.Core.PageResponseEnumerator.FuncPageable`1.<AsPages>d__2.MoveNext() at 
Azure.Pageable`1.<GetEnumerator>d__8.MoveNext() at 
Microsoft.Configuration.ConfigurationBuilders.AzureKeyVaultConfigBuilder.GetAllKeys() at 
Microsoft.Configuration.ConfigurationBuilders.AzureKeyVaultConfigBuilder.LazyInitialize(String name, NameValueCollection config) at 
Microsoft.Configuration.ConfigurationBuilders.KeyValueConfigBuilder.EnsureInitialized() at 
Microsoft.Configuration.ConfigurationBuilders.KeyValueConfigBuilder.EnsureGreedyInitialized() **Invalid response, the authentication response was not in the expected format.** at 
Azure.Identity.ManagedIdentityClient.Deserialize(JsonElement json) at 
Azure.Identity.ManagedIdentityClient.Deserialize(Stream content) at 
Azure.Identity.ManagedIdentityClient.Authenticate(String[] scopes, CancellationToken cancellationToken) at 
Azure.Identity.ManagedIdentityCredential.GetTokenImpl(TokenRequestContext requestContext, CancellationToken cancellationToken)

Thanks

[jsquire]

//cc: @schaabs

[heaths]

DefaultAzureCredential and from what I can make out in your stack (can you re-paste that into ``` code blocks for readability?) show you're using at least the Azure.Identity package, but those links you supplied use the older Microsoft.Azure.KeyVault packages for their samples. Can you confirm if you're only using Azure.* packages for build configuration builder and identity? Do you have some sample code that shows how you're using it as well?

Also see https://github.com/Azure/azure-sdk-for-net/blob/master/sdk/keyvault/Azure.Security.KeyVault.Secrets/README.md#troubleshooting for how to troubleshoot requests and get more information that can help diagnose, and if Application Monitoring is enabled for your app you should get more information as well. It would be helpful to see other calls being made, since I'm wondering if this could be a dup of #9737 since our Azure.* builders make simultaneous async calls.

/cc @pakrym for configuration builder.

[maldago]

Thanks for looking at it, looks like pakrym beat me to formatting the stack.

We don't have any code blocks. It is the ConfigBuilder that makes calls to the KeyVault to retrieve the secrets using the system assigned managed identity.

Azure.Identity, Version 1.1.1.0
Azure.Security.KeyVault.Secrets, Version 4.0.2.0
*for the configuration builders see in config below

An extract of the web.config is as follows:

  <configSections>
    <section name="configBuilders" type="System.Configuration.ConfigurationBuildersSection, System.Configuration, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" restartOnExternalChanges="false" requirePermission="false" />
  </configSections>
  <configBuilders>
    <builders>
      <add name="Environment" mode="Greedy" type="Microsoft.Configuration.ConfigurationBuilders.EnvironmentConfigBuilder, Microsoft.Configuration.ConfigurationBuilders.Environment, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
      <add name="Secrets" userSecretsId="" type="Microsoft.Configuration.ConfigurationBuilders.UserSecretsConfigBuilder, Microsoft.Configuration.ConfigurationBuilders.UserSecrets, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
      <add name="KeyVault" mode="Greedy" vaultName="${KEY_VAULT_NAME}" type="Microsoft.Configuration.ConfigurationBuilders.AzureKeyVaultConfigBuilder, Microsoft.Configuration.ConfigurationBuilders.Azure, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
      <add name="ConfigStore" mode="Greedy" connectionString="${CS_AAC}" type="Microsoft.Configuration.ConfigurationBuilders.AzureAppConfigurationBuilder, Microsoft.Configuration.ConfigurationBuilders.AzureAppConfiguration, Version=1.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35" />
    </builders>
  </configBuilders>
  <appSettings configBuilders="Environment,KeyVault,ConfigStore,Secrets">
    <add key="CS_AAC" value="" />
    <add key="KEY_VAULT_NAME" value="" />
  </appSettings>
   ...
</configuration>```

[heaths]

Based on their source, this is probably the same as #9737. As a temporary workaround - and if you only have 2 values (assuming that matches your use case) - can you disable greedy mode? From what I'm seeing in their implementation, it might alleviate the problem and probably not affect performance. Getting all secrets from a Key Vault still has to call to get each actual secret individual, though they run that in parallel. But depending on the size of the vault and given its low rate limit, may still be as slow or slower than trying to potentially fetch 2 secrets individually.

[maldago]

@heaths

Thanks for looking into this, we will try disabling Greedy. Will let you know how we get on.

[heaths]

I also have a fix prepared and will get a servicing update out soon.

[maldago]

Thanks very much for the fix.

Just to note we tried changing Greedy to Strict but we still observed the same behaviour.

[heaths]

I believe I have a fix and was hoping you could try it to verify before we release it, given the nature of this problem is impacted by machine and scenario differences.

1. Register our dev feed in your nuget configuration. Typically, you'd run the following in your solution or project root (can also be machine wide, but any CI/CD will also need it):

   dotnet nuget add source -n AzureSDK https://azuresdkartifacts.blob.core.windows.net/azure-sdk-for-net/index.json
   

2. Install the dev package into your project:

   dotnet add package Azure.Security.KeyVault.Secrets --version 4.0.3-dev.20200318.2
   

3. Rebuild your project. Make sure the right DLL was copied to your output. It should have the file version 4.0.320.16802 and product version 4.0.3-dev.20200318.2+3a643510066880ee8bffe38c55662e29a6ea6ea4.

Please let me know if this solves your problem and we'll get a release out on nuget.org. Thank you.

[heaths]

We had one customer from the other bug verify the fix works, so you can use that for now. We want to do some additional testing to make sure we didn't regress anything and fixed all the related issues here, and will get a servicing release out on nuget.org shortly.

[maldago]

Hello, apologies for the delay, I had been distracted by another problem. Deployed the new package and we are, unfortunately, observing, the same error.

I verified the binary is being deployed and can see it being referenced in the latest Azure Diagnostics.

[heaths]

Can you verify you are seeing our pipeline trace messages in your diagnostics (if not, see my link to troubleshooting steps above), and send my any events named Azure* via email so we can diagnose further? My email is my GitHub username @microsoft.com.