All Function Keys Regenerate Constantly

[jonarmstrong]

Investigative information
Please provide the following:

Timestamp:
Function App version: 2.0.13351.0
Function App name: automation-lnpojzrfgkuqy
Function name(s) (as appropriate): function invocation not required
Invocation ID: function invocation not required
Region: East US

Repro steps

I have a function app, which any time any of the function keys pages are loaded or key is used by performing an API call, such as to an HTTP trigger, the keys are immediately regenerated, which is preventing the calls from being successful.

For instance, if I go to this website https://portal.azure.com/#{DOMAIN}/resource/subscriptions/{SUBSCRIPTION}/resourceGroups/{RESOURCE_GROUP}/providers/Microsoft.Web/sites/{SITENAME}/functionsAppKeys and view the function keys, I'll see one version of the key. I can then reload the page and see another value of the function of the key right away. I can also view the keys refreshing if I just hit the the refresh button.

Expected behavior

When refreshing the page, I expect that the host function/master keys should not change w/o explicitly renewing them

Actual behavior

The host function/master key or the function key of any implemented function renews anytime they are accessed (viewed or a function is called which attempts to authenticate with them)

Known workarounds

none, I can't get the function keys to stop regenerating

Related information

Here are some possibly relevant facts to my setup with the function key management

• AzureWebJobsSecretStorageType = keyvault
  • The access to the kevault is working, when I view the keys, can I go look at the head version of the secret in the keyvault and see the same value. Then; when reloading the page, the backend keyvault setting is also being update.
• Function App is an I1 SKu in an ILB App Service Environment, DNS & SSL certs are setup so the function host is available from another machine on a peered vnet. E.g. the kudu console is available
  • There are 2 instances of the azure function apps running. But they are in different app service plans under different App Service Environments, within the same Azure Region. They are running the exact same version of code and share the same configuration (think DR instances as a warm spare that is ready to be switch in if the primary app service environment failed)
• FUNCTIONS_WORKER_RUNTIME = dotnet
• FUNCTIONS_EXTENSION_VERSION = ~2

[jonarmstrong]

Actually, I think I've figured out the problem in the host code. I pasted the function from the source file I've referenced. The issue boils down to the GetSecretsAsync (line 3 below) doesn't go through all of the paginated results when you use the .FirstOrDefault extension method. If you review the documentation for GetSecretsAsync, it details that if the maxresults parameter isn't specified it limits the secret retrieval to only 25. It looks like if you want to traverse the paginated results, you need to use the GetSecretsNextAsync method to first get all of the possible secrets (until all of the next page links are exhausted). Alternatively, if you know the exact name of the secrets your looking for use GetSecretAsync and retrieve the secret value immediately.

The reason this breaks in my environment, is the keyvault in question has over >100 secrets present and the functions keys aren't returned in the first 25 results

KeysVaultSecretsRepository.cs

        private async Task<ScriptSecrets> ReadHostSecrets()
        {
            // ONLY HAS 25 RESULTS 
            IPage<SecretItem> secretItems = await _keyVaultClient.Value.GetSecretsAsync(GetVaultBaseUrl());
            List<Task<SecretBundle>> tasks = new List<Task<SecretBundle>>();

            // Add master key task
            SecretItem masterItem = secretItems.FirstOrDefault(x => x.Identifier.Name.StartsWith(MasterKey));
            if (masterItem != null)
            {
                tasks.Add(_keyVaultClient.Value.GetSecretAsync(GetVaultBaseUrl(), masterItem.Identifier.Name));
            }
            else
            {
                return null;
            }

            // Add functionKey tasks
            foreach (SecretItem item in secretItems.Where(x => x.Identifier.Name.StartsWith(FunctionKeyPrefix)))
            {
                tasks.Add(_keyVaultClient.Value.GetSecretAsync(GetVaultBaseUrl(), item.Identifier.Name));
            }

            // Add systemKey tasks
            foreach (SecretItem item in secretItems.Where(x => x.Identifier.Name.StartsWith(SystemKeyPrefix)))
            {
                tasks.Add(_keyVaultClient.Value.GetSecretAsync(GetVaultBaseUrl(), item.Identifier.Name));
            }

            await Task.WhenAll(tasks);

            HostSecrets hostSecrets = new HostSecrets()
            {
                FunctionKeys = new List<Key>(),
                SystemKeys = new List<Key>()
            };

            foreach (Task<SecretBundle> task in tasks)
            {
                SecretBundle item = task.Result;
                if (item.SecretIdentifier.Name.StartsWith(MasterKey))
                {
                    hostSecrets.MasterKey = SecretBundleToKey(item, MasterKey);
                }
                else if (item.SecretIdentifier.Name.StartsWith(FunctionKeyPrefix))
                {
                    hostSecrets.FunctionKeys.Add(SecretBundleToKey(item, FunctionKeyPrefix));
                }
                else if (item.SecretIdentifier.Name.StartsWith(SystemKeyPrefix))
                {
                    hostSecrets.SystemKeys.Add(SecretBundleToKey(item, SystemKeyPrefix));
                }
            }

            return hostSecrets;
        }

[soninaren]

@fabiocav any thoughts on this?

[jonarmstrong]

@fabiocav @soninaren

Any thoughts on this?

[mathewc]

@alrod to comment on the potential paging issue

[fabiocav]

This looks lit a valid bug.

@jonarmstrong are you sharing this KV instance with other apps/processes? Would it be a viable mitigation option for you to reduce the number of secrets on that KV instance (perhaps scoping to this app only) while we investigate and provide a fix?

[jonarmstrong]

@fabiocav It's OK right now, because the functions that we use this as the primary security method are infrequently used in our application, so it isn't a huge headache for us most days. At least, it isn't such an issue that it was desirable for us to make the exact change that you mentioned. I hope it can be resolved quickly! Thank you for taking the time to look into it.

[fabiocav]

This has been assigned to the next sprint. Thanks for bringing it up!

[alrod]

From the issue description:

There are 2 instances of the azure function apps running. But they are in different app service plans under different App Service Environments, within the same Azure Region. They are running the exact same version of code and share the same configuration

It looks like both function apps use the same keyvault. Keyvault keys feature was implemented to use one function app with one key vault.

[jonarmstrong]

From the issue description:

There are 2 instances of the azure function apps running. But they are in different app service plans under different App Service Environments, within the same Azure Region. They are running the exact same version of code and share the same configuration

It looks like both function apps use the same keyvault. Keyvault keys feature was implemented to use one function app with one key vault.

@alrod It's interesting that you state that because the Keyvault secret store is the only one which supports this particular setup. The blob store almost does it, but it prefixes all of the secret storage with instance/host specific identifiers, which prevents multiple instances sharing the same secret keys. The Keyvault secret storage doesn't add any sort of prefixing to the naming of the secrets, so it allow multiple hosts which point to the same key vault to share the same secrets.

There a need to do this when you have function apps which are in different app service plans and you need to coordinate them so that if http traffic is redirected from one instance to another, it doesn't require a new set of credentials to be provided to integrators, just to access the same functions which have been redeployed in a different app service instance.

By and large, the functions apps in my setup have worked very well with coordinating work across multiple instances with the limited set of triggers that I'm currently using (blob, queue, http). The shared secret storage across multiple instances was also working very well, until enough secrets accumulated in the keyvault which caused the lack of paging to become problematic.

[fabiocav]

The point made by @alrod is a good one. Secrets are unique for a given app, and scoped to that app by default, for all providers. The prefixing approach used with the storage provider is the strategy used to scope those secrets while allowing storage account reuse.

Even with the clarification here, there's likely some work needed to ensure this scenario is properly handled and provides helpful error messages when dealing with a setup similar to yours.

[fabiocav]

This is in validation, PR should be landing in sprint 85

[jonarmstrong]

@fabiocav

Thank you for your help! Is there anything I can help validate or test with this?