[Azure Identity] Python app running in the k8s pod failed while retrieving the token from MSI

[Shuanglu]

Describe the bug
Invoked below to retrieve the token for keyvault but failed at 'get_token()'. Use 'curl' to directly access the MSI endpoint works and 3rd party SDK works as well https://pypi.org/project/keyvaultlib/

    credential = DefaultAzureCredential()
    print("\nPass Default credential configuration")
    #print out the token value used to access the keyvault
    try:
        tokenValue = credential.credentials[1].get_token('https://vault.azure.net/.default')
    except Exception as ex:
        print("Failed to get token: {}".format(ex))
    else:
        print(tokenValue)

Exception or Stack Trace

string indices must be integers

Pass Default credential configuration
ManagedIdentityCredential.get_token failed: string indices must be integers
Failed to get token: string indices must be integers
Pass tokenValue
EnvironmentCredential.get_token failed: EnvironmentCredential authentication unavailable. Environment variables are not fully configured.
ManagedIdentityCredential.get_token failed: string indices must be integers
DefaultAzureCredential.get_token failed: ManagedIdentityCredential raised unexpected error "string indices must be integers"
DefaultAzureCredential failed to retrieve a token from the included credentials.
Attempted credentials:
        EnvironmentCredential: EnvironmentCredential authentication unavailable. Environment variables are not fully configured.
        ManagedIdentityCredential: **_string indices must be integers_**
Failed to get secret: DefaultAzureCredential failed to retrieve a token from the included credentials.
Attempted credentials:
        EnvironmentCredential: EnvironmentCredential authentication unavailable. Environment variables are not fully configured.
        ManagedIdentityCredential: string indices must be integers

To Reproduce
only can be reproduced in the specific environment

Code Snippet

    credential = DefaultAzureCredential()
    print("\nPass Default credential configuration")
    #print out the token value used to access the keyvault
    try:
        tokenValue = credential.credentials[1].get_token('https://vault.azure.net/.default')
    except Exception as ex:
        print("Failed to get token: {}".format(ex))
    else:
        print(tokenValue)

Expected behavior
The approach can get the access token

Screenshots
N/A

Setup (please complete the following information):

• Python Version: [e.g. Python 3.8]
  Python 3.7(docker image: mcr.microsoft.com/azure-functions/python:3.0-python3.7-slim)
• SDK Version: [e.g. azure-mgmt-resource-15.0.0b1]
  azure-identity==1.4.0
  azure-keyvault-secrets==4.2.0

Additional context
Add any other context about the problem here.

Information Checklist
Kindly make sure that you have added all the following information above and checkoff the required fields otherwise we will treat the issuer as an incomplete report

• Bug Description Added
• Repro Steps Added
• Setup information Added

[yunhaoling]

thanks @Shuanglu for reaching out. We'll investigate this asap.
adding @chlowell who could further help on the issue.

[chlowell]

Thanks for opening this issue. I haven't seen this error before. My first thought is, the credential may be getting an unexpected payload from the managed identity endpoint. That is to say, not JSON. I would have expected a different error in such a case but if the credential somehow got a string instead of a dictionary (deserialized JSON), "string indices must be integers" makes sense.

Could you please collect some debugging information? I'm curious about the original exception, and the managed identity endpoint's response. If you configure a logger for DEBUG logging, you'll get a traceback of the original exception. If you pass logging_enable=True to the credential, it will log its HTTP sessions including any authentication secrets in them, so only do this if your log output is secure:

logger = logging.getLogger('azure')
logger.setLevel(logging.DEBUG)
credential = DefaultAzureCredential(logging_enable=True)

The log output will look something like this:

Request URL: 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https://vault.azure.net'
Request method: 'GET'
Request headers:
    'Metadata': 'true'
    'User-Agent': 'azsdk-python-identity/1.4.0 Python/3.6.9 (Linux-5.4.0-1040-azure-x86_64-with-Ubuntu-18.04-bionic)'
Request body:
This request has no body
Response status: 200
Response headers:
    'Content-Type': 'application/json; charset=utf-8'
    'Server': 'IMDS/150.870.65.492'
    'Date': 'Fri, 12 Mar 2021 00:26:16 GMT'
    'Content-Length': '1645'
Response content:
    ...

If debug-level logging isn't safe for your application, you can also simply configure an "azure" logger for INFO messages. The output won't include response bodies then, but it will at least show the content type of the managed identity response.

[Shuanglu]

Thanks for opening this issue. I haven't seen this error before. My first thought is, the credential may be getting an unexpected payload from the managed identity endpoint. That is to say, not JSON. I would have expected a different error in such a case but if the credential somehow got a string instead of a dictionary (deserialized JSON), "string indices must be integers" makes sense.

Could you please collect some debugging information? I'm curious about the original exception, and the managed identity endpoint's response. If you configure a logger for DEBUG logging, you'll get a traceback of the original exception. If you pass logging_enable=True to the credential, it will log its HTTP sessions including any authentication secrets in them, so only do this if your log output is secure:

logger = logging.getLogger('azure')
logger.setLevel(logging.DEBUG)
credential = DefaultAzureCredential(logging_enable=True)

The log output will look something like this:

Request URL: 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https://vault.azure.net'
Request method: 'GET'
Request headers:
    'Metadata': 'true'
    'User-Agent': 'azsdk-python-identity/1.4.0 Python/3.6.9 (Linux-5.4.0-1040-azure-x86_64-with-Ubuntu-18.04-bionic)'
Request body:
This request has no body
Response status: 200
Response headers:
    'Content-Type': 'application/json; charset=utf-8'
    'Server': 'IMDS/150.870.65.492'
    'Date': 'Fri, 12 Mar 2021 00:26:16 GMT'
    'Content-Length': '1645'
Response content:
    ...

If debug-level logging isn't safe for your application, you can also simply configure an "azure" logger for INFO messages. The output won't include response bodies then, but it will at least show the content type of the managed identity response.

Thanks a lot! will provide the debug log once I get that.

[ghost]

Hi, we're sending this friendly reminder because we haven't heard back from you in a while. We need more information about this issue to help address it. Please be sure to give us your input within the next 7 days. If we don't hear back from you within 14 days of this comment the issue will be automatically closed. Thank you!

[Shuanglu]

Client discontinued. closing... thanks a ton for the support!