Credentials reloading is not working in the recent versions (2.20.x)

[munendrasn]

Describe the bug

Support to reload credentials was added in 2.19.x (maybe in 2.19.33), related to PR - #3712

Expected Behavior

Expected to reload the credentials when there is a change in credentials

Current Behavior

When recently upgraded to 2.20.x, encountering the token expired error, Same error for multiple AWS Services. Tried with 2.20.112

software.amazon.awssdk.services.athena.model.AthenaException: The security token included in the request is expired

Reproduction Steps

Upgrade to latest version of aws sdk, and update the credentials, it should pick it but not working

Possible Solution

Maybe this is occurring post this change - 8e55ccc

ie, from 2.20.54
Here, should it be 8e55ccc#diff-20afd65c6f0a6de7d5d0669cbd441b1c14afcdf694c379387f57e8399f93c8fcR269

ProfileSupplier.defaultSupplier() instead of fixedSupplier?

Additional Information/Context

No response

AWS Java SDK version used

2.20.112

JDK version used

11

Operating System and version

Mac OS - 13.5

[munendrasn]

@dave-fn FYI

[debora-ito]

@munendrasn can you provide a sample code we can use to reproduce?
We need more context on which and how you are using the credentials.

[munendrasn]

Creating AWS Clients, without specifying any explicit credentials provider
Example,

return AthenaClient.builder().build()

Steps to reproduce:

K8s (Preferred way to reproduce, mimics the normal usage)

java -version
11.0.19

• Create K8s Secrets containing credentials which would be periodically rotated
  • for reproduction, this can be configMap too
• Add the secrets as volume and mount on the Java service container pod (readonly volume mount)
• set AWS_SHARED_CREDENTIALS_FILE env variable to point the volume mount location.
• deploy the the service

Mac

• update the temporary credentials in ~/.aws/credentials
• Start the java service from command line
• update it invalid credentials
• make request to java service (should get error due to expired credentials)
• Update the credentials file again with valid credentials

This flows works with 2.19.33 but fails with recent version of aws java sdk for credentials reloading

@debora-ito Kindly, let me know if any additional info is required. As there is no specific logic in creating the client, provided the sample one-liner code for AthenaClient creation

[turaleck]

@debora-ito any update on this? We are facing the same issue with sdk 2.20.81

[max-peroch]

Same thing for us on v2.20.149, would love an update on this if possible

[DocX]

Hello, I prepared sample that reproduces this issue:

SDK version: v2.25.25

Create two files /some/path/credentials.test_a:

[default]
aws_access_key_id=test_aaaaaaaa
aws_secret_access_key=test_aaaaaaaa
aws_session_token=test_aaaaaaaa

and /some/path/credentials.test_b:

[default]
aws_access_key_id=test_bbbbbbbb
aws_secret_access_key=test_bbbbbbbb
aws_session_token=test_bbbbbbbb

Then run following java program with AWS_SHARED_CREDENTIALS_FILE=/some/path/credentials

public class MyClass {
    public static void main(String[] args) {
        DefaultCredentialsProvider credentialsWithProfileFile = DefaultCredentialsProvider.builder()
                .profileFile(ProfileFileSupplier.defaultSupplier())
                .build();
        DefaultCredentialsProvider credentialsDefault = DefaultCredentialsProvider.create();

        Path credentialsFilePath = FileSystems.getDefault().getPath(System.getenv("AWS_SHARED_CREDENTIALS_FILE"));

        List<Path> testCredentials = new ArrayList<>();
        testCredentials.add(credentialsFilePath.resolveSibling(credentialsFilePath.getFileName() + ".test_a"));
        testCredentials.add(credentialsFilePath.resolveSibling(credentialsFilePath.getFileName() + ".test_b"));

        int fileIndex = 0;

        while(true) {
            Path currentPath = testCredentials.get(fileIndex++ % testCredentials.size());
            System.out.println("Writing credentials file from " + currentPath);
            try {
                Files.copy(currentPath, credentialsFilePath, REPLACE_EXISTING);
            } catch (IOException e) {
                System.out.println("ERROR " + e.getMessage());
            }

            tryCredentials(credentialsDefault,         "Default                 ");
            tryCredentials(credentialsWithProfileFile, "With ProfileFileSupplier");

            System.out.println("---");
            try {
                Thread.sleep(1000);
            } catch(InterruptedException e) {
                System.exit(0);
            }
        }
    }

    private static void tryCredentials(AwsCredentialsProvider provider, String name) {
        String accessKeyId = "";
        try {
            var credentials = provider.resolveCredentials();
            accessKeyId = credentials.accessKeyId();
        } catch(Exception e) {
            accessKeyId = "ERROR " + e.getMessage();
        }
        System.out.format("Access Key ID (%s): %s*****\n", name, accessKeyId.substring(0, accessKeyId.length() - 5));
    }
}

The output is

Writing credentials file from ****/credentials.test_a
Access Key ID (Default                 ): test_aaa*****
Access Key ID (With ProfileFileSupplier): test_aaa*****
---
Writing credentials file from ****/credentials.test_b
Access Key ID (Default                 ): test_aaa*****
Access Key ID (With ProfileFileSupplier): test_bbb*****
---
Writing credentials file from ****/credentials.test_a
Access Key ID (Default                 ): test_aaa*****
Access Key ID (With ProfileFileSupplier): test_aaa*****
---
...

As you can see the default builder for the DefaultCredentialsProvider uses fixedProfileFile() supplier, instead of the defaultSupplier()