Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: support IRSA for aws s3 provider #749

Merged
merged 3 commits into from
Jan 3, 2025
Merged

chore: support IRSA for aws s3 provider #749

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
7f41551
chore: support IRSA for aws s3 provider
Anhui-tqhuang Dec 27, 2024
e2ff635
chore: support IRSA for aws s3 provider
Anhui-tqhuang Jan 2, 2025
37bf233
chore: support IRSA for aws s3 provider
Anhui-tqhuang Jan 3, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -394,6 +394,11 @@
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>com.amazonaws</groupId>
<artifactId>aws-java-sdk-sts</artifactId>
<version>1.12.261</version>
</dependency>
<dependency>
<groupId>args4j</groupId>
<artifactId>args4j</artifactId>
Expand Down Expand Up @@ -510,7 +515,6 @@
<groupId>org.slf4j</groupId>
<artifactId>jcl-over-slf4j</artifactId>
<version>${slf4j.version}</version>
<scope>test</scope>
</dependency>
<!-- tests dependencies -->
<dependency>
Expand Down
36 changes: 35 additions & 1 deletion src/main/java/org/gaul/s3proxy/Main.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -36,15 +36,21 @@
import java.util.concurrent.TimeUnit;
import java.util.regex.Pattern;

import com.amazonaws.auth.AWSCredentials;
import com.amazonaws.auth.AWSCredentialsProvider;
import com.amazonaws.auth.AWSSessionCredentials;
import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;
import com.google.common.base.Strings;
import com.google.common.collect.ImmutableBiMap;
import com.google.common.collect.ImmutableMap;
import com.google.common.io.Files;
import com.google.common.util.concurrent.ThreadFactoryBuilder;
import com.google.common.base.Supplier;

import org.jclouds.Constants;
import org.jclouds.ContextBuilder;
import org.jclouds.JcloudsVersion;
import org.jclouds.aws.domain.SessionCredentials;
import org.jclouds.blobstore.BlobStore;
import org.jclouds.blobstore.BlobStoreContext;
import org.jclouds.concurrent.DynamicExecutors;
Expand All @@ -53,6 +59,7 @@
import org.jclouds.logging.slf4j.config.SLF4JLoggingModule;
import org.jclouds.openstack.swift.v1.blobstore.RegionScopedBlobStoreContext;
import org.jclouds.s3.domain.ObjectMetadata.StorageClass;
import org.jclouds.domain.Credentials;
import org.kohsuke.args4j.CmdLineException;
import org.kohsuke.args4j.CmdLineParser;
import org.kohsuke.args4j.Option;
Expand Down Expand Up @@ -377,7 +384,6 @@ private static BlobStore createBlobStore(Properties properties,

ContextBuilder builder = ContextBuilder
.newBuilder(provider)
.credentials(identity, credential)
.modules(List.of(
new SLF4JLoggingModule(),
new ExecutorServiceModule(executorService)))
Expand All @@ -386,6 +392,34 @@ private static BlobStore createBlobStore(Properties properties,
builder = builder.endpoint(endpoint);
}

if ((identity.isEmpty() || credential.isEmpty()) && provider.equals("aws-s3")) {
Supplier<Credentials> credentialsSupplier = new Supplier<Credentials>() {
@Override
public Credentials get() {
AWSCredentialsProvider authChain = DefaultAWSCredentialsProviderChain.getInstance();
AWSCredentials newCreds = authChain.getCredentials();
Credentials jcloudCred = null;

if (newCreds instanceof AWSSessionCredentials) {
jcloudCred = SessionCredentials.builder()
.accessKeyId(newCreds.getAWSAccessKeyId())
.secretAccessKey(newCreds.getAWSSecretKey())
.sessionToken(((AWSSessionCredentials) newCreds).getSessionToken())
.build();
} else {
jcloudCred = new Credentials(
newCreds.getAWSAccessKeyId(), newCreds.getAWSSecretKey()
);
}

return jcloudCred;
}
};
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you use the Apache jclouds STS supplier for consistency:

https://stackoverflow.com/questions/23520216/using-aws-s3-via-jclouds-how-to-assume-role

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hey, i am still testing this change, btw, the sts api only supports assume role not assume role with web identity 🤔

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you expand on your comment? Do you mean that the AWS STS library only supports assume role? Or do you mean the jclouds code only allows it? If the former, it would be better to remove the aws-java-sdk-sts dependency. But if the latter, I am willing to take this if you can explain more clearly what this means.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sure

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

assume role is:

  • you have your own credentials (access key id and access secret key), then you have the role arn that you want to assume
  • assume role with web identity is, you have only one role arn and one web identity token

jclouds doesn't have a way to let you load you default aws credentials including web identity token

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and this is something like what we have done for azure

s3proxy/src/main/java/org/gaul/s3proxy/azureblob/AzureBlobStore.java

Line 133 in da99e5b

new DefaultAzureCredentialBuilder().build());

Copy link
Contributor Author

@Anhui-tqhuang Anhui-tqhuang Jan 2, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

btw, default aws credential provider chan here could help the program to load default credentials from many kind of aws sdk compatible env like

  • aws credentials: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY
  • iam role service account: AWS_ROLE_ARN, AWS_WEB_IDENTITY_TOKEN_FILE
  • etc ...
    public DefaultAWSCredentialsProviderChain() {
        super(new EnvironmentVariableCredentialsProvider(),
              new SystemPropertiesCredentialsProvider(),
              WebIdentityTokenCredentialsProvider.create(),
              new ProfileCredentialsProvider(),
              new EC2ContainerCredentialsProviderWrapper());
    }

builder = builder.credentialsSupplier(credentialsSupplier);
} else {
builder = builder.credentials(identity, credential);
}

BlobStoreContext context = builder.build(BlobStoreContext.class);
BlobStore blobStore;
if (context instanceof RegionScopedBlobStoreContext &&
Expand Down