[destination-Redshift] S3 Assumed role (instead of key-id/access-key)

[sharon-clue]

Connector Name

destination-redshift

Connector Version

0.6.11

What step the error happened?

Configuring a new connector

Relevant information

I'm trying to setup the connector while using the S3 Staging option.
the aws key-id and access are mandatory and it's not required since my my implementation is using a service account that has the needed permissions to the s3 bucket.

please change it to allow using the s3 staging option without the aws key-id and access key

Relevant log output

No response

Contribute

• Yes, I want to contribute

[pquadri]

same issue here, i think it's not only a matter of removing the required fields as the COPY command is explicitly using credentials (i guess it would be needed to add the service account name there)

[sharon-clue]

the service account is attached to the pod, so no need to do anything besides allowing the whole thing work without having the secret/access key.
If i have time i'll change it and create a pull request

[hvignolo87]

Same issue here. Do you have an estimated timeframe for this? I would love to help, but I know nothing about Java 😩

[alexlopes]

I'm facing the same problem using k8s service accounts only. Looking here I think this is not possible, because the function expects a access/secret key environment variables.

I'm getting this error

Stack Trace: java.util.NoSuchElementException: Key AWS_ACCESS_KEY_ID is missing in the map.
        at kotlin.collections.MapsKt__MapWithDefaultKt.getOrImplicitDefaultNullable(MapWithDefault.kt:24)
        at kotlin.collections.MapsKt__MapsKt.getValue(Maps.kt:360)
        at io.airbyte.cdk.integrations.destination.s3.credential.S3AssumeRoleCredentialConfig$Companion.getCredentialProvider(S3AssumeRoleCredentialConfig.kt:65)

@stephane-airbyte can you confirm please?

[evantahler]

For the moment, we don't have answers to these questions, as we are focused on enabling this feature for users of Airbyte Cloud. We will later explore if/how we want to enable this for self-hosted users. #39371

[alexlopes]

Thanks for the clarification @evantahler

[assaadhjb]

Hello @evantahler, I can see that you are tackling this for S3 destination, however the same applies to Redshift destination with COPY strategy (S3 Staging), it makes sense enabling passing ARN Role for both, WDYT? Especially that we only have that option now that standard inserts have been removed

[jacoblElementor]

@evantahler
I see that your PR is closed in favor of this PR #39468
but the new PR is still open.
Is the connector with Role ARN authentication still not available for Airbyte OSS?
Is it available for Airbyte Enterprise?