Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

s3: bucket replication #1680

Closed
eladb opened this issue Feb 5, 2019 · 14 comments · Fixed by #30966
Closed

s3: bucket replication #1680

eladb opened this issue Feb 5, 2019 · 14 comments · Fixed by #30966
Assignees
@vinayak-kukreja
Labels
@aws-cdk/aws-s3 Related to Amazon S3 effort/small Small work item – less than a day of effort feature/coverage-gap Gaps in CloudFormation coverage by L2 constructs feature-request A feature should be added or improved. p1

Comments

@eladb
Copy link
Contributor

eladb commented Feb 5, 2019

PR #184 has some initial work around S3 bucket replication
@eladb eladb mentioned this issue Feb 5, 2019
Closed
@eladb eladb added feature-request A feature should be added or improved. @aws-cdk/aws-s3 Related to Amazon S3 labels Feb 5, 2019
@eladb eladb assigned eladb and unassigned eladb Aug 12, 2019
@kadishmal
Copy link

Is the bucket replication planned in the near future?

@pierreis
Copy link

pierreis commented Sep 3, 2019

Would be great to see this landing in CDK.

@hleb-albau
Copy link

Any progress on issue?

@eladb eladb assigned iliapolo and unassigned eladb Jan 22, 2020
@iliapolo iliapolo added the effort/medium Medium work item – several days of effort label Mar 9, 2020
@mvanzanten
Copy link

Also running into the same issue, any update?

@denissimonovski
Copy link

Also running into the same issue. Any progress on this?

@mugfordsfcc
Copy link

Blocked we had to do a work around for our client for this. Please update.

@UnbiasedGoat
Copy link

UnbiasedGoat commented Jul 2, 2020 •
edited
Loading

Also really need this, another upvote for priority :)

@mulvaney
Copy link

We could really use this too, cross-region replication is a requirement for us

@iliapolo iliapolo added the p2 label Aug 29, 2020
@techcoderunner
Copy link

We used heavily cdk for development and this is one of the core feature requiremet most of the people has. Another upvote for priority

@jpSimkins
Copy link

Need replication for cross accounts.

@jpSimkins
Copy link

I was able to use CfnBucket to have cross account replication. It seems like this may be possible for other scenarios too. Docs are not that helpful (many props were not needed and there is no explanation of what props are needed for what replication type) but I was able to get it working.

@cmorgia
Copy link

cmorgia commented Apr 10, 2021

Planning to heavily use replication, any update on this issue?

@ericzbeard ericzbeard added the feature/coverage-gap Gaps in CloudFormation coverage by L2 constructs label Apr 20, 2021
@af-tomwilkins
Copy link

Would be great to have this available on the high level Bucket construct. In the meantime, creating a subclass will let you to retain the benefits of the Bucket construct while adding support for replicationConfiguration via the underlying CfnBucket, as below:

import { Construct } from '@aws-cdk/core';
import { Bucket, BucketProps, CfnBucket, CfnBucketProps } from '@aws-cdk/aws-s3';

export class BucketWithReplication extends Bucket {
  constructor(
    scope: Construct,
    id: string,
    props: BucketProps & Required<Pick<CfnBucketProps, 'replicationConfiguration'>>
  ) {
    const { replicationConfiguration, ...bucketProps } = props;
    super(scope, id, bucketProps);
    (this.node.defaultChild as CfnBucket).replicationConfiguration = replicationConfiguration;
  }
}

@iliapolo iliapolo removed their assignment Jun 27, 2021
@rix0rrr rix0rrr added p1 and removed p2 labels Mar 16, 2022
@TheRealAmazonKendra TheRealAmazonKendra added effort/small Small work item – less than a day of effort and removed effort/medium Medium work item – several days of effort labels Jan 24, 2023
@vinayak-kukreja vinayak-kukreja self-assigned this Apr 14, 2023
@vinayak-kukreja vinayak-kukreja removed their assignment Jul 3, 2023
@colifran colifran self-assigned this Jul 13, 2023
@vinayak-kukreja vinayak-kukreja self-assigned this Oct 9, 2023
@badmintoncryer badmintoncryer mentioned this issue Jul 28, 2024
Merged
1 task
@mergify mergify bot closed this as completed in #30966 Jan 16, 2025
mergify bot pushed a commit that referenced this issue Jan 16, 2025
@badmintoncryer
feat(s3): replicating objects (#30966)
9d8a7e2 
### Issue # (if applicable)

Closes #1680.

### Reason for this change

AWS S3 supports configuring [object replication](https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication.html) , but the `s3.Bucket` construct does not support it.

### Description of changes

Added `replicationRules` to `BucketProps`.

#### Replication configuration version

There are two versions of [replication configuration](https://docs.aws.amazon.com/AmazonS3/latest/userguide/replication-add-config.html#replication-backward-compat-considerations). This PR uses only the V2 replication configuration to enable the specification of the Filter element and S3 Replication Time Control (S3 RTC).

To use V2 replication configuration, this PR explicitly specifies [Filter.Prefix](https://docs.aws.amazon.com/ja_jp/AWSCloudFormation/latest/UserGuide/aws-properties-s3-bucket-replicationrulefilter.html#cfn-s3-bucket-replicationrulefilter-prefix) property.
```ts
        const prefix = rule.prefixFilter ?? '';
        const filter = isAndFilter ? {
          and: {
            prefix,
            tagFilters: rule.tagFilter,
          },
        } : {
          prefix,
        };
```

V2 replication configuration has some restriction:
- Must specify [DeleteMarkerReplication](https://docs.aws.amazon.com/ja_jp/AWSCloudFormation/latest/UserGuide/aws-properties-s3-bucket-replicationrule.html#cfn-s3-bucket-replicationrule-deletemarkerreplication)
```sh
ReplicationStack | 4/7 | 9:22:08 PM | CREATE_FAILED        | AWS::S3::Bucket  | SourceBucket (SourceBucketDDD2130A) Resource handler returned message:
Delete marker replication is not supported if any Tag filter is specified. Please refer to S3 Developer Guide for more information. (Service: S3, Status Code: 400, Request ID: XXX, Extended Request ID: XXX)
```
- Must specify [Priority](https://docs.aws.amazon.com/ja_jp/AWSCloudFormation/latest/UserGuide/aws-properties-s3-bucket-replicationrule.html#cfn-s3-bucket-replicationrule-priority)
```sh
ReplicationStack | 4/7 | 9:12:08 PM | CREATE_FAILED        | AWS::S3::Bucket  | SourceBucket (SourceBucketDDD2130A) Resource handler returned message:
Priority must be specified for this version of Cross Region Replication configuration schema. Please refer to S3 Developer Guide for more information. (Service: S3, Status Code: 400, Request ID: XXX, Extended Request ID: XXX)
```

These restriction is not documented but there are some posts about these points.
- https://repost.aws/questions/QUiEc8wFE_Q16fX5WG-YWnrA/cloudformation-support-for-s3-replication-to-multiple-destination-buckets

To resolve these problems,I made the `priority` required and explicitly set the `deleteMarkerReplication`.

```ts
       const prefix = rule.prefixFilter ?? ''; // set empty string to use V2 replication configuration
        const filter = isAndFilter ? {
          and: {
            prefix,
            tagFilters: rule.tagFilter,
          },
        } : {
          prefix,
        };

        return {
          id: rule.id,
          priority: rule.priority,
          status: 'Enabled',
          destination: {
            bucket: rule.destination.bucket.bucketArn,
            account: rule.destination.account,
            storageClass: rule.storageClass?.toString(),
            accessControlTranslation: rule.destination.accessControlTransition ? {
              owner: 'Destination',
            } : undefined,
            encryptionConfiguration: rule.kmsKey ? {
              replicaKmsKeyId: rule.kmsKey.keyArn,
            } : undefined,
            replicationTime: rule.replicationTimeControl !== undefined ? {
              status: rule.replicationTimeControl ? 'Enabled' : 'Disabled',
              time: {
                minutes: 15,
              },
            } : undefined,
            metrics: rule.replicationTimeControlMetrics !== undefined ? {
              status: rule.replicationTimeControlMetrics ? 'Enabled' : 'Disabled',
              eventThreshold: {
                minutes: 15,
              },
            } : undefined,
          },
          filter,
          // To avoid deploy error when there are multiple replication rules with undefined deleteMarkerReplication,
          // CDK explicitly set the deleteMarkerReplication if it is undefined.
          deleteMarkerReplication: {
            status: rule.deleteMarkerReplication ? 'Enabled' : 'Disabled',
          },
          sourceSelectionCriteria,
        };
```

#### IAM permission

There is a [documentation to setup IAM permissions for service role](https://docs.aws.amazon.com/AmazonS3/latest/userguide/setting-repl-config-perm-overview.html).

```json
{
   "Version":"2012-10-17",
   "Statement":[
      {
         "Effect":"Allow",
         "Action":[
            "s3:GetReplicationConfiguration",
            "s3:ListBucket"
         ],
         "Resource":[
            "arn:aws:s3:::SRC-BUCKET"
         ]
      },
      {
         "Effect":"Allow",
         "Action":[
            "s3:GetObjectVersionForReplication",
            "s3:GetObjectVersionAcl",
            "s3:GetObjectVersionTagging"
         ],
         "Resource":[
            "arn:aws:s3:::SRC-BUCKET/*"
         ]
      },
      {
         "Effect":"Allow",
         "Action":[
            "s3:ReplicateObject",
            "s3:ReplicateDelete",
            "s3:ReplicateTags"
         ],
         "Resource":"arn:aws:s3:::DST-BUCKET/*"
      }
   ]
}
```

However, there are discrepancies between the automatically generated IAM policies in the management console and the IAM policies in the documentation.

Generated Policy:

```json
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "s3:ListBucket",
                "s3:GetReplicationConfiguration",
                "s3:GetObjectVersionForReplication",
                "s3:GetObjectVersionAcl",
                "s3:GetObjectVersionTagging",
                "s3:GetObjectRetention",
                "s3:GetObjectLegalHold"
            ],
            "Effect": "Allow",
            "Resource": [
                "arn:aws:s3:::SRC-BUCKET",
                "arn:aws:s3:::SRC-BUCKET/*"
            ]
        },
        {
            "Action": [
                "s3:ReplicateObject",
                "s3:ReplicateDelete",
                "s3:ReplicateTags",
                "s3:GetObjectVersionTagging",
                "s3:ObjectOwnerOverrideToBucketOwner"
            ],
            "Effect": "Allow",
            "Condition": {
                "StringLikeIfExists": {
                    "s3:x-amz-server-side-encryption": [
                        "aws:kms",
                        "aws:kms:dsse",
                        "AES256"
                    ]
                }
            },
            "Resource": [
                "arn:aws:s3:::DST-BUCKET/*"
            ]
        },
        {
            "Action": [
                "kms:Decrypt"
            ],
            "Effect": "Allow",
            "Condition": {
                "StringLike": {
                    "kms:ViaService": "s3.ap-northeast-1.amazonaws.com",
                    "kms:EncryptionContext:aws:s3:arn": [
                        "arn:aws:s3:::SRC-BUCKET/*"
                    ]
                }
            },
            "Resource": [
                "arn:aws:kms:ap-northeast-1:123456789012:key/hogehuga"
            ]
        },
        {
            "Action": [
                "kms:Encrypt"
            ],
            "Effect": "Allow",
            "Condition": {
                "StringLike": {
                    "kms:ViaService": [
                        "s3.ap-northeast-1.amazonaws.com"
                    ],
                    "kms:EncryptionContext:aws:s3:arn": [
                        "arn:aws:s3:::DST-BUCKET*"
                    ]
                }
            },
            "Resource": [
                "arn:aws:kms:ap-northeast-1:123456789012:key/hogefuga"
            ]
        }
    ]
}
```

I adopted the policy from the document. I look forward to hearing your thoughts on this matter.

### Description of how you validated changes

Added both unit and integ tests.

### Checklist
- [x] My code adheres to the [CONTRIBUTING GUIDE](https://github.com/aws/aws-cdk/blob/main/CONTRIBUTING.md) and [DESIGN GUIDELINES](https://github.com/aws/aws-cdk/blob/main/docs/DESIGN_GUIDELINES.md)

----

*By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
@github-actions GitHub Actions
Copy link

Comments on closed issues and PRs are hard for our team to see.
If you need help, please open a new issue that references this one.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jan 16, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@vinayak-kukreja vinayak-kukreja

Labels
@aws-cdk/aws-s3 Related to Amazon S3 effort/small Small work item – less than a day of effort feature/coverage-gap Gaps in CloudFormation coverage by L2 constructs feature-request A feature should be added or improved. p1
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat(s3): replicating objects badmintoncryer/aws-cdk