BucketDeployment: retainOnDelete: false not working when stack deleted

[mikebroberts]

Describe the bug

With this example stack I would expect the BucketDeployment to delete contents of the associated bucket when the stack is deleted because retainOnDelete is set to false, allowing the bucket itself to be deleted. However the contents of the bucket are not deleted, and so the stack fails to delete because the bucket can't be deleted:

class Example extends Stack {
  constructor(scope: Construct, id: string, props: StackProps) {
    super(scope, id, props);

    const bucket = new Bucket(this, 'SiteBucket', {
      removalPolicy: RemovalPolicy.DESTROY
    });

    new BucketDeployment(this, 'Deploy', {
      sources: [Source.asset('../site')],
      destinationBucket: bucket,
      retainOnDelete: false,
      logGroup: new LogGroup(this, 'bucketDeploymentLogs', {retention: RetentionDays.ONE_WEEK}),
    });
  }
}

Regression Issue

• Select this option if this issue appears to be a regression.

Last Known Working CDK Version

No response

Expected Behavior

I expected the contents of the bucket to be deleted to allow the stack to be deleted successfully

Current Behavior

The contents of the bucket are not deleted. In the logs for the underlying Custom Resource Lambda function I see this:

[INFO]	2025-02-11T18:18:02.567Z	f730c674-c871-4396-ace6-6eba722d8b7a	{'RequestType': 'Delete', 'ServiceToken': 'arn:aws:lambda:us-east-1:XXXXXXXXXXXX:function:coffee-store-web-CustomCDKBucketDeployment8693BB64-Xoo2NONG8SqU', 'StackId': 'arn:aws:cloudformation:us-east-1: XXXXXXXXXXXX:stack/coffee-store-web/fbed4940-e8a3-11ef-b96b-123141793963', 'RequestId': '61374908-00e5-4cdd-a5ff-d5a3919514c4', 'LogicalResourceId': 'DeployCustomResource218AF6A4', 'PhysicalResourceId': 'aws.cdk.s3deployment.e0df1d06-479a-4083-80a1-589ca71e6f6f', 'ResourceType': 'Custom::CDKBucketDeployment', 'ResourceProperties': {'ServiceToken': 'arn:aws:lambda:us-east-1: XXXXXXXXXXXX:function:coffee-store-web-CustomCDKBucketDeployment8693BB64-Xoo2NONG8SqU', 'RetainOnDelete': 'false', 'Prune': 'true', 'OutputObjectKeys': 'true', 'SourceBucketNames': ['cdk-hnb659fds-assets-XXXXXXXXXXXX-us-east-1'], 'DestinationBucketName': 'coffee-store-web-sitebucket397a1860-s6537k3vma3v', 'SourceObjectKeys': ['d3967fb79345a78bec13705c7c17d25b9e319ac215814492f4c64a15db5601dd.zip']}}

[INFO]	2025-02-11T18:18:02.567Z	f730c674-c871-4396-ace6-6eba722d8b7a	| s3_dest: s3%3A//coffee-store-web-sitebucket397a1860-s6537k3vma3v/

[INFO] 2025-02-11T18:18:02.567Z f730c674-c871-4396-ace6-6eba722d8b7a | old_s3_dest: None

[INFO]	2025-02-11T18:18:02.839Z	f730c674-c871-4396-ace6-6eba722d8b7a	| response body:
{
    "Status": "SUCCESS",
    "Reason": "See the details in CloudWatch Log Stream: 2025/02/11/coffee-store-web-CustomCDKBucketDeployment8693BB64-Xoo2NONG8SqU[$LATEST]3c51c64ca11e4b51bb85bb0f26eb3708",
    "PhysicalResourceId": "aws.cdk.s3deployment.e0df1d06-479a-4083-80a1-589ca71e6f6f",
    "StackId": "arn:aws:cloudformation:us-east-1: XXXXXXXXXXXX:stack/coffee-store-web/fbed4940-e8a3-11ef-b96b-123141793963",
    "RequestId": "61374908-00e5-4cdd-a5ff-d5a3919514c4",
    "LogicalResourceId": "DeployCustomResource218AF6A4",
    "NoEcho": false,
    "Data": {
        "DestinationBucketArn": null,
        "SourceObjectKeys": [
            "d3967fb79345a78bec13705c7c17d25b9e319ac215814492f4c64a15db5601dd.zip"
        ]
    }
}

Reproduction Steps

See example in summary

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.178.1 (build ae342cb)

Framework Version

No response

Node.js Version

22

OS

Mac OS 15

Language

TypeScript

Language Version

No response

Other information

No response

[pahud]

Root Cause:

• The BucketDeployment construct uses a custom resource backed by a Lambda function to manage the bucket contents
• During stack deletion, the custom resource receives a Delete event with RetainOnDelete: false
• However, based on the logs, the Lambda function appears to be returning SUCCESS without actually emptying the bucket
• This causes the stack deletion to fail because the bucket still contains objects when CloudFormation tries to delete it

Workarounds

1. Manually empty the bucket before stack deletion
2. Use the bucket's autoDeleteObjects: true property instead:

const bucket = new s3.Bucket(this, 'SiteBucket', {
  removalPolicy: RemovalPolicy.DESTROY,
  autoDeleteObjects: true  // Use this instead of retainOnDelete: false
});

To fix this issue:

Fix the BucketDeployment custom resource Lambda function to properly handle the RetainOnDelete: false case by emptying the bucket during Delete events.

Looks like the problem is in the bucket_owned function that we should further investigate.

Making this a P2 and we welcome PRs.

[mikebroberts]

FWIW I was originally using the Bucket's autoDeleteObjects, but the Lambda Function under the covers for that construct doesn't currently allow setting a log group, or log expiration, and so if I use that capability I end up with a log group with no retention setting. Which is why I was trying to use BucketDeployment instead

[himanshu-jain16]

@pahud looks like we add the tag to the deployment bucket here: bucket-deployment and in the bucket_owner method, we only check the keyPrefix.

But looks like we do not consider the case where there is just a single bucket deployment with no keyPrefix. (In which case we just check if there are any tags starting with "aws-cdk:cr-owned")

I think another workaround for now might be to just provide a keyPrefix for now.

Would it make sense to pass the custom resource node addr to the lambda handler as props? Can try to work on a fix for this soon. not sure if I have understood this right.