S3 bucket slow to delete when destroyed during an apply #12146

atoms42 · 2020-02-24T18:52:18Z

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform Version

Terraform version: 0.12.9
AWS provider version: 2.42

Affected Resource(s)

aws_s3_bucket

Terraform Configuration Files

resource "aws_s3_bucket" "cdn_logs_bucket" {
  bucket        = "cdn.logs.${local.fqdn}"
  acl           = "private"
  force_destroy = "true"
  tags          = "${var.tags}"

  server_side_encryption_configuration {
    rule {
      apply_server_side_encryption_by_default {
        sse_algorithm = "AES256"
      }
    }
  }
}

Expected Behavior

S3 bucket destroyed in a timely manner (e.g., within minutes)

Actual Behavior

S3 bucket takes multiple hours to destroy

Steps to Reproduce

Create an S3 bucket via Terraform
terraform apply
Put some objects in the bucket
Remove S3 bucket from Terraform

Important Factoids

I've encountered this a few times and was able to delete the S3 bucket manually via the AWS console before Terraform finished deleting it.
S3 bucket destroys during a terraform destroy seem to work as expected.

The text was updated successfully, but these errors were encountered:

ewbankkit · 2020-02-24T21:09:47Z

@MrAtoms Of what order of magnitude is the number of objects (including all versions) in the bucket?
Internally the provider ends up listing all the object versions in a bucket and deleting them one at a time when the bucket is deleted.
We could look at deleting objects in batches via the DeleteObjects API.

atoms42 · 2020-02-24T21:24:46Z

@ewbankkit If I recall, the buckets that were destroyed had hundreds of thousands of objects in them.

tiny-dancer · 2020-02-25T18:19:05Z

We're encountering this as well

aws_s3_bucket.account_logging_s3_pr[0]: Still destroying... [id=bucketname, 402h21m30s elapsed]

dead10ck · 2020-03-06T19:21:52Z

I'm seeing this issue in 0.12.21. The order of magnitude of the objects in the bucket is hundreds of thousands. Deleting the bucket through the web console works just fine, but Terraform just says "Still destroying..." forever.

nithinvenugopal · 2020-03-10T14:52:36Z

In my case the issue can be replicated with just 5000 objects in bucket without versioning enabled. Took around 30 minutes. Used 0.12.21 terraform and AWS terraform provider v2.49.0

timleecasey · 2020-07-30T02:00:36Z

I have had this happen with ~100 items ina bucket.
module.aws_s3_bucket_NAME.aws_s3_bucket.this[0]: Still destroying... [id=NAME, 58m51s elapsed]
source = "terraform-aws-modules/s3-bucket/aws"
version = "1.9.0"

jurajseffer · 2020-08-17T12:22:22Z

The performance is terrible especially when deleting a bucket geographically located across the world with a higher latency. It's deleting about 40 objects per minute if deleting a Sydney bucket from London.

mlushpenko · 2020-09-14T15:55:59Z

Same problem here. While deleting in UI, we can see progress. So, would be great to see some sort of progress indicator here as well.

Terraform v0.12.29

provider.aws v2.61.0

nithinvenugopal · 2020-11-25T16:57:15Z

Hi,
Was there any update on this? I haven't tried the new version of the provider but this is really blocking us to migrate to new version and try out the AWS features released in newer versions.

ghost · 2021-02-22T19:03:34Z

This problem is still available! Does anyone know how the current state?
When is the problem resolved?

bassmanitram · 2021-04-09T06:52:36Z

We found a really simple workaround that I have documented here: https://gist.github.com/bassmanitram/53a57988d9f4e4a2ffc94aed789bbf9e

When testing with a bucket containing 190,000 objects, this technique deleted the content and the bucket in about 30 minutes. Without this workaround, the terraform destroy command was still running 12 hours later with still over 75000 objects left in the bucket (I got bored waiting and deleted them via the AWS S3 console just to put TF out of its misery).

NOTE that this WON'T work for buckets with object versioning enabled since the CLI s3 rm command simply places a deleted marker in the object history rather than actually getting rid of the object.

sbarakat · 2021-04-12T10:56:53Z

This issue first appears in v2.29.0 and looks related to the changes in #9942 - where resource/aws_s3_bucket_object locks were introduced. For v2.28.1 and below, objects were deleted in batch API calls. From v2.29.0 onwards each object is deleted one at a time, leading to far more API calls and poorer efficiency.

The changelog for v2.29.0 doesn't suggest there's any change in behaviour for resource/aws_s3_bucket, only enhancements to resource/aws_s3_bucket_object. So I assume this was unintended.

This issue can even be seen with as few as 200 objects in a bucket. terraform-provider-aws v2.28.1 takes ~11 seconds to terraform destroy, whereas v2.29.0 takes considerably longer at ~38 seconds. And testing with 5,000 objects took 15 seconds on v2.28.1, and now takes 12 minutes on v2.29.0.

Debug logging also shows v2.28.1 is deleting multiple objects in a single s3/DeleteObjects API call:

2021-04-12T10:16:05.878Z [DEBUG] plugin.terraform-provider-aws_v2.28.1_x4: 2021/04/12 10:16:05 [DEBUG] [aws-sdk-go] DEBUG: Request s3/DeleteObjects Details:
2021-04-12T10:16:05.878Z [DEBUG] plugin.terraform-provider-aws_v2.28.1_x4: ---[ REQUEST POST-SIGN ]-----------------------------
2021-04-12T10:16:05.878Z [DEBUG] plugin.terraform-provider-aws_v2.28.1_x4: POST /?delete= HTTP/1.1
2021-04-12T10:16:05.878Z [DEBUG] plugin.terraform-provider-aws_v2.28.1_x4:
ionId>null</VersionId></Object><Object><VersionId>null</VersionId><Key>e0997074-639e-4c5c-a763-155dfa597d3d</Key></Object><Object><Key>e116db4f-1191-4329-9938-dd37c4dc9c70</Key><VersionId>null</VersionId></Object><Object><Key>e2947b2e-e90b-4b56-bf29-a75
48e010804</Key><VersionId>null</VersionId></Object><Object><Key>e4cdb020-0b40-4228-a4b2-b76f54557fd8</Key><VersionId>null</VersionId></Object><Object><Key>e6af51d1-b887-4579-b566-76d8f88ccbfc</Key><VersionId>null</VersionId></Object><Object><VersionId>n
ull</VersionId><Key>e9705515-8ea6-4152-b32d-633c5652b888</Key></Object><Object><Key>ea494902-8784-4b37-a6a8-ff1551c42b90</Key><VersionId>null</VersionId></Object><Object><Key>ea7b7af1-8446-4c17-b3cb-01f347be8f73</Key><VersionId>null</VersionId></Object>
<Object><VersionId>null</VersionId><Key>ec5d3a8d-da24-4d8b-a554-009db9d1bfcd</Key></Object><Object><Key>ed3971c1-1c6d-4b94-bb4e-90dc0e7ee765</Key><VersionId>null</VersionId></Object><Object><Key>f4fce423-3875-4f55-a19c-05d85fcedbeb</Key><VersionId>null<
/VersionId></Object><Object><Key>f57e2fec-8900-4ed4-b221-1d0a2bd20b65</Key><VersionId>null</VersionId></Object><Object><Key>f5959153-bac6-4f6c-a5c1-48e1553913d3</Key><VersionId>null</VersionId></Object><Object><Key>f7131b6d-2b6d-4162-8b61-77d0228bfa0e</
Key><VersionId>null</VersionId></Object><Object><Key>f876c127-9062-4704-b043-135c4dedb291</Key><VersionId>null</VersionId></Object><Object><Key>f878910f-2820-4a19-a216-5ad7aa5a42f3</Key><VersionId>null</VersionId></Object><Object><Key>fb33147f-a1d4-4747
-85b9-31b8940e56a2</Key><VersionId>null</VersionId></Object><Object><Key>fbc9fc53-cdb3-4e61-b939-d79c90b3bd19</Key><VersionId>null</VersionId></Object><Object><Key>fbd9dbd4-91dd-4bd4-baee-11e65763bd6f</Key><VersionId>null</VersionId></Object><Object><Ke
y>fbfbd9e7-9752-4f03-93b0-15002ac1c0ca</Key><VersionId>null</VersionId></Object><Object><Key>fe8eaffa-3787-4f6e-a3ea-4eaf926f67d4</Key><VersionId>null</VersionId></Object><Object><Key>feb38f85-76e8-48fd-92a8-ae86fb75c18f</Key><VersionId>null</VersionId>
</Object></Delete>
04-12T10:16:05.878Z [DEBUG] plugin.terraform-provider-aws_v2.28.1_x4: -----------------------------------------------------
2021-04-12T10:16:07.012Z [DEBUG] plugin.terraform-provider-aws_v2.28.1_x4: 2021/04/12 10:16:07 [DEBUG] [aws-sdk-go] DEBUG: Response s3/DeleteObjects Details:
2021-04-12T10:16:07.013Z [DEBUG] plugin.terraform-provider-aws_v2.28.1_x4: ---[ RESPONSE ]--------------------------------------
2021-04-12T10:16:07.022Z [DEBUG] plugin.terraform-provider-aws_v2.28.1_x4: HTTP/1.1 200 OK
2021-04-12T10:16:07.022Z [DEBUG] plugin.terraform-provider-aws_v2.28.1_x4: -----------------------------------------------------
2021-04-12T10:16:07.022Z [DEBUG] plugin.terraform-provider-aws_v2.28.1_x4: 2021/04/12 10:16:07 [DEBUG] [aws-sdk-go] <?xml version="1.0" encoding="UTF-8"?>
d88-ab70-bb8bc387c060</Key><VersionId>null</VersionId></Deleted><Deleted><Key>a663cc6b-eeeb-4827-8189-f4f516e77d03</Key><VersionId>null</VersionId></Deleted><Deleted><Key>169ab779-95e4-4d9e-9420-c092097456cf</Key><VersionId>null</VersionId></Deleted><De
leted><Key>815052cd-e7cb-45b4-85e3-06d62b98b9ff</Key><VersionId>null</VersionId></Deleted><Deleted><Key>cc791a1e-cfa7-4e8d-9359-772a26f4e96a</Key><VersionId>null</VersionId></Deleted><Deleted><Key>1e5d6bd0-7043-42b4-8088-3d531881f2f5</Key><VersionId>nul
l</VersionId></Deleted><Deleted><Key>7ed89cb9-a91e-459a-9122-f4d11b9366de</Key><VersionId>null</VersionId></Deleted><Deleted><Key>08d222ea-3381-4b00-a166-7e01fe074311</Key><VersionId>null</VersionId></Deleted><Deleted><Key>061434d7-abd5-41ae-ba92-f3ba5e
4b110d</Key><VersionId>null</VersionId></Deleted><Deleted><Key>2895928d-9a62-41a3-8027-261e69db58b5</Key><VersionId>null</VersionId></Deleted><Deleted><Key>216ed0f2-85e3-46c5-af60-0eecb8a21b65</Key><VersionId>null</VersionId></Deleted><Deleted><Key>2b54
0f88-6a38-4fe5-b5e5-ad50795c17fe</Key><VersionId>null</VersionId></Deleted><Deleted><Key>0142421a-12ea-4a9e-861d-2b3543c162ba</Key><VersionId>null</VersionId></Deleted><Deleted><Key>9fa5d246-7dde-4320-9299-26dfd875aded</Key><VersionId>null</VersionId></
Deleted><Deleted><Key>feb38f85-76e8-48fd-92a8-ae86fb75c18f</Key><VersionId>null</VersionId></Deleted><Deleted><Key>ae107035-e1d1-4bf4-99b5-c9c1deabeebb</Key><VersionId>null</VersionId></Deleted><Deleted><Key>e6af51d1-b887-4579-b566-76d8f88ccbfc</Key><Ve
rsionId>null</VersionId></Deleted><Deleted><Key>74eaa3b9-447e-4cc8-8c9b-96858d8b5b01</Key><VersionId>null</VersionId></Deleted><Deleted><Key>fe8eaffa-3787-4f6e-a3ea-4eaf926f67d4</Key><VersionId>null</VersionId></Deleted></DeleteResult>

Whereas, terraform-provider-aws v2.29.0 is seen deleting single objects in multiple s3/DeleteObjects API call (there's 200 s3/DeleteObjects API calls in the 200 objects example):

2021-04-12T10:21:42.792Z [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: 2021/04/12 10:21:42 [INFO] Deleting S3 Bucket (test-bucket) Object (0557c246-7d7e-4701-bd7c-493322160f6d) Version: null
2021-04-12T10:21:42.792Z [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: 2021/04/12 10:21:42 [DEBUG] [aws-sdk-go] DEBUG: Request s3/DeleteObject Details:
2021-04-12T10:21:42.792Z [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: ---[ REQUEST POST-SIGN ]-----------------------------
2021-04-12T10:21:42.792Z [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: DELETE /0557c246-7d7e-4701-bd7c-493322160f6d?versionId=null HTTP/1.1
2021-04-12T10:21:42.792Z [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: -----------------------------------------------------
2021-04-12T10:21:42.941Z [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: 2021/04/12 10:21:42 [DEBUG] [aws-sdk-go] DEBUG: Response s3/DeleteObject Details:
2021-04-12T10:21:42.942Z [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: ---[ RESPONSE ]--------------------------------------
2021-04-12T10:21:42.942Z [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: HTTP/1.1 204 No Content
2021-04-12T10:21:42.942Z [DEBUG] plugin.terraform-provider-aws_v2.29.0_x4: -----------------------------------------------------

There doesn't seem to be an elegant workaround for this issue and its limiting our ability to upgrade.

gwvandesteeg · 2021-06-23T05:32:05Z

This has been a constant thorn in my side during development of an environment, if left overnight (and the bucket is receiving VPC flow logs, or access logs of other buckets) then the delete/teardown will take hours. Generally this is in the 20k objects region including versioned files and it'll take even longer if the bucket has an object_lock set to GOVERNANCE.
To work around this i've implemented a quick script (i've got one for ruby or python) that is run prior to the teardown of the stack.

To find the active items, we use the ListObjectsV2 API call (iterating over blocks of 1000 items), and when deleting them we only specify the key in the list of ObjectIdentifier structs to DeleteObjects API call (and providing the BypassGovernanceRetention flag in the delete if object lock is enabled in the bucket).

After those have been deleted in batches of 1000
Do the same using the ListObjectVersions

Then this process is repeated for the DeleteMarkers and Versions lists in the ListObjectVersionsOutput response to the ListObjectVersions PAI call, again using blocks of 1000 objects.

You probably also want to deal with the multi-part uploads (not in use in our case).
This can take some time but at least it works in a timely manner (and saves on API calls).

breathingdust · 2022-02-03T14:19:42Z

Hi all 👋 Just letting you know that this is issue is featured on this quarters roadmap. If a PR exists to close the issue a maintainer will review and either make changes directly, or work with the original author to get the contribution merged. If you have written a PR to resolve the issue please ensure the "Allow edits from maintainers" box is checked. Thanks for your patience and we are looking forward to getting this merged soon!

ojongerius · 2022-04-01T00:57:42Z

@ewbankkit are you working on this atm?

ewbankkit · 2022-04-01T11:20:59Z

@ojongerius Yes, it's in my queue.

github-actions · 2022-04-07T22:38:42Z

This functionality has been released in v4.9.0 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

github-actions · 2022-05-08T02:31:18Z

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

ghost added the service/s3 Issues and PRs that pertain to the s3 service. label Feb 24, 2020

github-actions bot added the needs-triage Waiting for first response or review from a maintainer. label Feb 24, 2020

ewbankkit mentioned this issue Mar 10, 2020

S3 bucket deletion takes a lot of time for buckets with large number of objects #12323

Closed

justinretzolk removed the needs-triage Waiting for first response or review from a maintainer. label Oct 26, 2021

ewbankkit self-assigned this Mar 30, 2022

ewbankkit added the bug Addresses a defect in current functionality. label Apr 2, 2022

ewbankkit mentioned this issue Apr 4, 2022

r/aws_s3_bucket: Speed up deletion #24020

Merged

ewbankkit closed this as completed in #24020 Apr 5, 2022

github-actions bot added this to the v4.9.0 milestone Apr 5, 2022

github-actions bot locked as resolved and limited conversation to collaborators May 8, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

S3 bucket slow to delete when destroyed during an apply #12146

S3 bucket slow to delete when destroyed during an apply #12146

atoms42 commented Feb 24, 2020 •

edited

Loading

ewbankkit commented Feb 24, 2020 •

edited

Loading

atoms42 commented Feb 24, 2020

tiny-dancer commented Feb 25, 2020

dead10ck commented Mar 6, 2020

nithinvenugopal commented Mar 10, 2020 •

edited

Loading

timleecasey commented Jul 30, 2020 •

edited

Loading

jurajseffer commented Aug 17, 2020

mlushpenko commented Sep 14, 2020 •

edited

Loading

nithinvenugopal commented Nov 25, 2020 •

edited

Loading

ghost commented Feb 22, 2021

bassmanitram commented Apr 9, 2021 •

edited

Loading

sbarakat commented Apr 12, 2021

gwvandesteeg commented Jun 23, 2021

breathingdust commented Feb 3, 2022

ojongerius commented Apr 1, 2022

ewbankkit commented Apr 1, 2022

github-actions bot commented Apr 7, 2022

github-actions bot commented May 8, 2022

S3 bucket slow to delete when destroyed during an apply #12146

S3 bucket slow to delete when destroyed during an apply #12146

Comments

atoms42 commented Feb 24, 2020 • edited Loading

Community Note

Terraform Version

Affected Resource(s)

Terraform Configuration Files

Expected Behavior

Actual Behavior

Steps to Reproduce

Important Factoids

ewbankkit commented Feb 24, 2020 • edited Loading

atoms42 commented Feb 24, 2020

tiny-dancer commented Feb 25, 2020

dead10ck commented Mar 6, 2020

nithinvenugopal commented Mar 10, 2020 • edited Loading

timleecasey commented Jul 30, 2020 • edited Loading

jurajseffer commented Aug 17, 2020

mlushpenko commented Sep 14, 2020 • edited Loading

nithinvenugopal commented Nov 25, 2020 • edited Loading

ghost commented Feb 22, 2021

bassmanitram commented Apr 9, 2021 • edited Loading

sbarakat commented Apr 12, 2021

gwvandesteeg commented Jun 23, 2021

breathingdust commented Feb 3, 2022

ojongerius commented Apr 1, 2022

ewbankkit commented Apr 1, 2022

github-actions bot commented Apr 7, 2022

github-actions bot commented May 8, 2022

atoms42 commented Feb 24, 2020 •

edited

Loading

ewbankkit commented Feb 24, 2020 •

edited

Loading

nithinvenugopal commented Mar 10, 2020 •

edited

Loading

timleecasey commented Jul 30, 2020 •

edited

Loading

mlushpenko commented Sep 14, 2020 •

edited

Loading

nithinvenugopal commented Nov 25, 2020 •

edited

Loading

bassmanitram commented Apr 9, 2021 •

edited

Loading