BucketDeployment: SIGKILL 9 during S3 sync step

[TrevorBurnham]

Describe the bug

After a recent CDK update, I started seeing this error on some deploys when using BucketDeployment:

cfn_error: b"Command '['/opt/awscli/aws', 's3', 'sync', '/tmp/tmps02annce/contents', 's3://beta-us-east-1-main-assets-v1/']' died with <Signals.SIGKILL: 9>."

The error seems to occur at random, forcing me to frequently retry failed deploys.

I've searched for this same error message but haven't been able to find it. The closest I've found is the issue described at #29862, which is also a SIGKILL but occurs when running the 'aws s3 cp' command rather than 'aws s3 sync'. Also, I'm not sure why I'd be running out of memory: The total size of the files I'm deploying is only about 30MB, so I'd expect the 128MB default to be plenty.

I can't say definitively whether this error is caused by a change in CDK or in some other piece of AWS infrastructure.

Regression Issue

• Select this option if this issue appears to be a regression.

Last Known Working CDK Version

No response

Expected Behavior

Deploys using BucketDeployment should either work or provide a clear error message in the logs when something goes wrong.

Current Behavior

The error message above is all I see in the logs. It's followed up by a report that says:

Memory Size: 128 MB Max Memory Used: 128 MB

That suggests that the default memory limit may be the issue… except that I see the same values in the logs when the deploy succeeded. And, again, the files I'm uploading total only 30MB, so what could be using all that memory?

Reproduction Steps

It's unclear how to reproduce, since the bug is non-deterministic.

Possible Solution

The improvement proposed at #29898 might address this.

Additional Information/Context

I'm going to try raising memoryLimit to see if it prevents the issue for me. At this point, I don't know.

CDK CLI Version

2.178.2 (build 89c49cc)

Framework Version

No response

Node.js Version

20.16.0

OS

macOS

Language

Java

Language Version

Java (17)

Other information

No response

[pahud]

Yes, related to #29898

I think the long-term solution is to implement it using SDK with better retry/backoff control.

[TrevorBurnham]

Update: After adding memoryLimit(1024), I got a successful deploy with this message in the REPORT:

Memory Size: 1024 MB Max Memory Used: 272 MB

That certainly suggests that the default 128MB memory limit was exceeded. But if that's the case, that raises a few mysteries:

1. Why is uploading ~30MB of files using 272MB of memory in the Lambda?
2. Why is it happening inconsistently (that is, uploading the same files uses <128MB of memory sometimes)?
3. Why can't I find any instances of the same error message on the internet? (I can find plenty of reports of a SIGKILL during aws s3 cp, but none during aws s3 sync.)
4. Why did this only start happening in my project last week?