DockerImageCode: docker build fails on Apple silicon

[xkjjx]

Describe the bug

The following code:

    const dockerLambda = new DockerImageFunction(this, 'dockerFunction', {
      code: DockerImageCode.fromImageAsset('path',{
        cacheDisabled: true,
        platform: Platform.LINUX_AMD64
      }),
      memorySize: 3008,
      timeout: Duration.seconds(30)
    });

works randomly about 1/5 of the time on my macbook, but seems to work reliably on an ubuntu x86 VM.

The error is regarding how one specific layer cannot be pushed, and how lambda can't use it since it hasn't stabilized yet.

Regression Issue

• Select this option if this issue appears to be a regression.

Last Known Working CDK Version

No response

Expected Behavior

Expected to build my docker from the Dockerfile and deploy with no errors.

Current Behavior

On Apple Silicon there seems to be an error.

First time:

<dockerFunction> (<dockerFunction>347AB729) Resource handler returned message: "Lambda function <Stack>-<dockerFunction>347AB7-CJVpjHMzkl8f reached terminal FAILED state due to InvalidImage(ImageLayerFailure: UnsupportedImageLayerDetected - Layer Digest sha256:<hash>) and failed to stabilize" (RequestToken: 1d2edfb3-09ef-a4a6-b356-72029656351a, HandlerErrorCode: NotStabilized)

After the first time:

<Stack> | 9:22:59 AM | CREATE_FAILED        | AWS::Lambda::Function       | <dockerFunction> (<dockerFunction>347AB729) Resource handler returned message: "Lambda function <lambdaFunction>-d39w2Cx1DtJT reached terminal FAILED state due to InvalidImage(ImageLayerFailure: UnsupportedImageLayerDetected - Layer Digest sha256:<hash>) and failed to stabilize" (RequestToken: 60b31552-2d36-a508-741f-99c9ef057c50, HandlerErrorCode: NotStabilized)
❌  MdhostingStack failed: Error: The stack named <Stack> failed to deploy: UPDATE_ROLLBACK_COMPLETE: Resource handler returned message: "Lambda function <lambdaFunction> reached terminal FAILED state due to InvalidImage(ImageLayerFailure: UnsupportedImageLayerDetected - Layer Digest sha256:<hash>) and failed to stabilize" (RequestToken: 60b31552-2d36-a508-741f-99c9ef057c50, HandlerErrorCode: NotStabilized)

Reproduction Steps

Create a CDK with a Lambda Function that uses DockerImageFunction that uses DockerImageCode on Apple Silicon and try deploying it.

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.173.2

Framework Version

No response

Node.js Version

22.0.0

OS

OSX 15.3.1

Language

TypeScript

Language Version

TypeScript (5.6.3)

Other information

No response

[pahud]

Generally, I would recommend bundling on a x86 environment or pipeline if you are deploying to a x86 runtime.

I have been using this to make sure the lambda architecture will be consistent with my bundling platform.

const handler = new lambda.DockerImageFunction(this, 'Func', {
      code: lambda.DockerImageCode.fromImageAsset(local_path, {
        file: 'Dockerfile'
      }),
      architecture: os.arch() === 'arm64' ? lambda.Architecture.ARM_64 : undefined,
    });

[xkjjx]

Generally, I would recommend bundling on a x86 environment or pipeline if you are deploying to a x86 runtime.

I have been using this to make sure the lambda architecture will be consistent with my bundling platform.

const handler = new lambda.DockerImageFunction(this, 'Func', {
code: lambda.DockerImageCode.fromImageAsset(local_path, {
file: 'Dockerfile'
}),
architecture: os.arch() === 'arm64' ? lambda.Architecture.ARM_64 : undefined,
});

This did not fix the issue - I suspect it is something wrong with Apple Silicon in particular?

[BwL1289]

I am also experiencing this. I am deploying ARM on ARM so I don't know why this would occur.

EDIT:

In my case this was due to the image not being built at all (even though it looked like it had been built). And therefore, the layers were invalid because there were no layers and it couldn't be uploaded.

[pahud]

@xkjjx @BwL1289

Are you able to provide a self-sustained CDK app that I can try in my MBP M4? I'll need to view the Dockerfile as well.

[xkjjx]

@pahud

The code that I initially had the bug on was extremely complex and heavy, with around 450MB in the node_modules directory. Regardless, I tried to reproduce the bug in a simple environment, and was not able to. Everything deployed successfully through my Macbook without issue.

One important thing to note is that online, many people have detailed the many settings that one has to change in order to deploy from an Apple Silicon Macbook without issue, but in a separate branch, I pretty much removed all of these, only to have:

  const exampleLambda = new DockerImageFunction(this, 'exampleLambda', {
    code: DockerImageCode.fromImageAsset('docker'),
  });

and the deployment still worked flawlessly.

HERE is the repo.

and HERE is the branch without any options.

I have moved on to using a different architecture for my project that doesn't involve Docker Lambda functions, but when free I will try to make any changes to see if the bug can be reproduced, and will update this thread.