CdkPipelines: UpdatePipeline stage still running standard:6.0 w/ node16 and fails

[ahammond]

Describe the bug

This breaks cdk pipelines at the UpdatePipeline stage:

[Container] 2025/02/17 00:54:20.400751 Running command cdk -a . deploy InfratestServicePipeline --require-approval=never --verbose
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!                                                                            !!
!!  Node 16 has reached end-of-life on 2023-09-11 and is not supported.       !!

Regression Issue

• Select this option if this issue appears to be a regression.

Last Known Working CDK Version

none

Expected Behavior

The buildspec of the Build stage of cdk pipelines includes a "test -f .nvmrc && n auto || true",, but the UpdatePipeline stage does not. I would expect ALL stages of the pipeline to respect the .nvmrc file, when it exists.

I would expect ALL stages to run supported versions of node by default.

Current Behavior

Attempts to run node16 and breaks.

Reproduction Steps

Deploy a CDK pipeline, trigger a run and watch it fail. I'm seeing failures across > 100 pipelines in our org at this time.

Possible Solution

Add "test -f .nvmrc && n auto || true", to the UpdatePipeline stage.

Default to using supported node (ideally latest node)

Additional Information/Context

No response

CDK CLI Version

2.1000.0 (build 1c60bc6) --- uh... this is suprising to me.

Framework Version

tested on 2.179.0

Node.js Version

22.14.0 on laptop, apparently 16.? in CodePipeline

OS

MacOS

Language

TypeScript

Language Version

TypeScript (5.7.3)

Other information

No response

[hbracy]

+1 We are also running into this

[ahammond]

I just realize that we're already customizing the installCommands for all our Builds. That's where the respect .nvmrc stuff is coming from. We'll probably workaround this by applying that for all ShellStep stages.

[rix0rrr]

The warning banner is definitely new, that was not intentional. It should not actually cause anything to fail though.

[ahammond]

Here's the contents of the logs from the UpdatePipeline stage:

[Container] 2025/02/18 19:42:39.295548 Phase complete: INSTALL State: SUCCEEDED
[Container] 2025/02/18 19:42:39.295577 Phase context status code:  Message: 
[Container] 2025/02/18 19:42:39.330582 Entering phase PRE_BUILD
[Container] 2025/02/18 19:42:39.332300 Phase complete: PRE_BUILD State: SUCCEEDED
[Container] 2025/02/18 19:42:39.332314 Phase context status code:  Message: 
[Container] 2025/02/18 19:42:39.367742 Entering phase BUILD
[Container] 2025/02/18 19:42:39.368978 Running command cdk -a . deploy InfratestServicePipeline --require-approval=never --verbose
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!!                                                                            !!
!!  Node 16 has reached end-of-life on 2023-09-11 and is not supported.       !!
!!  Please upgrade to a supported node version as soon as possible.           !!
!!                                                                            !!
!!  This software is currently running on node v16.20.2.                      !!
!!  As of the current release of this software, supported node releases are:  !!
!!  - ^22.0.0 (Planned end-of-life: 2027-04-30)                               !!
!!  - ^20.0.0 (Planned end-of-life: 2026-04-30)                               !!
!!  - ^18.0.0 (Planned end-of-life: 2025-04-30)                               !!
!!                                                                            !!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
/usr/local/lib/node_modules/aws-cdk/lib/index.js:334202
    var wasmModule = new WebAssembly.Module(bytes);
                     ^

CompileError: WebAssembly.Module(): invalid value type 'externref', enable with --experimental-wasm-reftypes @+139
    at ../../node_modules/cdk-from-cfn/index.js (/usr/local/lib/node_modules/aws-cdk/lib/index.js:334202:22)
    at __require (/usr/local/lib/node_modules/aws-cdk/lib/index.js:16:50)
    at lib/commands/migrate.ts (/usr/local/lib/node_modules/aws-cdk/lib/index.js:336442:28)
    at __init (/usr/local/lib/node_modules/aws-cdk/lib/index.js:13:56)
    at lib/cli/cdk-toolkit.ts (/usr/local/lib/node_modules/aws-cdk/lib/index.js:337442:5)
    at __init (/usr/local/lib/node_modules/aws-cdk/lib/index.js:13:56)
    at lib/cli/cli.ts (/usr/local/lib/node_modules/aws-cdk/lib/index.js:345803:5)
    at __init (/usr/local/lib/node_modules/aws-cdk/lib/index.js:13:56)
    at Object.<anonymous> (/usr/local/lib/node_modules/aws-cdk/lib/index.js:346028:14)
    at Module._compile (node:internal/modules/cjs/loader:1198:14)

[Container] 2025/02/18 19:42:40.259284 Command did not exit successfully cdk -a . deploy InfratestServicePipeline --require-approval=never --verbose exit status 1
[Container] 2025/02/18 19:42:40.262560 Phase complete: BUILD State: FAILED
[Container] 2025/02/18 19:42:40.262574 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: cdk -a . deploy InfratestServicePipeline --require-approval=never --verbose. Reason: exit status 1
[Container] 2025/02/18 19:42:40.300969 Entering phase POST_BUILD
[Container] 2025/02/18 19:42:40.302492 Phase complete: POST_BUILD State: SUCCEEDED
[Container] 2025/02/18 19:42:40.302504 Phase context status code:  Message: 
[Container] 2025/02/18 19:42:40.351476 Set report auto-discover timeout to 5 seconds
[Container] 2025/02/18 19:42:40.351511 Expanding base directory path:  .
[Container] 2025/02/18 19:42:40.352991 Assembling file list
[Container] 2025/02/18 19:42:40.353003 Expanding .
[Container] 2025/02/18 19:42:40.354556 Expanding file paths for base directory .
[Container] 2025/02/18 19:42:40.354566 Assembling file list
[Container] 2025/02/18 19:42:40.354570 Expanding **/*
[Container] 2025/02/18 19:42:40.359511 No matching auto-discover report paths found
[Container] 2025/02/18 19:42:40.359528 Report auto-discover file discovery took 0.008051 seconds
[Container] 2025/02/18 19:42:40.359539 Phase complete: UPLOAD_ARTIFACTS State: SUCCEEDED
[Container] 2025/02/18 19:42:40.359544 Phase context status code:  Message: 

This pipeline hasn't had any updates applied to it between this run and the previous successful run.

The failed run is me manually triggering the pipeline against the current main HEAD. The prior, succeeding run, is when that main HEAD was merged on GH.

We were guessing that it's exploding because of node16 hitting some kind of end-of-life. We know that this repo has a .nvmrc setting node 22.14.0 (with the corepack fix from last week). We know that it is running that in all CI workflows and in the Build step.

[ahammond]

For context, we aren't married to .nvmrc, and also have the following in all our .npmrc files:

# ~~ Generated by projen. To modify, edit .projenrc.ts and run "npx projen".

resolution-mode=highest
package-manager-strict=false
use-node-version=22.14.0

[mrlikl]

The issue is not reproducible with the latest version of aws-cdk-lib: 2.179.0 which is using aws/codebuild/standard:7.0 (node default v18) for the updatepipeline build project. Older versions of cdk are using aws/codebuild/standard:6.0 for which the default node version is 16.

[ashishdhingra]

The issue is not reproducible with the latest version of aws-cdk-lib: 2.179.0 which is using aws/codebuild/standard:7.0 (node default v18) for the updatepipeline build project. Older versions of cdk are using aws/codebuild/standard:6.0 for which the default node version is 16.

@mrlikl Thankd for the verification. @ahammond Please validate this using latest version of aws-cdk-lib and see if issue goes away. Regarding AWS CDK CLI version 2.1000.0 (build 1c60bc6), please refer pinned issue #32775.

Thanks,
Ashish

[ahammond]

Oh, it's nice to see the cli and the lib finally separated. As noted above, I first attempted to upgrade aws-cdk-lib to latest. It's currently runnign 2.179.0. I have also grovelled my pnpm-lock.yaml to confirm that is the only version installed. When I run

❯ pnpm cdk diff InfratestServicePipeline
start: Building 09dcb4a24c3139d15f13df6845605afa65ba6e8bb0abdba97dd8cc3f5c0ad21b:013819181968-us-east-1
success: Built 09dcb4a24c3139d15f13df6845605afa65ba6e8bb0abdba97dd8cc3f5c0ad21b:013819181968-us-east-1
start: Publishing 09dcb4a24c3139d15f13df6845605afa65ba6e8bb0abdba97dd8cc3f5c0ad21b:013819181968-us-east-1
success: Published 09dcb4a24c3139d15f13df6845605afa65ba6e8bb0abdba97dd8cc3f5c0ad21b:013819181968-us-east-1
Hold on while we create a read-only change set to get a diff with accurate replacement information (use --no-change-set to use a less accurate but faster template-only diff)
Stack InfratestServicePipeline
There were no differences

✨  Number of stacks with differences: 0

I confirm that this is what is deployed.

When I look at the deployed template in Cfn console, I see

    "InfratestServicePipelineUpdatePipelineSelfMutationCCE1DB9A": {
      "Type": "AWS::CodeBuild::Project",
      "Properties": {
        "Artifacts": {
          "Type": "CODEPIPELINE"
        },
        "Cache": {
          "Type": "NO_CACHE"
        },
        "Description": "Pipeline step InfratestServicePipeline/Pipeline/UpdatePipeline/SelfMutate",
        "EncryptionKey": "arn:aws:kms:us-east-1:013819181968:key/f4e3acee-d676-4116-a4f7-9194c2a46f71",
        "Environment": {
          "ComputeType": "BUILD_GENERAL1_SMALL",
          "Image": "aws/codebuild/standard:6.0",
          "ImagePullCredentialsType": "CODEBUILD",
          "PrivilegedMode": true,
          "Type": "LINUX_CONTAINER"
        },
        "Name": "InfratestServicePipeline-selfupdate",

So... It look like this isn't getting updated.

[ahammond]

I'm going to tear down and re-deploy this stack to see if it get the standard:7.0. I have plenty of other stacks with this issue. I'll get pretty major pushback around history loss, but it's fine for this one repo for testing.

[ahammond]

Oh! Found it! We have a library which wraps the cdk-pipelines stuff to handle a whole bunch of things including bumping the default for the BuildImage. We did this when the default was 5 to pick up 6 earlier... and haven't updated it since. Fixed.

[github-actions]

Comments on closed issues and PRs are hard for our team to see.
If you need help, please open a new issue that references this one.