Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

aws-cdk synth: increased synth time on Node22 runtime #33576

Open
1 task
justin-masse opened this issue Feb 20, 2025 · 9 comments
Open
1 task

aws-cdk synth: increased synth time on Node22 runtime #33576

justin-masse opened this issue Feb 20, 2025 · 9 comments
Assignees
Labels

Comments

@justin-masse
Copy link

Describe the bug

When updating from Node18 or Node20 on MacOS (15.3) to Node22 (22.13.1) synth times grow exponentially based on the size of the service being synthed.

I am seeing increased synth times across our services when increasing from Node20 to Node22 on our github actions and locally. Some benchmarks below:

Service A (small service, maybe 50 resources total):

  • Node18: 15s
  • Node20: 15s
  • Node22: 20s

Service B (larger service, hundreds of resources):

  • Node20: 2m
  • Node22: 15m+

Our synth times are generally longer because we synth stacks for multiple regions and multiple accounts at the same time. For service B if I reduce this to just one environment then it will synth on Node20 in about ~20s and takes 2m 10s on Node22. So I expect that this impact is amplified with the more stacks you are sything and may not be noticeable when synthing only one environment at a time with a small cdk app?

For reference we are using bootstrapless synthesizer.

Regression Issue

  • Select this option if this issue appears to be a regression.

Last Known Working CDK Version

No response

Expected Behavior

No change in synth times when using newer versions of node

Current Behavior

Synth not going brrr

Reproduction Steps

  1. Synth in node18/20 on local env
  2. nvm use 22
  3. run synth again and it takes much longer even for very large services.

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.1000.2

Framework Version

No response

Node.js Version

22

OS

MacOS 15.3

Language

TypeScript

Language Version

5.7.3

Other information

No response

@justin-masse
Copy link
Author

Verified this is happening on latest of V22 node and V23 as well.

@pahud
Copy link
Contributor

pahud commented Feb 20, 2025

Thank you for the report. We'll investigate this issue and please help us prioritize with 👍 if anyone is having the same issues. We'll bring this up to CDK engineering team for visibility.

@pahud pahud added the p2 label Feb 20, 2025
@funador
Copy link

funador commented Feb 21, 2025

Also experiencing this with updates to node 22 on our larger services

@HeatherFlux
Copy link

Been having this issue as well lately, it is stopping us from migrating some of our repos to node 22.

@pahud pahud added p1 and removed p2 labels Feb 24, 2025
@godwingrs22 godwingrs22 transferred this issue from aws/aws-cdk Feb 24, 2025
@mrgrain
Copy link
Contributor

mrgrain commented Feb 25, 2025

What operating systems and architectures are you all on?

@mrgrain
Copy link
Contributor

mrgrain commented Feb 25, 2025

Current Findings

  • I have not been able reproduce the extreme slow-downs that are reported.
  • From my benchmarks with CDK apps, Node22 seems to be consistently slower by 5-10%
  • Initially we thought Maglev (new JIT compiler in V8) could be the culprit. However after running benchmarks for node 20 and 22 with and without Maglev, there is no evidence it is the root cause. I've noticed some performance differences, but nothing substantial and the pattern is mostly the same that Node20 is just a little faster than Node22. Mind you that this is still without a proper reproduction.

@mrgrain mrgrain transferred this issue from aws/aws-cdk-cli Feb 25, 2025
@rix0rrr
Copy link
Contributor

rix0rrr commented Feb 25, 2025

We'd also be interested in if you're running compiled JavaScript, or are running TypeScript directly through ts-node or tsx.

Are you using Node from Homebrew or elsewhere?

Of course, any reproducing repository is appreciated even more dearly.

@mrgrain
Copy link
Contributor

mrgrain commented Feb 25, 2025

Current reports indicate the issue is related to ts-node. Especially when then CDK app imports large packages, the slowdown gets much worse.

I would appreciate confirmation by anyone who is effected:

  • Do you use ts-node? Check the content of cdk.json. If the app value containts ts-node, you are using it.
  • Does the slowness persist when you run the app directly (without cdk synth)? Take the app value from cdk.json and run the command in your terminal. This is expected to be a little faster anyway, however the node22 slowness should still be noticeable.
  • If you are able to, build your app (e.g. by running npx tsc) and then execute the JS files directlly: node bin/app.js. Is this still slow?

@justin-masse
Copy link
Author

We were able to get this resolved in our app with a few changes:

  1. Stopped importing a large package with many modules into our CDK app. This was an ~800kb commonJS package that was importing everything into the app because there was no tree shaking.
  2. We also swapped from ts-node to tsx to improve performance.

On node20 we were seeing ~150-180s synth times with our previous setup and when swapping to node22 it catapulted to 15-20 mins. When we removed this package and swapped to tsx we are seeing ~30s synth times on both node20 and 22.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

7 participants