Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(build): fast fail PR builds if rosetta extraction fails #14946

Merged
merged 15 commits into from
Jun 9, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 16 additions & 4 deletions build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -2,13 +2,15 @@
set -euo pipefail

bail="--bail"
runtarget="build+test"
runtarget="build"
run_tests="true"
check_prereqs="true"
check_compat="true"
extract_snippets="false"
while [[ "${1:-}" != "" ]]; do
case $1 in
-h|--help)
echo "Usage: build.sh [--no-bail] [--force|-f] [--skip-test]"
echo "Usage: build.sh [--no-bail] [--force|-f] [--skip-test] [--skip-prereqs] [--skip-compat] [--extract]"
exit 1
;;
--no-bail)
Expand All @@ -18,14 +20,17 @@ while [[ "${1:-}" != "" ]]; do
export CDK_BUILD="--force"
;;
--skip-test|--skip-tests)
runtarget="build"
run_tests="false"
;;
--skip-prereqs)
check_prereqs="false"
;;
--skip-compat)
check_compat="false"
;;
--extract)
extract_snippets="true"
;;
*)
echo "Unrecognized parameter: $1"
exit 1
Expand All @@ -35,7 +40,7 @@ while [[ "${1:-}" != "" ]]; do
done

export PATH=$(npm bin):$PATH
export NODE_OPTIONS="--max-old-space-size=4096 ${NODE_OPTIONS:-}"
export NODE_OPTIONS="--max-old-space-size=4096 --experimental-worker ${NODE_OPTIONS:-}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We probably need to be a bit careful here.

I believe this option is required only for Node 10. In later versions of node, this has graduated out of experimental. And not supported in earlier versions of node.

What's the experience when this option is set for version > node 10? Fail or ignore?

We could be super precise and say - if node 10 and snippets extraction is set, then use --experiemental-worker and if < node10 and snippet extraction is set, fail?

WDYT? too pedantic?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The following is just speculation on how node should behave since I haven't been able to easily find node's policies on experimental features.

Providing flags that enable experimental features after the feature has stabilized should not have any effect on behaviour (confirmed for v10, v12, and v14). Further, the flag should not be reused for another purpose. Any other experience would be very frustrating to customers, forcing them to configure different versions in different ways and breaking legacy scripts.

So I don't think it's necessary to be careful with adding this flag. I'm of course happy to do so if you feel otherwise!


I also don't think we should fail if the node version is < 10, since rosetta itself still supports single-threaded extraction (though it is quite slow, especially for the monolithic modules). Rosetta already emits a warning in this case.


if ! [ -x "$(command -v yarn)" ]; then
echo "yarn is not installed. Install it from here- https://yarnpkg.com/en/docs/install."
Expand Down Expand Up @@ -73,6 +78,13 @@ rm -rf $BUILD_INDICATOR
export MERKLE_BUILD_CACHE=$(mktemp -d)
trap "rm -rf $MERKLE_BUILD_CACHE" EXIT

if [ "$run_tests" == "true" ]; then
runtarget="$runtarget+test"
fi
if [ "$extract_snippets" == "true" ]; then
runtarget="$runtarget+extract"
fi

echo "============================================================================================="
echo "building..."
time lerna run $bail --stream $runtarget || fail
Expand Down
2 changes: 1 addition & 1 deletion buildspec-pr.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -14,4 +14,4 @@ phases:
- yarn --version || npm -g install yarn
build:
commands:
- /bin/bash ./build.sh && git diff-index --exit-code --ignore-space-at-eol --stat HEAD
- /bin/bash ./build.sh --extract && git diff-index --exit-code --ignore-space-at-eol --stat HEAD
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Out of curiosity, how many packages do we currently have rosetta extract enabled?

And what is the cost of this extra work on our build time? The PR build for this PR should give you that, baselined against any other PR build.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

203 + monocdk + aws-cdk-lib = 205 have an extraction script. It doesn't look like any of the packages have strict extraction enabled currently.

As I commented above, it seems to be about a 9 minute increase in build time. Not horrible but also not great. From a cursory investigation, it seems to take <50ms on my local machine to bring each worker online, so we don't seem to be losing much to overhead subprocess costs. I can do a deeper dive on how to speed it up if you think it's warranted.

Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,9 @@
"awslint": "cdk-awslint",
"build+test+package": "yarn build+test && yarn package",
"build+test": "yarn build && yarn test",
"compat": "cdk-compat"
"compat": "cdk-compat",
"build+extract": "yarn build",
"build+test+extract": "yarn build+test"
},
"keywords": [
"aws",
Expand Down
4 changes: 3 additions & 1 deletion packages/@aws-cdk/alexa-ask/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,9 @@
"build+test": "yarn build && yarn test",
"compat": "cdk-compat",
"gen": "cfn2ts",
"rosetta:extract": "yarn --silent jsii-rosetta extract"
"rosetta:extract": "yarn --silent jsii-rosetta extract",
"build+extract": "yarn build && yarn rosetta:extract",
"build+test+extract": "yarn build+test && yarn rosetta:extract"
},
"cdk-build": {
"cloudformation": "Alexa::ASK",
Expand Down
4 changes: 3 additions & 1 deletion packages/@aws-cdk/app-delivery/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,9 @@
"build+test+package": "yarn build+test && yarn package",
"build+test": "yarn build && yarn test",
"compat": "cdk-compat",
"rosetta:extract": "yarn --silent jsii-rosetta extract"
"rosetta:extract": "yarn --silent jsii-rosetta extract",
"build+extract": "yarn build && yarn rosetta:extract",
"build+test+extract": "yarn build+test && yarn rosetta:extract"
},
"dependencies": {
"@aws-cdk/aws-cloudformation": "0.0.0",
Expand Down
4 changes: 3 additions & 1 deletion packages/@aws-cdk/assert-internal/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,9 @@
"pkglint": "pkglint -f",
"package": "cdk-package",
"build+test+package": "yarn build+test && yarn package",
"build+test": "yarn build && yarn test"
"build+test": "yarn build && yarn test",
"build+extract": "yarn build",
"build+test+extract": "yarn build+test"
},
"author": {
"name": "Amazon Web Services",
Expand Down
4 changes: 3 additions & 1 deletion packages/@aws-cdk/assert/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,9 @@
"pkglint": "pkglint -f",
"package": "cdk-package",
"build+test+package": "yarn build+test && yarn package",
"build+test": "yarn build && yarn test"
"build+test": "yarn build && yarn test",
"build+extract": "yarn build",
"build+test+extract": "yarn build+test"
},
"cdk-build": {
"jest": true,
Expand Down
10 changes: 8 additions & 2 deletions packages/@aws-cdk/assets/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,9 @@
"build+test+package": "yarn build+test && yarn package",
"build+test": "yarn build && yarn test",
"compat": "cdk-compat",
"rosetta:extract": "yarn --silent jsii-rosetta extract"
"rosetta:extract": "yarn --silent jsii-rosetta extract",
"build+extract": "yarn build && yarn rosetta:extract",
"build+test+extract": "yarn build+test && yarn rosetta:extract"
},
"cdk-build": {
"pre": [
Expand Down Expand Up @@ -98,7 +100,11 @@
"announce": false
},
"nozem": {
"ostools": ["rm", "tar", "gzip"]
"ostools": [
"rm",
"tar",
"gzip"
]
},
"publishConfig": {
"tag": "latest"
Expand Down
4 changes: 3 additions & 1 deletion packages/@aws-cdk/aws-accessanalyzer/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,9 @@
"build+test": "yarn build && yarn test",
"build+test+package": "yarn build+test && yarn package",
"gen": "cfn2ts",
"rosetta:extract": "yarn --silent jsii-rosetta extract"
"rosetta:extract": "yarn --silent jsii-rosetta extract",
"build+extract": "yarn build && yarn rosetta:extract",
"build+test+extract": "yarn build+test && yarn rosetta:extract"
},
"cdk-build": {
"cloudformation": "AWS::AccessAnalyzer",
Expand Down
4 changes: 3 additions & 1 deletion packages/@aws-cdk/aws-acmpca/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,9 @@
"build+test": "yarn build && yarn test",
"compat": "cdk-compat",
"gen": "cfn2ts",
"rosetta:extract": "yarn --silent jsii-rosetta extract"
"rosetta:extract": "yarn --silent jsii-rosetta extract",
"build+extract": "yarn build && yarn rosetta:extract",
"build+test+extract": "yarn build+test && yarn rosetta:extract"
},
"cdk-build": {
"cloudformation": "AWS::ACMPCA",
Expand Down
4 changes: 3 additions & 1 deletion packages/@aws-cdk/aws-amazonmq/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,9 @@
"build+test": "yarn build && yarn test",
"compat": "cdk-compat",
"gen": "cfn2ts",
"rosetta:extract": "yarn --silent jsii-rosetta extract"
"rosetta:extract": "yarn --silent jsii-rosetta extract",
"build+extract": "yarn build && yarn rosetta:extract",
"build+test+extract": "yarn build+test && yarn rosetta:extract"
},
"cdk-build": {
"cloudformation": "AWS::AmazonMQ",
Expand Down
4 changes: 3 additions & 1 deletion packages/@aws-cdk/aws-amplify/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,9 @@
"build+test+package": "yarn build+test && yarn package",
"compat": "cdk-compat",
"gen": "cfn2ts",
"rosetta:extract": "yarn --silent jsii-rosetta extract"
"rosetta:extract": "yarn --silent jsii-rosetta extract",
"build+extract": "yarn build && yarn rosetta:extract",
"build+test+extract": "yarn build+test && yarn rosetta:extract"
},
"cdk-build": {
"cloudformation": "AWS::Amplify",
Expand Down
4 changes: 3 additions & 1 deletion packages/@aws-cdk/aws-apigateway/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,9 @@
"build+test": "yarn build && yarn test",
"compat": "cdk-compat",
"gen": "cfn2ts",
"rosetta:extract": "yarn --silent jsii-rosetta extract"
"rosetta:extract": "yarn --silent jsii-rosetta extract",
"build+extract": "yarn build && yarn rosetta:extract",
"build+test+extract": "yarn build+test && yarn rosetta:extract"
},
"cdk-build": {
"cloudformation": "AWS::ApiGateway",
Expand Down
4 changes: 3 additions & 1 deletion packages/@aws-cdk/aws-apigatewayv2-authorizers/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,9 @@
"compat": "cdk-compat",
"build+test": "yarn build && yarn test",
"build+test+package": "yarn build+test && yarn package",
"rosetta:extract": "yarn --silent jsii-rosetta extract"
"rosetta:extract": "yarn --silent jsii-rosetta extract",
"build+extract": "yarn build && yarn rosetta:extract",
"build+test+extract": "yarn build+test && yarn rosetta:extract"
},
"cdk-build": {
"jest": true,
Expand Down
4 changes: 3 additions & 1 deletion packages/@aws-cdk/aws-apigatewayv2-integrations/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,9 @@
"compat": "cdk-compat",
"build+test": "yarn build && yarn test",
"build+test+package": "yarn build+test && yarn package",
"rosetta:extract": "yarn --silent jsii-rosetta extract"
"rosetta:extract": "yarn --silent jsii-rosetta extract",
"build+extract": "yarn build && yarn rosetta:extract",
"build+test+extract": "yarn build+test && yarn rosetta:extract"
},
"cdk-build": {
"jest": true,
Expand Down
4 changes: 3 additions & 1 deletion packages/@aws-cdk/aws-apigatewayv2/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,9 @@
"build+test": "yarn build && yarn test",
"build+test+package": "yarn build+test && yarn package",
"gen": "cfn2ts",
"rosetta:extract": "yarn --silent jsii-rosetta extract"
"rosetta:extract": "yarn --silent jsii-rosetta extract",
"build+extract": "yarn build && yarn rosetta:extract",
"build+test+extract": "yarn build+test && yarn rosetta:extract"
},
"cdk-build": {
"cloudformation": "AWS::ApiGatewayV2",
Expand Down
4 changes: 3 additions & 1 deletion packages/@aws-cdk/aws-appconfig/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,9 @@
"build+test": "yarn build && yarn test",
"compat": "cdk-compat",
"gen": "cfn2ts",
"rosetta:extract": "yarn --silent jsii-rosetta extract"
"rosetta:extract": "yarn --silent jsii-rosetta extract",
"build+extract": "yarn build && yarn rosetta:extract",
"build+test+extract": "yarn build+test && yarn rosetta:extract"
},
"cdk-build": {
"cloudformation": "AWS::AppConfig",
Expand Down
4 changes: 3 additions & 1 deletion packages/@aws-cdk/aws-appflow/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,9 @@
"build+test": "yarn build && yarn test",
"compat": "cdk-compat",
"gen": "cfn2ts",
"rosetta:extract": "yarn --silent jsii-rosetta extract"
"rosetta:extract": "yarn --silent jsii-rosetta extract",
"build+extract": "yarn build && yarn rosetta:extract",
"build+test+extract": "yarn build+test && yarn rosetta:extract"
},
"cdk-build": {
"cloudformation": "AWS::AppFlow",
Expand Down
4 changes: 3 additions & 1 deletion packages/@aws-cdk/aws-appintegrations/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,9 @@
"build+test+package": "yarn build+test && yarn package",
"compat": "cdk-compat",
"gen": "cfn2ts",
"rosetta:extract": "yarn --silent jsii-rosetta extract"
"rosetta:extract": "yarn --silent jsii-rosetta extract",
"build+extract": "yarn build && yarn rosetta:extract",
"build+test+extract": "yarn build+test && yarn rosetta:extract"
},
"cdk-build": {
"cloudformation": "AWS::AppIntegrations",
Expand Down
4 changes: 3 additions & 1 deletion packages/@aws-cdk/aws-applicationautoscaling/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,9 @@
"build+test": "yarn build && yarn test",
"compat": "cdk-compat",
"gen": "cfn2ts",
"rosetta:extract": "yarn --silent jsii-rosetta extract"
"rosetta:extract": "yarn --silent jsii-rosetta extract",
"build+extract": "yarn build && yarn rosetta:extract",
"build+test+extract": "yarn build+test && yarn rosetta:extract"
},
"cdk-build": {
"cloudformation": "AWS::ApplicationAutoScaling",
Expand Down
4 changes: 3 additions & 1 deletion packages/@aws-cdk/aws-applicationinsights/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,9 @@
"build+test": "yarn build && yarn test",
"compat": "cdk-compat",
"gen": "cfn2ts",
"rosetta:extract": "yarn --silent jsii-rosetta extract"
"rosetta:extract": "yarn --silent jsii-rosetta extract",
"build+extract": "yarn build && yarn rosetta:extract",
"build+test+extract": "yarn build+test && yarn rosetta:extract"
},
"cdk-build": {
"cloudformation": "AWS::ApplicationInsights",
Expand Down
4 changes: 3 additions & 1 deletion packages/@aws-cdk/aws-appmesh/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,9 @@
"build+test": "yarn build && yarn test",
"compat": "cdk-compat",
"gen": "cfn2ts",
"rosetta:extract": "yarn --silent jsii-rosetta extract"
"rosetta:extract": "yarn --silent jsii-rosetta extract",
"build+extract": "yarn build && yarn rosetta:extract",
"build+test+extract": "yarn build+test && yarn rosetta:extract"
},
"cdk-build": {
"cloudformation": "AWS::AppMesh",
Expand Down
4 changes: 3 additions & 1 deletion packages/@aws-cdk/aws-apprunner/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -52,7 +52,9 @@
"build+test+package": "yarn build+test && yarn package",
"compat": "cdk-compat",
"gen": "cfn2ts",
"rosetta:extract": "yarn --silent jsii-rosetta extract"
"rosetta:extract": "yarn --silent jsii-rosetta extract",
"build+extract": "yarn build && yarn rosetta:extract",
"build+test+extract": "yarn build+test && yarn rosetta:extract"
},
"cdk-build": {
"cloudformation": "AWS::AppRunner",
Expand Down
4 changes: 3 additions & 1 deletion packages/@aws-cdk/aws-appstream/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,9 @@
"build+test": "yarn build && yarn test",
"compat": "cdk-compat",
"gen": "cfn2ts",
"rosetta:extract": "yarn --silent jsii-rosetta extract"
"rosetta:extract": "yarn --silent jsii-rosetta extract",
"build+extract": "yarn build && yarn rosetta:extract",
"build+test+extract": "yarn build+test && yarn rosetta:extract"
},
"cdk-build": {
"cloudformation": "AWS::AppStream",
Expand Down
4 changes: 3 additions & 1 deletion packages/@aws-cdk/aws-appsync/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,9 @@
"build+test": "yarn build && yarn test",
"compat": "cdk-compat",
"gen": "cfn2ts",
"rosetta:extract": "yarn --silent jsii-rosetta extract"
"rosetta:extract": "yarn --silent jsii-rosetta extract",
"build+extract": "yarn build && yarn rosetta:extract",
"build+test+extract": "yarn build+test && yarn rosetta:extract"
},
"cdk-build": {
"cloudformation": "AWS::AppSync",
Expand Down
4 changes: 3 additions & 1 deletion packages/@aws-cdk/aws-athena/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -56,7 +56,9 @@
"build+test": "yarn build && yarn test",
"compat": "cdk-compat",
"gen": "cfn2ts",
"rosetta:extract": "yarn --silent jsii-rosetta extract"
"rosetta:extract": "yarn --silent jsii-rosetta extract",
"build+extract": "yarn build && yarn rosetta:extract",
"build+test+extract": "yarn build+test && yarn rosetta:extract"
},
"keywords": [
"aws",
Expand Down
4 changes: 3 additions & 1 deletion packages/@aws-cdk/aws-auditmanager/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,9 @@
"build+test": "yarn build && yarn test",
"compat": "cdk-compat",
"gen": "cfn2ts",
"rosetta:extract": "yarn --silent jsii-rosetta extract"
"rosetta:extract": "yarn --silent jsii-rosetta extract",
"build+extract": "yarn build && yarn rosetta:extract",
"build+test+extract": "yarn build+test && yarn rosetta:extract"
},
"cdk-build": {
"cloudformation": "AWS::AuditManager",
Expand Down
4 changes: 3 additions & 1 deletion packages/@aws-cdk/aws-autoscaling-common/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,9 @@
"build+test+package": "yarn build+test && yarn package",
"build+test": "yarn build && yarn test",
"compat": "cdk-compat",
"rosetta:extract": "yarn --silent jsii-rosetta extract"
"rosetta:extract": "yarn --silent jsii-rosetta extract",
"build+extract": "yarn build && yarn rosetta:extract",
"build+test+extract": "yarn build+test && yarn rosetta:extract"
},
"keywords": [
"aws",
Expand Down
4 changes: 3 additions & 1 deletion packages/@aws-cdk/aws-autoscaling-hooktargets/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,9 @@
"build+test+package": "yarn build+test && yarn package",
"build+test": "yarn build && yarn test",
"compat": "cdk-compat",
"rosetta:extract": "yarn --silent jsii-rosetta extract"
"rosetta:extract": "yarn --silent jsii-rosetta extract",
"build+extract": "yarn build && yarn rosetta:extract",
"build+test+extract": "yarn build+test && yarn rosetta:extract"
},
"keywords": [
"aws",
Expand Down
4 changes: 3 additions & 1 deletion packages/@aws-cdk/aws-autoscaling/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,9 @@
"build+test": "yarn build && yarn test",
"compat": "cdk-compat",
"gen": "cfn2ts",
"rosetta:extract": "yarn --silent jsii-rosetta extract"
"rosetta:extract": "yarn --silent jsii-rosetta extract",
"build+extract": "yarn build && yarn rosetta:extract",
"build+test+extract": "yarn build+test && yarn rosetta:extract"
},
"cdk-build": {
"cloudformation": "AWS::AutoScaling",
Expand Down
4 changes: 3 additions & 1 deletion packages/@aws-cdk/aws-autoscalingplans/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,9 @@
"build+test": "yarn build && yarn test",
"compat": "cdk-compat",
"gen": "cfn2ts",
"rosetta:extract": "yarn --silent jsii-rosetta extract"
"rosetta:extract": "yarn --silent jsii-rosetta extract",
"build+extract": "yarn build && yarn rosetta:extract",
"build+test+extract": "yarn build+test && yarn rosetta:extract"
},
"cdk-build": {
"cloudformation": "AWS::AutoScalingPlans",
Expand Down
Loading