Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Move ICFY stat generation to Github Actions #39646

Merged
merged 26 commits into from
Mar 2, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
63 changes: 0 additions & 63 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@ references:
name: Create Directories for Results and Artifacts
command: |
mkdir -p \
"$CIRCLE_ARTIFACTS/icfy" \
"$CIRCLE_ARTIFACTS/notifications-panel" \
"$CIRCLE_ARTIFACTS/translate" \
"$CIRCLE_ARTIFACTS/screenshots" \
Expand Down Expand Up @@ -464,65 +463,6 @@ jobs:
- save_cache: *save-jest-cache
- store-artifacts-and-test-results

icfy-stats:
<<: *defaults
steps:
- prepare
- restore_cache: *restore-babel-client-cache
- restore_cache: *restore-terser-cache
- run:
name: Build Stats
environment:
NODE_ENV: 'production'
BROWSERSLIST_ENV: 'defaults'
command: npm run analyze-icfy
- save_cache: *save-terser-cache
- save_cache: *save-babel-client-cache
- store-artifacts-and-test-results
- run:
name: Notify ICFY of failed build
when: on_fail
command: |
#
# This block should not cause a test failure and block PRs.
# The shell should never error and exit 0 to indicate success.
#
set +o errexit
curl -X POST \
"http://api.iscalypsofastyet.com:5000/submit-stats-failed?secret=$ICFY_SECRET" \
-H 'Cache-Control: no-cache' \
-H 'Content-Type: application/json' \
-d '{
"payload": {
"branch": "'"$CIRCLE_BRANCH"'",
"build_num": '"$CIRCLE_BUILD_NUM"',
"sha": "'"$CIRCLE_SHA1"'"
}
}'
- run:
name: Notify ICFY
command: |
#
# This block should not cause a test failure and block PRs.
# The shell should never error and exit 0 to indicate success.
#
set +o errexit
if [ -e "$CIRCLE_ARTIFACTS/icfy/stats.json" ] && [ -e "$CIRCLE_ARTIFACTS/icfy/chart.json" ]; then
ANCESTOR_SHA1=$(git merge-base HEAD origin/master)
curl -X POST \
"http://api.iscalypsofastyet.com:5000/submit-stats?secret=$ICFY_SECRET" \
-H 'Cache-Control: no-cache' \
-H 'Content-Type: application/json' \
-d '{
"payload": {
"branch": "'"$CIRCLE_BRANCH"'",
"build_num": '"$CIRCLE_BUILD_NUM"',
"sha": "'"$CIRCLE_SHA1"'",
"ancestor": "'"$ANCESTOR_SHA1"'"
}
}'
fi

# Prime calypso.live so it has a build ready
#
# We can send a request to calypso.live so that it gets a build ready.
Expand Down Expand Up @@ -674,9 +614,6 @@ workflows:
- build-wpcom-block-editor:
requires:
- setup
- icfy-stats:
requires:
- setup
- lint-and-translate:
requires:
- setup
Expand Down
52 changes: 52 additions & 0 deletions .github/workflows/icfy-stats.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
name: Calculate ICFY stats

on: [push]

jobs:
build:
name: Build ICFY stats

runs-on: ubuntu-latest

steps:
- name: Set up Node
uses: actions/setup-node@v1
with:
node-version: '>=12.15.0'
- name: Checkout code
uses: actions/checkout@v2
- name: Fetch git history
run: git fetch --prune --unshallow
- name: Install dependencies
env:
PUPPETEER_SKIP_CHROMIUM_DOWNLOAD: true
run: npm ci
- name: Build ICFY stats
env:
NODE_ENV: production
BROWSERSLIST_ENV: defaults
WORKERS: 2
run: npm run analyze-icfy
- run: mkdir icfy-stats && mv client/{chart,stats}.json icfy-stats
- uses: actions/upload-artifact@v1
with:
name: icfy
path: icfy-stats
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we publish the artifacts as two plain files, stats.json and chart.json? That would be much more convenient for the ICFY client. It now downloads the files directly into memory. Now, it would need to unpack the ZIP to some tmp directory, read the JSON files from there, cleanup at the end...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This uses https://github.com/actions/upload-artifact. Docs make it sound like it will always be a zip:

You must upload artifacts during a workflow run. GitHub provides two actions that you can use to upload and download build artifacts. Files uploaded to a workflow run are archived using the .zip format. For more information, see the actions/upload-artifact and download-artifact actions.

https://help.github.com/en/actions/configuring-and-managing-workflows/persisting-workflow-data-using-artifacts

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Doesn't seem like it. https://github.com/actions/upload-artifact

It wants a path to a directory and a name

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure it helps, but zips can be used as streams with https://www.npmjs.com/package/node-stream-zip

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

node-stream-zip can only read from files 🙁 I need something that can read from a generic stream. Will continue looking for another package.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Found https://www.npmjs.com/package/unzipper which works great so far 👍

- name: Upload build artifact
env:
ICFY_SECRET: ${{ secrets.ICFY_SECRET }}
run: |
ANCESTOR_SHA1=$(git merge-base HEAD origin/master)
CURRENT_BRANCH=$(git rev-parse --abbrev-ref HEAD)
curl -X POST --globoff \
"http://api.iscalypsofastyet.com:5000/submit-stats?from=github&secret=$ICFY_SECRET" \
-H 'Cache-Control: no-cache' \
-H 'Content-Type: application/json' \
-d '{
"payload": {
"branch": "'"$CURRENT_BRANCH"'",
"build_num": '"$GITHUB_RUN_ID"',
"sha": "'"$GITHUB_SHA"'",
"ancestor": "'"$ANCESTOR_SHA1"'"
}
}'
2 changes: 1 addition & 1 deletion package.json
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@
},
"scripts": {
"analyze-bundles": "npm run build-client-stats && webpack-bundle-analyzer client/stats.json public/evergreen -h 127.0.0.1 -p 9898 -s gzip",
"analyze-icfy": "npm run build-client-stats && node --max-old-space-size=4096 bin/icfy-analyze.js && mv client/stats.json client/chart.json \"$CIRCLE_ARTIFACTS/icfy\"",
"analyze-icfy": "npm run build-client-stats && node --max-old-space-size=4096 bin/icfy-analyze.js",
"autoprefixer": "postcss -u autoprefixer -r --no-map --config packages/calypso-build/postcss.config.js",
"postcss": "postcss -r --config packages/calypso-build/postcss.config.js",
"prebuild": "npm run install-if-deps-outdated",
Expand Down