Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: better caching of Rust deps #733

Merged
merged 5 commits into from
Sep 20, 2024
Merged

fix: better caching of Rust deps #733

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
68c1bc0
Test try-runtime stuff
ntn-x2 Sep 18, 2024
d071844
Final config
ntn-x2 Sep 19, 2024
a90d9f7
Last fixes
ntn-x2 Sep 19, 2024
8d9b0b2
Fix volume mounting for cargo-test
ntn-x2 Sep 19, 2024
e12617a
Test cache retrieval
ntn-x2 Sep 19, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
129 changes: 78 additions & 51 deletions .github/workflows/check-code.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,13 @@ jobs:
cargo-clippy:
name: Run Clippy checks
runs-on: ubuntu-latest
container:
image: paritytech/ci-unified:bullseye-1.74.0
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's out of scope for this PR, but do you think it would be worth changing the image to an environment variable? We have multiple workflow files, and using an environment variable would be less error-prone.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This would need to be a repo-wide variable, which should be updated once we bump our dependencies to a new version, if we want to share it across workflows. I agree it's out of scope, but I cc @ggera to see if there's anything low effort we can do to minimize duplication.

env:
# Configured by the Docker image. We can't change this unless the image does it.
CARGO_HOME: /usr/local/cargo
RUSTFLAGS: -D warnings
SKIP_WASM_BUILD: 1
needs: get-commit-head
if: ${{ !contains(needs.get-commit-head.outputs.headCommitMsg, 'ci-skip-rust') }}

Expand All @@ -56,28 +63,25 @@ jobs:
uses: actions/cache@v4
with:
path: |
~/.cargo/bin/
~/.cargo/registry/index/
~/.cargo/registry/cache/
~/.cargo/git/db/
${{ env.CARGO_HOME }}/bin/
${{ env.CARGO_HOME }}/registry/index/
${{ env.CARGO_HOME }}/registry/cache/
${{ env.CARGO_HOME }}/git/db/
key: ${{ github.job }}-${{ github.ref }}-${{ matrix.features }}-${{ hashFiles('**/Cargo.lock') }}
save-always: true

- name: Run `cargo clippy`
run: |
docker run --rm \
-v "${GITHUB_WORKSPACE}:/workspace" \
-v "${HOME}/.cargo:/root/.cargo" \
-w /workspace \
-e SKIP_WASM_BUILD=1 \
paritytech/ci-unified:bullseye-1.74.0 \
bash -c "cargo clippy --all-targets --locked ${{ matrix.features }} -- -D warnings"
run: cargo clippy --all-targets --locked ${{ matrix.features }}

cargo-fmt:
name: Check formatting
runs-on: ubuntu-latest
container:
image: paritytech/ci-unified:bullseye-1.74.0
env:
# Configured by the Docker image. We can't change this unless the image does it.
CARGO_HOME: /usr/local/cargo
# Latest nightly version matching the base rustc version (1.74.0).
RUSTUP_NIGHTLY_VERSION: nightly-2023-10-02
needs: get-commit-head
if: ${{ !contains(needs.get-commit-head.outputs.headCommitMsg, 'ci-skip-rust') }}
Expand All @@ -86,11 +90,21 @@ jobs:
- name: Checkout repository
uses: actions/checkout@v4

- name: Set up Cargo cache
uses: actions/cache@v4
with:
path: |
${{ env.CARGO_HOME }}/bin/
${{ env.CARGO_HOME }}/registry/index/
${{ env.CARGO_HOME }}/registry/cache/
${{ env.CARGO_HOME }}/git/db/
key: ${{ github.job }}-${{ github.ref }}-${{ hashFiles('**/Cargo.lock') }}
save-always: true

- name: Install nightly toolchain
run: rustup toolchain add ${{ env.RUSTUP_NIGHTLY_VERSION }}

- name: Run `cargo fmt`
# Latest nightly version matching the base rustc version (1.74.0)
run: cargo +${{ env.RUSTUP_NIGHTLY_VERSION }} fmt -- --check

- name: Run `taplo`
Expand All @@ -106,8 +120,7 @@ jobs:
defaults:
run:
working-directory: ${{ env.working-dir }}
needs:
- get-commit-head
needs: get-commit-head
if: ${{ !contains(needs.get-commit-head.outputs.headCommitMsg, 'ci-skip-integration-tests') }}

steps:
Expand Down Expand Up @@ -136,12 +149,14 @@ jobs:
- name: Set up Cargo cache
uses: actions/cache@v4
with:
# Not executed in a container, so we cache the local paths as on the GH worker.
path: |
~/.cargo/bin/
~/.cargo/registry/index/
~/.cargo/registry/cache/
~/.cargo/git/db/
${HOME}/.cargo/bin/
${HOME}/.cargo/registry/index/
${HOME}/.cargo/registry/cache/
${HOME}/.cargo/git/db/
key: ${{ github.job }}-${{ github.ref }}-${{ hashFiles('**/Cargo.lock') }}
save-always: true

- name: Build Peregrine runtime
run: cargo build -p peregrine-runtime
Expand All @@ -152,6 +167,9 @@ jobs:
cargo-test:
name: Run Cargo tests
runs-on: ubuntu-latest
env:
# Configured by the Docker image. We can't change this unless the image does it.
CARGO_HOME: /usr/local/cargo
needs: cargo-clippy

strategy:
Expand All @@ -173,25 +191,38 @@ jobs:
- name: Set up Cargo cache
uses: actions/cache@v4
with:
# These paths are mounted inside the Docker container.
# We cannot mount the `.cargo/bin` folder since the container already contains binaries, and overriding with an empty one breaks compilation.
path: |
~/.cargo/bin/
~/.cargo/registry/index/
~/.cargo/registry/cache/
~/.cargo/git/db/
~/.cargo/registry/index
~/.cargo/registry/cache
~/.cargo/git/db
key: ${{ github.job }}-${{ github.ref }}-${{ matrix.features }}-${{ hashFiles('**/Cargo.lock') }}
save-always: true

- name: Run `cargo test`
# We cannot use the default `container:` component because it runs out of memory, so we resort to using this solution instead.
# Maybe re-evaluate this job if anything changes GH side.
run: |
docker run --rm \
-v "${GITHUB_WORKSPACE}:/workspace" \
-v "${HOME}/.cargo:/root/.cargo" \
-v "${HOME}/.cargo/registry/index:${{ env.CARGO_HOME }}/registry/index" \
-v "${HOME}/.cargo/registry/cache:${{ env.CARGO_HOME }}/registry/cache" \
-v "${HOME}/.cargo/git/db:${{ env.CARGO_HOME }}/git/db" \
-w /workspace \
paritytech/ci-unified:bullseye-1.74.0 \
bash -c "cargo test --all-targets --locked ${{ matrix.features }}"

cargo-doc:
name: Check Rustdoc
runs-on: ubuntu-latest
container:
image: paritytech/ci-unified:bullseye-1.74.0
env:
# Configured by the Docker image. We can't change this unless the image does it.
CARGO_HOME: /usr/local/cargo
SKIP_WASM_BUILD: 1
RUSTDOCFLAGS: -D warnings
needs: cargo-clippy

strategy:
Expand All @@ -214,31 +245,26 @@ jobs:
uses: actions/cache@v4
with:
path: |
~/.cargo/bin/
~/.cargo/registry/index/
~/.cargo/registry/cache/
~/.cargo/git/db/
key: ${{ github.job }}-${{ github.ref }}-${{ hashFiles('**/Cargo.lock') }}
${{ env.CARGO_HOME }}/bin/
${{ env.CARGO_HOME }}/registry/index/
${{ env.CARGO_HOME }}/registry/cache/
${{ env.CARGO_HOME }}/git/db/
key: ${{ github.job }}-${{ github.ref }}-${{ matrix.features }}-${{ hashFiles('**/Cargo.lock') }}
save-always: true

- name: Run `cargo doc`
run: |
docker run --rm \
-v "${GITHUB_WORKSPACE}:/workspace" \
-v "${HOME}/.cargo:/root/.cargo" \
-w /workspace \
-e RUSTDOCFLAGS='-D warnings' \
-e SKIP_WASM_BUILD=1 \
paritytech/ci-unified:bullseye-1.74.0 \
bash -c "cargo doc --no-deps --locked ${{ matrix.features }}"
run: cargo doc --no-deps --locked ${{ matrix.features }}

try-runtime:
name: Run try-runtime
runs-on: ubuntu-latest
needs: cargo-clippy
env:
TRY_RUNTIME_CLI_VERSION_TAG: v0.7.0
container:
image: paritytech/ci-unified:bullseye-1.74.0
env:
# Configured by the Docker image. We can't change this unless the image does it.
CARGO_HOME: /usr/local/cargo
TRY_RUNTIME_CLI_VERSION_TAG: v0.7.0
needs: cargo-clippy

strategy:
matrix:
Expand All @@ -256,22 +282,23 @@ jobs:
with:
tool-cache: true

- name: Set up Cargo cache
uses: actions/cache@v4
with:
path: |
${{ env.CARGO_HOME }}/bin/
${{ env.CARGO_HOME }}/registry/index/
${{ env.CARGO_HOME }}/registry/cache/
${{ env.CARGO_HOME }}/git/db/
key: ${{ github.job }}-${{ github.ref }}-${{ matrix.runtime }}-${{ hashFiles('**/Cargo.lock') }}
save-always: true

- name: Install try-runtime
run: |
curl -sL https://github.com/paritytech/try-runtime-cli/releases/download/${{ env.TRY_RUNTIME_CLI_VERSION_TAG }}/try-runtime-x86_64-unknown-linux-musl -o try-runtime
chmod +x ./try-runtime
./try-runtime --version

- name: Set up Cargo cache
uses: actions/cache@v4
with:
path: |
~/.cargo/bin/
~/.cargo/registry/index/
~/.cargo/registry/cache/
~/.cargo/git/db/
key: ${{ github.job }}-${{ github.ref }}-${{ hashFiles('**/Cargo.lock') }}

- name: Build runtime
run: cargo build --release --locked -p ${{ matrix.runtime }}-runtime --features try-runtime

Expand Down
Loading