Skip to content

Commit

Permalink
[SPARK-44688][INFRA] Add a file existence check before executing `fre…
Browse files Browse the repository at this point in the history
…e_disk_space` and `free_disk_space_container`

### What changes were proposed in this pull request?
This pr add a file existence check before executing `dev/free_disk_space` and `dev/free_disk_space_container`

### Why are the changes needed?
We added `free_disk_space` and `free_disk_space_container` to clean up the disk, but because the daily tests of other branches and the master branch share the yml file, we should check if the file exists before execution, otherwise it will affect the daily tests of other branches.

- branch-3.5: https://github.com/apache/spark/actions/runs/5761479443
- branch-3.4: https://github.com/apache/spark/actions/runs/5760423900
- branch-3.3: https://github.com/apache/spark/actions/runs/5759384052

<img width="1073" alt="image" src="https://github.com/apache/spark/assets/1475305/6e46b34b-645a-4da5-b9c3-8a89bfacabcb">

### Does this PR introduce _any_ user-facing change?
No

### How was this patch tested?
- Pass GitHub Action

Closes #42359 from LuciferYang/test-free_disk_space-exist.

Authored-by: yangjie01 <yangjie01@baidu.com>
Signed-off-by: yangjie01 <yangjie01@baidu.com>
  • Loading branch information
LuciferYang authored and pull[bot] committed Mar 14, 2024
1 parent 41fb4d3 commit 2016101
Showing 1 changed file with 15 additions and 4 deletions.
19 changes: 15 additions & 4 deletions .github/workflows/build_and_test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -241,7 +241,10 @@ jobs:
restore-keys: |
${{ matrix.java }}-${{ matrix.hadoop }}-coursier-
- name: Free up disk space
run: ./dev/free_disk_space
run: |
if [ -f ./dev/free_disk_space ]; then
./dev/free_disk_space
fi
- name: Install Java ${{ matrix.java }}
uses: actions/setup-java@v3
with:
Expand Down Expand Up @@ -419,7 +422,9 @@ jobs:
# uninstall libraries dedicated for ML testing
python3.9 -m pip uninstall -y torch torchvision torcheval torchtnt tensorboard mlflow
fi
./dev/free_disk_space_container
if [ -f ./dev/free_disk_space_container ]; then
./dev/free_disk_space_container
fi
- name: Install Java ${{ matrix.java }}
uses: actions/setup-java@v3
with:
Expand Down Expand Up @@ -519,7 +524,10 @@ jobs:
restore-keys: |
sparkr-coursier-
- name: Free up disk space
run: ./dev/free_disk_space_container
run: |
if [ -f ./dev/free_disk_space_container ]; then
./dev/free_disk_space_container
fi
- name: Install Java ${{ inputs.java }}
uses: actions/setup-java@v3
with:
Expand Down Expand Up @@ -629,7 +637,10 @@ jobs:
restore-keys: |
docs-maven-
- name: Free up disk space
run: ./dev/free_disk_space_container
run: |
if [ -f ./dev/free_disk_space_container ]; then
./dev/free_disk_space_container
fi
- name: Install Java 8
uses: actions/setup-java@v3
with:
Expand Down

0 comments on commit 2016101

Please sign in to comment.