Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FVTs don't run due to "No space left on device" error #367

Closed
ckadner opened this issue May 5, 2023 · 0 comments · Fixed by #368
Closed

FVTs don't run due to "No space left on device" error #367

ckadner opened this issue May 5, 2023 · 0 comments · Fixed by #368
Labels
test testing related bugs and fixes

Comments

@ckadner
Copy link
Member

ckadner commented May 5, 2023

Describe the bug

https://github.com/kserve/modelmesh-serving/actions/runs/4831653258/jobs/8609373544#step:11:367

failed to register layer: ApplyLayer exit status 1 stdout:  stderr: write /usr/local/lib64/python3.8/site-packages/tensorflow/libtensorflow_cc.so.2: no space left on device
Error: Process completed with exit code 1.

We should investigate what is using up the additional disk space and follow some best practices to reduces disk usage for GitHub action runners: https://github.com/marketplace/actions/maximize-build-disk-space

Or here: actions/runner-images#2840 (comment)

sudo rm -rf /usr/share/dotnet
sudo rm -rf /opt/ghc
sudo rm -rf "/usr/local/share/boost"
sudo rm -rf "$AGENT_TOOLSDIRECTORY"

To Reproduce
Steps to reproduce the behavior:

  1. Go to the FVT jobs list
  2. Click on any of the most recent failures, e.g. run #768
  3. Scroll down to see the build failure
  4. See error no space left on device

Expected behavior

FVT run should complete successfully, or at the very least run some of the actuall FVT tests

Additional context

These errors first appeared during the HPA PR checks after the minikube setup was changed

@Jooho @rafvasq @tjohnson31415 @njhill

@ckadner ckadner added the bug Something isn't working label May 5, 2023
ckadner added a commit to ckadner/modelmesh-serving that referenced this issue May 5, 2023
Resolves kserve#367

Signed-off-by: Christian Kadner <ckadner@us.ibm.com>
ckadner added a commit that referenced this issue May 9, 2023
Fix "no space left on device" error for FVTs on GitHub action runners
by deleting libraries that are not needed for dotnet, Android, PowerShell,
Swift which increases available disk space by 25 GB plus an additional
reclaimed space of 2.6 GB after pruning docker images.

Resolves #367

Signed-off-by: Christian Kadner <ckadner@us.ibm.com>
@ckadner ckadner added test testing related bugs and fixes and removed bug Something isn't working labels May 9, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
test testing related bugs and fixes
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant