[Cluster] ray running inside docker on a cloud VM losing GPU access after few hours

[lobo1586]

What happened + What you expected to happen

I believe i am currently affected by NVIDIA/nvidia-docker#1671

We deployed ray serve to a cloud compute VM using cluster_config.yaml, after running couple hours, the docker suddenly lost access to the host GPU.

This post seems to published some workaround: NVIDIA/nvidia-docker#1730

However i am not sure how can i integrate any of those solutions into the ray up cluster_config.yaml process

Versions / Dependencies

our container base image is from rayproject/ray-ml:2.24.0-py310-gpu
our GCP boot disk source image is from: projects/ml-images/global/images/c0-deeplearning-common-cu118-v20240613-debian-11-py310

Reproduction script

see linked nvidia issue

Issue Severity

High: It blocks me from completing my task.

[lobo1586]

ok, looks like it can be fixed using nvidia's posted workaround mentioned in NVIDIA/nvidia-docker#1730, however this would require a one time fix effort, but not sure if feasible it is when there are multiple worker node with GPU

[rynewang]

If I read this correctly, the workaround involves running a script on each node. Can you try to put the command here in the cluster_config.yaml? https://docs.ray.io/en/latest/cluster/vms/references/ray-cluster-configuration.html#initialization-commands

Generally speaking we expect NVIDIA to fix the issue for good, such that we don't need extra commands to fix.