Implement GPU passthrough functionality to provide hw-acceleration to inner-containers #50

rodnymolina · 2020-09-06T00:08:11Z

Our goal here is to allow Sysbox container hierarchies to make use of hardware-acceleration capabilities offered by system devices. Device 'passthrough' is a concept that applies naturally to system-containers, and as such, it has been on our mind since Sysbox's early days, but it recently came up as part of a conversation with @jamierajewski (thanks for that).

A couple of scenarios where this would be useful:

Development environments: A sysbox container would wrap inner containers with GUI requirements.
K8s environments: A sysbox container could act as a K8s node and have PODs running apps with GUI demands.

Even though most of the concepts described here are applicable to any GPU, we will limit the scope of this issue to Nvidia GPU's; let's create separate issues for other GPUs.

At high-level, these are some of the requirements that Sysbox would need to meet:

Sysbox should identify the GPU devices in the host and expose them automatically to the sysbox containers (through 'devices' oci-spec attribute).
Sysbox should provide a mechanism that allows cuda-toolkit and related nvidia tools, which are required at host-level, to be shared (bind-mounted?) with sysbox containers. This would address two problems:
1. Cuda-toolkit and drivers installed within sysbox containers must fully match those installed in the host (end-user wouldn't know which version to fetch at image build-time).
2. Cuda packages are quite large, which would bloat sysbox images.
Sysbox should allow proper execution of the nvidia-container-runtime within the system-containers, which should expose all the required abstractions for nvidia-runtime to operate as if it were running in the host.

This list of requirements will obviously change as we further understand the problem.

rodnymolina · 2020-09-06T00:45:25Z

As expected, gpu-related resources are properly passed inside the sys container ...

$ docker run --runtime=sysbox-runc -it --rm --name test-1 --device=/dev/dri --device=/dev/nvidia0 --device=/dev/nvidiactl --device=/dev/nvidia-modeset --device=/dev/nvidia-uvm --device=/dev/nvidia-uvm-tools --device=/dev/vga_arbiter nestybox/ubuntu-focal-systemd-docker

admin@d9e318c31bec:~$ ls -lrt /dev/dri /dev/nvidia*
crw-rw-rw- 1 nobody nogroup 195, 255 Sep  5 20:30 /dev/nvidiactl
crw-rw-rw- 1 nobody nogroup 195,   0 Sep  5 20:30 /dev/nvidia0
crw-rw-rw- 1 nobody nogroup 235,   1 Sep  5 20:30 /dev/nvidia-uvm-tools
crw-rw-rw- 1 nobody nogroup 235,   0 Sep  5 20:30 /dev/nvidia-uvm
crw-rw-rw- 1 nobody nogroup 195, 254 Sep  5 20:30 /dev/nvidia-modeset

/dev/dri:
total 0
crw-rw----+ 1 nobody nogroup 226, 128 Sep  5 20:30 renderD128
crw-rw----+ 1 nobody nogroup 226,   0 Sep  5 20:30 card0
admin@d9e318c31bec:~$

<-- Install same nvidia-driver as in the host (i.e. "440"):

rodny@nestybox-srv-01:~$ sudo nvidia-smi
[sudo] password for rodny:
Sat Sep  5 14:17:57 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.100      Driver Version: 440.100      CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Quadro K620         Off  | 00000000:03:00.0 Off |                  N/A |
| 35%   49C    P8     1W /  30W |    177MiB /  1994MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      1438      G   /usr/lib/xorg/Xorg                            31MiB |
|    0      3138      G   /usr/lib/xorg/Xorg                            51MiB |
|    0      3343      G   /usr/bin/gnome-shell                          83MiB |
+-----------------------------------------------------------------------------+
rodny@nestybox-srv-01:~$

admin@d9e318c31bec:~$ sudo apt install nvidia-driver-440

<-- Device/driver properly seen within sys container:

admin@d9e318c31bec:~$ sudo nvidia-smi
[sudo] password for admin:
Sat Sep  5 21:11:47 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 440.100      Driver Version: 440.100      CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  Quadro K620         Off  | 00000000:03:00.0 Off |                  N/A |
| 34%   48C    P8     1W /  30W |    177MiB /  1994MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
+-----------------------------------------------------------------------------+
admin@d9e318c31bec:~$

<-- Install nvidia-runtime within sys container:

admin@d9e318c31bec:~$ curl -s -L https://nvidia.github.io/nvidia-docker/gpgkey | \
sudo apt-key add -

admin@d9e318c31bec:~$ distribution=$(. /etc/os-release;echo $ID$VERSION_ID)

admin@d9e318c31bec:~$ curl -s -L https://nvidia.github.io/nvidia-docker/$distribution/nvidia-docker.list | \
sudo tee /etc/apt/sources.list.d/nvidia-docker.list

admin@d9e318c31bec:~$ sudo apt-get update

admin@d9e318c31bec:~$ sudo apt-get install -y nvidia-docker2

admin@d9e318c31bec:~$ sudo pkill -SIGHUP dockerd

<-- Check everything's looking good so far:

admin@d9e318c31bec:~$ sudo nvidia-container-cli --load-kmods info
NVRM version:   440.100
CUDA version:   10.2

Device Index:   0
Device Minor:   0
Model:          Quadro K620
Brand:          Quadro
GPU UUID:       GPU-6c5b3240-538b-0d41-f327-285da7535a9c
Bus Location:   00000000:03:00.0
Architecture:   5.0
admin@d9e318c31bec:~$

<-- Finally, let's launch a cuda-app as an L2 container:

admin@d9e318c31bec:~$ docker run --rm --runtime=nvidia -ti nvidia/cuda
Unable to find image 'nvidia/cuda:latest' locally
latest: Pulling from nvidia/cuda
3ff22d22a855: Pull complete
e7cb79d19722: Pull complete
323d0d660b6a: Pull complete
b7f616834fd0: Pull complete
c2607e16e933: Pull complete
46a16da628dc: Pull complete
4871b8b75027: Pull complete
e45235afa764: Pull complete
250da266cf64: Pull complete
78f4b6d02e6c: Pull complete
ebf42dcedf4b: Pull complete
Digest: sha256:0fe0406ec4e456ae682226751434bdd7e9b729a03067d795f9b34c978772b515
Status: Downloaded newer image for nvidia/cuda:latest
docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"process_linux.go:432: running prestart hook 1 caused \\\"error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: requirement error: unsatisfied condition: cuda>=11.0, please update your driver to a newer version, or use an earlier cuda container\\\\n\\\"\"": unknown.
admin@d9e318c31bec:~$

<-- Let's pick an earlier cuda-app image -- looks like our nvidia driver (440) may not be the latest.

admin@d9e318c31bec:~$ docker run --rm --runtime=nvidia -ti nvidia/cuda:10.0-base
Unable to find image 'nvidia/cuda:10.0-base' locally
10.0-base: Pulling from nvidia/cuda
7ddbc47eeb70: Pull complete
c1bbdc448b72: Pull complete
8c3b70e39044: Pull complete
45d437916d57: Pull complete
d8f1569ddae6: Pull complete
de5a2c57c41d: Pull complete
ea6f04a00543: Pull complete
Digest: sha256:e6e1001f286d084f8a3aea991afbcfe92cd389ad1f4883491d43631f152f175e
Status: Downloaded newer image for nvidia/cuda:10.0-base
docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"process_linux.go:432: running prestart hook 1 caused \\\"error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: mount error: write error: /sys/fs/cgroup/devices/docker/27422711f8b99961b9ac6f6758f2016dc930a4704b2655bcc3943fdfb1eb72df/devices.allow: operation not permitted\\\\n\\\"\"": unknown.
admin@d9e318c31bec:~$

Hmm, need to check what is this nvidia prestart hook doing to trigger an EPERM while mounting this resource ...

evberrypi · 2020-11-16T05:13:09Z

Will this work for multiple GPUs?
eg. --device=/dev/nvidia{0,1,2,3}

rodnymolina · 2020-11-16T19:57:06Z

@evberrypi, we haven't initiated this effort yet, so we may find surprises that could easily influence this feature's scope, but yes, we do intend to support multiple GPUs.

Ideally, our runtime should be capable of detecting the available GPUs and expose them automatically within the system-container. Alternatively, the user should be able to specify which GPU to utilize inside the container, and have only that one exposed in the container's rootfs.

Would this meet your requirements? Also, if you don't mind, can you please describe what's the use-case / setup you have in mind? Thanks.

evberrypi · 2020-11-16T22:25:42Z

The use case would be running sysbox to replace some CI build/test steps that need to be ran on special, pet like servers with multiple GPUs in favor of something that can run across various servers. Support for 2 or more GPUs is desired. Auto detecting the # of GPUs and allocating them could work, but preferably specifying the number of GPUs to pass to the container would be ideal and would absolutely meet requirements. Thanks for the prompt reply!

SoloGao · 2020-11-26T06:31:21Z

We are also working on this. Nvidia-docker is unable to deal with sysbox-modified userns-remap or cgroups. But no-cgroups = true in /etc/nvidia-container-toolkit/config.toml will cause Failed to initialize NVML: Unknown Error in system docker. As for now, there's a very dirty solution.
First, according to the debug logs provided by nvidia-container-toolkit, Nvidia didn't only mount the devices to the system container. Instead of installing driver with the same version, they just bind mount them. However, when we try to mount the drivers in /usr/lib/x86_64-linux-gnu, it simply tells us operation not permitted.
So, we firstly copy everything end with .so.{nv_drv_ver} to /usr/lib/x86_64-cuda(like here is *.so.455.32.00), then run with this:

docker run --detach --interactive --runtime=sysbox-runc \
--mount type=tmpfs,destination=/proc/driver/nvidia \
--mount type=bind,source=/usr/bin/nvidia-smi,target=/usr/bin/nvidia-smi \
--mount type=bind,source=/usr/bin/nvidia-debugdump,target=/usr/bin/nvidia-debugdump \
--mount type=bind,source=/usr/bin/nvidia-persistenced,target=/usr/bin/nvidia-persistenced \
--mount type=bind,source=/usr/bin/nvidia-cuda-mps-control,target=/usr/bin/nvidia-cuda-mps-control \
--mount type=bind,source=/usr/bin/nvidia-cuda-mps-server,target=/usr/bin/nvidia-cuda-mps-server \
--mount type=bind,source=/usr/lib/x86_64-cuda/libnvidia-ml.so.455.32.00,target=/usr/lib/x86_64-linux-gnu/libnvidia-ml.so.455.32.00 \
--mount type=bind,source=/usr/lib/x86_64-cuda/libnvidia-cfg.so.455.32.00,target=/usr/lib/x86_64-linux-gnu/libnvidia-cfg.so.455.32.00 \
--mount type=bind,source=/usr/lib/x86_64-cuda/libcuda.so.455.32.00,target=/usr/lib/x86_64-linux-gnu/libcuda.so.455.32.00 \
--mount type=bind,source=/usr/lib/x86_64-cuda/libnvidia-opencl.so.455.32.00,target=/usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.455.32.00 \
--mount type=bind,source=/usr/lib/x86_64-cuda/libnvidia-ptxjitcompiler.so.455.32.00,target=/usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.455.32.00 \
--mount type=bind,source=/usr/lib/x86_64-cuda/libnvidia-allocator.so.455.32.00,target=/usr/lib/x86_64-linux-gnu/libnvidia-allocator.so.455.32.00 \
--mount type=bind,source=/usr/lib/x86_64-cuda/libnvidia-compiler.so.455.32.00,target=/usr/lib/x86_64-linux-gnu/libnvidia-compiler.so.455.32.00 \
--mount type=bind,source=/run/nvidia-persistenced/socket,target=/run/nvidia-persistenced/socket \
--device /dev/nvidiactl:/dev/nvidiactl \
--device /dev/nvidia-uvm:/dev/nvidia-uvm \
--device /dev/nvidia-uvm-tools:/dev/nvidia-uvm-tools \
--device /dev/nvidia0:/dev/nvidia0 \
--device /dev/nvidia1:/dev/nvidia1 \
--device /dev/nvidia2:/dev/nvidia2 \
--device /dev/nvidia3:/dev/nvidia3 \
--device /dev/nvidia4:/dev/nvidia4 \
--device /dev/nvidia5:/dev/nvidia5 \
--device /dev/nvidia6:/dev/nvidia6 \
--device /dev/nvidia7:/dev/nvidia7 \
nestybox/ubuntu-focal-systemd-docker:latest

It works but we don't know whether we missed something or something is not useful. Besides, /proc/driver/nvidia should be mounted read only with an overlay, but tmpfs is working too (but nvidia-docker run inside needs /proc/driver/nvidia on the host, need further investigation).
At first logging in, a sudo ldconfig.real is needed, which is also come from the log.
Then, to start inner docker, we also use a similar dirty script:

docker run \
--mount type=tmpfs,destination=/proc/driver/nvidia \
--mount type=bind,source=/usr/bin/nvidia-smi,target=/usr/bin/nvidia-smi \
--mount type=bind,source=/usr/bin/nvidia-debugdump,target=/usr/bin/nvidia-debugdump \
--mount type=bind,source=/usr/bin/nvidia-persistenced,target=/usr/bin/nvidia-persistenced \
--mount type=bind,source=/usr/bin/nvidia-cuda-mps-control,target=/usr/bin/nvidia-cuda-mps-control \
--mount type=bind,source=/usr/bin/nvidia-cuda-mps-server,target=/usr/bin/nvidia-cuda-mps-server \
--mount type=bind,source=/usr/lib/x86_64-linux-gnu/libnvidia-ml.so.455.32.00,target=/usr/lib/x86_64-linux-gnu/libnvidia-ml.so.455.32.00 \
--mount type=bind,source=/usr/lib/x86_64-linux-gnu/libnvidia-cfg.so.455.32.00,target=/usr/lib/x86_64-linux-gnu/libnvidia-cfg.so.455.32.00 \
--mount type=bind,source=/usr/lib/x86_64-linux-gnu/libcuda.so.455.32.00,target=/usr/lib/x86_64-linux-gnu/libcuda.so.455.32.00 \
--mount type=bind,source=/usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.455.32.00,target=/usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.455.32.00 \
--mount type=bind,source=/usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.455.32.00,target=/usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.455.32.00 \
--mount type=bind,source=/usr/lib/x86_64-linux-gnu/libnvidia-allocator.so.455.32.00,target=/usr/lib/x86_64-linux-gnu/libnvidia-allocator.so.455.32.00 \
--mount type=bind,source=/usr/lib/x86_64-linux-gnu/libnvidia-compiler.so.455.32.00,target=/usr/lib/x86_64-linux-gnu/libnvidia-compiler.so.455.32.00 \
--mount type=bind,source=/run/nvidia-persistenced/socket,target=/run/nvidia-persistenced/socket \
--device /dev/nvidiactl:/dev/nvidiactl \
--device /dev/nvidia-uvm:/dev/nvidia-uvm \
--device /dev/nvidia-uvm-tools:/dev/nvidia-uvm-tools \
--device /dev/nvidia0:/dev/nvidia0 \
--device /dev/nvidia1:/dev/nvidia1 \
--device /dev/nvidia2:/dev/nvidia2 \
--device /dev/nvidia3:/dev/nvidia3 \
--device /dev/nvidia4:/dev/nvidia4 \
--device /dev/nvidia5:/dev/nvidia5 \
--device /dev/nvidia6:/dev/nvidia6 \
--device /dev/nvidia7:/dev/nvidia7 \
nvcr.io/nvidia/cuda:11.1-devel-ubuntu20.04

After starting the inner docker, a ldconfig.real is also needed. Then, gpu can run in the inner docker flawlessly.
Since we only have tesla cards, the graphics function is not taken into consideration.

ctalledo · 2020-11-26T15:21:57Z

Hi @SoloGao, thanks for sharing that info, much appreciated. We've not had the cycles to add Nvidia GPU support to Sysbox yet, but your findings will certainly help when we do so.

Is this something that is high priority for you? What is the use-case you have in mind?

rodnymolina · 2020-11-26T18:49:52Z

@SoloGao, that's excellent feedback, thanks for that!

First, according to the debug logs provided by nvidia-container-toolkit, Nvidia didn't only mount the devices to the system container. Instead of installing driver with the same version, they just bind mount them. However, when we try to mount the drivers in /usr/lib/x86_64-linux-gnu, it simply tells us operation not permitted.
So, we firstly copy everything end with .so.{nv_drv_ver} to /usr/lib/x86_64-cuda(like here is *.so.455.32.00), then run with this.

We are also thinking about bind-mounting these libraries to avoid copying all these content back and forth. Ideally, sysbox should be able to hide all this complexity from the user, but i'm not sure how far we can go with this zero-touch approach.

The EPERM you are getting is expected as you're attempting to access a host-owned resource while being in a separate user-namespace. We should be able to fix that too.

Then, to start inner docker, we also use a similar dirty script:

Right, we also need to think about how we simplify this process for the user, doesn't look like an easy task given that we are relying on the regular oci-runc at this level.

Let's keep this communication channel open to exchange notes on your findings as well as our planning in this area. This is an important feature for us, we will start working on it asap.

One question for you. Have you tried to share one nvidia device across two separate sys-containers + inner-containers?

SoloGao · 2020-11-27T02:04:54Z

Hello @ctalledo, just a short answer to your question.

Is this something that is high priority for you? What is the use-case you have in mind?

GPGPU support is the must-have features for us. Basically, we are using docker to run GPGPU-intensive tasks like deep-learning on servers. NVIDIA offers lots of pre-configured docker images in https://ngc.nvidia.com and has built a workflow to run different versions of CUDA or DL frameworks easily. To safely start/stop/remove containers by user rather than admin, Podman and Sysbox is the only choices. Moreover, all containers need to expose ports for services like Tensorboard. So in order to make port arrangement manually, we opt to docker in docker with Sysbox.

SoloGao · 2020-11-27T02:22:20Z

Hi @rodnymolina, thanks for the reply and explanation.

Ideally, sysbox should be able to hide all this complexity from the user, but i'm not sure how far we can go with this zero-touch approach.
Right, we also need to think about how we simplify this process for the user, doesn't look like an easy task given that we are relying on the regular oci-runc at this level.

In my opinion, forking the https://github.com/NVIDIA/nvidia-container-toolkit might be a good starting point. NVIDIA offers a set of tools to start docker with GPU support. The docs: https://docs.nvidia.com/datacenter/cloud-native/index.html. Users might only need a patched nvidia-container-toolkit to run system/inner docker with GPGPU support. They use GoLang to do the stuffs, a superset of what I currently did with lots of validation works. However, I don't have that much time digging into the code, so I just grab the results and reproduce them now. Besides, the cgroups problem might need to solve for nvidia-container-toolkit.

Have you tried to share one nvidia device across two separate sys-containers + inner-containers?

Yes, that works flawlessly even on [sys-A(inner-a, inner-b), sys-B(inner-c, inner-d)] scheme.

rodnymolina · 2020-11-27T09:28:59Z

Thanks for your detailed responses @SoloGao, it all makes sense. Btw, I've already looked at nvidia-toolkit in the past and that's something that will certainly keep in mind.

One last thing. If possible, could you please ping me when have a chance? (rmolina@nestybox.com) There are a couple of points that I would like to clarify about your setup to make sure that we fully address your use-case.

Thanks!

shinji62 · 2021-09-16T09:19:17Z

@rodnymolina any effort on the GPU side ? We will love to see that too.

rodnymolina · 2021-09-17T01:31:31Z

@shinji62, this is one of the top items in our todo-list, but we haven't had cycles for it yet. It would certainly help if you could provide a brief description of your use-case as that help us prioritize features accordingly (please ping us on Slack if you'd rather discuss this topic there).

ctalledo · 2021-09-22T05:33:21Z

FYI: Another user for Sysbox is looking to use hardware accelerators with Sysbox towards the end of 2021.

elgalu · 2021-10-05T19:27:55Z

Me too

ctalledo · 2022-04-15T16:52:59Z

FYI: some GPU functionality does work inside a Sysbox container currently, as described in this comment in issue #452.

r614 · 2022-09-06T18:51:34Z

Are there any updates on this issue? Our use case is running heavy inference/scientific workflows inside nested containers on a Kubernetes pod.

ctalledo · 2022-09-07T15:59:34Z

Hi Roshan (@r614), unfortunately no updates yet.

As Docker recently acquired Nestybox, we are currently busy integrating Sysbox into Docker Desktop but should get some more cycles to work on Sysbox improvements within a couple of months (and GPU passthrough is one of the top items).

Thanks for your patience!

r614 · 2022-09-12T16:44:55Z

Thanks for the prompt reply - appreciate it! Would love to know to find out when you guys start work on this, happy to help test it out.

kkangle · 2023-01-17T02:22:27Z

So looking forward to seeing this feature! Is there any plan to release this in 2023?

ctalledo · 2023-01-18T05:45:01Z

Hi @kkangle, we are hoping to get some cycles to work on this soon.

What's the use case you have in mind, if you don't mind sharing?

zhongcloudtian · 2023-02-10T10:13:48Z

@ctalledo Sysbox supports the gpu function. Can we speed up the solution? Now there is a urgent project

zhongcloudtian · 2023-02-10T10:18:40Z

@ctalledo we use sysbox k8s in docker ，hoping k8s runing gpu，but it is errors ：
docker: Error response from daemon: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: Running hook #1:: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: initialization error: load library failed: libnvidia-ml.so.1: cannot open shared object file: no such file or directory: unknown.

Darker7 · 2023-02-17T14:10:27Z

Another Use Case

My university wants to offer ML execution as a CI service in its private GitLab instance, so that it can offer more ML projects in the future.

I'm working on this for my Bachelor's thesis, so I'll have to just accept the inherent danger of --privileged dind or docker.sock mounting, but I'll include a link here for future upgrades :Ü™

ctalledo · 2023-02-19T18:23:09Z

Can we speed up the solution? Now there is a urgent project

Unfortunately we still haven't had the cycles to work on this.

However, there have been some users that have had some limited success exposing GPUs inside Sysbox containers. See here for example.

zhongcloudtian · 2023-02-20T08:53:46Z

@SoloGao
Call in the tensorflow:2.9.1-gpu container in sysbox to check the number of gpus. The execution fails .
Refer to these cases #50 (comment)

christopherhesse · 2023-09-21T05:32:15Z

@ctalledo We would like to be able to use sysbox to run containerized tests on Nvidia GPUs under kubernetes without privileged mode. I'm curious, do you have an estimate on the amount of work required to add this sort of feature to sysbox?

ctalledo · 2023-09-22T22:31:48Z

Hi @christopherhesse, unfortunately I don't have an estimate at this time for a couple of reasons:

I need to fully size-up the work that is required, given the latest changes in Nvidia-Docker integration as well as advancements in the Linux kernel.
We've not yet scheduled the work inside Docker, due to other priorities.

If I may ask, what's the big advantage of using Sysbox in your scenario (e.g., why not use regular containers)?

christopherhesse · 2023-09-25T15:12:40Z

We're unable to launch regular containers inside a kubernetes pod without privileged mode. Is that supported already somehow?

ctalledo · 2023-09-25T16:18:49Z

We're unable to launch regular containers inside a kubernetes pod without privileged mode. Is that supported already somehow?

With Sysbox as the runtime for the pod yes, without it I don't believe so.

christopherhesse · 2023-09-27T18:43:57Z

@ctalledo Great, then IIUC this would be the big advantage of using Sysbox. Does that answer your question? Within a kubernetes pod, without privileged mode, we want to run a transient container for our tests and it seems like this is the most promising option there.

ctalledo · 2023-09-28T22:47:07Z

@ctalledo Great, then IIUC this would be the big advantage of using Sysbox. Does that answer your question? Within a kubernetes pod, without privileged mode, we want to run a transient container for our tests and it seems like this is the most promising option there.

Hi @christopherhesse, yes that's a very common use case: running Docker inside an unprivileged K8s pod; the pod is launched with K8s + Sysbox. This way you get a lot more isolation / security around the pod as opposed to using a privileged one. Hope that helps; if you run into any trouble let us know please. Thanks.

christopherhesse · 2023-09-29T23:41:20Z

@ctalledo Thanks for the confirmation! I am running into the issue that GPUs are not supported, since the tests require GPUs.

traverseda · 2023-11-17T18:18:29Z

We're using this feature to simulate flocks of robots, where each sysbox runs a replica of our software, as it would run on the robot.

This would be a very useful feature for us, as it lets our simulations stay closer to what our real hardware presents. Essentially we're using a nestybox runtime to simulate a complete robot stack, so we can test things like swarm-SLAM.

This is, presumably, possible without implementing nvidia-container-runtime capabilities, but it makes the job a lot harder and means porting to another compute cluster becomes a nightmare.

dray92 · 2024-05-14T22:56:10Z

@ctalledo, wanted to check to see if there was any appetite for either scoping this work out, or helping build out the capability.
We've been using Sysbox in some areas and are currently exploring sharing GPUs in pods running with K8s+Sysbox. We're looking to support some CUDA workflows that require access to GPUs at runtime.
Happy to contribute and/or help investigate potential solutions if you're open to it.

ccll · 2024-05-20T06:53:12Z

I'm using Coder as online development environment, and using sysbox in the dev container to enable dockerd-in-docker, now I need to work on a new project that do Machine Learning accelerated by GPU, hope this would be supported.

leakingtapan · 2024-09-25T22:19:36Z

looking forward to this features too, we are going to use sysbox as the runtime for CI workload on top of kubernetes, the workload needs GPU, so this will be a blocker for us. And BTW, if there is a scoping of the work, or changes required to make this happen, I will be happy to contribute too

chris-gputrader · 2025-01-11T01:16:11Z

Would be great to see this feature, what are the chances of this getting worked on in the near future?

rodnymolina added the enhancement New feature or request label Sep 6, 2020

rodnymolina self-assigned this Sep 6, 2020

0x2b3bfa0 mentioned this issue Nov 24, 2021

Standardize on container images instead of machine images iterative/terraform-provider-iterative#146

Open

rodnymolina mentioned this issue Feb 8, 2022

How to use Nvidia inside a Sysbox container #470

Closed

ctalledo added this to the v0.6 milestone Apr 26, 2022

ctalledo mentioned this issue Mar 29, 2023

How does sysbox k8s in docker schedule tensorflow/tensorflow:2.9.1-gpu？ #643

Closed

ctalledo mentioned this issue Jun 4, 2024

sysbox-fs high cpu usage, infinite unmount calls on shared device #808

Closed

thesofakillers mentioned this issue Jan 6, 2025

Clarification on Resource Utilization and Runtime Compatibility for Agents openai/mle-bench#25

Closed

Implement GPU passthrough functionality to provide hw-acceleration to inner-containers #50

Implement GPU passthrough functionality to provide hw-acceleration to inner-containers #50

Comments

rodnymolina commented Sep 6, 2020

rodnymolina commented Sep 6, 2020

evberrypi commented Nov 16, 2020

rodnymolina commented Nov 16, 2020 • edited Loading

evberrypi commented Nov 16, 2020

SoloGao commented Nov 26, 2020

ctalledo commented Nov 26, 2020

rodnymolina commented Nov 26, 2020 • edited Loading

SoloGao commented Nov 27, 2020 • edited Loading

SoloGao commented Nov 27, 2020 • edited Loading

rodnymolina commented Nov 27, 2020

shinji62 commented Sep 16, 2021

rodnymolina commented Sep 17, 2021

ctalledo commented Sep 22, 2021

elgalu commented Oct 5, 2021

ctalledo commented Apr 15, 2022

r614 commented Sep 6, 2022

ctalledo commented Sep 7, 2022

r614 commented Sep 12, 2022

kkangle commented Jan 17, 2023

ctalledo commented Jan 18, 2023

zhongcloudtian commented Feb 10, 2023

zhongcloudtian commented Feb 10, 2023

Darker7 commented Feb 17, 2023

Another Use Case

ctalledo commented Feb 19, 2023

zhongcloudtian commented Feb 20, 2023 • edited Loading

christopherhesse commented Sep 21, 2023

ctalledo commented Sep 22, 2023

christopherhesse commented Sep 25, 2023

ctalledo commented Sep 25, 2023

christopherhesse commented Sep 27, 2023

ctalledo commented Sep 28, 2023 • edited Loading

christopherhesse commented Sep 29, 2023

traverseda commented Nov 17, 2023

dray92 commented May 14, 2024

ccll commented May 20, 2024

leakingtapan commented Sep 25, 2024

chris-gputrader commented Jan 11, 2025

rodnymolina commented Nov 16, 2020 •

edited

Loading

rodnymolina commented Nov 26, 2020 •

edited

Loading

SoloGao commented Nov 27, 2020 •

edited

Loading

SoloGao commented Nov 27, 2020 •

edited

Loading

zhongcloudtian commented Feb 20, 2023 •

edited

Loading

ctalledo commented Sep 28, 2023 •

edited

Loading