A caral server cannot use multiple GPUs at the same time

[yuexishuihan]

A caral server cannot use multiple GPUs at the same time.

carla 0.9.11 docker+nvidia docker mod.
I can use one of the GPUs, which I can control, but I can’t use more than one at the same time, and the current GPU utilization rate is not high, only 30-40%.
command

sudo docker run -d -p 2000-2002:2000-2002 --runtime=nvidia -e NVIDIA_VISIBLE_DEVICES=0 carlasim/carla:0.9.11 /bin/bash -c 'SDL_VIDEODRIVER=offscreen SDL_HINT_CUDA_DEVICE=0 ./CarlaUE4.sh -ResX=800 -ResY=600 -nosound -windowed -opengl'

NVIDIA_VISIBLE_DEVICES
Does anyone have the same problem as me？
Platform:Ubuntu 20.04.1 LTS
Carla version: 0.9.11
NVRM version: NVIDIA UNIX x86_64 Kernel Module 460.32.03 Sun Dec 27 19:00:34 UTC 2020
1d:00.0 VGA compatible controller: NVIDIA Corporation Device 2204 (rev a1)
1d:00.1 Audio device: NVIDIA Corporation Device 1aef (rev a1)
b4:00.0 VGA compatible controller: NVIDIA Corporation Device 2204 (rev a1)
b4:00.1 Audio device: NVIDIA Corporation Device 1aef (rev a1)

[qhaas]

I have noticed that a single carla process cannot effectively use multiple GPUs at once. I'm not sure if SLI is supported on Linux like it is on Windows with the underlying UnrealEngine, which would explain this.

Worse still, it will not automatically 'load balance' on multi-gpu systems and will just throw multiple server runs onto gpu 0, even if gpu 0 is pegged at 100% and the other GPUs are idle. As you discovered, you can use docker or SDL environment variables (outside of docker) to manually restrict it when running with OpenGL. Unfortunately, using environment variables appears broken with vulkan, at least when using UE 4.25 (which, to be fair, is unsupported by CARLA).

To be fair, I think this is more of an UnrealEngine thing than a CARLA thing.

[yuexishuihan]

Okay, thank you for your answer. The bad thing is that it seems impossible to use multiple GPUs in a short time.

[qhaas]

Might want to ask over at UnrealEngine Answers, they recently helped me with UnrealEngine GPU selection questions.

[yuexishuihan]

Okay.This is a good idea, I will try

[corkyw10]

@bernatx could you have a look at this please?

[qhaas]

@yuexishuihan , did you discover anything about leveraging multiple GPUs for CARLA? The closest we have come is just running multiple client/servers on the same GPU (i.e. larger number of independent simulations).

[yuexishuihan]

Not yet, I am currently exploring adding/modifying assets in carla, including cars,
Maps, weather, etc.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[jimmyw404]

I'm very interested in multi GPU or multi computer CARLA simulation! If anyone has gotten multippe gpus to work or has tried to distribute the carla computing please let us know!!!!

[gouchaoer]

any progress?
I am doing realtime 4 camera output in HIL, and I found it's impossible

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[jimmyw404]

I'm still very interested in this! Unreal doesn't seem to handle multi-GPU simulation very well, so solving this problem is non-trivial.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[jimmyw404]

Still very interested in this feature and looking forward to https://github.com/carla-simulator/carla/tree/bernat/multi_gpu !!!

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.