Need assistance helping an HPC user get Visit GPU rendering setup inside OOD remote desktop (non parallel)

[DevinBayly]

Hello there!

I work as a data and visualization consultant at the university of arizona, and I have a user who needs to be able to use Visit from our system in a GPU accelerated capacity. I know this question has been asked and answered in many variations, but I'm having a little bit of trouble discerning what the correct approach is.

Most importantly, I'm not a privileged user on my system and attempts to create a GPU accelerated display haven't traditionally worked out. Is there still a way (perhaps using singularity containers?) that I can take care of this? We have Open On Demand and the remote desktop application configured for this, but that display is software rendering, and so the Visit GUI doesn't automatically make use of a GPU when I open it up normally.

I took a look at http://www.visitusers.org/index.php?title=Parallel_Hardware_Acceleration, and it made me feel like there's perhaps just a set of command line arguments to try to make this happen, but with a single p100 gpu and a display running on another number (:4.0 i believe) running visit -launch-x opens the gui, but no new X processes are started that connect to the existing GPU.

I'll also try to specify the other discussions I've read and how they don't quite get me where i need to go.

• Using hardware acceleration on a single node #19625: this one is pretty recent and seems like it would be a great place to start, but they are also interested in parallel gpu usage and I'm less interested in that at the moment
• hardware acceleration help #15976: is actually from 2015 unless I'm mistaken? also they are able to get a display started that shows gpu speed ups for glxgears, and I'm not able to do that so I figure the solutions provided might be different than my situation.

Thanks so much everybody! I'll try to explain more if there's any confusion about what I'm trying to do.

[markcmiller86]

@DevinBayly sorry for delay in responding.

Before I think I can provide a useful response, I think we need some clarification.

What VisIt client are you running? What version of VisIt and on what platform (macOS, Windows, Linux)? Did you build VisIt from sources or are you using a pre-built binary downloaded from our website?

When you say you want "GPU accelerated" rendering, that is pretty much what you get out of the box if you run one of our pre-built binaries. If you can run glxgears on your client, then you should be able to run VisIt with GPU accleration there.

OTOH, if you have some parallel cluster and you have GPUs on each node of that cluster and you want VisIt to use those GPUs, we rely upon VTKm for that and, in particular, we don't render that way. We do (some) data processing that way (e.g. compute an iso-contour), but we don't use parallel node-level GPUs to render...at least not to my knowledge.

That old 2015 material is woefully out of date and unhelpful to current design and behavior.

[DevinBayly]

Hey there! happy to clarify things.

What VisIt client are you running?

the version number is 3.4.1 and I'm using it in linux.

Did you build VisIt from sources or are you using a pre-built binary downloaded from our website?

I installed it within a container according to the steps in this recipe file so I think it's a pre-built binary
https://github.com/DevinBayly/github-ci/blob/main/visit_hpc

When you say you want "GPU accelerated" rendering, that is pretty much what you get out of the box if you run one of our pre-built binaries. If you can run glxgears on your client, then you should be able to run VisIt with GPU accleration there.

Shoot, ok. So there isn't some sort of setting visit can be downloaded/built with that makes use of some sort of library like egl to make use of a card if the display isn't using it (our OOD desktops are all software rendering).

OTOH, if you have some parallel cluster and you have GPUs on each node of that cluster and you want VisIt to use those GPUs, we rely upon VTKm for that and, in particular, we don't render that way. We do (some) data processing that way (e.g. compute an iso-contour), but we don't use parallel node-level GPUs to render...at least not to my knowledge.

I'll make sure to come back when I'm trying to set that up, because I imagine that will be the next thing my users expect to have working with their Visit sessions, but perhaps that's a better set of questions for the folks who support VTKm.

I think perhaps this answers my question: if there's no existing gpu accelerated display for rendering the program set up then visit will make use of cpu rendering, and if there is one then it doesn't require a separate pathway for configuration

Is that right?

[markcmiller86]

I think perhaps this answers my question: if there's no existing gpu accelerated display for rendering the program set up then visit will make use of cpu rendering, and if there is one then it doesn't require a separate pathway for configuration

Is that right?

Mostly. VisIt relies upon an implementation of GL there that will drive the graphics hardware available. So, VisIt is using (normal) GL for hardware accelerated graphics and Mesa GL for software graphics.

We have yet to fully investigate getting hardware accelerated graphics while running in a container. We've been playing with Docker for this but have yet to actual drive the host machine's graphics hardware from within the container doing that. I do not think it is easy to set that up in general.

[DevinBayly]

We have yet to fully investigate getting hardware accelerated graphics while running in a container. We've been playing with Docker for this but have yet to actual drive the host machine's graphics hardware from within the container doing that. I do not think it is easy to set that up in general.

I'd be happy to share some of my recent momentum on this, the special twist being that I'm targeting apptainer which makes the already murky subject of hardware accelerated container vis even more mysterious.

I did happen to get a few different programs up and running today, but visit gave me this message

Apologies for the quality, my current hack is a remote desktop inside another (I'm working on trying to remove this dependency not just because it'll make for better discussion screenshots)

https://paste.rs/cx0xE <-- same error lines

Are you familiar with this?

[biagas]

@DevinBayly I am in early stages with trying Apptainer for VisIt. I currently cannot build from a .def file, so am trying to build from a pre-built docker archive. Any pointers (docs/tips/tricks) appreciated. Have been browsing Apptainer's RTD.
You are aware of their GPU docs ?

As for the error message, Haven't seen that before, but it may be that VTK was configured incorrectly for the Apptainer/GPU use-case. Either that or VisIt isn't using the correct GL. We haven't fully looked into supporting GLVND, which I think would be needed. It's easy to get VTK to build with it, but VisIt may require some minor mods to how it looks for and utilizes GL.

[DevinBayly]

I've been making use of a docker file that still just uses the prebuilt binary and that's been enough to get visit up and running within a software rendering desktop. I haven't dived into the apptainer .def direction but I've come across that gpu doc page and lightly browsed it once or twice. If you want to test an apptainer from docker conversion then the following might work as long as you're not looking to build from source.

singularity build visit.sif docker://ghcr.io/devinbayly/visit_hpc:latest

Ok interesting. Another idea that I had was to see if I can get the remote desktop system to work outside of a container, and then try the visit binaries so that we can narrow down whether it's the container that's causing the problem or not. I'll start there, but if you are interested in trying to recreate the bug I'd be happy to walk you through the steps I took.

[biagas]

I had success with converting a docker-archive (.tar file) into an Apptainer .sif file, then running the .sif and successfully running VisIt (at least with a serial engine). I made the mistake of installing our standard host profiles, so I had to modify the profile to point to the location of VisIt in the .sif.

When building the docker archive, I installed glvnd and other similar support libraries, then ran build_visit without --mesagl and without --osmesa so that VTK and VisIt would pick up the system GL.

Only tried a serial engine, just testing concept at this point. May have to build with OSMesa as well in order to run on our batch nodes (headless).

I have been reading about run scripts that allow the apptainer to run a specific app rather than having to run the apptainer and then execute the app once inside. What do you think your preferred use-case would be?

[DevinBayly]

I think your success is probably very similar to the approach that I tried of letting singularity sort out how to do the conversion on its own.

When building the docker archive, I installed glvnd and other similar support libraries, then ran build_visit without --mesagl and without --osmesa so that VTK and VisIt would pick up the system GL.

Do you have a docker build script that I could take a look at? It would be nice to see if that gets me around the error I'm seeing when I launch inside the gpu accelerated desktop.

Serial is fine with us for a start, really just trying to get some of our users past the point where we usually tell them to run visit on their local desktops after the simulations are done.

I have been reading about run scripts that allow the apptainer to run a specific app rather than having to run the apptainer and then execute the app once inside. What do you think your preferred use-case would be?

This is a neat find, and very timely since I'm trying to make HPC modules out of many of these containers, and it's ideal if the container can start up and automatically launch the program they've module loaded. My preference in this particular moment would be to have run scripts

[biagas]

This is based off the Ubuntu22 dockerfile I used for 3.4.1.
Mods include installing glvnd and a few extra supports for gl development.
Removed setting of USER and group stuff.
Do everything in /build_visit instead of /home/visit
Also I modified the run_build_visit script to remove --mesagl and add --no-llvm.
Modified the build_test_visit script as well to work from the new WORKDIR and to install VisIt to /visit, and not create a binary visit distro tarball. HTH.

Ubuntu22-For_Apptainer.zip

I think supporting parallel on clusters/batch is going to be a much bigger fish to fry, based on the Apptainer MPI docs.

Also, interactivity with the GUI was rather slow for me.

[biagas]

FYI, our current docker files and support scripts are available here:
https://github.com/visit-dav/visit/tree/develop/scripts/docker