gamescope having CAP_SYS_NICE break the steam overlay in nested mode

[Oschowa]

I've seen that gamescope now re-nices itself and uses realtime priority compute if CAP_SYS_NICE is available, so I tried to add this capability to the gamescope executable with either:
sudo setcap cap_sys_nice+pie or
sudo setcap cap_sys_nice+pe
While this does give gamescope the ability to re-nice itself as confirmed by the log output, it seems to break the steam overlay and consequently steam-input when running games for steam in gamescope via launch options.
Removing the capability from the gamescope executable makes the overlay work again.

[Plagman]

Oh, thanks for catching that. It's likely causing LD_PRELOAD to get skipped, hm.

[emersion]

Hm, but LD_PRELOAD should work fine in the game right? (Because it doesn't have CAP_SYS_NICE?)

[Plagman]

We want the overlay hooked to gamescope, not the game, when running it as a sub-command. Otherwise the overlay isn't 1:1 screen-space, and you get input hooking at the wrong level.

[emersion]

Hmm, I see. I don't see an easy solution.

[Samsagax]

I tested OP's capabilities setting and seem to ignore all environment variables.

I usually do:

VK_ICD_FILENAMES=/usr/share/vulkan/icd.d/intel_icd.x86_64.json gamescope -- glxgears

to use the iGPU (since Nvidia wont work) and with the set CAP_SYS_NICE capability it won't select the intel gpu so it wont start with nvidia errors.

Would it be better to use pam_limit module for renice itself?

[kasha695]

I'm also having a similar issue.

Entering sudo setcap 'CAP_SYS_NICE=eip' /usr/bin/gamescope in a Terminal allows gamescope to re-nice itself but whenever I launch a game through steam the overlay doesn't work.

Once I've set gamescope back to what it was before, sudo setcap 'CAP_SYS_NICE-eip' /usr/bin/gamescope, it works again through Steam no problem.

[dgmvecuador]

I have the same issue, is there a workaround to this??

[DisplayTalk]

Still happens here too.

[abigrock]

Running the setcap command shown in the Arch Wiki, I can confirm the steam overlay is broken for me as well.
I'm using the following launch options for games under steam:

gamescope -w 3440 -h 1440 -e -- gamemoderun %command%

[InsaneCake]

Same here. Overlay in gamescope doesn't work after executing sudo setcap 'CAP_SYS_NICE=eip' /usr/bin/gamescope. Works again after sudo setcap 'CAP_SYS_NICE-eip' /usr/bin/gamescope

[abigrock]

This has been driving me nuts so I looked into other methods to allow automatically renicing gamescope. I installed ananicy-cpp and created a simple rules file under /etc/ananicy.d/gamescope.rules with the following:

{"name": "gamescope", "nice": -20}

I then started/enabled the ananicy-cpp systemd service. Now I can launch a game using gamescope and the steam overlay still works. As expected gamescope is running at nice level -20 as it does when CAP_SYS_NICE=eip is enabled for it. Definitely more of a workaround than a full fix.

[igo95862]

This most likely because of the dynamic linker secure mode: https://man.archlinux.org/man/ld.so.8#Secure-execution_mode

Because linker detects the extra capabilities it will prevent LD_PRELOAD from working.

I wonder if running entire Steam with the CAP_SYS_NICE instead of setting capabilities on gamescope can be a work-around. (for some reason the overlay gets disabled in gamescope no matter the capabilities on my system)

[ammgws]

Just wondering, is CAP_SYS_NICE set on the Steam Deck when in game mode?

[misyltoad]

Yes, but the overlay on Steam Deck uses an overlay plane and doesn't hook into the game like on desktop

[ammgws]

If you launch gamescope from a tty on a desktop does it also do that, or is the only specific to the Steam Deck?

[adlainewson]

just to add that indeed on a PRIME setup setting CAP_SYS_NICE=eip will make gamescope ignore the variables I use to force GPU on the GPU-attached monitor (thus instead using intel and breaking).

In particular, with CAP_SYS_NICE=eip: VK_SELECT_FILENAMES=(nvidia) MESA_VK_DEVICE_SELECT=(nvidia) gamescope -- vkcube still attempts to use igpu

[DianaNites]

Mesa, and all spec compliant vulkan loaders ignore such variables in contexts with "elevated privileges", such as binaries with setuid or with capabilities, for security reasons.

Gamescope isnt intentionally ignoring anything, security features of mesa and vulkan loaders out of its control are doing it. I have never been able to find any alternative functionality or settings in "secure" locations you can use to get around this.

[sharkautarch]

@Joshua-Ashton
I have an idea for how to get steamoverlay to be able to hook into gamescope when gamescope has CAP_SYS_NICE

But it is a bit complicated, because it would require work on both the steam client and gamescope. Also I’m not sure how this would work in flatpak, but I guess that’s just life lol

On steam side, (if this isn’t a thing yet) add an explicit vulkan layer which either has steam overlay built into it, or is dynamically linked to it.
Then have steam set some environment variable like STEAM_WANTS_STEAMOVERLAY or something whenever steam launches a game with steam overlay enabled

On the gamescope side, just check for the presence of said STEAM_WANTS_STEAMOVERLAY envvar and then add the name of said explicit vulkan layer to the list of enabled layers when gamescope creates a vulkan instance.

I imagine that it’d take a lot of work to make said explicit layer if steam overlay wasn’t simply hooking into vulkan functions

The only other thing I could think of would be to have gamescope directly dlopen() steam overlay, but somehow I get the feeling that surely it wouldn’t be that simple?

[fritz-fritz]

Hey, stumbled upon this while trying to figure out how to get my overlay working again.

Previously I had been blocking the steam overlay from being loaded and explicitly loading a cloned version of it as steam would try to implicitly load it. Not sure why that stopped working exactly.

But I can indeed confirm that removing the CAP_SYS_NICE allows the overlay to work properly! Running gamescope 3.14.24