Vulkan: Shader compilation stutter when materials enter the view frustum for the first time (unless cached)

[Calinou]

---

Bugsquad note: This issue has been confirmed several times already. No need to confirm it further.

---

Godot version

4.0.alpha8

System information

Fedora 36, GeForce GTX 1080 (NVIDIA 510.68.02)

Issue description

A noticeable shader compilation stutter occurs when materials enter the view frustum for the first time (unless cached). This is more visible on lower-end CPUs/GPUs which take a longer time to compile shaders.

Labeling regression, as ubershaders make it possible to avoid the initial stutter (on cold cache) in 3.5 with GLES3.

Steps to reproduce

• Open the minimal reproduction project. If you can't notice shader compilation stuttering in that project, try Add a 3D antialiasing demo godot-demo-projects#743 instead.
• Run the project and wait for stuttering to occur due to shader compilation.
• Run the project a second time and notice there won't be stuttering anymore.
• Remove the shader cache folder (.godot/shader_cache/ within the project folder).
• Run the project again, and notice stuttering occurring once again.

Minimal reproduction project

test_shader_compilation_stutter.zip

[Riteo]

What will be the solution to this problem on Vulkan? Will Ubershaders be implemented? A friend of mine working on a game in Godot 4 stumbled on this stuttering and was very confused about its nature until I pointed it out.

I recall seeing some discussion about it, with the only one behind a ticket that comes to mind being godotengine/godot-proposals#4754, but I think that it has more specific usecases.

Edit: oof, it didn't took so long to find an explaination that ubershaders aren't good enough: godotengine/godot-docs#6217 (comment) what now?

[TokisanGames]

GD4 Beta4 still stutters quite a bit on load with a lot of assets. You need to force precompilation of shaders.

#53411 (comment)

#46330 (comment)

[Calinou]

what now?

I think adding a true asynchronous shader compilation mode is the way to go – essentially mimicking dxvk-async behavior. It causes objects to be temporarily invisible until their shaders are compiled, but this is far preferable to stutter in 99% of situations.

dxvk-async is effectively being used to "fix" AAA releases that suffer from shader compilation stutter, without having access to the game's source code.

[Riteo]

I think adding a true asynchronous shader compilation mode is the way to go.

@Calinou why couldn't this be integrated with Ubershaders? I recall the dudes over at Dolphin having issues with plain asynchronous compilation, as some shaders depended on being shown right away.

[clayjohn]

Primarily this will be solved with a combination of ubershaders (we already have the structure set up), pipeline caching, and asynchronous shader creation/compiling.

In Vulkan, the story is much more complicated as pipeline creation is also super expensive. Right now we compile shaders which are quick to load from disk, but we still have to create pipelines in the first frame an object is used which is most likely where the reported stutter is coming from. We also have tools in place to use a more ubershader like approach (so we can quickly compile one variant of the shader/pipeline and then replace it with a more specialized variant later), but we need to make better use of them.

[Riteo]

@clayjohn is this planned by the stable release or will this require enough work to warrant being in 4.1?

[clayjohn]

Hopefully we can do most of it in time for the stable release.

[warriormaster12]

Since VkGraphicsLibrary extension is available almost with every single gpu vendor, could that solution be considered for stuttering problem 🤔

[warriormaster12]

Would be interesting to try and implement it since I've been doing multiple personal vulkan projects and I'd like to put it to use.

[jcarlosrc]

The problem is still present in 4.1 (Mobile at least).

[cosmoddd]

Still an issue on desktop as well. Able to reproduce by clearing the cache shaders in my project's AppData/Roming folder and the NVIDIA cache folder, then restarting the game. Only happens the first time the game.

[elvisish]

Is it not possible for Godot 4.x to use the asynchronous compilation technique 3.5 does? I'm having huge stutters whenever I turn a light on or add a light to the scene (even when using preload()) and I'm not sure what the current workaround is. This happens when the game is run from Steam, so I'm not entirely sure why this would be happening.

[warriormaster12]

I don’t think that adding a light to the scene causes a pipeline compilation🤔 though someone more knowledgeable can correct me. Does the stutter happen every single time when you add a light?

[elvisish]

I don’t think that adding a light to the scene causes a pipeline compilation🤔 though someone more knowledgeable can correct me. Does the stutter happen every single time when you add a light?

It happens the first time a new light is added or turned on, but then it's fine each time you do it after (as if it's become cached).

[warriormaster12]

It is cached. Pipeline caching was introduced in 4.1 on top of shader caching. You can find it in user://vulkan folder. Deleting that folder will most likely trigger the stutter again unless your gpu driver has made its own cache.

[cosmoddd]

I can retrigger the stutter by deleting specific cache files in my graphics card settings folders. This resets the process of re-caching. When a new shader or post processing effect is enabled it happens then, too. When the game is launched on a version update, it happens though not as severe. Would need to test further to provide feedback there.

I look longingly at Steam's "precompile vulkan shaders" message which comes up when I launch certain games through that platform. I'm wondering if there's something to be gleaned from that approach.

[elvisish]

It is cached. Pipeline caching was introduced in 4.1 on top of shader caching. You can find it in user://vulkan folder. Deleting that folder will most likely trigger the stutter again unless your gpu driver has made its own cache.

Are lights considered shaders to be cached, though? I'm only having this issue with lights, I'm not even having trouble with changing materials it just seems to happen when lights are enabled or added to the scene enabled.

[Calinou]

Is it not possible for Godot 4.x to use the asynchronous compilation technique 3.5 does?

The ubershader approach in 3.x had many issues that made it not ideal in my experience:

• Performance suffered quite a bit when initially starting a scene until most shaders were compiled.
• It didn't completely resolve stutter, only reduce it.
• It ran into driver issues on many platforms and GPUs. This may be less of an issue with Vulkan, but I wouldn't be surprised if it still happened with a Vulkan implementation.

I would prefer looking at an easier way to create shader precompilation dialogs like this, which seems to be the more common trend in AAA games lately. Asynchronous shader compilation can be used as a fallback to prevent stutter (as a last resort), even if this means having certain objects be momentarily invisible¹. Not displaying ubershaders prevents performance issues arising from their use.

I look longingly at Steam's "precompile vulkan shaders" message which comes up when I launch certain games through that platform. I'm wondering if there's something to be gleaned from that approach.

This works by making users compile and upload shaders after they're done playing a game. It works because Steam has such a large network of users that you're almost guaranteed to be able to download a compiled shader that matches your OS, GPU and driver version (all 3 need to match exactly). If you see high CPU usage after exiting a game on Steam, that's why 🙂

It is possible for Godot games on Steam to use this approach if they are on Steam and use a Vulkan-based rendering method.

Footnotes

1. This is fine for most kinds of games in practice. It can be an issue for competitive multiplayer games, but stutter will be just as much of an issue in that kind of game anyway (and this would be a feature you could turn off if not desired). ↩

[warriormaster12]

Geaphics pipeline library could help with invisible meshes in async compilation by creating a fast linked pipeline first before doing final optimized version and by recycling pipeline stages as much as possible. But it would require first to have async compilation infrastructure in place if we want to take advantage of that extension.

[warriormaster12]

At least that’s my understanding that I got after writing GPL draft pr.

[elvisish]

Here's a clip someone sent me of them playing a game I'm beta testing through Steam (so it's built and run via Steam), you can see a huge lag when the flashlight goes on the first time and not subsequent times (the flashlight is already added to the scene on ready, this is enabling it):

iasryu.2.mp4

Here's another when a light is removed from the scene, and one after when a light behind the wall is instatiated (it's not seen in this clip):

fvuxcv.mp4

This seems like an issue with compilation stutter, is this expected behavior with lights being added to the scene/enabled?

[TokisanGames]

Shaders are compiled per lighting setup, or at least they were in gd3. The only reliable way we've found to reduce stutter is force precompilation. I linked instructions in my comment above.

[elvisish]

Shaders are compiled per lighting setup, or at least they were in gd3. The only reliable way we've found to reduce stutter is force precompilation. I linked instructions in my comment above.

Essentially the 4.x version of this? https://github.com/Brandt-J/ShaderPrecompiler

I think for the time being a built-in node or even an addon like this would be a good immediate solution so people can avoid as much stutter as possible out-of-the-box (as you mentioned in your comment, 99% of the stuttering #46330 (comment))

[elvisish]

I forked Brandt-J's to 4.x: https://github.com/elvisish/ShaderPrecompiler4x

It seems to work okay from what I can tell, but this should hopefully serve as a bandaid until something can be implemented in 4.x.