Custom shader templates

[BastiaanOlij]

Custom shader templates is a new feature in the rendering engine that allows you to override the built-in shaders into which user shaders are injected. While the inputs and outputs are fixed, this gives full control of the executed shader code.

NOTE This is an advanced feature that requires in-depth knowledge of the rendering engine to use. We should consider hiding this feature by hiding the property on the Shader resource unless a project setting is enabled. It should still load/save the property so advanced users can make plugins that make materials available that use a custom shader template. Also note that this feature is likely very version sensitive, while the include system (see #94193 which this PR is dependent on) will likely soften this, we can't completely remove this.
Also custom shader templates are renderer dependent, you can likely not use a template created for the Forward+ renderer on the mobile renderer or compatibility renderer.

Todos:

• Add shader template to material storage.
• Link shader to shader template in material storage.
• Implement new material storage implementation for RD renderer.
• Modify forward+ renderer scene shader to use shader template and create our default shader template.
• Modify mobile renderer scene shader to use shader template and create our default shader template.
• Create Shader Template resource
• Modify Shader resource to assign a shader template (see Note above)
• Modify Material resource classes so we can nominate a shader template that will be used to generate built-in shaders

Add into separate PR for enabling this on compatibility renderer:

• Implement new material storage implementation for compatibility renderer.
• Modify compatibility renderer scene shader to use shader template and create our default shader template.

Add into additional follow up PR(s):

• Modify canvas shaders to use shader template and create our default shader template
• Modify sky shaders to use shader template and create our default shader template
• Modify fog shaders to use shader template and create our default shader template
• Rejig built-in include shader files for ease of creating a custom shader template (follow up PR)

Known issues:

• If you have shader templates for a different renderer than selected in your renderer, and there are shaders using those even if not currently loaded, Godot creating previews will result in errors as those templates can't be compiled.

Very simple test project can be found here:
https://github.com/BastiaanOlij/test-custom-shader-templates
The readme on this project contains extra information about writing custom shader templates.

Live streams about this feature:

• https://youtube.com/live/2hoC35pR6-s

[RedMser]

Should this feature really be called shader templates? It might cause confusion with "script templates" which work completely differently (I assumed this PR would allow a newly created shader to start with a different code preset).
Maybe since it's not (yet?) exposed in the UI, it doesn't matter as much, but could be worth thinking about alternatives while the PR is in development.

[JoNax97]

I agree with the comment above. I feel something like "scaffolding" could communicate the idea.

[Calinou]

I agree with the comment above. I feel something like "scaffolding" could communicate the idea.

Scaffolding generally refers to a template system that creates files/folders for you (like good old Yeoman), so I'd try to find a different term for that instead.

The concept of overriding is more accurate here, although calling it a shader override may be mistaken for a material override (which is a different concept).

I think it's important to highlight the fact that you are replacing core shaders here, not user-provided shaders. The core shaders are always written in GLSL, not the Godot shader language. Maybe core shader override?

[JoNax97]

Your last paragraph made me think, if this is for overriding core shaders, why is it set per-shader instead of globally in project settings?

[BastiaanOlij]

Your last paragraph made me think, if this is for overriding core shaders, why is it set per-shader instead of globally in project settings?

Because you may want to combine different shaders. For instance you may want to only use this for your terrain shader.

(that said, having a project settings that sets a default could be an interesting option)

[mrjustaguy]

something that's Kind of like the Default Environment and Default Environment Background Color you can set in the project settings would be great..

[BastiaanOlij]

something that's Kind of like the Default Environment and Default Environment Background Color you can set in the project settings would be great..

Hmm, we could add it to the compositor, that might be a good place for this.

[BastiaanOlij]

something that's Kind of like the Default Environment and Default Environment Background Color you can set in the project settings would be great..

Hmm, we could add it to the compositor, that might be a good place for this.

Actually, thinking about this more, this is not possible because we could have the same shader compiled with multiple templates and we have no support for that. So we can do an all or nothing project setting, or keep it as is right now where we identify the template to use for each user shader.

[BastiaanOlij]

Basics are starting to work, we can now build ShaderMaterial based shaders using a custom template. I've linked a very simple demo up in the OP.

The user shader has a new optional keyword shader_template that allows you to specify the custom template to use:

shader_type spatial;
shader_template "res://test.gdtemplate";
render_mode blend_mix, depth_draw_opaque, cull_back, diffuse_burley, specular_schlick_ggx;

...

I've also done a basic rejig of the include files for the built-in template for the forward+ renderer. This allows us to include all the core inputs and outputs. See the test.gdtemplate in the example linked in the OP to see how this is used.
At this moment this test project simply creates an unshaded templated that only supports basic objects and shaders (misses support for motion vectors, multimesh, bones and other features), for that we need to expose much more of the built in logic. In discussions with Clay we figured it may be smarter to do this rejig after the main logic is merged.

(left cube is built-in template, right cube is custom template)

[BastiaanOlij]

Fixed the issues around reloading the shader template, now when you modify the shader template, all shaders that use the template get recompiled automatically.

[BastiaanOlij]

Cleaned up some more code and added support for setting the shader material on StandardMaterial3D resources:

[BastiaanOlij]

This may be a breaking change, I am not sure what the implications of changing the material key are. However seeing shader templates effect the compiled shader, it has to be part of the key.

[BastiaanOlij]

@clayjohn this is an interesting difference between the mobile and clustered implementation. We check more defines in the mobile renderer, but we're not consistent with it. I wonder if this is correct or left over from an older implementation?

[BastiaanOlij]

Added some extra logic so you can optionally specific templates for multiple renderers. A template in the format of:

shader_type spatial;

#[vertex]

// vertex shader code goes here

#[fragment]

// fragment shader code goes here

Will be used regardless of the renderer selected and thus will fail when used with the wrong renderer.

Alternatively you can create blocks for each renderer:

shader_type spatial;

#[forward_plus]

#[vertex]

// vertex shader code for the forward+ renderer goes here

#[fragment]

// fragment shader code for the forward+ renderer goes here

#[mobile]

#[vertex]

// vertex shader code for the mobile renderer goes here

#[fragment]

// fragment shader code for the mobile renderer goes here

In this scenario, if the current renderer does not have any code in the template, a missing code error will be given.

[Calinou]

Tested locally (rebased on top of master 20f4d26), it works as expected.

Some feedback:

• There is no shader compilation error when the shader template file doesn't exist:

You do get this message in the editor Output panel Resource file not found: res://notfound.gdtemplate (expected type: ), but no error highlighting in the shader editor due to this.

• The .gdtemplate file extension is pretty generic. It could be a script template, a project template, anything – it's not very descriptive. I suggest using .gdshadertpl which would make it more consistent with the existing .gdshaderinc extension.

As a reminder, we'll need to contact KDE/Dolphin developers to add an icon for the new file extension we choose to use:

• I've gotten this error message from making a change in a .gdtemplate file then reloading the project:

  Unknown shader type in shader template.

Could more context be added to this message (such as the type that was specified, and the line of the error)? The change was on https://github.com/BastiaanOlij/test-custom-shader-templates/blob/master/test.gdtemplate#L323, where I replaced the line with:

	albedo_output_buffer = vec4(albedo * vec3(1.0, 0.0, 0.0), alpha);

(I'm not sure why that caused an error too, either way. It seems to be valid GLSL.)

I also tried reverting my change and reloading the project again, but I still get the error message and the material is no longer using the custom shader template:

The cube on the right should appear unshaded, but it appears shaded.

[BastiaanOlij]

@Calinou agree with all you said there and I'll look into the error you reported, though I'm not planning on spending a lot of time on this until after 4.3 stable is out, I want to just gather feedback and then do a whole bunch of changes at once.

When the shader template can't be loaded, it just defaults back to the built in template, hence your right cube showing up correctly. I think it would be better if the compilation just fails and it just defaults back to our white shader.

I need to figure out how to make that happen because if the fault lies in the template, it finds it out too late, but when the fault lies in the template not being found we can piggy back on the existing logic for this.