Terrain rendering issues with TAA/FSR (flickering)

[DevLewa]

Terrain3D version

v0.9.0

System information

Godot v4.2.1 - Windows 10, Forward+/Vulkan GTX 1060 6GB

Is the issue reproducable in the demo?

Not applicable

Issue description

Jittering/flickering can be seen/observed on the Terrain if TAA or FSR 2.0 is enabled. (With TAA it is less noticeable, with FSR its more pronounced)

Here a demonstration:
I set 3D scaling to FSR 2.0 and reduced the resolution scale to 0.1 (to make this really pronounced.)
https://www.youtube.com/watch?v=mKjPz8nHW2U

At first thought that this is a general Godot issue, howerver in the video only the terrain surfaces are flickering. (Other static geometry looks fine.)
I assume that this is due to the snapping/reposition behaviour of the 3D terrain renderer given that this flicker happens in a fixed intervall while moving. At least FSR to my knowledge requires a velocity buffer to reconstruct the image. The repositioning of the terrain might interfere with the FSR implementation casing incorrect reconstruction.

Not entirely sure how the velocity tracking in Godot is implemented. If we store the previous transform of the last frame (to calculate the delta for the velocity buffer each frame) one fix would be to artifically override the last transform state in case of a reposition to trick the image reconstruction on applying the correct velocity. (Have to check how Godot handles this internally.)

Can't test this with the demo project as of right now. (Having issues with godot not wanting to enable the addon on new projects due to parsing errors in the addons gd scripts. Will update this once i figure out what is going on.)

[TokisanGames]

Thanks for the report. I haven't used either yet. Is it the mesh or the textures flickering? Show it in wireframe mode, and checkerboard and grey debug views. Your trees flicker just as much.

[DevLewa]

Here a video with the default terrain grid texture and with enabled collision mesh:
https://www.youtube.com/watch?v=JJZRwbQwQiI

The trees flicker constantly (even stationary) as they have a material with alpha transparency + alpha clipping enabled. (without FSR it's already very aliased + i'm using a rather low resolution scale for demo purposes so it looks especially bad here.) However the terrain flickering shows a different behaviour. (It flickers only while moving in fixed intervals.)

You can also see that the grey grid texture on the cubes to the left are always stable while the floor is constantly jittering while moving. (it's stable if the camera doesn't change position in worldspace. Rotating the camera also works as expected.)
Hence my assumption that this is related to the velocity buffer/repositioning issue.

[InitialCon]

It's very likely that this is a motion vector problem. The height of the vertex is determined by its world position so when the terrain moves with the camera the position of the vertex will move and this will not write to the motion vector buffer. This is a problem since the fragment sample position moves too. Since TAA and FSR2 expect pixel colours to remain constant with a constant velocity it's reprojecting the previous sample from before the camera moved causing the blurring/flickering. I'm not sure if it's possible to fix this without outputting the velocity in the shader which is not currently supported. My TAA understanding isn't the best so I may be incorrect here or get some terminology wrong.

[TokisanGames]

I just tried FSR 1.0 and 2.2 in Godot 4.2.1 with the Terrain3D demo and they display perfectly fine on my NVidia 3070.

However when trying to load a project or a new terrain on my system it reports errors:

ERROR: Condition "p_job.uavMip[i] >= mip_slice_rids.size()" is true. Returning: FFX_ERROR_INVALID_ARGUMENT
   at: execute_gpu_job_compute_rd (servers/rendering/renderer_rd/effects/fsr2.cpp:418)

It also repeats these when moving the mouse around, and the decal mouse cursor doesn't show.

I just helped a user on discord who was having Godot crash with FSR 2.2 enabled on his NVidia 3080 while adding Terrain3D, with these errors:

ERROR: Width must be equal or greater than 1 for all textures
   at: (drivers/vulkan/rendering_device_vulkan.cpp:1722)
ERROR: Condition "texture.is_null()" is true. Returning: FFX_ERROR_BACKEND_API_ERROR
   at: create_resource_rd (servers/rendering/renderer_rd/effects/fsr2.cpp:240)

He probably needs to upgrade his drivers.

So, in conclusion, FSR 2.2 in Godot is not ready for production use. Even if it works for you, you have no idea if it will be stable on end user machines. We could possibly put some test conditions in Terrain3D to avoid problems if we know the triggers. However, the back end in the engine clearly has bugs which need to be identified, reported, and fixed on their end.

My recommendation is to not use it until it's much more stable.

[TokisanGames]

Regarding TAA: I do see occasional flickering in the demo in some sections of the rock texture, not all. It's minor. If I zoom up the textures so they're pixelated and set the filtering to nearest, it triggers TAA artifacts much more.

In Op's case, it looks like all of the textures are chosen specifically to conflict with TAA. If I put the texture on a mesh it looks fine. As @InitialCon wrote, the difference is that the terrain mesh moves around. Put the viewport in wireframe mode and you can see that when you laterally move the camera the mesh moves constantly. There's, very little TAA artifacts if the camera moves vertically only.

So TAA is likely not going to work. FSR seems fine in the demo, but maybe not with other textures, and is unstable anyway.

For now use MSAA or FXAA.

[mrjustaguy]

I've observed that the motion vectors just shoot all over the place with the terrain (and nothing else), so it's a MV issue that's breaking TAA and FSR

[TokisanGames]

Wrobot's latest jungle demo uses TAA in 4.3 with Terrain3D and it looks great. Terrain textures are stable. Very few artifacts in lighting only, unrelated to Terrain3D.

It also tests fxaa, msaa, and FSR. No issues on my 3070.

[mrjustaguy]

I'm looking in the Motion Vector Debug view in 4.3 on an AMD Radeon RX 6600, and I'm getting this when moving forward:

The MVs randomly flicker like that, and it's only happening for the terrain, while other things provide proper motion vectors such as that box

Do also note, that at times the terrain seems to be providing proper MVs too, it just keeps going completely crazy every handful of frames

This basically forces TAA to disable itself causing a very jittery feel when moving, and FSR Upscaling just straight up shits itself, though FSR native AA acts similarly to TAA

Also note that only the Terrain itself has issues, not any of the foliage instances and the like

Baked mesh as expected doesn't have any issues

[mrjustaguy]

Interesting fact I've just discovered, the Motion vectors go crazy differently for different LODs, if you scroll from side to side when looking from top down on a flat terrain and look at the MV view, you'll be able to see the different LODs represented in the Motion vectors

This combined with the fact this is showing up mostly during actual camera movement, and only when the Terrain is aware of the camera, I'd say the culprit of these wacky MVs is the Clip map poisition updating

[TokisanGames]

https://www.reddit.com/r/godot/comments/1bmcv9x/shader_motion_vectors_godot_421/

https://docs.godotengine.org/en/stable/contributing/development/core_and_modules/internal_rendering_architecture.html#temporal-antialiasing

Motion vectors are generated on the GPU in the main material shader. This is done by running the vertex shader corresponding to the previous rendered frame (with the previous camera transform) in addition to the vertex shader for the current rendered frame, then storing the difference between them in a color buffer.

Perhaps we can look at the motion vector code in the main shader and invert it, or empty the color buffer when TAA is enabled.

Or perhaps we need a godot proposal to expose an API in the renderer to control the motion vectors to either:

• insert an inverse vector to neutralize the motion
• override the vector
• apply a mask to cull the terrain from motion vector calculation

[mrjustaguy]

godotengine/godot#77523 may be the answer to this problem, as it'd disable motion vector generation, which if I understand it correctly, in the case of FSR would mean it'd Fall back to Derived Motion Vectors that wouldn't suffer from this problem.

On the other hand this looks like a possible candidate for resolving this too - godotengine/godot#82733

[TokisanGames]

77523, disabling motion vectors does not help and makes things worse.

[mrjustaguy]

You tested it with TAA or FSR?

TAA to my knowledge doesn't derivate invalid MVs, and lack of MVs would mess things up

FSR does so if no MVs are provided, it'll derivate them and it should work fine.

if it was tested with TAA, please retest with FSR 2.2