Fix synchronization validation error in water example on Vulkan

[ErichDonGubler]

GPU-based validation was implemented for the Vulkan backend in #5046 (🎉). With it, we began to run into a validation error for the water example:

[2024-02-02T20:25:58Z ERROR wgpu_test::expectations] Validation Error: Validation Error: [ SYNC-HAZARD-WRITE-AFTER-READ ] Object 0: handle = 0xa808d50000000033, type = VK_OBJECT_TYPE_RENDER_PASS; | MessageID = 0x376bc9df | vkCmdEndRenderPass: Hazard WRITE_AFTER_READ in subpass 0 for attachment 1 depth aspect during store with storeOp VK_ATTACHMENT_STORE_OP_STORE. Access info (usage: SYNC_LATE_FRAGMENT_TESTS_DEPTH_STENCIL_ATTACHMENT_WRITE, prior_usage: SYNC_FRAGMENT_SHADER_SHADER_SAMPLED_READ, read_barriers: VkPipelineStageFlags2(0), command: vkCmdDraw, seq_no: 2, reset_no: 1)

Because of the value of validation, that PR was merged without a complete fix to water, changing the outcome we expect of CI testing to failure. We should fix it, and adjust the expected test outcome of that example back to fully passing.

Additional historical context: #5046 (comment)

[jimblandy]

In Fedora's vulkan-validation-layers-1.3.275.0-1.fc39.x86_64, this also produces a message (line-wrapped):

[2024-04-09T21:02:01Z ERROR wgpu_test::expectations] Validation Error:
Validation Error: [ SYNC-HAZARD-WRITE-AFTER-WRITE ] Object 0: handle =
0x7f2fb53d44e0, type = VK_OBJECT_TYPE_QUEUE; | MessageID = 0x5c0ec5d6
| vkQueueSubmit(): Hazard WRITE_AFTER_WRITE for entry 7,
VkCommandBuffer 0x7f2fb6fd5b40[], Submitted access info
(submitted_usage:
SYNC_LATE_FRAGMENT_TESTS_DEPTH_STENCIL_ATTACHMENT_WRITE, command:
vkCmdEndRenderPass, seq_no: 3, renderpass: VkRenderPass
0x2d000000002d[], reset_no: 1). Access info (prior_usage:
SYNC_IMAGE_LAYOUT_TRANSITION, write_barriers:
SYNC_VERTEX_SHADER_SHADER_SAMPLED_READ|SYNC_VERTEX_SHADER_SHADER_STORAGE_READ|SYNC_VERTEX_SHADER_UNIFORM_READ|SYNC_FRAGMENT_SHADER_DEPTH_STENCIL_ATTACHMENT_READ|SYNC_FRAGMENT_SHADER_SHADER_SAMPLED_READ|SYNC_FRAGMENT_SHADER_SHADER_STORAGE_READ|SYNC_FRAGMENT_SHADER_UNIFORM_READ|SYNC_EARLY_FRAGMENT_TESTS_DEPTH_STENCIL_ATTACHMENT_READ|SYNC_LATE_FRAGMENT_TESTS_DEPTH_STENCIL_ATTACHMENT_READ|SYNC_COMPUTE_SHADER_SHADER_SAMPLED_READ|SYNC_COMPUTE_SHADER_SHADER_STORAGE_READ|SYNC_COMPUTE_SHADER_UNIFORM_READ,
queue: VkQueue 0x7f2fb53d44e0[], submit: 0, batch: 0, batch_tag: 28,
command: vkCmdPipelineBarrier, command_buffer: VkCommandBuffer
0x7f2fb6fd43c0[Main Command Encoder], seq_no: 1, VkImage
0x1c000000001c[Reflection Render Texture], VkImage
0x1f000000001f[Depth Buffer], reset_no: 1).

[zhanghe9702]

as i known , command_encoder_run_render_pass_impl looks like has auto insert barrier in command buffer, in water example, create the depth texture view has add all possible TextureUsages,

wgpu/examples/src/water/mod.rs

Lines 207 to 209 in cc0ee7b

	usage: wgpu::TextureUsages::TEXTURE_BINDING
	| wgpu::TextureUsages::COPY_DST
	| wgpu::TextureUsages::RENDER_ATTACHMENT,

so why validation layer still report WAR ?

[MatusT]

I can reliably reproduce this error every single frame on Windows 11, NVIDIA RTX 2070 Mobile.

The issue is fixed by either:

• adding VK_ACCESS_DEPTH_STENCIL_ATTACHMENT_WRITE_BIT to the barrier of the last pass
• changing render pass' VkAttachmentStoreOp to VK_ATTACHMENT_STORE_OP_NONE

See https://registry.khronos.org/vulkan/specs/1.3-extensions/man/html/VkAttachmentStoreOp.html. Any other VkAttachmentStoreOp than VK_ATTACHMENT_STORE_OP_NONE requires VK_ACCESS_DEPTH_STENCIL_ATTACHMENT_WRITE_BIT .

Seems to have been discussed here KhronosGroup/Vulkan-ValidationLayers#2349. The current situations is this I believe: KhronosGroup/Vulkan-ValidationLayers#2349 (comment)

[JMS55]

I did some researching, and I think we need to replace the following line

wgpu/wgpu-hal/src/vulkan/conv.rs

Line 462 in 0fc0b35

vk::AttachmentStoreOp::DONT_CARE

• On platforms that support one of VK_ATTACHMENT_STORE_OP_NONE, VK_ATTACHMENT_STORE_OP_NONE_KHR, VK_ATTACHMENT_STORE_OP_NONE_QCOM, or VK_ATTACHMENT_STORE_OP_NONE_EXT, we should use that
• On other platforms use VK_ATTACHMENT_STORE_OP_DONT_CARE, hope for the best, and maybe suppress the validation error.

[cwfitzgerald]

You need to use Store not Dont_care, but yes we need to either use None or suppress the error.

[cwfitzgerald]

@JMS55 can you try to fix this?

[JMS55]

No sorry. The dance to enable vulkan extensions within wgpu is hard to understand, I don't really want to go through that again. I had too much trouble figuring it out when doing u64 atomics and eventually just tried stuff until I got it running, but never really figured out how it's supposed to be setup.

[cwfitzgerald]

Alright. I was going to silence this validation error quickly, but this is harder to match on than I expected. If anyone is interested in adding this exception you'd need to silence the following, without silencing too many other validation errors.

 Validation Error: [ SYNC-HAZARD-WRITE-AFTER-WRITE ] Object 0: handle = 0x202bb03b510, type = VK_OBJECT_TYPE_QUEUE; | MessageID = 0x5c0ec5d6 | vkQueueSubmit():  Hazard WRITE_AFTER_WRITE for entry 7, VkCommandBuffer 0x202bb1292e0[], Submitted access info (submitted_usage: SYNC_LATE_FRAGMENT_TESTS_DEPTH_STENCIL_ATTACHMENT_WRITE, command: vkCmdEndRenderPass, seq_no: 3, reset_no: 1). Access info (prior_usage: SYNC_IMAGE_LAYOUT_TRANSITION, write_barriers: SYNC_VERTEX_SHADER_SHADER_BINDING_TABLE_READ|SYNC_VERTEX_SHADER_SHADER_SAMPLED_READ|SYNC_VERTEX_SHADER_SHADER_STORAGE_READ|SYNC_FRAGMENT_SHADER_DEPTH_STENCIL_ATTACHMENT_READ|SYNC_FRAGMENT_SHADER_SHADER_BINDING_TABLE_READ|SYNC_FRAGMENT_SHADER_SHADER_SAMPLED_READ|SYNC_FRAGMENT_SHADER_SHADER_STORAGE_READ|SYNC_EARLY_FRAGMENT_TESTS_DEPTH_STENCIL_ATTACHMENT_READ|SYNC_LATE_FRAGMENT_TESTS_DEPTH_STENCIL_ATTACHMENT_READ|SYNC_COMPUTE_SHADER_SHADER_BINDING_TABLE_READ|SYNC_COMPUTE_SHADER_SHADER_SAMPLED_READ|SYNC_COMPUTE_SHADER_SHADER_STORAGE_READ, queue: VkQueue 0x202bb03b510[], submit: 0, batch: 0, batch_tag: 28, command: vkCmdPipelineBarrier, command_buffer: VkCommandBuffer 0x202bb1e4890[(wgpu internal) Pre Pass], seq_no: 1, reset_no: 1)