feat(client): ✨ Make lobby use passthrough

[zmerp]

Tested on Quest 3 and Focus 3

Maybe should we add a background to the text?

[The-personified-devil]

Tbh I would quite like a toggle. But there's no real way to implement that ig

[zmerp]

Personally i'd do without toggle, because that's simpler. I think passthrough is an improvement from the previous lobby in any way, except maybe contrast of the text that can be improved.

Any specific reason you prefer a toggle to switch to the previous lobby?

[The-personified-devil]

It's a weird personal preference for no passthrough, but tbh that's not a real reason not to do it, as generally it's more practical

[zmerp]

Ok, green light then?

[The-personified-devil]

yes but not a pr review lol

[zmerp]

Ok then, I'll wait your review when you can

[ShootingKing-AM]

Tbh I would quite like a toggle. But there's no real way to implement that ig

Yes I would like a toggle too, atleast on the AVLR settings, defaulted to passthough ?

[zmerp]

@ShootingKing-AM for PhoneVR you could have your own background. You can for example have a draw pass where you clear with a set color, then draw on top the texture from client core lobby. I would be fine with just leaving it as it is with black background for the headsets that don't support passthrough.

[ShootingKing-AM]

@ShootingKing-AM for PhoneVR you could have your own background. You can for example have a draw pass where you clear with a set color, then draw on top the texture from client core lobby. I would be fine with just leaving it as it is with black background for the headsets that don't support passthrough.

As far as i understand, i(client_core implementation) make the texture and give it to client_core to render lobby on that. Pass the texture to Carboard API to distort and render onto screen. This shader code is not exposed to client_core implementation. Where should the draw pass come it.

How would you know whether a headset supports passthrough or not and what is the passthough specification, what do you need from client implementation ? I mean i can give the camera frames to client_core to draw the ALVR lobby text on.

[zmerp]

The way I implemented it is that client core does not deal with passthrough. In fact OpenXR apps don't have access to the passthrough data, but they can tell OpenXR where and how to draw it as a separate layer. While OpenXR provides a compositor, for cardboard you will have to provide your own. It doesn't have to be anything fancy, just draw full screen textures on top of each other. So basically you have to make a render pass that first draws a camera surface, then draws the texture provided by client core. You cannot draw on the client core texture directly because we clear it every time before rendering (to avoid the classic trippy ghosting)

[ShootingKing-AM]

So basically you have to make a render pass that first draws a camera surface, then draws the texture provided by client core.

So i need to make a texture draw camera image per eye and pass the texture to alvr_client_core and it will write lobby text on it ? I can only transfer textures, presently drawing is solely done by client_core. Cardboard API takes a texture as input.

[zmerp]

No, I said the opposite. You should not pass the camera texture to alvr client core because it would be overwritten. Also, the camera texture would have a private format (Android Surface) which requires the special samplerExternalOES in the shader, while the lobby texture should be a normal OpenGL texture.

[ShootingKing-AM]

Oh so I have to get the lobby text from alvr, discard its lobby gfx part, and write the text onto camera frame and pass this texture to cardboard. So now the client_core lobby gfx is of no use, am i correct ?

... then draws the texture provided by client core ...
by texture you mean the client_core lobby texture ? wouldn't i be overwriting the camera image data with the client_core lobby texture ? what will client_core lobby texture would contain, the black/blue grid lobby or just the text (before/after this pr) ?

[zmerp]

@ShootingKing-AM i mean you can, but the lobby code is still available if you implement a compositor. I highly suggest you implement the compositor because i will do the same thing with the stream view (i will add some config so you know when to activate the camera view).

More precisely, you need to create a render pass that samples first the camera pass, then sample the lobby texture, and use the alpha channel of the lobby texture to do the blending (0 alpha means only passthrough, 1 alpha means only lobby). You will be able to reuse the same compositor shader once we implement the stream chroma keying, because i will pre-apply the color keying

[The-personified-devil]

@zmerp Are you sure that all headsets that alvr supports have passthrough quality that's good enough to warrant this? Quest 1 looks pretty damn awful, so maybe the old lobby would be prettier?

[Vixea]

It could be a toggle

[The-personified-devil]

Fwiw I really think we should keep the code for the old lobby in the codebase and architect it so that we can change a single parameter to switch it. What we do with that we can decide later, maybe we'll just make a normal setting that gets cached on the client upon first connection. Then we could also just have it as a fallback if we don't get the passtrough permissions (not sure if the user can actually disable that tho).

[The-personified-devil]

lgmt, see my objections about keeping both possibilities tho

[zmerp]

@The-personified-devil Man, i know you're in favor of no passthrough but i'm strongly in favor on passthrough. So what should we use as default?

[zmerp]

Are you sure that all headsets that alvr supports have passthrough quality that's good enough to warrant this? Quest 1 looks pretty damn awful, so maybe the old lobby would be prettier?

I'm not seeing this a matter of how pretty the passthrough is, but how the user feels when entering ALVR. I feel like passthrough is more welcoming regardless

[The-personified-devil]

@zmerp I already conceded that passthru is the better default. The matter is more whether to keep the old lobby in the codebase

[The-personified-devil]

I'm not seeing this a matter of how pretty the passthrough is, but how the user feels when entering ALVR. I feel like passthrough is more welcoming regardless

Sure, pretty moot point from me

[zmerp]

I already conceded that passthru is the better default. The matter is more whether to keep the old lobby in the codebase

So, i can keep the code, add an option to in client core, and leave integration as todo (with passthrough default)? Or you want to have the switch in this PR?

[The-personified-devil]

So, i can keep the code, add an option to in client core, and leave integration as todo (with passthrough default)?

Yes, that'd be great

[zmerp]

@ShootingKing-AM I've updated the C API:

fn alvr_render_lobby_opengl(
    view_inputs: *const AlvrViewInput,
    render_background: bool,
)

Now this function takes a render_background so you can decide to render the background (floor + sky) when the camera feed is not available.

[The-personified-devil]

Note to self: Need to change this when adding back some form of toggling

[zmerp]

Technically you don't have to but might be good to set to false if you know the layer is opaque. The runtime might do some optimizations

[The-personified-devil]

lgtm, if you tested background still working if you change the params you can merge

[ShootingKing-AM]

Now this function takes a render_background so you can decide to render the background (floor + sky) when the camera feed is not available.

Nice, but when cam feed is available i need compositor to blend in both camera texture and alvr_lobby texture ?
What would be availble as texture if the render_background true, the current grid lobby ?

[zmerp]

Nice, but when cam feed is available i need compositor to blend in both camera texture and alvr_lobby texture ?
What would be availble as texture if the render_background true, the current grid lobby ?

If render_background is true, nothing changes from before. The lobby texture is opaque and will render the floor and sky. If it's false, there is no floor and the background is transparent