WebXR support for Quest 3

[dlazares]

Hey great work here so far @mkkellogg!

I'd love to help out where we can on a WebXR viewer for Splats.
We have our own internal renderer right now running on WebXR but it's closed source. Ideally we could support a open source renderer in a popular framework like Three.js to make them more accessible.

We have optimized splats to make them really small while retaining most of the quality. We have the splat here in the repo running at 90FPS on Quest 3 in our WebXR renderer. Hope this can be useful for testing out in the headset since most of the data out there is too large to be useful.
https://github.com/gmix-tech/small_splats

Let me know how I can help out.
Feel free to ping me at diego@gmix.ai

[mkkellogg]

Hey Diego, thanks for sharing! WebXR support is definitely something I'd like to add at some point. I don't think it should be too much extra work because it seems to be a matter of setting a flag on the Three.js renderer (I bet it turns out to be more complicated than that 🙂). An example here shows that it should be pretty straight forward though.

Do you happen to have some documentation on the format of your splats? (or is it the same as produced by tools like the PlayCanvas editor?)

[dlazares]

Same format as the splats in the antimatter15 repo with precomputed colors. I also have the .ply with the spherical harmonics for these scenes if that'd be useful to add to the repo.

Format is roughly:

// XYZ - Position (Float32)
// XYZ - Scale (Float32)
// RGBA - colors (uint8)
// IJKL - quaternion/rot (uint8)

[mkkellogg]

Yeah if you have the .ply version, that would be helpful. I currently don't have code to import the antimatter15 .splat format. I should really make an importer for it since it's becoming so commonly used.

[dlazares]

Added it to the repo!

[chrisirhc]

Fyi, I did some exploration of getting WebXR to work while trying to get the Gaussian Splat to render on the Looking Glass (still WIP).

I did it via the dropin.html . It is pretty straightforward, though I needed to set local development server to https, and change the rendering from requestAnimationFrame to renderer.setAnimationLoop. I tested things via the WebXR Chrome extension since I don't have a Quest device.

Ref: a44ae40...chrisirhc:GaussianSplats3D:8783d3e85426ee594998e50aaa5b6e22a0f0132f

[mkkellogg]

@chrisirhc Thanks so much for doing this! @dlazares Sorry for the delay in looking into this, I've got a lot on my plate these days.

[dlazares]

I tried @chrisirhc code on my Quest 3 and it works for rendering it but there was a couple issues. one had to with coordinate frames which I sorta fixed by adding

    renderer.xr.addEventListener('sessionstart', (e) => {

			controls.update();

            const baseReferenceSpace = renderer.xr.getReferenceSpace();
            
            const offsetPosition = camera.position;

		
			const offsetRotation = camera.quaternion;

            const transform = new XRRigidTransform( offsetPosition, { x: offsetRotation.x, y: -(offsetRotation.y), z: offsetRotation.z, w: offsetRotation.w } ); 

            const teleportSpaceOffset = baseReferenceSpace.getOffsetReferenceSpace( transform );

            renderer.xr.setReferenceSpace( teleportSpaceOffset );

		});

the major issue left is that the immersive scene is blurry. It's not blurry in the emulator and it's not blurry on the preview canvas on the Quest 3, but once you click enter VR and enter the immersive scene it becomes blurry. It's not blurry per se, but the splats become chunkier.

My theory is it has something to do with the change in camera. I know from previous development that the projection matrices for the webxr emulator and the actual devices are not the same. I also know we had a bug around focal length due to webxr.

@mkkellogg any thoughts on what it could be?

[dlazares]

okay turns out the blur was related to focal length/viewport

I bumped the version number to 1.6.0 for Three.js so that it included this issue fix
mrdoob/three.js#21188 which was addressed by this PR.
mrdoob/three.js#26905

got it running 90 FPS on the Quest 3 :)
still very minor issues around rotation

[mkkellogg]

Glad you got it working! Looks like I should incorporate the three.js version bump into the release. Thanks again @chrisirhc for helping with this.

[chrisirhc]

@dlazares Thanks so much for pointing out the blur issue and the fix!
I think that was an issue I ran into but was stuck for a long time (and gave up). (Was getting blurry results on the Looking Glass)

@mkkellogg Looking forward to the release! :)

[mkkellogg]

@dlazares Would you be willing to share the code you used to get it running at 90 fps? I have a Quest 3 and I've managed to sort through the rotation issues so it technically works... the performance is just bad for some reason. (Not to mention when I exit immersive mode, the device basically freezes up). Also, are you using the Quest's built-in browser?

[Bishadkoju]

hi @mkkellogg could you update the current progress in web-xr branch ? Thank you.

[mkkellogg]

@dlazares I have included built-in WebXR support in this branch: https://github.com/mkkellogg/GaussianSplats3D/tree/web_xr. Would you be able to try out the included demo in demo/vr.html to see how it works for you? Thanks!

[mkkellogg]

Going to close this one since WebXR support is now built in. Let me know if you run into any issues with it!

[EliCDavis]

I'm probably getting like, ~1 fps in WebXR on Quest 2 on my splat of 857805 points. What is your splat point count to get 90fps on quest 3 @dlazares?