Skip to content
This repository has been archived by the owner on Feb 1, 2024. It is now read-only.

Invalid eye transform #32

Closed
m4gr3d opened this issue Jul 29, 2019 · 16 comments
Closed

Invalid eye transform #32

m4gr3d opened this issue Jul 29, 2019 · 16 comments
Milestone
1.0.0

Comments

@m4gr3d
Copy link
Collaborator

m4gr3d commented Jul 29, 2019

The plugin doesn't seem to properly set the eye transforms as the rendered 3D content is not stereoscopic.
@NeoSpark314
Copy link
Collaborator

I can observer the same issue when running the demo application on my quest. In the attached screenshot with only the table it looks like no stereo separation at all.

quest_bug_noStereo_01 09 2019_small

@m4gr3d m4gr3d mentioned this issue Aug 1, 2019
Closed
@NeoSpark314
Copy link
Collaborator

Hi,

i tried your changes that you mentioned previously from your branch at https://github.com/m4gr3d/godot_oculus_mobile/tree/fix_invalid_eye_transform.

I also tried to rewrite the the OvrMobileSession::get_transform_for_eye function to use directly the data provided in head_tracker.Eye[eye].ViewMatrix from the ovr SDK. This is how the VrCubeWorld_NativeActivity sample renders. This is also how godot_open_vr extracts the eye transform.
You can see my test implementation in NeoSpark314@ae2ad9c

I then compared the resulting eye transform using godot_transform_as_string from your code and from using my code (ViewMatrix directly) rotation is identical; but the eye position is slightly different:

here is the Log output (first transform is my code; second transform line is your code; your code seems to only move the eye along x-axis independent of head rotation...)

08-04 17:04:08.408 14144 14160 V GodotOVRMobile: Transform Comparison for godot_eye 1

08-04 17:04:08.408 14144 14160 V GodotOVRMobile: 0.715116, -0.121121, 0.688432, 0.216813, 0.974734, -0.053725, -0.664531, 0.18768, 0.723308 - 0.163868, 1.011757, -0.242753

08-04 17:04:08.408 14144 14160 V GodotOVRMobile: 0.715116, -0.121121, 0.688432, 0.216813, 0.974734, -0.053725, -0.664531, 0.18768, 0.723308 - 0.154765, 1.018685, -0.263985

08-04 17:04:08.408 14144 14160 V GodotOVRMobile: Transform Comparison for godot_eye 2

08-04 17:04:08.409 14144 14160 V GodotOVRMobile: 0.715116, -0.121121, 0.688432, 0.216813, 0.974734, -0.053725, -0.664531, 0.18768, 0.723308 - 0.209564, 1.025612, -0.285218

08-04 17:04:08.409 14144 14160 V GodotOVRMobile: 0.715116, -0.121121, 0.688432, 0.216813, 0.974734, -0.053725, -0.664531, 0.18768, 0.723308 - 0.218667, 1.018685, -0.263985

From testing using the ViewMatrix directly feels quite good; but hard to tell if it is correct.
Regarding the other issue you mentioned of objects disappearing; it is still there also with my code; maybe a clipping issue?

@m4gr3d
Copy link
Collaborator Author

m4gr3d commented Aug 5, 2019

@NeoSpark314 Good catch! Using the ViewMatrix is definitely the way to go.
I'm making headway with the clipping issue. The visual_server_scene has some strange logic to calculate the camera frustum when in VR, and disabling that logic fixes the clipping issue (here's the relevant commit.

@BastiaanOlij any idea why that logic would be an issue for the oculus_mobile plugin, and not for the other VR plugins?

@m4gr3d
Copy link
Collaborator Author

m4gr3d commented Aug 5, 2019

@BastiaanOlij In addition, that change seems to drastically improve performance for the oculus mobile plugin. To a point where the requirement to enable extra latency mode (added by PR #24) is no longer necessary.

@BastiaanOlij
Copy link
Member

BastiaanOlij commented Aug 5, 2019 via email

@BastiaanOlij
Copy link
Member

BastiaanOlij commented Aug 5, 2019 via email

@Schroedi
Copy link
Contributor

Schroedi commented Aug 5, 2019 via email

@m4gr3d
Copy link
Collaborator Author

m4gr3d commented Aug 5, 2019

Can you clarify how stereo rendering is turned off?

Before the change, the logic would call _prepare_scene once with the combined frustum, and call _render_scene twice, once for each eye.
With the disabling change, the logic is calling both _prepare_scene and _render_scene twice, once for each eye.

When testing on the device, all 3d objects (controllers, table) are rendered at the proper depth with the correct stereo separation.

@m4gr3d
Copy link
Collaborator Author

m4gr3d commented Aug 5, 2019

@Schroedi looking at the comments I would assume so.

@BastiaanOlij
Copy link
Member

Can you clarify how stereo rendering is turned off?

Before the change, the logic would call _prepare_scene once with the combined frustum, and call _render_scene twice, once for each eye.
With the disabling change, the logic is calling both _prepare_scene and _render_scene twice, once for each eye.

When testing on the device, all 3d objects (controllers, table) are rendered at the proper depth with the correct stereo separation.

Thats what I get for reading it on a mobile phone :)

Yes you are right, it will do prepare and render twice, however it doesn't make sense that it would render differently when you prepare once with the combined frustum and then have problems calling render_scene. It is render_scene that will define the projection matrix and render out the scene. all prepare scene does is the culling and shadow updates and stuff like that.
So while the culling may fail and not cull the right objects, render_scene should still render with the same stereoscopic parameters. There must be something else going on here.

Keep in mind, this is working fine for every other headset so if it fails, it fails because we're not returning the correct matrices

@BastiaanOlij
Copy link
Member

One thing that may throw things off is that it is expecting the camera left/right offset for each eye to be returned in get_transform_for_eye not mixed in with the projection matrix returned by get_projection_for_eye

@NeoSpark314
Copy link
Collaborator

Hi,

I think I found the issue with the clipping:

In OvrMobileSession::fill_projection_for_eye the oculus provided projection matrix is directly used and the given function parameters z_near and z_far are ignored.
This is fine in theory because the function VisualServerScene::render_camera in godot takes the projection matrix and computes near and far values from it for the merged frustum computation (the code is in camera_matrix.cpp CameraMatrix::get_z_far())

The problem now is that oculus uses an +infinite far plane for their projection (the reason is documented here: https://developer.oculus.com/reference/mobile/1.24/vr_api_helpers_8h/ in the documentation of the function static ovrMatrix4f ovrMatrix4f_CreateProjection: "...except for things right up against the near plane, it always provides better precision...")

But in this case the godot z far computation code in CameraMatrix::get_z_far() returns 0.0 instead of +inf.

One solution to this problem would be to add to visual_server_scene.cpp in the function VisualServerScene::render_camera
directly after the near and far computation (line 1799) a check and set it to +infinity like this for example
if (z_far <= z_near) z_far = 1.0f/0.0f;
this then produces correct results in the following computations.

An alternative is to respect the given near and far values in vrMobileSession::fill_projection_for_eye
and overwrite them in the projection matrix returned by doing

matrix.M[2][2] = -(z_far + z_near) / (z_far - z_near);
matrix.M[2][3] = -(2.0f * z_far * z_near) / (z_far - z_near);

at line 198 after getting the ovrMatrix4f matrix from the tracking state.

@BastiaanOlij
Copy link
Member

Indeed, having an invalid far plane value would screw up the entire calculation.

Interesting choice from Oculus because the reason you don't have an infinite far plane is because the near<->far range directly impacts the precision of your Z-buffer. With an infinite far plane the depth buffer would fail to work properly.

So they're obviously doing more here.

@BastiaanOlij
Copy link
Member

(btw, if we check for z_far == 0 in the code that merges the frustums and change it to something arbitrary like 4096.0 (max value for the far plane property in Godot) in theory the logic will work.

@NeoSpark314
Copy link
Collaborator

The logic seems work even when setting z_far to +infinity if it was 0 but I did not check the result in detail.

Oculus references this paper to explain the choice https://www.cs.cornell.edu/~paulu/tightening.pdf and writes in there documentation
https://developer.oculus.com/reference/mobile/1.24/vr_api_helpers_8h/
"""
The projection matrix transforms -Z=forward, +Y=up, +X=right to the appropriate clip space for the graphics API. The far plane is placed at infinity if farZ <= nearZ. An infinite projection matrix is preferred for rasterization because, except for things right up against the near plane, it always provides better precision: "Tightening the Precision of Perspective Rendering" Paul Upchurch, Mathieu Desbrun Journal of Graphics Tools, Volume 16, Issue 1, 2012
"""

@m4gr3d
Copy link
Collaborator Author

m4gr3d commented Aug 7, 2019

Fixed by PR #36

@m4gr3d m4gr3d closed this as completed Aug 7, 2019
@m4gr3d m4gr3d added this to the 1.0.0 milestone Aug 26, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
1.0.0
Development

No branches or pull requests
4 participants
@Schroedi @m4gr3d @NeoSpark314 @BastiaanOlij