Align depth stream to external camera stream

[abuzaina]

I have a D410 on a Jetson and an external 4K camera (MIPI interface). I can capture depth frames from the D410 and RGB frames from the 4K camera concurrent and in real time.
I would like to align the depth frame to the color frame of the external camera, similr to the rs-align sample.
Is it possible to create a synthetic stream and pass the intrinsics and extrinsics of the external camera to it ? If so how to do it?

Thanks

[ev-mp]

@abuzaina hello, this scenario is handled with the software-device class that the SDK provides.
Using this construct allows to bind and handle otherwise non-related sensors with a mockup "device", like in your case. Other cases include synchronizing depth streams of multiple depth cameras to generate a panoramic point cloud.
The software-device demo shows how to use this class, including building up the virtual sensors, generate the "software sensor" options, intrinsic and inter-stream extrinsic and then sending "user-defined" frames to Librealsense pipe - the core blocks to implement your use-case :
https://github.com/IntelRealSense/librealsense/tree/master/examples/software-device

This class is extensively utilized internally for unit-testing, see usages here and here

[ev-mp]

@abuzaina hello.
One thing that I didn't mention which is, imo the real challenge in this scenario is obtaining the intrinsic matrix for the external sensor, and the extrinsic matrix between D410 and the external sensor.

• The camera intrinsic matrix could probably be provided by the external sensor/OEM API or spec.
  It should include the focal and principal points, the distortion model and its coefficients. You'll have to check that Librealsense supports the distortion model exhibited by the 3rd-party sensor.
• The extrinsic matrix between D400 and the external sensor is the one that you'll have to generate on your own.

[abuzaina]

hi @ev-mp ,
Thank you for your reply, the second usage link in your first reply does not go anywhere. I would appreciate if you send it again.
I am still not sure how to inject the depth frames from the D410 and the RGB frames from the camera to the software device, are there any examples of it ?

[RealSenseCustomerSupport]

---

@abuzaina Are you asking about the links in "This class is extensively utilized internally for unit-testing, see usages here and here"?

Please find them as blow.
https://github.com/IntelRealSense/librealsense/blob/master/unit-tests/unit-tests-post-processing.cpp#L220-L265
#4202

[RealSenseCustomerSupport]

---

@abuzaina Any other questions about this? Thanks!

[jb455]

I'm not OP but I have a related question (I can post a new issue if you prefer). I had a similar thing to this (using software device to align depth stream with other camera) working with an older version of the SDK (2.14) and I have recently updated to the latest SDK (2.24) and now when I align, the depth stream is blank (all zeroes). Has there been a regression or is there something else I now need to do to get it working? Attached is modified Window.xaml.cs from cs-tutorial-4-software-dev to reproduce the issue.
Window.xaml.cs.txt

Edit: I realised the depth data isn't all zeroes; the first pixel is nonzero then the rest are always zero.

[ev-mp]

@jb455 , please open a separate ticket for the C# issue

[RealSenseCustomerSupport]

---

@abuzaina Any other questions about this ticket? Looking forward to your reply. Thanks!

[RealSenseCustomerSupport]

---

@abuzaina Sorry that we didn't get feedback from you for weeks. Will close this issue accordingly. Please feel free to create new ticket if you still have issues/questions. Thanks!