Get depth pixel to corresponding color pixel in Unity without aligning

[Niclas1997]

Required Info
Camera Model	D435i
Firmware Version	5.15.1
Operating System & Version	Win 11
Platform	PC
SDK Version	v2.54.2
Language	C#/Unity
Segment	AR

Issue Description

Hello,

I'm using the Realsense camera in Unity and try to get the depth pixel to the corresponding color pixel without aligning them. I can't align them, because I use the Vuforia framework and need the RGB stream for that. In Python and C++ there is a rs2_project_color_pixel_to_depth_pixel method, but I couldn't find it in the Unity package and not in the C# wrapper.
Does someone have an idea how I can get the depth pixel without aligning the streams in Unity or C#?
Can I compute it manually?
Or is it possible to align depth to color and then stop the color stream without losing the aligned depth stream?

Thanks in advance

[MartyG-RealSense]

Hi @Niclas1997 One possible approach would be to call the C++ rs2_project_color_pixel_to_depth_pixel instruction from C# via JNI. Please see the comment at #5440 (comment) and the comments beneath it regarding doing so.

An alternative approach would be a RealSense C# tutorial at the link below that has scripting under the Mapping 2D to 3D heading for transforming 2D XY coordinates to 3D XYZ coordinates.

https://lightbuzz.com/intel-realsense-coordinate-mapping/

[Niclas1997]

Hi @MartyG-RealSense,

Thanks for the quick reply.

The comment you linked talks about Android and Java, so I don't think that I can use JNI (Java Native Interface) for C#.

The second link sadly doesn't help, because I want to project a pixel to another pixel and in the article it says that you should use align for that.

Is there a reason why these mathematical helper functions are not a part of the C# and Unity wrapper?

[MartyG-RealSense]

I see, #5440 began as a C# case and turned into an Android case. My apologies.

I believe that an equivalent of the Android extrinsics script in that link would be one in the RealSense SDK's C# Cookbook page of C# example code at the link below that calculates the extrinsics between the depth and color streams in order to obtain and print XYZ instead of using depth-color alignment.

https://github.com/IntelRealSense/librealsense/blob/master/wrappers/csharp/Documentation/cookbook.md#get-and-apply-depth-to-color-extrinsics

C# is not as widely used a language for RealSense applications compared to C++ and Python, and its instructions for performing a particular action tend to be quite different compared to Python, where each C++ instruction should automatically have a Python equivalent. For this reason, a C# script for performing a certain function may not be available unless somebody has already written a script that demonstrates it.

---

It is possible to access C++ instructions in the RealSense DLL library through the C# wrapper's Platform Invoke (pinvoke) system described at the link below.

https://github.com/IntelRealSense/librealsense/blob/master/wrappers/csharp/Documentation/pinvoke.md

[MartyG-RealSense]

Hi @Niclas1997 Do you require further assistance with this case, please? Thanks!

[Niclas1997]

Hey @MartyG-RealSense,
I tried out the method in the C# Cookbook but the pixels sadly didn't match.
Did you try it out before? I'm transforming the point like in the example.
My code looks like this:

  private Pipeline _pipeline;
  private Extrinsics _e;

  void Start()
  {
    _pipeline = new Pipeline();

    var config = new Config();
    config.EnableStream(Stream.Depth, 640, 480, Format.Z16, 30);
    config.EnableStream(Stream.Color, 640, 480, Format.Rgb8, 30);

    PipelineProfile selection = _pipeline.Start(config);
    var depth_stream = selection.GetStream(Stream.Depth);
    var color_stream = selection.GetStream(Stream.Color);
    _e = depth_stream.GetExtrinsicsTo(color_stream);
  }

  void Update()
  {
    using (var frames = _pipeline.WaitForFrames())
    using (FrameSet frameSet = frames.As<FrameSet>())
    {
      var colorFrame = frameSet.ColorFrame.DisposeWith(frameSet);
      var depthFrame = frameSet.DepthFrame.DisposeWith(frameSet);

      Vector c = new Vector();
      c.x = 500;
      c.y = 360;
      c.z = 0;

      var d = Transform(_e, c);

      ProcessColorFrameAndDrawPoint(colorFrame, c);
      ProcessDepthFrameAndDrawPoint(depthFrame, d);
    }
  }

  private Vector Transform(Extrinsics extrin, Vector from_point)
  {
    Vector to_point;
    to_point.x = extrin.rotation[0] * from_point.x + extrin.rotation[3] * from_point.y + extrin.rotation[6] * from_point.z + extrin.translation[0];
    to_point.y = extrin.rotation[1] * from_point.x + extrin.rotation[4] * from_point.y + extrin.rotation[7] * from_point.z + extrin.translation[1];
    to_point.z = extrin.rotation[2] * from_point.x + extrin.rotation[5] * from_point.y + extrin.rotation[8] * from_point.z + extrin.translation[2];
    return to_point;
  }

Thank you and best Regards

[MartyG-RealSense]

I cannot see any obvious problems in your code.

Does it make a difference if your var d line is altered to use the structure of the C# Cookbook example that puts e on the outside of the bracket?

var d = _e.Transform(c);

[Niclas1997]

Hey,
I altered the structure but it doesn't make any difference.
Does the code work for you?

[MartyG-RealSense]

I am not able to test code on my computer, unfortunately. I do apologize.

[MartyG-RealSense]

Hi @Niclas1997 Do you require further assistance with this case, please? Thanks!

[MartyG-RealSense]

Case closed due to no further comments received.