Transforming the coordinates of points from camera coordinate system to world coordinate system

[koalaaaaaaaaa]

Required Info
Camera Model	D400
Firmware Version	-
Operating System & Version	Win 10
Kernel Version (Linux Only)	-
Platform	PC
SDK Version	-
Language	python
Segment	-

Issue Description

Hi, I want to transform the coordinates of points from camera coordinate system to world coordinate system. I'm intended to build a world coordinate system whose coordinate axis are Perpendicular to four wall planes, floor plane and ceiling plain of a room.
I've roughly read Intel RealSense D400 Series Custom Calibration Whitepaper, and I'm not sure about whether contents in Chapter 5 is what I need. Could you give me some guidance about that?
Thank you!

[MartyG-RealSense]

Hi @koalaaaaaaaaa In the RealSense SDK, coordinates can be converted from 2D pixel coordinates to 3D world coordinates by first aligning the depth stream to the color stream with the align_to instruction and then 'deprojecting' 2D to 3D using the instruction rs2_deproject_pixel_to_point. The link below, a conversion to Python of the SDK's 'rs-measure' C++ example program, demonstrates using align_to and then deproject_pixel_to_point.

https://github.com/soarwing52/RealsensePython/blob/master/separate%20functions/measure_new.py

Alternatively, instead of converting the entire image to 3D points, you could obtain the 3D XYZ of a specific coordinate with the instruction rs2_project_color_pixel_to_depth_pixel. A Python example of this instruction is at #5603 (comment)

[koalaaaaaaaaa]

thanks, Marty.
I'm afraid that I didn't make myself understood. What I'm intended to do is actually a transformation from 3D to 3D points. To be specific, I want to transform the coordinates of pointcloud generated by Realsense dynamic calibration system (which I called camera coordinate system) to a fixed coordinates in real world (which i describe in the question above), could you kindly give me some instructions? Thanks a lot~

[MartyG-RealSense]

Generation of 3D point clouds is not involved in the operating principles of the Dynamic Calibration tool.

It sounds as though what you are aiming to achieve may be suited to the SDK instruction rs2_transform_point_to_point, which allows you to transform points from one coordinate space to another. This instruction is an 'affine transform' using a 3x3 rotation matrix and a 3-component translation vector. #5583 discusses the instruction in regard to Python.

An example of its use is stitching together multiple individual point clouds into a single large cloud by setting a common position and rotation in 3D space for each of the individual clouds to append them together.

[koalaaaaaaaaa]

I appreciate your help~
I once read the statement of function "rs2_transform_point_to_point()", but what puzzles me is how can I get the actual extrinsic parameters between camare coordinate system and world coordinate system (as I mentioned in very first question)? the link you gave (#5583) has demonstrated that I can use "rs2_get_extrinsics(...)" to acquire extrinsic parameters. However when I read the source code of that function I noticed that it can only calculate the extrinsic parameters between two streams which are both captured by realsense camera (say, depth stream and color stream) but maybe I'm wrong.. if so, please point out.
so how can I accomplish my goal, do I need a chessboard or something else? I have totally no idea due to my little knowledge... please give me so guidance ...

[MartyG-RealSense]

What may work better than a chessboard for obtaining a single specific real-world XYZ coordinate is to point the camera at a print-out of an ArUco image tag attached to an object or a wall, as described in #8961 (comment)

[koalaaaaaaaaa]

I've roughly read the link you provided, and found it's undoubtedly helpful, thanks a lot, Marty~
Please let the issue remain open until I finally work out.

[MartyG-RealSense]

Thanks very much, @koalaaaaaaaaa - it is no problem to keep the case open. Good luck!

[roshan1208]

@koalaaaaaaaaa, Please let me know if you were able to solve this issue. I am also doing a similar project where I have to transform the RealSense camera(D435i) coordinate system to a world coordinate system.

[koalaaaaaaaaa]

@roshan1208 I've roughly read the link above that Marty has provided, and found that possibly useful. But I haven't tried myself, you may check that.

[koalaaaaaaaaa]

@roshan1208 this link
https://www.programmersought.com/article/31073681999/

[roshan1208]

@koalaaaaaaaaa thanks I will look and let you know

[koalaaaaaaaaa]

@roshan1208 Thank you~

[MartyG-RealSense]

Hi @koalaaaaaaaaa Do you require further assistance with this case, please? Thanks!

[koalaaaaaaaaa]

@MartyG-RealSense Thanks, Marty, I have set this problem aside for days to work on other things, so I didn't update my news.

[MartyG-RealSense]

Okay, thanks very much for the update. I will keep this case open for a further time period. Good luck!

[MartyG-RealSense]

Hi @koalaaaaaaaaa If you are not currently able to work on your RealSense issue, please feel free to temporarily close the case using the Close Issue button under the comment writing box and re-open it at a future date or create a new issue if you need to. Thanks!

[koalaaaaaaaaa]

ok，thank you~