Camera Extrinsics Issue

[kudansam]

Required Info
Camera Model	D435i
Firmware Version	05.12.03.00
Operating System & Version	MacOS
Kernel Version (Linux Only)	(e.g. 4.14.13)
Platform	PC / Android
SDK Version	2
Language	C
Segment	Robot

Issue Description

I'm having trouble understanding the output extrinsics of the realsense SDK.

I'm attempting to obtain the extrinsics of the RGB stream with respect to the depth stream as follows:

rs2::stream_profile depthStream = pipeline.get_active_profile().get_stream(RS2_STREAM_DEPTH);
rs2::stream_profile colorStream = pipeline.get_active_profile().get_stream(RS2_STREAM_COLOR);

rs2_extrinsics rsExtrinsics = depthStream.get_extrinsics_to(colorStream);

Given the documentation suggests the depth coordinate system has the x-axis pointing to left of the camera when viewed from the front:

And that the RGB module is to the right of the depth module on the D435i:

I would expect the x component of the output extrinsics to be negative. However, when I query from the device I get a result with a positive x component:

translation: [0.014835646376013756, 0.0003800476551987231, 0.0001869368425104767]

Am I misunderstanding the coordinate system of the realsense or the API in some way?

[MartyG-RealSense]

I hope that the discussion in the link below will be useful to you.

#2327

[kudansam]

Thanks for the information.

That link is helpful, and matches what I'm observing. This means the streams are not in the point coordinate space. Is there any further documentation for the coordinate space of the individual camera streams?

Also, will the accelerometer and gyroscope poses be in the same coordinate space as the depth frame with the motion correction turned off?

[MartyG-RealSense]

The SDK has a 'Projection' documentation page that describes the coordinate systems.

https://github.com/IntelRealSense/librealsense/wiki/Projection-in-RealSense-SDK-2.0

The 'world origin' location of the 400 Series cameras is the left IR imager.

The axes of the D435i's IMU are described in the link below:

https://www.intelrealsense.com/how-to-getting-imu-data-from-d435i-and-t265/

The purpose of the motion correction is to correct for a known inaccuracy factor in the IMU hardware component, so that the final calculation for gravity acceleration is closer to the '9.81' gravity value of the real world after adjustment than the "raw" gravity values before adjustment.

[ev-mp]

@kudansam , the Depth->RGB extrinsic, or RGB extrinsic in short does not represents the position of the RGB sensor in World CS (Depth sensor) but the inverse - the origin of the world as observed from the RGB sensor's base. It is often a source of confusion when dealing with the camera matrix.

A very elaborated explanation about the camera matrix can be found at Kyle Simek's site : Quote

[The extrinsic] matrix describes how to transform points in world coordinates to camera coordinates. The vector T can be interpreted as the position of the world origin in camera coordinates, and the columns of R represent represent the directions of the world-axes in camera coordinates.
.
The important thing to remember about the extrinsic matrix is that it describes how the world is transformed relative to the camera. This is often counter-intuitive, because we usually want to specify how the camera is transformed relative to the world.

[ev-mp]

@kudansam , do you have further questions or this can be closed?.

[kudansam]

No. That's very useful and has clarified things. Thanks everyone!