Discrepancy between raw depth frame values when multiplied by units and getDistance function

[blakat360]

I'm trying to understand the way the raw depth values are stored. This is in order for me to be able to bind the depth texture to a shader and then later query it for distance values.

    auto [width, height] = cam.getResolution();

    rs2::depth_frame frame = cam.getDepthFrame();
    auto units = frame.get_units();
    auto arr = reinterpret_cast<const uint16_t*>(frame.get_data());

    double cumulative_error = 0;

    for (int row = 0; row < height; row++) {
        for (int col = 0; col < width; col++) {
            cumulative_error += std::abs(frame.get_distance(col, row) - (double)arr[col + row * col] * units);
        }
    }

    std::cerr << "avg error: " << cumulative_error / (width * height) << '\n';

On some frames the error I'm getting is in excess of 1m.

Is this because get distance linearises the depth values before converting them to a distance? If so, how do I get the near and far planes of the realsense camera?

Edit: the first test script was wrong and added more code for context

[MartyG-RealSense]

Hi @blakat360 The original depth pixel value is the uint16_t value. The pixel value is then multiplied by the depth unit scale provided by get_depth_scale() to convert the pixel value to meters.

#2348 (comment)

The documentation also provides an example of getting the distance in meters using get_distance()

https://dev.intelrealsense.com/docs/projection-in-intel-realsense-sdk-20#section-depth-image-formats

A list of further links to discussions about this subject can be found by scrolling to the bottom of the page at #2348

[blakat360]

I am taking it as a uint_16, note: auto arr = reinterpret_cast<const uint16_t*>(frame.get_data());
I've tested and depth scale is the same as the units given by querying rs2::depth_frame::get_units()

I'm confused as to why cumulative error does not stay at 0. I'm assuming I'm indexing into the image incorrectly but can't seem to come up with something that works.

[MartyG-RealSense]

RealSense 400 Series cameras have a phenomenon called RMS Error where the amount of error in measurements increases over distance. Error is less than 1% of the distance of the object from the camera. So for example, if the camera is 1 meter away then an accuracy of 2.5 mm to 5 mm is expected.

[blakat360]

I understand that - I'm wondering why I get a difference in measurement when I pull the distance out of the raw data using 2 different methods.

[MartyG-RealSense]

I am trying to understand your script, so thank you for your patience. It looks as though you are calculating the distance in two ways - through uint16 and through frame.get_distance() - and subtracting one from the other. And that you expect that if both methods read the same distance then the result should be zero (e.g 1.5 - 1.5 = 0).

Is my interpretation correct, please?

I note that in the uint16 part of the calculation, uint16 seems to be multiplied by frame.get_data(). I would have thought that it should be multiplied by frame.get_depth_scale() to get the distance value in meters with that method.

[blakat360]

That is correct!

This line;
auto arr = reinterpret_cast<const uint16_t*>(frame.get_data());
gets a pointer to the start of the depth frame (through frame.get_data()) and casts this to an array of unsigned 16 bit integers (raw depth values). I also grab the amount I need to multiply these by to get distances using rs2::depth_frame::get_units().

I then loop over the entire frame using the nested for loops and attempt to calculate the distance by
a) calling rs2::depth_frame::get_distance
b) getting the raw value, and multiplying that by units myself

I expect the value to be the same. However, it is not. This leads me to believe I'm indexing into the frame the wrong way.

I have tested get_depth_scale (through querying the device and then accessing its streams) and in this case it is the same as get_units (0.001). This leads me to believe that this is not the problem. Furthermore, the errors are uncorrelated in each iteration of the loop are uncorrelated which further suggests that the multiplier is not the issue.

[ev-mp]

@blakat360 , can you take a second look into the code?

for (int row = 0; row < height; row++) {
        for (int col = 0; col < width; col++) {
            cumulative_error += std::abs(frame.get_distance(col, row) - (double)arr[col + row * col] * units);
        }
    }

Spoiler

arr[col + row * col width] * units

[blakat360]

I'm an idiot, thank you @ev-mp!