D435 Align High CPU #5440

amatabuena · 2019-12-12T16:13:14Z

Required Info
Camera Model	D435
Firmware Version	05.12.00
Operating System & Version	Windows 10
Kernel Version (Linux Only)
Platform	PC
SDK Version	2.19.0
Language	C#
Segment	others

Issue Description

Hi,

I'm trying to use D435 with lastest Production SDK, and I have realized that CPU usage is quite high (almost x2) when frame alignment is done. This is aprox de CPU usage:

Without aligment: 30% to 32%
With alignment (Depth alignment to Color): 55% to 60%

This usage is for my laptop, an Intel Core i5-6300U, 2.40Ghz, x64 architecture and 8Gb RAM.

As I need alignment in order to process both Color and Depth Frames together, ¿There is any way to reduce this consumption, or impplement any oter kind of aligment?

Thanks,

MartyG-RealSense · 2019-12-12T18:21:50Z

You may be able to reduce the load on the CPU if you offload some of the processing onto your computer's GPU.

If you have an Nvidia GPU then you can build Librealsense with CUDA support. If your GPU is not Nvidia, you could alternatively offload to the GPU with a GLSL processing block, which is vendor-neutral. Details of both methods can be found in the link below.

#4905 (comment)

If your recording sessions are short (about 10 seconds) then another option might be to use an instruction called Keep() to store the frames in memory and then do the alignment on the frames all at once as a batch operation when the stream is closed and save the results. The requirement for a short recording duration is because the Keep() process consumes the computer's available memory resources over time.

#1000

amatabuena · 2019-12-12T19:11:01Z

Hi MartyG-Realsense

My GPU is Intel UHD Graphics 620, so I would have to use GLSL processing block. ¿Does the C# Wrapper implements GLSL processing block? My C++ knowledge is very poor.

About the recording session, the camera will be running and processing continually (24x7), so apart from the fact that cpu consumption is important, I can't use keep.

MartyG-RealSense · 2019-12-12T19:34:26Z

I had a look at the GLSL example program for adapting a project for GLSL use, but my programming knowledge isn't sufficiently advanced to offer advice about how convertible it is to C#. Hopefully one of the Intel RealSense staff on this forum can help with this.

https://github.com/dorodnic/librealsense/tree/glsl_extension/examples/gl

kafan1986 · 2019-12-15T08:42:19Z

@amatabuena I am not sure how you are using it. But if there is a possibility for merging the final result of image processing of colour and depth (both non-aligned), by say simple overlap of (x,y) coordinates or minimum euclidean distance etc. approach, then you can just align the coordinates obtained from colour image processing to coordinates system of depth image or vice-versa. For around 100 different pixels, it will be done in less than millisecond time.

dorodnic · 2019-12-16T20:01:05Z

Hi @amatabuena
Unfortunately, CPU (or GPU) utilization of aligned frames will always be greater compared to non-aligned, since they come out not aligned from the sensor. We are trying to provide reasonably optimized implementations, but it is possible alignment can be further optimized.

ankittecholution · 2019-12-17T06:40:14Z

@amatabuena I am not sure how you are using it. But if there is a possibility for merging the final result of image processing of colour and depth (both non-aligned), by say simple overlap of (x,y) coordinates or minimum euclidean distance etc. approach, then you can just align the coordinates obtained from colour image processing to coordinates system of depth image or vice-versa. For around 100 different pixels, it will be done in less than millisecond time.

Did anyone tried this method??

amatabuena · 2019-12-17T08:14:59Z

Hi @ankittecholution, @kafan1986,

To put you in context, I process Color image to detect faces and then I search a series of characteristics in exactly same region of depth image. So really I would need only (more or less) the coordinates of the square generated for the face detected).

¿How can I calculate the matching of color and depth points using C# Wrapper?

kafan1986 · 2019-12-17T14:35:39Z

Yes, for my own use case I have done the same. There is no ready made solution atleast not in the Android wrapper. One need to call inbuilt function through own code, which my Android I do via JNI. I can post the code if you want, it is already present in the C++ code in librealsense SDK.

amatabuena · 2019-12-17T15:20:55Z

Hi @kafan1986 ,

It would be nice If you could give me an example. I'm not used to working with C++ and it's a bit difficult to me for understanding it.

Thanks.

kafan1986 · 2019-12-17T15:35:09Z

I store the extrinsic and intrinsic camera parameters of the depth and colour frame. Do this after all the decimation filter step (if any). This will not only help you to map from one coordinate to another coordinate system, it will work even if the resolution of camera image and depth image is different.

bool setDepthCamVals(long depthFrameAddress, long colourFrameAddress) {
        if (depthCamValAlreadySet){
            return true;
        }

        rs2::frame depthFrame = reinterpret_cast<rs2_frame *>(depthFrameAddress);
        rs2::frame colourFrame = reinterpret_cast<rs2_frame *>(colourFrameAddress);

        auto depth_profile = depthFrame.get_profile().as<rs2::video_stream_profile>();
        auto colour_profile = colourFrame.get_profile().as<rs2::video_stream_profile>();

        _depth_intrin = depth_profile.get_intrinsics();
        _depth_to_color_extrin = depth_profile.get_extrinsics_to(colour_profile);
        _color_intrin = colour_profile.get_intrinsics();
        _colour_to_depth_extrin = colour_profile.get_extrinsics_to(depth_profile);
        if (finalModeLogging) {
            __android_log_print(ANDROID_LOG_VERBOSE, APPNAME, "_depth_intrin: width %d , height %d , ppx %f , ppy %f , fx %f , fy %f , c0 %f , c1 %f , c2 %f , c3 %f , c4 %f",
                                _depth_intrin.width,_depth_intrin.height, _depth_intrin.ppx,_depth_intrin.ppy,_depth_intrin.fx,_depth_intrin.fy,_depth_intrin.coeffs[0],_depth_intrin.coeffs[1],_depth_intrin.coeffs[2],_depth_intrin.coeffs[3],_depth_intrin.coeffs[4]);


            __android_log_print(ANDROID_LOG_VERBOSE, APPNAME, "_color_intrin: width %d , height %d , ppx %f , ppy %f , fx %f , fy %f , c0 %f , c1 %f , c2 %f , c3 %f , c4 %f",
                                _color_intrin.width,_color_intrin.height, _color_intrin.ppx,_color_intrin.ppy,_color_intrin.fx,_color_intrin.fy,_color_intrin.coeffs[0],_color_intrin.coeffs[1],_color_intrin.coeffs[2],_color_intrin.coeffs[3],_color_intrin.coeffs[4]);

            __android_log_print(ANDROID_LOG_VERBOSE, APPNAME, "_depth_to_color_extrin Translation Vector: [ %f , %f , %f ]",
                                _depth_to_color_extrin.translation[0],_depth_to_color_extrin.translation[1],_depth_to_color_extrin.translation[2]);
            __android_log_print(ANDROID_LOG_VERBOSE, APPNAME, "_depth_to_color_extrin Rotation Matrix: [ %f , %f , %f , %f , %f , %f , %f , %f , %f ]",
                                _depth_to_color_extrin.rotation[0],_depth_to_color_extrin.rotation[1],_depth_to_color_extrin.rotation[2],_depth_to_color_extrin.rotation[3],_depth_to_color_extrin.rotation[4],_depth_to_color_extrin.rotation[5],_depth_to_color_extrin.rotation[6],_depth_to_color_extrin.rotation[7],_depth_to_color_extrin.rotation[8]);

            __android_log_print(ANDROID_LOG_VERBOSE, APPNAME, "_colour_to_depth_extrin Translation Vector: [ %f , %f , %f ]",
                                _colour_to_depth_extrin.translation[0],_colour_to_depth_extrin.translation[1],_colour_to_depth_extrin.translation[2]);
            __android_log_print(ANDROID_LOG_VERBOSE, APPNAME, "_colour_to_depth_extrin Rotation Matrix: [ %f , %f , %f , %f , %f , %f , %f , %f , %f ]",
                                _colour_to_depth_extrin.rotation[0],_colour_to_depth_extrin.rotation[1],_colour_to_depth_extrin.rotation[2],_colour_to_depth_extrin.rotation[3],_colour_to_depth_extrin.rotation[4],_colour_to_depth_extrin.rotation[5],_colour_to_depth_extrin.rotation[6],_colour_to_depth_extrin.rotation[7],_colour_to_depth_extrin.rotation[8]);
        }
        depthCamValAlreadySet = true;
        return true;
    }

The below code changes depth coordinate to colour coordinate. z value is supposed to be in meters by default.

PixelOwn mapDepthToColourPixel(double xVal, double yVal, double zVal){

        if(zVal < 0.1) {//zVal can never be less than 10 cm. You can return 0 as well or try to find best candidate z value from nearest valid pixel. Here I am returning same coordinates as received after simple scaling.
            PixelOwn tmpPixelOwn;
            tmpPixelOwn.x = (int)(xVal*_color_intrin.width/_depth_intrin.width);
            tmpPixelOwn.y = (int)(yVal*_color_intrin.height/_depth_intrin.height);
            return tmpPixelOwn;
        }

        //https://github.com/IntelRealSense/librealsense/issues/1890
        float depth_pixel[2] = {(float)(xVal-0.5), (float)(yVal-0.5)};
        float depth = (float)zVal;

        float depth_point[3], colour_point[3], colour_pixel[2];
        rs2_deproject_pixel_to_point(depth_point, &_depth_intrin, depth_pixel, depth);
        rs2_transform_point_to_point(colour_point, &_depth_to_color_extrin, depth_point);
        rs2_project_point_to_pixel(colour_pixel, &_color_intrin, colour_point);
        PixelOwn tmpPixelOwn;
        tmpPixelOwn.x = min(max(0,static_cast<int>(colour_pixel[0] + 0.5f)),_colourCamWidth-1);
        //tmpPixelOwn.x = min(max(0,static_cast<int>(colour_pixel[0])),_colourCamWidth);
        tmpPixelOwn.y = min(max(0,static_cast<int>(colour_pixel[1] + 0.5f)),_colourCamHeight-1);
        //tmpPixelOwn.y = min(max(0,static_cast<int>(colour_pixel[1])),_colourCamHeight);
        return tmpPixelOwn;
    }

Below function is used to convert colour coordinate to depth coordinate. You need the actual depth frame to be passed to function for this to work. So if you are doing this at a later step, clone the particular depth frame.

PointOwn MapColourToDepthPoint(double xVal, double yVal, long depthFrameAddress, double depthMinMeters, double depthMaxMeters){

        float colour_pixel[2] = {(float)xVal, (float)yVal};
        float depth_pixel[2];
        rs2::frame depth = reinterpret_cast<rs2_frame *>(depthFrameAddress);
        rs2_project_color_pixel_to_depth_pixel(depth_pixel, reinterpret_cast<const uint16_t*>(depth.get_data()), _depthScale, float(depthMinMeters), (float)depthMaxMeters, &_depth_intrin, &_color_intrin, &_colour_to_depth_extrin, &_depth_to_color_extrin, colour_pixel);
        PointOwn tmpPointOwn;
        tmpPointOwn.x = min(max(0,static_cast<int>(depth_pixel[0])),_depthCamWidth);
        tmpPointOwn.y = min(max(0,static_cast<int>(depth_pixel[1])),_depthCamHeight);
        return tmpPointOwn;
    }

ankittecholution · 2019-12-19T10:58:40Z

I store the extrinsic and intrinsic camera parameters of the depth and colour frame. Do this after all the decimation filter step (if any). This will not only help you to map from one coordinate to another coordinate system, it will work even if the resolution of camera image and depth image is different.

bool setDepthCamVals(long depthFrameAddress, long colourFrameAddress) {
        if (depthCamValAlreadySet){
            return true;
        }

        rs2::frame depthFrame = reinterpret_cast<rs2_frame *>(depthFrameAddress);
        rs2::frame colourFrame = reinterpret_cast<rs2_frame *>(colourFrameAddress);

        auto depth_profile = depthFrame.get_profile().as<rs2::video_stream_profile>();
        auto colour_profile = colourFrame.get_profile().as<rs2::video_stream_profile>();

        _depth_intrin = depth_profile.get_intrinsics();
        _depth_to_color_extrin = depth_profile.get_extrinsics_to(colour_profile);
        _color_intrin = colour_profile.get_intrinsics();
        _colour_to_depth_extrin = colour_profile.get_extrinsics_to(depth_profile);
        if (finalModeLogging) {
            __android_log_print(ANDROID_LOG_VERBOSE, APPNAME, "_depth_intrin: width %d , height %d , ppx %f , ppy %f , fx %f , fy %f , c0 %f , c1 %f , c2 %f , c3 %f , c4 %f",
                                _depth_intrin.width,_depth_intrin.height, _depth_intrin.ppx,_depth_intrin.ppy,_depth_intrin.fx,_depth_intrin.fy,_depth_intrin.coeffs[0],_depth_intrin.coeffs[1],_depth_intrin.coeffs[2],_depth_intrin.coeffs[3],_depth_intrin.coeffs[4]);


            __android_log_print(ANDROID_LOG_VERBOSE, APPNAME, "_color_intrin: width %d , height %d , ppx %f , ppy %f , fx %f , fy %f , c0 %f , c1 %f , c2 %f , c3 %f , c4 %f",
                                _color_intrin.width,_color_intrin.height, _color_intrin.ppx,_color_intrin.ppy,_color_intrin.fx,_color_intrin.fy,_color_intrin.coeffs[0],_color_intrin.coeffs[1],_color_intrin.coeffs[2],_color_intrin.coeffs[3],_color_intrin.coeffs[4]);

            __android_log_print(ANDROID_LOG_VERBOSE, APPNAME, "_depth_to_color_extrin Translation Vector: [ %f , %f , %f ]",
                                _depth_to_color_extrin.translation[0],_depth_to_color_extrin.translation[1],_depth_to_color_extrin.translation[2]);
            __android_log_print(ANDROID_LOG_VERBOSE, APPNAME, "_depth_to_color_extrin Rotation Matrix: [ %f , %f , %f , %f , %f , %f , %f , %f , %f ]",
                                _depth_to_color_extrin.rotation[0],_depth_to_color_extrin.rotation[1],_depth_to_color_extrin.rotation[2],_depth_to_color_extrin.rotation[3],_depth_to_color_extrin.rotation[4],_depth_to_color_extrin.rotation[5],_depth_to_color_extrin.rotation[6],_depth_to_color_extrin.rotation[7],_depth_to_color_extrin.rotation[8]);

            __android_log_print(ANDROID_LOG_VERBOSE, APPNAME, "_colour_to_depth_extrin Translation Vector: [ %f , %f , %f ]",
                                _colour_to_depth_extrin.translation[0],_colour_to_depth_extrin.translation[1],_colour_to_depth_extrin.translation[2]);
            __android_log_print(ANDROID_LOG_VERBOSE, APPNAME, "_colour_to_depth_extrin Rotation Matrix: [ %f , %f , %f , %f , %f , %f , %f , %f , %f ]",
                                _colour_to_depth_extrin.rotation[0],_colour_to_depth_extrin.rotation[1],_colour_to_depth_extrin.rotation[2],_colour_to_depth_extrin.rotation[3],_colour_to_depth_extrin.rotation[4],_colour_to_depth_extrin.rotation[5],_colour_to_depth_extrin.rotation[6],_colour_to_depth_extrin.rotation[7],_colour_to_depth_extrin.rotation[8]);
        }
        depthCamValAlreadySet = true;
        return true;
    }

The below code changes depth coordinate to colour coordinate. z value is supposed to be in meters by default.

PointOwn mapDepthToColourPixel(double xVal, double yVal, double zVal){

        //https://github.com/IntelRealSense/librealsense/issues/1890
        //float depth_pixel[2] = {(float)(xVal-0.5), (float)(yVal-0.5)};
        float depth_pixel[2] = {(float)(xVal), (float)(yVal)};
        float depth = (float)zVal;

        float depth_point[3], colour_point[3], colour_pixel[2];
        rs2_deproject_pixel_to_point(depth_point, &_depth_intrin, depth_pixel, depth);
        rs2_transform_point_to_point(colour_point, &_depth_to_color_extrin, depth_point);
        rs2_project_point_to_pixel(colour_pixel, &_color_intrin, colour_point);
        PointOwn tmpPointOwn;
        //tmpPointOwn.x = min(max(0,static_cast<int>(colour_pixel[0] + 0.5f)),_colourCamWidth);
        tmpPointOwn.x = min(max(0,static_cast<int>(colour_pixel[0])),_colourCamWidth);
        //tmpPointOwn.y = min(max(0,static_cast<int>(colour_pixel[1] + 0.5f)),_colourCamHeight);
        tmpPointOwn.y = min(max(0,static_cast<int>(colour_pixel[1])),_colourCamHeight);
        return tmpPointOwn;
    }

Below function is used to convert colour coordinate to depth coordinate. You need the actual depth frame to be passed to function for this to work. So if you are doing this at a later step, clone the particular depth frame.

PointOwn MapColourToDepthPoint(double xVal, double yVal, long depthFrameAddress, double depthMinMeters, double depthMaxMeters){

        float colour_pixel[2] = {(float)xVal, (float)yVal};
        float depth_pixel[2];
        rs2::frame depth = reinterpret_cast<rs2_frame *>(depthFrameAddress);
        rs2_project_color_pixel_to_depth_pixel(depth_pixel, reinterpret_cast<const uint16_t*>(depth.get_data()), _depthScale, float(depthMinMeters), (float)depthMaxMeters, &_depth_intrin, &_color_intrin, &_colour_to_depth_extrin, &_depth_to_color_extrin, colour_pixel);
        PointOwn tmpPointOwn;
        tmpPointOwn.x = min(max(0,static_cast<int>(depth_pixel[0])),_depthCamWidth);
        tmpPointOwn.y = min(max(0,static_cast<int>(depth_pixel[1])),_depthCamHeight);
        return tmpPointOwn;
    }

Thanks for the code. I understood the code and how to do it. I have only one doubt what are these parameter and how to get those:-
double depthMinMeters, double depthMaxMeters

kafan1986 · 2019-12-19T17:31:47Z

It is the depth range in which it searches for the best match. Anyways according the SDK, you can use a suitable value around 0.1 meter for depthMinMeters and 10 meter for depthMaxMeters.

ankittecholution · 2019-12-19T17:42:52Z

Thanks, @kafan1986 for your help. I am trying to make an android application with this where I can find the dept of a frame corresponding to the colour frame. If you have any information about how to get a bitmap from the colour frame that would be great but other than converting GLRenderer into Bitmap.

amatabuena · 2019-12-19T18:06:57Z

Thanks @ankittecholution,

I will try it as soon as I can. I will have to find out how to create the methods you mentioned in C++ and the C# Wrapper for calling it.

kafan1986 · 2019-12-19T18:21:27Z

Thanks @ankittecholution,

I will try it as soon as I can. I will have to find out how to create the methods you mentioned in C++ and the C# Wrapper for calling it.

All the best. Anyways, you generally need either of the two functions. Either you want everything to be represented in depth coordinate system or color coordinate system. I use Colour coordinate system, so don't need the depth Frame.

kafan1986 · 2019-12-20T14:19:32Z

Thanks, @kafan1986 for your help. I am trying to make an android application with this where I can find the dept of a frame corresponding to the colour frame. If you have any information about how to get a bitmap from the colour frame that would be great but other than converting GLRenderer into Bitmap.

I believe you have 2 questions:
A) You can use the MapColourToDepthPoint function above to get the (x,y) coordinate in depth field corresponding to colour coordinate and then you can use getDistance function which returns the distance in meters by default.
B) If you have used stream format as RGB for colour stream, then you can use above code to convert it into bitmap as it is already in 8UC3 format. In case, you are using native YUYV format, then it needs to be converted into RGB format first.

RealSenseCustomerSupport · 2020-04-29T21:18:41Z

Hi @amatabuena

Will you be needing further assistance with this?

Please note that if we don’t hear from you in 7 days, this issue will be closed.

Thank you

amatabuena · 2020-04-30T07:06:56Z

Hi,

I couldn't find out a optimal solution, so I had to increase hardware specifications in order to ensure performance. (Cost increase is another matter).

Thanks.

MartyG-RealSense · 2020-04-30T07:25:57Z

@amatabuena Pairing the RealSense with hardware acceleration from an Intel Neural Compute Stick 2 in another USB port may be a less costly method to implement your project. The Stick is sold in the official RealSense online store.

https://store.intelrealsense.com/buy-intel-neural-compute-stick-2.html

https://www.intelrealsense.com/depth-camera-and-ncs2/

https://github.com/movidius/ncappzoo/tree/master/apps/realsense_object_distance_detection

RealSenseCustomerSupport · 2020-05-03T19:44:07Z

Hi @amatabuena

Do you need any further assistance with your alignment/cpu-usage question or would you say we can we close this issue?

Thanks

RealSenseCustomerSupport · 2020-05-11T03:57:00Z

Hi.

Will you be needing further help with this? If we don’t hear from you in 7 days, this issue will be closed.

Thanks

amatabuena · 2020-05-11T06:57:03Z

Hi,

I won`t need more help for this issue. Thanks.

MartyG-RealSense · 2020-05-11T08:27:43Z

Thanks! :)

MartyG-RealSense mentioned this issue Dec 24, 2019

color cam scale is smaller than depth cam scale ? IntelRealSense/realsense-ros#1035

Closed

kafan1986 mentioned this issue Jan 10, 2020

[Question] How to use rs2_project_color_pixel_to_depth_pixel in python to convert 1 point from color coordinate into depth coordinate #5603

Closed

kafan1986 mentioned this issue Feb 12, 2020

filter pointcloud from binary image #5822

Closed

kafan1986 mentioned this issue Apr 17, 2020

How to get distance on Android phone #6249

Closed

RealSenseSupport added .NET D400 Series Windows question labels May 3, 2020

amatabuena closed this as completed May 11, 2020

MartyG-RealSense mentioned this issue Oct 27, 2023

Get depth pixel to corresponding color pixel in Unity without aligning #12324

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

D435 Align High CPU #5440

D435 Align High CPU #5440

amatabuena commented Dec 12, 2019 •

edited

Loading

MartyG-RealSense commented Dec 12, 2019 •

edited

Loading

amatabuena commented Dec 12, 2019

MartyG-RealSense commented Dec 12, 2019

kafan1986 commented Dec 15, 2019 •

edited

Loading

dorodnic commented Dec 16, 2019

ankittecholution commented Dec 17, 2019 •

edited

Loading

amatabuena commented Dec 17, 2019

kafan1986 commented Dec 17, 2019

amatabuena commented Dec 17, 2019

kafan1986 commented Dec 17, 2019 •

edited

Loading

ankittecholution commented Dec 19, 2019

kafan1986 commented Dec 19, 2019

ankittecholution commented Dec 19, 2019

amatabuena commented Dec 19, 2019

kafan1986 commented Dec 19, 2019

kafan1986 commented Dec 20, 2019

RealSenseCustomerSupport commented Apr 29, 2020

amatabuena commented Apr 30, 2020

MartyG-RealSense commented Apr 30, 2020

RealSenseCustomerSupport commented May 3, 2020

RealSenseCustomerSupport commented May 11, 2020

amatabuena commented May 11, 2020

MartyG-RealSense commented May 11, 2020

D435 Align High CPU #5440

D435 Align High CPU #5440

Comments

amatabuena commented Dec 12, 2019 • edited Loading

Issue Description

MartyG-RealSense commented Dec 12, 2019 • edited Loading

amatabuena commented Dec 12, 2019

MartyG-RealSense commented Dec 12, 2019

kafan1986 commented Dec 15, 2019 • edited Loading

dorodnic commented Dec 16, 2019

ankittecholution commented Dec 17, 2019 • edited Loading

amatabuena commented Dec 17, 2019

kafan1986 commented Dec 17, 2019

amatabuena commented Dec 17, 2019

kafan1986 commented Dec 17, 2019 • edited Loading

ankittecholution commented Dec 19, 2019

kafan1986 commented Dec 19, 2019

ankittecholution commented Dec 19, 2019

amatabuena commented Dec 19, 2019

kafan1986 commented Dec 19, 2019

kafan1986 commented Dec 20, 2019

RealSenseCustomerSupport commented Apr 29, 2020

amatabuena commented Apr 30, 2020

MartyG-RealSense commented Apr 30, 2020

RealSenseCustomerSupport commented May 3, 2020

RealSenseCustomerSupport commented May 11, 2020

amatabuena commented May 11, 2020

MartyG-RealSense commented May 11, 2020

amatabuena commented Dec 12, 2019 •

edited

Loading

MartyG-RealSense commented Dec 12, 2019 •

edited

Loading

kafan1986 commented Dec 15, 2019 •

edited

Loading

ankittecholution commented Dec 17, 2019 •

edited

Loading

kafan1986 commented Dec 17, 2019 •

edited

Loading