Aligned depth jumping when low light

[redvinaa]

Dear realsense developers,

We have this issue with the realsense d435 cameras, that the resolution of the aligned depth image
keeps changing when there is low light (relatively). We tried changing the raw resolution of the depth image
using parameters, but the resolution stays the same.

To use the images, we need the same aspect ratio for the color and depth images.

To launch the node, we used the following command:
ros2 launch realsense2_camera rs_launch.py align_depth:=true depth_width:=640 depth_height:=480

Here is a demonstration video:
video

We use ros-galactic.

Thanks

[MartyG-RealSense]

Hi @redvinaa When defining a custom configuration in the roslaunch instruction, three factors should be specified for each stream type defined - width, height and FPS. If these three values are not provided then the custom configuration is deemed to be invalid and the default stream configuration of the camera would be applied by the launch instead. The custom configuration above would be invalid because it does not provide an FPS value.

In #1751 (comment) Doronhi the RealSense ROS wrapper developer provides a custom aligned 640x480 depth and color configuration for ROS2's rs_launch.py instruction.

ros2 launch realsense2_camera rs_launch.py align_depth:=true depth_width:=640 depth_height:=480 depth_fps:=30.0 color_width:=640 color_height:=480 color_fps:=30.0

[redvinaa]

You are right, that solved the aspect ratio problem.

But the aligned image is still jumping. As you can see in the new video,
now the resolution stays the same, but the image only stays aligned if we shine a light into the cam.

Thanks for your help.

[emilnovak]

We concluded that the exposure time of the color camera was messing with the color - depth alignment. Since the color image took more time to make than available between two frames (at 30 fps) the aligned image came out strange. After decreasing the color camera expo time and limiting the fps to 15 the glitchiness disappeared.

Cheers

[MartyG-RealSense]

Thanks very much for the update!

[redvinaa]

Sorry, but the problem still persists.

We tried to set both color and depth fps to 6, set auto_exposure to false and exposure as low as 100.
(The settings do take effect now.) But the aligned image is still jumping. Do you have any idea what could go wrong?

Thanks

[MartyG-RealSense]

As you are using both depth and color and have auto-exposure set to false and FPS set to '6', could you try using an RGB exposure of 70 instead of 100, please?

FPS = 6, auto-exposure = false and RGB manual exposure = 70 is a configuration that can resolve image problems caused by the RGB sensor of the D435 / D435i camera models having a slower 'rolling' shutter than the faster 'global' shutter on the depth sensor.

The D455 camera model has a fast global shutter on both the RGB and depth sensors.

[redvinaa]

Still no change, after setting the exposure to 70.

For the record, this is the command we are running now:

ros2 run realsense2_camera realsense2_camera_node --ros-args
  -p color_width:=640
  -p color_height:=480
  -p color_fps:=6.0
  -p depth_width:=640
  -p depth_height:=480
  -p depth_fps:=6.0
  -p align_depth:=true
  -p initial_reset:=true
  -p rgb_camera.enable_auto_exposure:=false
  -p rgb_camera.exposure:=70

[MartyG-RealSense]

Next, please try disabling depth auto-exposure too. This will test whether the jump is resulting from the depth image being automatically adjusted by the depth auto-exposure following close-range exposure to the bright light. This is because directing a bright light directly into the view of the left-right sensors can cause them to be saturated. Removing the light source or moving the camera away from viewing the light can then result in auto-exposure adjusting the image to try to auto-recover from the sensor saturation.

[redvinaa]

There is no such parameter: depth.exposure, did you mean stereo_module.exposure (and enable_auto_exposure)?
If yes, we tried setting that too, but it still doesn't work.

[MartyG-RealSense]

Yes, I meant stereo module exposure. Thanks for the confirmation that you already tried disabling stereo auto-exposure.

I carefully compared again the rect_raw image and the aligned image, Aside from the flicker, there does not seem to be a significant difference in appearance between the depth-only image and the aligned image.

Could you provide information please about the reason for using the aligned-depth topic instead of just the color and depth topics, please.

[redvinaa]

We are trying to do 3d slam using rtabmap and two d435 cameras.

To feed multiple cameras to rtabmap, we have to feed the color, depth and camera_info topics into a node called rgbd_sync
that publishes the data forward to rtabmap to process. We have two rgbd_sync processes, one per camera.

The problem is, rgbd_sync asserts that the two images (color and depth) has the same fov, width, height, etc., and
so it only takes one camera_info. So it can't work because even though the resolutions are the same,
the different fovs need to be compensated. This is why we need the aligned image.

The same setup works perfectly in simulation, where the same camera properties can be set, so it's probably not an rtabmap problem.

[MartyG-RealSense]

So your project is like a 2-camera version of Intel's D435i SLAM guide for performing SLAM with rtabmap_ros?

https://github.com/IntelRealSense/realsense-ros/wiki/SLAM-with-D435i

[redvinaa]

Exactly. Though that project uses ROS1.

[MartyG-RealSense]

Have you tried the rs_multi_camera_launch.py ROS2 launch file instead of rs_launch.py

https://github.com/IntelRealSense/realsense-ros/blob/ros2/realsense2_camera/launch/rs_multi_camera_launch.py

[redvinaa]

We are using a b450 chipset based x86-64 system. On the motherboard there are some blue and red (3.2 / 3.1 Gen2) usbs. After plugging both of our d435 cameras to the red ports, it seems to give a stable aligned image. I'm not sure about the exact usb-pcie chipset but it is possible that it is a usb issue.

We are still experimenting, I will update this thread if we find the culprit.
In the meantime thanks for the help.

[MartyG-RealSense]

You are very welcome, @redvinaa - it's great to hear that you made progress. :) Please do update here if you have more information to share. Good luck!