Aligning depth image (and pointcloud) to RGB image is slightly off in the vertical axis

[nancibles]

Required Info
Camera Model	{ D415 }
Firmware Version	(Open RealSense Viewer --> Click info)
Operating System & Version	{Win (10)
Platform	PC
SDK Version	{ legacy / 2.. }
Language	{python}
Segment	{Robot}

Issue Description

When I run the alignment script, align-depth2color.py, I'm getting an output that looks like this:

However, there's the area at the top of the depth map that, when aligned, is missing. From what I understand with the stereo mapping, any ambiguous areas should lie within the horizontal axis and there shouldn't be any misalignment in the vertical axis?

I'm having trouble with getting the pointcloud reprojection to perfectly align for an external camera, and I want to know if:

1. After running the rs.align function, it seems that the pointcloud also gets realigned to match the D415's RGB image BUT what happens to that area of no-data at the top of the image and is it a part of the pointcloud?
2. Does the pointcloud have the same frame as the RGB image alignment?

For reference, here is my code.

profile = pipe.start(cfg)
depth_frames = []
color_frames = []

align_to = rs.stream.color
align = rs.align(align_to)

for x in range(filter_stack_size):
    frameset = pipe.wait_for_frames()
    aligned_frames = align.process(frameset)
    depth_frames.append(aligned_frames.get_depth_frame())
    color_frames.append(aligned_frames.get_color_frame())
pipe.stop()

for x in range(filter_stack_size):
    frame = depth_frames[x]
    frame = decimation.process(frame)
    frame = depth_to_disparity.process(frame)
    frame = spatial.process(frame)
    frame = temporal.process(frame)
    frame = disparity_to_depth.process(frame)
    frame = hole_filling.process(frame)

pc.map_to(color_frames[0])
points = pc.calculate(frame)
vtx = np.asanyarray(points.get_vertices())
tex = np.asanyarray(points.get_texture_coordinates())

Thanks for the help!

[MartyG-RealSense]

The first point I would make is to recommend that alignment is performed after the post-processing filters are applied and not before the filters. This is how the RealSense Viewer's point cloud does it. Doing so helps to reduce the distortion effect known as aliasing.

It is also a good idea to calibrate the RealSense camera to eliminate that as a possible cause of mis-alignment.

The image seems to have particularly severe occlusion (a dark shadow around the body due to the depth and RGB cameras not having the same physical position). I would expect the shadow to be evenly distributed around the body like an outline, but on your image it is though it is pulled horizontally, putting the occlusion on the arm all on one side of it only. I can see it is a horizontal misalignment because the occlusion is correctly level with your waist and shoulders.

Please try a calibration first and see if that improves your results.

Edit: I note that you mention that you are trying to "align for an external camera". Is one of your streams coming from a separate non-RealSense camera please? If so, the link below may be useful.

#4202

[nancibles]

Thanks for your response. I ran the dynamic calibration tool and decided to hold off on the post-processing altogether for now. The images don't seem to be very much improved in terms of the severe occlusion, as this is what it looks like now. (I've put the same images twice horizontally/vertically just to visualize it better) I'm wondering if there's a way to improve this?

I might be mistaken but I would also assume that because of the way the D415 is set up there shouldn't have been any alignment needing to happen in the vertical axis as the cameras are in line on that axis?

The pointcloud reprojection also isn't working for my external camera but I'm doing that on one static image and it's not included here because I'm just trying to figure out how to align the depth/pointcloud to the D415's RGB first.

[MartyG-RealSense]

It may be best to use the non-RealSense camera for the RGB stream and the RealSense for the depth stream. This is because the RealSense 400 Series is not interfered with by non-RealSense cameras that are nearby, but RealSense 400 Series does interfere with non-RealSense IR cameras.

An example would be a laptop with a built in IR camera being interfered with by a RealSense camera using IR near the laptop. As far as I know, an RGB stream on a non-RealSense camera should be immune to interference.

If you are trying also to align depth and RGB in a point cloud on the same camera (D415), there is a method for doing it in the RealSense Viewer. If it looks fine in the Viewer then it would suggest that the issue is in the align script rather than a hardware issue. There is a guide to doing so in the link below. On the page, click Expand Post to see the guide.

https://forums.intel.com/s/question/0D70P000006AebbSAC/pixels-mismatch

[RealSenseCustomerSupport]

---

Please let us know if further assistance is needed.
Thank you.

[nancibles]

I'm just still not sure why there are data-less pixels at the top of the frame; shouldn't alignment (using rs.align) between the cameras only be happening in the horizontal axis, given the placement of the cameras? I've taken various images while ensuring that there are no nearby IR sources as suggested by MartyG with the same results.

On another note, I will try to play around with the filters in order to minimize the horizontal occlusion but if there are any other ways to improve this, please let me know.

[RealSenseCustomerSupport]

---

Since our RealSense D4 Processor uses the left IR imager as the reference to generate depth data, it could be helpful to compare Left IR to the RGB image of the same resolution and see the vertical differences, especially when the objects are close. However, it may or may not be what you've been observing.
Thank you.

Attachment(s):
rgb-leftIR_alignZ16_Z16.png - https://realsensesupport.intel.com/attachments/token/rpu80jMMHLFTzV7onmEQvv1A4/?name=rgb-leftIR_alignZ16_Z16.png

[RealSenseCustomerSupport]

---

Please let us know if further assistance is needed. Thank you.