Save pointcloud (RGB) without meshing?

[philianeles]

Required Info
Camera Model	D435i
Firmware Version	05.12.07.100
Operating System & Version	Win10
Platform	PC
SDK Version	2.38.1
Language	python
Segment	Robot

Issue Description

Is there an example (python) to save the pointcloud (RGB) without meshing?
Basically, this functionality of the viewer:

I have the following code (adapted from examples in the sdk) but does not return the point cloud with RGB info (but the mesh):

import numpy as np                     
import pyrealsense2 as rs             

import numpy as np

print("Environment Ready")

pc = rs.pointcloud()
points = rs.points()

pipe = rs.pipeline()

#Create a config and configure the pipeline to stream
#  different resolutions of color and depth streams
config = rs.config()
config.enable_stream(rs.stream.depth, 640, 480, rs.format.z16, 30)
config.enable_stream(rs.stream.color, 640, 480, rs.format.bgr8, 30)

# Start streaming
profile = pipe.start(config)

# Getting the depth sensor's depth scale (see rs-align example for explanation)
depth_sensor = profile.get_device().first_depth_sensor()
depth_scale = depth_sensor.get_depth_scale()
print("Depth Scale is: " , depth_scale)

# We will be removing the background of objects more than
#  clipping_distance_in_meters meters away
clipping_distance_in_meters = 1 #1 meter
clipping_distance = clipping_distance_in_meters / depth_scale

# Create an align object
# rs.align allows us to perform alignment of depth frames to others frames
# The "align_to" is the stream type to which we plan to align depth frames.
align_to = rs.stream.color
align = rs.align(align_to)


# Get frameset of color and depth
frames = pipe.wait_for_frames()
# frames.get_depth_frame() is a 640x360 depth image

# Align the depth frame to color frame
aligned_frames = align.process(frames)

# Get aligned frames
aligned_depth_frame = aligned_frames.get_depth_frame() # aligned_depth_frame is a 640x480 depth image
color_frame = aligned_frames.get_color_frame()

#depth_image = np.asanyarray(aligned_depth_frame.get_data())
#color_image = np.asanyarray(color_frame.get_data())

points = pc.calculate(aligned_depth_frame)
pc.map_to(color_frame)
print("Saving to 1.ply...")
points.export_to_ply("1.ply", color_frame)
print("Done")

[MartyG-RealSense]

Hi @philianeles The SDK provides OPTION_PLY configuration commands for enabling / disabling mesh, normals and binary point cloud settings.

#4906

A Python script in the opening comment of the discussion linked to below provides an example of defining the off / on state of these commands when exporting to a ply file.

#6194

[philianeles]

Yes, I have this now:

    ply.set_option(rs.save_to_ply.option_ply_mesh, False)
    ply.set_option(rs.save_to_ply.option_ply_binary, False)
    ply.set_option(rs.save_to_ply.option_ply_normals, True)
    ply.set_option(rs.save_to_ply.option_ignore_color, False)

which gives me the pointcloud, without normals and without RGB info - how do I include them?

[MartyG-RealSense]

I went through your code carefully, checking it against other examples. One trend I noticed is that pc_map_to preceded points:

pc.map_to(color_frame)
points = pc.calculate(aligned_depth_frame)

Whilst in your own script, you reverse that order:

points = pc.calculate(aligned_depth_frame)
pc.map_to(color_frame)

So it may be worth making that edit in your script just to eliminate it as a cause of your problems in getting RGB.

Also, my understanding is that the SDK's default is to export a ply in binary format, and binary = false is used if someone aims to export the data in textual / ASCII format. So can you try setting binary to True please?

#882 (comment)

[philianeles]

I reversed the order to your suggestion:

pc.map_to(color_frame)
points = pc.calculate(aligned_depth_frame)

However the result is still the mesh. No pointcloud with RGB info.

I tried also setting binary to True. But still no pointcloud with RGB info...

[MartyG-RealSense]

I compared your script to the SDK's export_ply_example Python example that can export color.

https://github.com/IntelRealSense/librealsense/blob/master/wrappers/python/examples/export_ply_example.py

It is possible that using points.export_to_ply to export the ply is not compatible with setting the ply options with ply.set_option(rs.save_to_ply ...).

In the export_to_ply example, which also supports setting the ply configuration options, it exports the ply with the rs.save_to_ply method instead of export_to_ply:

https://github.com/IntelRealSense/librealsense/blob/master/wrappers/python/examples/export_ply_example.py#L36

Notably, it also sets the options after the export instruction.

[philianeles]

I have a version with save_to_ply actually: (exactly like your suggestion - however does not return the RGB info. Only a point cloud that is missing the color info and the normals.

import numpy as np                       
import matplotlib.pyplot as plt          
import pyrealsense2 as rs                 

import numpy as np

print("Environment Ready")

def export ():
    # Setup:
    pipe = rs.pipeline()
    cfg = rs.config()
    # cfg.enable_stream(rs.stream.depth, 640, 480, rs.format.z16, 30)
    cfg.enable_stream(rs.stream.depth, 848, 480, rs.format.z16, 30)
    # cfg.enable_stream(rs.stream.depth)
    profile = pipe.start(cfg)

    # Skip 5 first frames to give the Auto-Exposure time to adjust
    for x in range(5):
        pipe.wait_for_frames()

    frames = []
    for x in range(15):
        frameset = pipe.wait_for_frames()
        frames.append(frameset.get_depth_frame())
    # print (frames)

    # Cleanup:
    pipe.stop()
    print("Frames Captured")

    ### POST PROCESSING FILTERS ###
    ##DECLARE FILTERS
    #Threshold filter
    threshold = rs.threshold_filter(min_dist = 0.15, max_dist = 2)
    #Decimation filter
    decimation = rs.decimation_filter()
    decimation.set_option(rs.option.filter_magnitude, 2)
    #Spatial filter
    spatial = rs.spatial_filter()
    spatial.set_option(rs.option.filter_magnitude, 1)#Filtering itterations from 1.000 to 5.000
    spatial.set_option(rs.option.filter_smooth_alpha, 1)
    spatial.set_option(rs.option.filter_smooth_delta, 3)# 8 is good
    spatial.set_option(rs.option.holes_fill, 2)
    #Temporal Filter
    temporal = rs.temporal_filter()
    temporal.set_option(rs.option.filter_smooth_alpha, 0.2)
    temporal.set_option(rs.option.filter_smooth_delta, 100)
    #Hole filling
    hole_filling = rs.hole_filling_filter()
    #Colorize
    colorizer = rs.colorizer()
    colorizer.set_option(rs.option.color_scheme, 2)
    
    depth_to_disparity = rs.disparity_transform(True)
    disparity_to_depth = rs.disparity_transform(True)
    #Apply filters
    for i in range(15):
        depth_frame = frames[i]
        depth_frame = threshold.process(depth_frame)
        depth_frame = temporal.process(depth_frame)
        depth_frame = decimation.process(depth_frame)
        depth_frame = spatial.process(depth_frame)
        # depth_frame = hole_filling.process(depth_frame)
        # depth_frame = colorizer.process(depth_frame)
        # colorized = colorizer.process(depth_frame)
        
    # Create save_to_ply object
        ply = rs.save_to_ply("pc.ply")

        # Set options to the desired values
        # Generate a textual PLY with normals (mesh is already created by default)
        ply.set_option(rs.save_to_ply.option_ply_mesh, False)
        ply.set_option(rs.save_to_ply.option_ply_binary, False)
        ply.set_option(rs.save_to_ply.option_ply_normals, True)
        ply.set_option(rs.save_to_ply.option_ignore_color, False)
    
    print("Saving ply...")
    # Apply the processing block to the frameset which contains the depth frame and the texture
    ply.process(depth_frame)
    print("Done")

export()

[MartyG-RealSense]

The next action that I would advise trying is to move the entire ply export code block to one line above the pipe.stop() instruction so that the export instruction is run whilst the pipeline is still active.

[philianeles]

Unfortunately if pipe.stop() comes after that block the code does not run without errors. No file saved at all.

[MartyG-RealSense]

Apologies for the delay in responding further. I was carefully considering the next step.

I have the feeling that there may be an issue related to how alignment is being set up. Looking at the align block setup in the code example below may give insights about something that may need to be changed / added in your own script.

#2769 (comment)

[MartyG-RealSense]

Hi @philianeles Do you have progress with this case to report, please? Thanks!

[philianeles]

Hi @MartyG-RealSense, yes, I also tried this suggestion however did not help.

[MartyG-RealSense]

There was another case recently from a RealSense user trying to export a colored point cloud to a ply with Python and not succeeding. There seems to be a historical pattern of cases in which RealSense users try to achieve this in Python without success.

#7770

[MartyG-RealSense]

Hi @philianeles Do you require further assistance with this case, please? Thanks!

[MartyG-RealSense]

Case closed due to no further cmmments received.

[sanjaiiv04]

@MartyG-RealSense Hey I am working on cropping depth map that I get from Intel RealSense D435 to get the depth map of a particular object in the frame and then process that cropped depth map into a point cloud. I tried it with the inbuilt functions but it only saved the entire frame as point cloud and not the cropped depth map. How do you think we can do that?

[MartyG-RealSense]

Hi @sanjaiiv04 The D435 and D455 camera model types support an 848x100 resolution where the depth data is cropped to a wide but thin horizontal strip at the center of the camera's field of view.

2D mode

3D point cloud mode

It is possible to define your own horizontal and vertical cropping, though it is a complex procedure. #2016 (comment) offers advice from a RealSense user about a method for doing so.