Using format z16h in pyrealsense2 #10644

jkenney9a · 2022-07-02T16:43:04Z

Required Info
Camera Model	{ D435 }
Firmware Version	05.13.00.50
Operating System & Version	Linux (Ubuntu 20.04)
Kernel Version (Linux Only)	5.4.0-26
Platform	PC
SDK Version	{ 2.50.0 }
Language	python}
Segment

Issue Description

We use intel realsense cameras in our research lab for 3D animal tracking (we have a pre-print up on how we've recently used this if you're curious (https://www.biorxiv.org/content/10.1101/2022.04.11.487928v2), so thank you to everyone at Intel for making such a wonderful tool that has really allowed us to level up our research!

Anyway, so the issue we're running into is one of file size, and we've been brainstorming ways to reduce file size. Currently, we take 6 minute videos at 30fps, 640 x 480 yielding ~10GB per file (of which most of this, 9GB, is taken up by the depth stream). To reduce filesize we're playing with reducing the fps to 15, and compressing the files using bz compression (instead of lz4) (changing the resolution of the depth stream seems to have little impact on file size, which I was surprised to see).

So a couple of related questions:

Is it possible to directly alter the 'shutter speed' when we reduce the frame rate? We use our videos to track zebrafish (about 2 cm in length) and our concern is that with the lower frame rate, our images will get blurred, which would affect our ability to track the animals accurately.
I saw that with SDK 2.32, there was the addition of an option to use a compressed format for the depth stream, z16h. However, when I incorporate this into my code I get the following error:

Traceback (most recent call last): File "record_video.py", line 508, in <module> bag_filename = record_and_display_video(camera_pair = camera_pair, File "record_video.py", line 154, in record_and_display_video pipeline_record_profile = pipeline_record.start(config_record) RuntimeError: Couldn't resolve requests

Here is the related code in python:
`
if camera_reset.lower() == 'y':
reset_camera(SN_D435[camera_pair])

#Turn on emitting camera
if emitting_camera_laser.lower() == 'y':
    pipeline_emit = rs.pipeline()
    config_emit = rs.config()
    config_emit.enable_device(SN_D415[camera_pair])
    pipeline_emit_profile = pipeline_emit.start(config_emit)
    
    #Change laser power of emitting laser
    device_emit = pipeline_emit_profile.get_device()
    depth_sensor_emit = device_emit.query_sensors()[0]
    depth_sensor_emit.set_option(rs.option.laser_power, emit_laser_power)

#Fire up recording camera
pipeline_record = rs.pipeline()
config_record = rs.config()
config_record.enable_device(SN_D435[camera_pair])

#Setup streaming and recording
if color_and_depth == 'n':
    config_record.enable_stream(rs.stream.color, resolution['x'], 
                            resolution['y'], rs.format.rgb8, fps)
else:
    config_record.enable_stream(rs.stream.depth, resolution['x'], 
                                resolution['y'], rs.format.z16h, fps)
    config_record.enable_stream(rs.stream.color, resolution['x'], 
                                resolution['y'], rs.format.rgb8, fps)


bag_filename = output_filename + ".bag"    
config_record.enable_record_to_file(bag_filename)

#Turn on recording pipeline
pipeline_record_profile = pipeline_record.start(config_record)
device_record = pipeline_record_profile.get_device()
device_recorder = device_record.as_recorder()

 #Turn off laser from recording camera
if recording_camera_laser.lower() == 'n':
    depth_sensor_record = device_record.query_sensors()[0]
    depth_sensor_record.set_option(rs.option.emitter_enabled, 0)
elif recording_camera_laser.lower() == 'y':
    depth_sensor_record = device_record.query_sensors()[0]
    depth_sensor_record.set_option(rs.option.emitter_enabled, 1)  

#Set autoexposure priority to False to improve frame rate acquisition
color_sensor_record = device_record.query_sensors()[1]
color_sensor_record.set_option(rs.option.auto_exposure_priority, False)`

I saw in a previous post that for z16h to be functional, it's when we're recording to file, which is indeed what I'm doing here. Is there something else that needs to be altered to get z16h to work?

Finally, any other suggestions for reducing file sizes? I'm not quite sure what to expect from using z16h, i.e. how big of a size savings we would get.

Thank you!

The text was updated successfully, but these errors were encountered:

MartyG-RealSense · 2022-07-02T22:19:43Z

Hi @jkenney9a Thanks very much for your questions!

The physical shutter speed of RealSense cameras cannot be changed, though you can instead set a custom exposure time.

The D415 and D435 camera models have a fast 'global shutter' on the depth sensor and a slower 'rolling shutter' on the RGB sensor, which can lead to RGB blurring when capturing fast motion. The D455 camera model has a global shutter on both the depth and RGB sensors.

If your project is not able to change camera models and you cannot use 60 FPS RGB to minimize RGB blur, an alternative method for blur reduction is to disable RGB auto-exposure and set FPS to only '6' and manual RGB exposure time to around '70'.

If you could record a bag at 6 FPS for both depth and RGB then this should have a positive impact on bag size. As small fish are typically slow moving, it should be a practical setting to use.

I see that you are using a two-pipeline, two-configuration method similar to the script at #1735 (comment)

The error Couldn't resolve requests indicates that when the pipeline was started, the program could not provide the requested stream configuration. This could be because there is an error in the formatting of the instruction that prevents the request from being understood or the requested mode is not supported by the camera at the time that the request is made.

Instead of using fps, resolution['x'] and resolution['y'] variables to configure the streams, I would recommend typing the values in directly to confirm whether the problem is with the variable names or the values stored in them. For example:

config_record.enable_stream(rs.stream.depth, 640, 480, rs.format.z16h, 30)

I have only seen examples of Z16H being used with C++ rather than Python, though it is listed as a supported format in the official pyrealsense2 wrapper documentation.

https://intelrealsense.github.io/librealsense/python_docs/_generated/pyrealsense2.format.html#pyrealsense2.format.z16h

In regards to how compression may reduce bag file size, you could test this by enabling recording compression in the RealSense Viewer's settings interface and recording a bag, as described at #10271 (comment)

jkenney9a · 2022-07-02T23:59:22Z

Hi MartyG,

Thanks so much for your response!

Thanks for explaining. I wonder, if I access a pre-recorded bag file via pyrealsense that is recorded at 30fps, is there a straightforward way to re-save it as 15 fps? I.e., basically dropping every other frame? We already run a script that pulls out the color frames and turn it into an .avi to track our animals, so we could include it there.

I should have mentioned, the code runs just fine if I use z16 instead of z16h:
config_record.enable_stream(rs.stream.depth, resolution['x'], resolution['y'], rs.format.z16, fps)

So I know it's not these variables.

So it seems like the pyrealsense2 wrapper doesn't properly implement the z16h? Or is there something else I need to include to use this? Or does the D435 not support this? I too had seen this implemented in C++, so I was hoping it would be simple enough to use with python.

As for compression, all of our recording comes from python scripts run on the command line. It does look like the files are compressed using lz4 compression already, which does a decent job (i.e. files are 10GB instead of 13GB). bz2 compression makes the files ~50% smaller, so we'll do that once we publish to archive the data.

Thanks again for your feedback!

MartyG-RealSense · 2022-07-03T07:20:40Z

As a bag file acts as an alternative source of camera data instead of a live camera and a script that enables from file should behave the same as when using a live camera, it should be possible for a bag-saving script to access a bag file for its streams and save them as a new bag.

It is also possible to retrieve frames at a slower rate than the original FPS by using only every 'nth' frame. For example, setting a stream at 30 FPS but having it effectively be 6 FPS by only using every 5th frame (as 6 x 5 = 30). A Python script where this method is demonstrated is at #3169

Does use of Z16H still cause an error if the enable_record_to_file instruction is moved to above the stream definitions instead of below them in the 'else' statement?

if color_and_depth == 'n':


config_record.enable_stream(rs.stream.color, resolution['x'], resolution['y'], rs.format.rgb8, fps)
else:

bag_filename = output_filename + ".bag"    
config_record.enable_record_to_file(bag_filename)

config_record.enable_stream(rs.stream.depth, resolution['x'], resolution['y'], rs.format.z16h, fps)
config_record.enable_stream(rs.stream.color, resolution['x'], resolution['y'], rs.format.rgb8, fps)

Some further suggestions for RealSense data compression with Python are discussed at #8117

Some file size reduction methods unrelated to compression are suggested at the link below.

https://support.intelrealsense.com/hc/en-us/community/posts/4415672297363/comments/4415672718995

jkenney9a · 2022-07-05T14:51:59Z

Hi Marty,

Thanks for the quick response. I'll play around with framerates and see how that goes as well as other compression options.

With respect to the z16h, changing the location of the enable_record_to_file(bag_filename) command does not make a difference. I still get the same error.

Is it possible the z16h was never properly implemented in Python like is was in C? If that's the case, is it possible to request this? It looks like it was meant to be brought into python.

Thanks!

MartyG-RealSense · 2022-07-05T15:20:16Z

As pyrealsense2 is a wrapper of the C++ SDK, there should be a Python equivalent of each C++ instruction. I will discuss Z16H's Python compatibility with the RealSense development team though.

jkenney9a · 2022-07-05T15:41:14Z

Gotcha'. I did some digging, and it looks like ja couple years ago you suggested that perhaps the firmware wasn't updated to be able to make use of the z16h. Perhaps that never got done?

jkenney9a · 2022-07-05T20:22:58Z

Another, related, question. I mentioned above about reducing framerate to reduce filesize. However, is there a way to reduce the fps within an already existing bagfile? I see that I can just take every other frame if I'm extracting frames, as you pointed out, but I'd like to keep the bag file intact and just change the frame rate within the already existing file (or potentially save it to a new bagfile). I couldn't find any examples and my playing around hasn't been very fruitful.

MartyG-RealSense · 2022-07-05T22:03:46Z

A Python bag recording script at #3671 (comment) uses a sleep instruction to only record a frame every 5 seconds with sleep(5000); so a similar approach with a smaller sleep-time value may be able to be taken with reading bag frames instead. For example, sleeping for '500' to playback a frame every 0.5 seconds if 1000 is equivalent to 1 second.

jkenney9a · 2022-07-05T22:28:31Z

Hi Marty,

Thank you for the suggestion. The issue I'm running into is that when I try to take a pre-existing bag file and record it into a new bag file, it doesn't appear to be supported:

`

pipeline_record = rs.pipeline()
config_record = rs.config()
config_record.enable_device_from_file(bag_file, repeat_playback=False)


#Setup streaming and recording
if color_and_depth == 'n':
    config_record.enable_stream(rs.stream.color, resolution['x'], 
                            resolution['y'], rs.format.rgb8, 15)
else:
    config_record.enable_stream(rs.stream.depth, resolution['x'], 
                                resolution['y'], rs.format.z16, 15)
    config_record.enable_stream(rs.stream.color, resolution['x'], 
                                resolution['y'], rs.format.rgb8, 15)

#bag_filename = output_filename + ".bag"    
config_record.enable_record_to_file('test_15fps.bag')

#Turn on recording pipeline
pipeline_record_profile = pipeline_record.start(config_record)
device_record = pipeline_record_profile.get_device()
device_recorder = device_record.as_recorder()

`

This gives me the error:
RuntimeError: Configuring both device from file, and record to file is unsupported

Maybe there is another way to 're-record' a bag file?

I did play with your 'sleep' suggestion. The issue is that it results in the blurring of color image recorded. I'm reluctant to give a specific autoexposure in case lighting conditions change; an upper autoexposure limit would be ideal, but I see that's not available for the color stream, only the depth stream.

Thanks for your suggestion though!

MartyG-RealSense · 2022-07-06T06:58:00Z

Z16H support was removed from the firmware in 2020, which is why it is not able to work now.

Using 15 FPS instead of 30 (which you have already been trying) should help in reducing file size as the data will be reduced by the same rate.

Furthermore, the sub-sampling filter can help to further reduce the depth map size but it will not work with rosbag recording.

https://dev.intelrealsense.com/docs/depth-post-processing#simple-post-processing

jkenney9a · 2022-07-06T18:00:13Z

Hi Marty,

Thanks for the info on z16h, that's unfortunate, but good to know!

I think I've finally solved the issue for recording of bags. The issue with going down to 15FPS is that the images get blurred because of the increased exposure time. I'm reluctant to give a specific exposure time in case the lighting changes a bit. However, I have come up with a workaround, inspired by some of you suggestions. I added the following code during recording to pause and resume the camera. Because the fps is set to 30, we get the exposure time as normal, but we only end up storing 1/2 or 1/3 of the frames, thereby reducing the file size:

`
#Reduce the number of frames kept. Note the file will 'look' shorter
#in the realsense viewer than it actually is by a factor of the
#size reduction factor

        if i % int(size_reduction_factor) == 0:
            rs.recorder.resume(device_recorder)
        else:
            rs.recorder.pause(device_recorder)
            
        i += 1

`

I'm still searching for a way to reduce the size of existing bag files. I'm thinking the only way to do this would be to use a filter with the rosbag tool, but I'm struggling to understand the right syntax. If I get it, I'll post here. Or if you know of any examples that use the rosbag to filter based on framenumber, I'd really appreciate it!

MartyG-RealSense · 2022-07-07T09:17:04Z

It's great to hear that you found a workaround for blurring - thanks so much for sharing the details with the RealSense community!

I did not find any examples of filtering by frame number other than the 'every nth frame' method discussed earlier.

As you found a way to use a lower FPS of 15 without blurring, could you reduce file size further by using 6 FPS mode with this technique?

jkenney9a · 2022-07-11T16:50:04Z

Hi Marty...so our approach works...kind of, but, unfortunately, we're finding it to be a bit inconsistent. The problem is that sometimes it ends up skipping several frames and yields files with very different numbers of frames captured each time, which is an issue for data collection. I've tried few things (e.g. time.sleep) and pausing/resuming, but cannot seem to find a way to make it consistent. I'm currently trying to see if there is a way I can alter the bag files after the fact with rosbag, but no luck yet.

Are you familiar with a way to use rosbag filter to pull out certain frames?

MartyG-RealSense · 2022-07-12T12:25:23Z

You can use rosbag filter to reduce bag size by filtering by time, as described at #1868

jkenney9a · 2022-07-12T15:01:32Z

Ok. Thank you. I think we finally figured out how best to get our file sizes smaller. Turns out it was the RGB color videos that were contributing most to the size of the bag file, not the depth video. So we've started recording using y16 instead of rgb8.

Thanks for all your help and suggestions!

MartyG-RealSense · 2022-07-12T15:04:28Z

You are very welcome. It's excellent to hear of the significant progress that you made in reducing file size. Thanks very much for the update!

MartyG-RealSense added D400 Series python Linux labels Jul 2, 2022

jkenney9a closed this as completed Jul 12, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Using format z16h in pyrealsense2 #10644

Using format z16h in pyrealsense2 #10644

jkenney9a commented Jul 2, 2022

MartyG-RealSense commented Jul 2, 2022 •

edited

Loading

jkenney9a commented Jul 2, 2022

MartyG-RealSense commented Jul 3, 2022

jkenney9a commented Jul 5, 2022

MartyG-RealSense commented Jul 5, 2022 •

edited

Loading

jkenney9a commented Jul 5, 2022

jkenney9a commented Jul 5, 2022

MartyG-RealSense commented Jul 5, 2022

jkenney9a commented Jul 5, 2022

MartyG-RealSense commented Jul 6, 2022

jkenney9a commented Jul 6, 2022

MartyG-RealSense commented Jul 7, 2022

jkenney9a commented Jul 11, 2022

MartyG-RealSense commented Jul 12, 2022

jkenney9a commented Jul 12, 2022

MartyG-RealSense commented Jul 12, 2022

Using format z16h in pyrealsense2 #10644

Using format z16h in pyrealsense2 #10644

Comments

jkenney9a commented Jul 2, 2022

Issue Description

MartyG-RealSense commented Jul 2, 2022 • edited Loading

jkenney9a commented Jul 2, 2022

MartyG-RealSense commented Jul 3, 2022

jkenney9a commented Jul 5, 2022

MartyG-RealSense commented Jul 5, 2022 • edited Loading

jkenney9a commented Jul 5, 2022

jkenney9a commented Jul 5, 2022

MartyG-RealSense commented Jul 5, 2022

jkenney9a commented Jul 5, 2022

MartyG-RealSense commented Jul 6, 2022

jkenney9a commented Jul 6, 2022

MartyG-RealSense commented Jul 7, 2022

jkenney9a commented Jul 11, 2022

MartyG-RealSense commented Jul 12, 2022

jkenney9a commented Jul 12, 2022

MartyG-RealSense commented Jul 12, 2022

MartyG-RealSense commented Jul 2, 2022 •

edited

Loading

MartyG-RealSense commented Jul 5, 2022 •

edited

Loading