Multiple problems while loading from .bag file.

[brunovollmer]

Required Info
Camera Model	D435
Firmware Version	05.12.09.00
Operating System & Version	Ubuntu 20.02
Kernel Version (Linux Only)	5.4.0-56-generic
Platform	PC/Laptop
SDK Version	legacy / 2.40.0
Language	python
Segment	Robot

Issue Description

Hey,

currently I'm struggling with loading a set of frames that I saved into a .bag file. I've included the code below. I struggle with two problems. If I just want to load the data without doing anything with it (see Loading Code) than after some frames I receive this error:

frameset = pipe.wait_for_frames()
RuntimeError: Frame didn't arrive within 5000

If I extend the loading code (see Loading Code Extended) to the actual use case I receive another error a lot earlier:

colorized_depth = np.asanyarray(colorizer.colorize(depth_frame).get_data())
RuntimeError: Error occured during execution of the processing block! See the log for more info

The extended version already crashes after 9 frames, while the first loading codes runs for about 100 frames.

Recording Code

 context = rs.context()
 config = rs.config()
 pipeline = rs.pipeline(context)
 config.enable_record_to_file(output_path)

 pipeline.start(config)

 time.sleep(record_time)

 pipeline.stop()

 del pipeline
 del config

Loading Code

    pipe = rs.pipeline()
    cfg = rs.config()
    cfg.enable_device_from_file(path, repeat_playback=False)
    profile = pipe.start(cfg)

    playback = profile.get_device().as_playback()
    playback.set_real_time(False)

    t = playback.get_duration().seconds + (
        playback.get_duration().microseconds / 1_000_000)
    num_frames = int(t * 30)

    print(f'"{path}" contains {num_frames} frames.')

    # Skip 5 first frames to give the Auto-Exposure time to adjust
    for _ in range(5):
        pipe.wait_for_frames()

    for i in range(num_frames - 5):
        print(i)
        frameset = pipe.wait_for_frames()

        playback.pause()

        depth_frame = frameset.get_depth_frame()
        color_frame = frameset.get_color_frame()

        playback.resume()

    # Cleanup:
    pipe.stop()

Loading Code Extended

    pipe = rs.pipeline()
    cfg = rs.config()
    cfg.enable_device_from_file(path, repeat_playback=False)
    profile = pipe.start(cfg)

    playback = profile.get_device().as_playback()
    playback.set_real_time(False)

    t = playback.get_duration().seconds + (
        playback.get_duration().microseconds / 1_000_000)
    num_frames = int(t * 30)

    print(f'"{path}" contains {num_frames} frames.')

    # Skip 5 first frames to give the Auto-Exposure time to adjust
    for _ in range(5):
        pipe.wait_for_frames()

    colorizer = rs.colorizer()
    align = rs.align(rs.stream.color)

    output_data = []

    for i in range(num_frames - 5):
        print(i)
        frameset = pipe.wait_for_frames()

        playback.pause()

        depth_frame = frameset.get_depth_frame()
        color_frame = frameset.get_color_frame()

        if not depth_frame or not color_frame:
            print("broken frame")
            continue

        # Align both images
        frameset = align.process(frameset)

        # Get depth image data
        aligned_depth_frame = frameset.get_depth_frame()

        colorized_depth = np.asanyarray(colorizer.colorize(depth_frame).get_data())
        colorized_aligned_depth = np.asanyarray(colorizer.colorize(aligned_depth_frame).get_data())
        color_frame = np.asanyarray(color_frame.get_data())

        output_data.append({'color': color_frame, 'aligned_depth': colorized_aligned_depth, 'depth': colorized_depth})
        playback.resume()


    # Cleanup:
    pipe.stop()

    return output_data

[MartyG-RealSense]

Hi @brunovollmer An auto-exposure frame skip mechanism would not usually be put in a playback mechanism. Instead, it would be put in the live capture script (the recording section) so that the initial frames are not captured. As it is playing back recorded data and not live frames in real-time from the camera, the bag file is not being affected by live auto-exposure.

What a skip mechanism in bag playback may accomplish is skipping past the initial several frames if they have been captured in the bag instead of being skipped at the time of capture in the recording script.

Jumping to a specific frame in a bag during playback does not work well when set_real_time = false. Skipping forward to the target frame is recommended instead. It looks as though you are using this method already though. This subject is discussed in the Python case in the link below.

#3121

[brunovollmer]

Thanks for answering @MartyG-RealSense,

Regarding the skip mechanism you are absolutely right. Haven't thought about this.

But my main problem is still open. My goal is simply to record and load all frames (depth and image) into and from the .bag file. I've tried multiple methods but all don't seem to work properly as they either produce the Frame didn't arrive error or the mentioned Runtime error.

What is the suggested way to record data and load it from a .bag file?

[MartyG-RealSense]

Your problem reminds me of a similar past Python case involving RuntimeError: Frame didn't arrive within 5000. The RealSense user in that case posted their script solution at the end of the discussion.

#6766

[brunovollmer]

So after reading your suggestions and searching a bit more I've found a working solution for both problems.

To avoid the RuntimeError: Frame didn't arrive within 5000 error I've used the suggested method from the other post (#6766) to compute the number of frames and I completely removed the playback code.

The other error I guess was related to performance issues or something as the error was resolved by replacing the line

output_data.append({'color': color_frame, 'aligned_depth': colorized_aligned_depth, 'depth': colorized_depth})

with this line

output_data.append({
            'color': color_frame.copy(),
            'aligned_depth': colorized_aligned_depth.copy(),
            'depth': colorized_depth.copy()
        })

Thanks for the quick help. I hope that this helps somebody else when he encounters similar problems!

[MartyG-RealSense]

Thanks so much for sharing your solution with the community @brunovollmer :) Great news that you were successful.