set_real_time = False still results in hanging during playback of bag file in python

[avasbr]

---

Required Info
Camera Model	D435i
Firmware Version	05.12.03.00
Operating System & Version	Debian
Kernel Version (Linux Only)	(e.g. 4.14.13)
Platform	PC
SDK Version	2.34
Language	python
Segment	{Robot/Smartphone/VR/AR/others }

Issue Description

I'm trying to loop through a number of bag files and dump all color frames to disk, among other things. However, i'm running into an issue where the program hangs, and it isn't consistent. Sometimes i'll get through 20, sometimes i'll get through 5, and it doesn't seem to hang on one particular bag file. See sample below:

# loop through a list of bag files
for b in bag_paths:
    
    pipe = rs.pipeline()
    cfg  = rs.config()
    cfg.enable_device_from_file(b, repeat_playback=False)
    cfg.enable_stream(rs.stream.color, 1280, 720, rs.format.rgb8, 30)
    
    profile = pipe.start(cfg)
    playback = profile.get_device().as_playback()
    playback.set_real_time(False)
    last_pos = playback.get_position()

    while True:
        _, frames = pipe.try_wait_for_frames(timeout_ms=500)
        curr_pos = playback.get_position()
        
        color = frames.get_color_frame()
        if not color_frame:
            continue
        np_color_frame = np.array(color.get_data(), copy=True)
        do_something(np_color_frame) # run detection and get bboxes
        write_to_disk(save_path, np_color_frame) # uses opencv
        
        if curr_pos < last_pos:
            break
        last_pos = curr_pos

I've written this loop in many different ways, using wait_for_frames, using try/except on RuntimeError when I run out of frames, calling pipe.stop() at the end, etc and I'm aware of issues in the past around the behavior of set_real_time=False & deadlocks (#3126, #3850, #4766) + the introduced fix (#4758). However, this problem persists for me in 2.34.

A couple observations/comments:

• if I get the color frame, convert to numpy and don't do anything else, i'm able to loop through all the bag files just fine. If i attempt to write to disk, or do anything else, like run a detection model to get bounding boxes of various objects, etc, before moving on, i'll get intermittent hanging. Finally, I tried also running time.sleep() just to see if it was time dependent - set it to about 30ms, and it starts to randomly hang again. Does this point to a possible race condition?

• setting real_time to True, I can loop through everything, but this is undesirable due to many dropped frames.

Any ideas as to why I would still get this kind of blocking behavior given that there seem to be fixes introduced to address this? should I be using a different version of the sdk? what else I can try?

[MartyG-RealSense]

Your code up until the point where the while statement begins looks consistent with other scripts that are reading through bag files, so the problems may be occurring due to something within the while section.

The most efficient way to save data in the RealSense SDK is to record bag files, so other methods of saving can have greater inefficiency to various degrees, depending on what type of format is being saved to.

Given the historical problems with using set_real_time(False), it may be better to set real_time to True (which you have found to work) and deal with the dropped frames problem by introducing latency into the frame processing.

One way to do this might be to configure the frame queue capacity to determine how the maximum number of frames that can be in circulation.

https://github.com/IntelRealSense/librealsense/wiki/Frame-Buffering-Management-in-RealSense-SDK-2.0#latency-vs-performance

This can be a tricky procedure to configure though, as explained in point 2 of the link below by @dorodnic:

#5041 (comment)

There is a documentation page at the link below for configuring the frame queue in Pyrealsense2:

https://intelrealsense.github.io/librealsense/python_docs/_generated/pyrealsense2.frame_queue.html

[avasbr]

hi @MartyG-RealSense, thanks for your response, though I'm not sure I fully understand if this applies to my situation.

The most efficient way to save data in the RealSense SDK is to record bag files, so other methods of saving can have greater inefficiency to various degrees, depending on what type of format is being saved to.

^This is what i'm doing. I've saved off a number of bag files and am now processing them offline, looping through them to extract the rgb/depth/ir frames. It's in this procedure of extraction that i'm running into this hanging issue.

Please correct me if i'm mistaken or am fundamentally misunderstanding something, but configuring frame queue capacity and balancing latency vs performance appear to be related to what I would need to do during live capture, and not offline processing of bag files.

[MartyG-RealSense]

Yes, I know you are already using bag files. Most methods of data writing are unlikely to achieve the same level of efficiency as bag creation and that slowdown when writing the color frames to disk in real-time is hard to avoid.

I would also be of the opinion that if using real_time = false gives many problems but setting it to true works fine aside from frame drop, the best path would be to use 'true' and put measures in place to minimize frame drop, such as prioritizing queue latency over performance (as favoring performance over latency increases the risk of frame drops occurring).

Also bear in mind that if you have missing frames, it may not be because the extract color frames to disk system is at fault but because the frames were absent from the original bag file due to being dropped during bag recording.

[avasbr]

Alright, I'll see what I can do by setting it to true and attempting to minimize frame drop. Thanks for the feedback & clarification.

Just as a closing thought, if I take a step back, I find this procedure quite unnatural. Rather than having a repeatable way of extracting all the frames I know to exist inside a bag file, i'm attempting to play it back in real time and deal with potential, inconsistent frame drops in other ways. It doesn't seem to address the core problem head on, but perhaps I can work w/ this solution in the meantime.

Thanks again.