Error when using pyrealsense2 with multithreading

[NatanelBirarov]

Hello, I'm trying to write a simple program in Python, where the main thread will read depth frames from the camera and put them in a queue,
and another thread that will run inference on them with a YoloV5 TensorRT model. The program runs on a Jetson Nano.

For some reason, after reading about 15 frames the program crashes with the following error:

Traceback (most recent call last):
  File "test2.py", line 59, in <module>
    img = np.asanyarray(c.colorize(DEPTH).get_data())
RuntimeError: Error occured during execution of the processing block! See the log for more info

Here is the full code:

from queue import Queue
import numpy as np 
from ObjectDetection.objectDetectionV2 import ODModel, letterbox
import torch
import time
from threading import Thread
import cv2
from Camera.Realsense import RealSense  # custom class for reading from Realsense camera

def detect(queue):
    while True:
        if not queue.empty():
            img0 = queue.get()
            if img0 is None: break
            img = letterbox(img0, 416, stride=32, auto=False)[0]
            img = img.transpose((2, 0, 1))[::-1]  # HWC to CHW, BGR to RGB
            img = np.ascontiguousarray(img)
            
            print("loading image...")
            img = torch.tensor(img)
            print("loaded image")
            img = img.float()  # uint8 to fp16/32
            img /= 255  # 0 - 255 to 0.0 - 1.0
            result = model(img)
            print(result)

if __name__ == '__main__':
    queue = Queue()
    print("loading model")
    model = ODModel()
    print("model loaded")
    rs = RealSense()
    p = Thread(target=detect, args=(queue,)) 
    c = rs.colorizer
    p.start()
    for i in range(100):
         RGB, DEPTH = rs.getData() 
         img = np.asanyarray(c.colorize(DEPTH).get_data())
         queue.put(img)
    queue.put(None)
    p.join()
    model.destroy()
    print("Exiting Main Thread")

I tried commenting everything out and checking line by line, and I think the error is because of the c.colorizer taking too much time? When I deleted it the error went away (but of course the inference failed). If I don't remove it then the error appears after the line img = np.ascontiguousarray(img). But then why is the error not on this line?
If I limit the size of the queue to at most 14, the problem stops, but then the queue is blocking so everything slows down. Also the error mentions a log, but I have no idea where it is.

Can anyone help me understand what I can do? Thank you in advance.

[MartyG-RealSense]

Hi @NatanelBirarov The case at #946 may be a helpful reference for your problem of the program crashing after 15 frames.

A workaround for the 15 frame issue can be to store frames in memory using the RealSense SDK's Keep() instruction, as described at #3164 (comment) and #3121 (comment)

[NatanelBirarov]

@MartyG-RealSense Thank you for the answer, I actually had a keep() call already but still it did not help. What DID help eventually (I hope) is adding np.copy() around the array that I store inside the queue, like this:

img = np.copy(np.asanyarray(c.colorize(DEPTH).get_data()))
queue.put(img)

I don't fully understand why, I guess it allows the program to get rid of the reference to the depth frame? not sure. I also don't know how it affects the runtime, but from the tests I made it didn't seem to affect it by much.
Anyway, thank you for the help!

[MartyG-RealSense]

You are very welcome, @NatanelBirarov - thanks very much for the update and for sharing your solution with the RealSense community!

[MartyG-RealSense]

Hi @NatanelBirarov Do you require further assistance with this case, please? Thanks!

[MartyG-RealSense]

Case closed due to no further comments received.