GPU hardware encoding: Option hwaccel (use HW accelerated decoding) cannot be applied to output url

[chgebhardt]

Great package. Saved us a lot of trouble trying to dig into PySpin and dealing with camera nodes etc ourselves. :)

We have several Blackfly S BFS-U3-51S5M FLIR cameras that in the far future we want to combine into a behavior setup plus open ephys system. So far we have managed to install the newest campy-master on a win 10 system with AMD Radeon Pro Wx 7100 without major hickups in a conda environment as per your instructions.

Due to the cameras being mono we had to comment out camera.BalanceWhiteAuto.SetValue in function ConfigurePixelFormat() and changed pixelFormatInput from "rgb24" to "gray" in the config.yaml. also StreamBufferCountManual.SetValue() only works with a value of 60 instead of 100 without crashing during acquisition. so far, we can stream from two cameras without GPU support.

We hit a bit of a wall though when we wanted to activate GPU support (using one camera atm). ffmpeg is installed in the environment and working but we get this error message:

`Opened: d:/data/20210910/Camera0\0.mp4 using GPU 0 to compress the stream.
WARNING: QApplication was not created in the main() thread.
Trigger source set to None...
Width set to 1280...
Height set to 1024...
Shutter time set to 10000 us...
Gain set to 1 dB.
Gamma correction disabled.
Buffer count now set to: 60
BufferMode has been set to Oldest First
Opened Camera0: flir Blackfly S BFS-U3-51S5M
Camera0 ready to trigger.
Option hwaccel (use HW accelerated decoding) cannot be applied to output url d:/data/20210910/Camera0\0.mp4 -- you are trying to apply an input option to an output file or vice versa. Move this option before the file it belongs to.
Error parsing options for output file d:/data/20210910/Camera0\0.mp4.
Error opening output files: Invalid argument
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
File "C:\Users\kim\anaconda3\envs\campy\lib\site-packages\imageio_ffmpeg_io.py", line 493, in write_frames
p.stdin.write(bb)
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:\Users\kim\anaconda3\envs\campy\lib\multiprocessing\pool.py", line 121, in worker
result = (True, func(*args, **kwds))
File "C:\Users\kim\anaconda3\envs\campy\lib\multiprocessing\pool.py", line 44, in mapstar
return list(map(*args))
File "c:\users\kim\campy-master\campy\campy.py", line 71, in AcquireOneCamera
writer.WriteFrames(cam_params, writeQueue, stopReadQueue, stopWriteQueue)
File "c:\users\kim\campy-master\campy\writer.py", line 153, in WriteFrames
writer.send(writeQueue.popleft())
File "C:\Users\kim\anaconda3\envs\campy\lib\site-packages\imageio_ffmpeg_io.py", line 500, in write_frames
raise IOError(msg)
OSError: [Errno 32] Broken pipe

FFMPEG COMMAND:
C:\Users\kim\anaconda3\envs\campy\Library\bin\ffmpeg.exe -y -f rawvideo -vcodec rawvideo -s 1280x1024 -pix_fmt gray -r 40.00 -an -i - -an -vcodec hevc_amf -pix_fmt yuv420p -qscale:v 16 -v warning -r:v 40 -usage lowlatency -rc cqp -qp_i 21 -qp_p 21 -qp_b 21 -bf:v 0 -hwaccel auto -hwaccel_device 0 d:/data/20210910/Camera0\0.mp4

FFMPEG STDERR OUTPUT:

"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "C:\Users\kim\anaconda3\envs\campy\Scripts\campy-acquire-script.py", line 33, in
sys.exit(load_entry_point('campy', 'console_scripts', 'campy-acquire')())
File "c:\users\kim\campy-master\campy\campy.py", line 78, in Main
p.map_async(AcquireOneCamera,range(params["numCams"])).get()
File "C:\Users\kim\anaconda3\envs\campy\lib\multiprocessing\pool.py", line 657, in get
raise self._value
OSError: [Errno 32] Broken pipe

FFMPEG COMMAND:
C:\Users\kim\anaconda3\envs\campy\Library\bin\ffmpeg.exe -y -f rawvideo -vcodec rawvideo -s 1280x1024 -pix_fmt gray -r 40.00 -an -i - -an -vcodec hevc_amf -pix_fmt yuv420p -qscale:v 16 -v warning -r:v 40 -usage lowlatency -rc cqp -qp_i 21 -qp_p 21 -qp_b 21 -bf:v 0 -hwaccel auto -hwaccel_device 0 d:/data/20210910/Camera0\0.mp4

FFMPEG STDERR OUTPUT:`

somehow hwaccel ends up as an output option in the ffmpeg call but is expected as input option? on a hunch we commented -hwaccel and hwaccel_device out in writer.py in lines 85 and 86. campy-acquire is running through now without error message. we are just not sure that it actually uses the GPU. also the framerate of the acquisition seems to be independent of what we specify in the config.yaml and is always 98fps (as confirmed by the timestamps). the correct number of frames (=frameRate*recTimeSec) is recorded though.

sorry if the solution is obvious but any pointer on how to get GPU encoding to run would be appreciated.
thank you so much,,
chris

[ksseverson57]

Good job hunting down the hard-coded bug in in the writer. To see if the GPU is being utilized, you can look at GPU0 tab's video encoding usage in Windows task manager. I haven't tested with AMD GPU for a while, so it may be time to refresh the AMD GPU options in campy.

The fps issue could be related to triggering. Seems like the camera is trying to run at the max frame rate given your 10 ms exposure time. Not sure why FLIR isn't setting the proper frame rate, but I can look into that with my FLIR testing camera. In the meantime, perhaps you can try hardware triggering to hard-cap the frame rate. See the trigger modules I wrote for Teensy and Arduino microcontrollers.

P.S. I apologize for the slow reply to your issue.

[chgebhardt]

no worries. we are all busy. :)

so here is what I learned in the mean time:

1. The current campy-master doesnt contain any functionality to change the framerate on FLIR cameras. I guess you could do this with hardware triggers (as you suggest) but I am not using those yet. So I added my own ConfigureFrameRate function that is being called within ConfigureCustomImageSettings() in flir.py:

def ConfigureFrameRate(camera, cam_params):
    """Test ConfigureFrameRate function."""
    try:        
        nodemap = camera.GetNodeMap()
        # Enable change of acquisition framerate
        nodeAcquisitionFramerateEnable = PySpin.CBooleanPtr(nodemap.GetNode("AcquisitionFrameRateEnable"))
        if (not PySpin.IsAvailable(nodeAcquisitionFramerateEnable)) or (not PySpin.IsWritable(nodeAcquisitionFramerateEnable)): 
             print('Unable to retrieve AcquisitionFrameRateEnable. Aborting...')
             return -1
        nodeAcquisitionFramerateEnable.SetValue(True)
        # Set acquisition framerate
        framerate_to_set = cam_params["frameRate"]
        nodeAcquisitionFramerate = PySpin.CFloatPtr(nodemap.GetNode("AcquisitionFrameRate"))
        if not PySpin.IsAvailable(nodeAcquisitionFramerate) and not PySpin.IsWritable(nodeAcquisitionFramerate):
           print('Unable to retrieve AcquisitionFrameRate. Aborting...')
           return -1
       
        nodeAcquisitionFramerate.SetValue(framerate_to_set)
        print('Frame rate set to {}fps...'.format(framerate_to_set))
        
    except Exception as e:
        logging.error("Caught exception at cameras/flir.py ConfigureFrameRate: {}".format(e))
    
    return cam_params

2. I ran into error messages with campy when repeatedly changing parameters without unplugging the cameras each time (err msg saying that nodes are not readable or writable). I guess that wont be an issue later on when I have all parameters optimized. in any case I added in OpenCamera() a call to reset default parameters each time the camera is initiated:

# load default camera configuration
    camera.UserSetSelector.SetValue(PySpin.UserSetSelector_Default)
    camera.UserSetLoad()

then the camera parameters are changed as per the users specs. maybe you dont want to do that each time but refreshes the camera such that changes to the nodes can be written.

3. the gpu seems to work as per windows manager. thanks for that suggestion. :) i can upload the code for that if you want as well. i added separate input and output arguments for the ffmpeg pipe to writer.py to retain functionality also for the other GPU types.

alright, hardware triggering is next. i ll report back if that works on my flir/amd combo.

cheers
chris

[histun]

Hi all. This thread has been very helpful for setting up my blackflyl s mono cameras.

1. I was having an issue with displaying streamed images. In case it's useful for someone, I fixed it by commenting out flir.py line 137-141 and modifying display.py line 22
   imageWindow = ax.imshow(np.zeros((1,1,3), dtype='uint8'), interpolation='none',cmap='gray', vmin=0, vmax=255)

2. I have two Nvidia GPUs in my computer. Using either card works for compressing upto 3 videos, but I couldn't get 5 videos (or 4). I was stuck for a while and found out that consumer nvidia gpu cards have a limit on the number of videos being encoded at the same time. To relieve this restriction, you need to do some patch work. Follow the instructions under this link. With this, 5 videos work fine with one GPU.
   https://github.com/keylase/nvidia-patch

[dankunk]

Hi all. This thread has been very helpful for setting up my blackflyl s mono cameras.

1. I was having an issue with displaying streamed images. In case it's useful for someone, I fixed it by commenting out flir.py line 137-141 and modifying display.py line 22
   imageWindow = ax.imshow(np.zeros((1,1,3), dtype='uint8'), interpolation='none',cmap='gray', vmin=0, vmax=255)
2. I have two Nvidia GPUs in my computer. Using either card works for compressing upto 3 videos, but I couldn't get 5 videos (or 4). I was stuck for a while and found out that consumer nvidia gpu cards have a limit on the number of videos being encoded at the same time. To relieve this restriction, you need to do some patch work. Follow the instructions under this link. With this, 5 videos work fine with one GPU.
   https://github.com/keylase/nvidia-patch

@histun It looks like the line numbers have changed a bit in the flir.py file. Would you be able to provide the specific line that needed commenting out in the flir.py file to fix plotting the video? I am currently troubleshooting my FLIR monochrome cam setup as well.

[dankunk]

Hi all. This thread has been very helpful for setting up my blackflyl s mono cameras.

1. I was having an issue with displaying streamed images. In case it's useful for someone, I fixed it by commenting out flir.py line 137-141 and modifying display.py line 22
   imageWindow = ax.imshow(np.zeros((1,1,3), dtype='uint8'), interpolation='none',cmap='gray', vmin=0, vmax=255)
2. I have two Nvidia GPUs in my computer. Using either card works for compressing upto 3 videos, but I couldn't get 5 videos (or 4). I was stuck for a while and found out that consumer nvidia gpu cards have a limit on the number of videos being encoded at the same time. To relieve this restriction, you need to do some patch work. Follow the instructions under this link. With this, 5 videos work fine with one GPU.
   https://github.com/keylase/nvidia-patch

@histun It looks like the line numbers have changed a bit in the flir.py file. Would you be able to provide the specific line that needed commenting out in the flir.py file to fix plotting the video? I am currently troubleshooting my FLIR monochrome cam setup as well.

Never mind! I found it! For those who are experiencing this issue; the correct lines to comment out in the flir.py file to make the video window show up is below, simply replace with this function with the correct lines commented out (also be sure that you follow @histun and @chgebhardt 's instructions regarding editing the flir.py and display.py files.

def DisplayImage(cam_params, dispQueue, grabResult):
	try:
		#if cam_params["pixelFormatInput"] == "bayer_rggb8" \
		#or cam_params["pixelFormatInput"] == "bayer_bggr8" \
		#or cam_params["pixelFormatInput"] == "gray":
			# Convert to RGB
			#grabResult = grabResult.Convert(PySpin.PixelFormat_RGB8, PySpin.HQ_LINEAR)

		# Convert to Numpy array
		img = GetImageArray(grabResult)

		# Downsample image
		img = img[::cam_params["displayDownsample"], ::cam_params["displayDownsample"]]

		# Send to display queue
		dispQueue.append(img)

	except Exception as e:
		logging.error('Caught exception at cameras/flir.py DisplayImage: {}.'.format(e))