Add Gallery VideoStream example

[MarcSkovMadsen]

Closes #4046

Server

The video shows the performance when running on the server. The key to getting this performance is setting the VideoStream format to jpeg. png is slow.

video-stream-server.mp4

You can get more than 10fps. This includes a round trip from DK to Amsterdam and back. That is pretty amazing. And it means you can use deep learning frameworks like PyTorch that are not available in Webassembly.

Webassembly

The video shows the performance when converted to webassembly. Its fucking fast! (sorry)

video-stream-webassembly2.mp4

Try it

Online

https://awesome-panel.github.io/examples/ (Works in Chrome, not firefox)

Panel Serve

pip install panel  numpy scikit-image

panel serve examples/gallery/streaming/video/app.py --autoreload

Panel Convert to webassembly

panel convert examples/gallery/streaming/video/app.py --to pyodide-worker --out pyodide --requirements examples/gallery/streaming/video/requirements.txt --watch

python -m http.server

open http://localhost:8000/pyodide/app.html.

Issues identified and solved

• The default .png format of the stream is slow to work with. .jpeg is much better.
• The server can get "behind" the videostream. This should be avoided.
• When using the face recognition algorithm the colors get strange. They are not like that in the Yuichiro example.

Potential future improvements

Someday someone might be willing to invest in

• [] Transfering reduced size video stream from client to server. Would require adjusting the videostream.ts file.
• [] There might be a more efficient form of communication than base64 data urls from client to server?
• [] There might be a more efficient form of communication than a jpeg file buffer from server to client embedded in a JPEG pane. Maybe a HoloViews Image is faster?
• [] Enabling interaction with the video output. Like clicking or drawing.
• Someone could build a panel-videostream package that provides the reusable code from this example such that for users they only have to implement a CustomTransform in 2-10 lines of code.
  • Gradio plans to provide this in Streaming video update gradio-app/gradio#1637
  • It will probable work similarly to https://gradio.app/real_time_speech_recognition/
• Getting it to work in Firefox. When loading scikit-image it throws an out of memory error.

Credits

Yuichiro for inspiration

[philippjfr]

The video shows the performance when converted to webassembly.

Looking good 😉 Definitely want to merge this one, can you remove the built app.html and app.js? They will get built automatically.

We do have to work out how to link the pyodide versions of all the gallery examples from the docs though.

[MarcSkovMadsen]

Yes. I will remove them. But also plan to convert to notebook. But need to solve some of the issues first

[MarcSkovMadsen]

The app is now working really great. Next step is to turn it into a Notebook.

[MarcSkovMadsen]

First version of notebook created.

[codecov]

Codecov Report

Merging #4047 (0dc4d4d) into main (9cbbd54) will increase coverage by 0.01%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main    #4047      +/-   ##
==========================================
+ Coverage   82.71%   82.72%   +0.01%     
==========================================
  Files         237      237              
  Lines       34276    34276              
==========================================
+ Hits        28350    28354       +4     
+ Misses       5926     5922       -4     

Flag	Coverage Δ
ui-tests	37.97% <ø> (+0.01%)	⬆️
unitexamples-tests	73.88% <ø> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
panel/util/__init__.py	86.66% <0.00%> (-0.45%)	⬇️
panel/io/server.py	75.70% <0.00%> (+0.56%)	⬆️
panel/tests/util.py	91.26% <0.00%> (+1.94%)	⬆️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[MarcSkovMadsen]

The notebook is now ready for review.

---

The assets below can be used for promotion if relevant.

streaming-videostream-speedup.mp4

[MarcSkovMadsen]

I can see that the tests fail because it tries to run the VideoStream notebook with the kernel conda-root-py. It seems not to exist. Where does this kernel name come from and what should I do to fix @philippjfr ?

[philippjfr]

I can see that the tests fail because it tries to run the VideoStream notebook with the kernel conda-root-py. It seems not to exist. Where does this kernel name come from and what should I do to fix @philippjfr ?

That's because you didn't clear out the metadata in the notebook. Will do that shortly.

[philippjfr]

pre-commit.ci autofix

[philippjfr]

☝️ If you decide that you can't be bothered to install pre-commit yourself you can apply it with a comment like above.

[philippjfr]

Amazing example and great work as always @MarcSkovMadsen, thanks!