native video decoding errors

[totaam]

Sometime since e0e5de3 (2016!)

I have added an example to enable fastdecode: ca3d1db
And the xpra server will now honour it, both with h264+mp4 and mpeg4+mp4:
Xpra-org/xpra@b10da3f

Note: to change which actual codec the server will choose for the html5 client in "native" video mode, using the "score-delta" for the codec chosen: 8cf87c2 (h264+mp4 is now ahead of mpeg4+mp4).

---

This command generates an h264 stream that browsers can play:

ffmpeg -f x11grab -i :99 -c:v libx264 -pix_fmt yuv420p \
    -profile:v baseline -tune fastdecode -preset ultrafast \
    -f mp4 -movflags frag_keyframe+default_base_moof -frag_duration 1 \
    pipe:1

Link: Fragmented MP4 - problem playing in browser

Comparing it with our enc_ffmpeg encoder:

• we use YUV420P (or NV12 with libva - which translates to the same subsampling)
• baseline profile since 8cf87c2
• fastdecode since b10da3f3291d492839843e12a08359c40cd40f8a
• ultrafast preset
• movflags: empty_moov+frag_keyframe+default_base_moof+faststart - IIRC, faststart is important too

---

There are a lot more fastdecode tweaks in the plain enc_x264 encoder:
https://github.com/Xpra-org/xpra/blob/b10da3f3291d492839843e12a08359c40cd40f8a/xpra/codecs/enc_x264/encoder.pyx#L594-L597
https://github.com/Xpra-org/xpra/blob/b10da3f3291d492839843e12a08359c40cd40f8a/xpra/codecs/enc_x264/encoder.pyx#L634-L638
removed B frames, sliced threads, b_weighted_bipred, i_weighted_pred.
Perhaps these tweaks would help with the enc_ffmpeg encoder?

---

Here is the cryptic browser error:

Error: Failed to execute 'appendBuffer' on 'SourceBuffer': This SourceBuffer has been removed from the parent media source.
    at XpraWindow._push_video_buffers (http://localhost:12000/js/Window.js:1223:7)
    at XpraWindow.paint [as do_paint] (http://localhost:12000/js/Window.js:1592:10)
    at XpraWindow.paint (http://localhost:12000/js/Window.js:1321:33)
    at XpraClient.do_process_draw (http://localhost:12000/js/Client.js:2990:7)
    at Worker.<anonymous> (http://localhost:12000/js/Client.js:433:7)
error painting mpeg4+mp4 InvalidStateError: Failed to execute 'appendBuffer' on 'SourceBuffer': This SourceBuffer has been removed from the parent media source.

[TijZwa]

I believe we push the frames to the MSE at the wrong time. We listen for waiting, but it seems to be a better practice to push again when updateend is firing.
I created a working proof of concept. Video is still a bit delayed but at least the error is gone.
Will rewrite and check in the code.

[TijZwa]

@totaam Check out https://github.com/TijZwa/xpra-html5/tree/%23114
I started to implement muxed/boxed video.

[TijZwa]

Did some work on the code. It works, but there are a few pitfalls.
Ie: we don't really know when a frame that is fed to the MSE is really rendered on screen. So if I visit a website like www.vpo.nl which has a video running and move the mouse to the top-menu, I receive a package with x and y set to zero.
If we draw at position 0,0 while the video element is still playing a frame of the video on the webpage, it is rendered a the wrong location.
Needs some more work.

[totaam]

Right. OTOH, there are things we could do to mitigate this - but none are going to be perfect.
From easy to extremely hard:

• associate each input frame with the location it should be painted at (and hope that the page is not scrolled up or down in the meantime..) then retrieve the paint location when it comes out (it should always be 1 in for 1 out)
• send the whole window as one video stream (and minimize decoder latency as much as possible), then there won't be paint synchonization issues... just extra latency for things that aren't video. (and switching to / from this mode would be tricky)
• composite the layers ourselves: keep track of what gets painted onto the video area after we submit a video frame to the decoder, then we can repaint whatever should be on top after painting the video frame that comes out - a bit hackish, and this would require telling the server to disable the exclude-region mechanism to prevent the overlays from being sent as part of the video (very tricky to identify what is and isn't video in this case! perhaps impossible)
• intercept the browser's own compositing (either modifying the browser or using a wayland based server and hope that the video will be using a separate surface) then we can do the compositing client side, effectively turning the video surface into an undecorated virtual window surface

This isn't a problem for broadway software decoding because that one is completely synchronous: we submit a frame and get the rgb data back.

[TijZwa]

@totaam The second option (send the whole window as one video stream) could be great for testing. Is there an Xpra switch I can use to achieve this?

[totaam]

@totaam The second option (send the whole window as one video stream) could be great for testing. Is there an Xpra switch I can use to achieve this?

Yes: encoding.full_frames_only=true : https://github.com/Xpra-org/xpra/blob/e645459d38599d6f2b13aec008b0b09c2f5db584/xpra/server/window/window_source.py#L705
Should work, but this has not been tested in years. YMMV.

[TijZwa]

Sending full frames works but makes text fuzzy (as expected). Other options might be a better choice.

[totaam]

Sending full frames works but makes text fuzzy (as expected)

Yes. And this limits our options: bumping up the quality (some codecs have a lossless mode enabled with quality=100) can be enough to make the text clear enough to read but then the bandwidth use and CPU consumption go way up. (and then we can't use scaling which also has a significant cost)

[TijZwa]

It seems that our best bet might be the MediaStreamTrackProcessor (https://developer.mozilla.org/en-US/docs/Web/API/MediaStreamTrackProcessor)
This interface can be used to capture video playback frame-by-frame.
PofA:

• Feed the MSE video chunks.
• Set the MSE as source of the video element
• Get the stream: this.video.captureStream().getVideoTracks()[0]
• Use the MediaStreamTrackProcessor to process the stream.
• Use TrackProcessor.readable.getReader() and TrackReader.read()
• Get a bitmap of the processed frame with the right widht & heigt set
• Get x & y coordinate from the package
• Paint
• Process next package.

Got the first 6 steps working, will commit the code after the last three steps.
This will not work on older browser, but they can fall back to broadway or non-video.

[totaam]

Wow. Awesome!

[TijZwa]

While working on the MSE solution I found something even more awesome. Chrome 94+ supports the new VideoDecoder in secure contexts (HTTPS).
So if we load Xpra-html5 over SSL in Chrome 94+ we have a native videodecoder (!!) which support raw H.264 (!!!!).

I've tested this and it works great!! This seems to be a drop-in replacement for broadway and we dont need to mux the video.

Refers:

• https://github.com/w3c/webcodecs/blob/main/explainer.md (section Example of video rendering to Canvas for extremely low-latency streaming (e.g. cloud gaming))
• https://developer.mozilla.org/en-US/docs/Web/API/VideoDecoder

To do:

• Implement this native decoder.
• Discuss if we still need support for muxed video for non-ssl / non-chrome users or wait till Mozzila/Apple adopts this tech.

[totaam]

Discuss if we still need support for muxed video for non-ssl / non-chrome users or wait till Mozzila/Apple adopts this tech.

I would guess not. The muxed video is just too hard to get right.
The fallback to broadway ought to be enough. Maybe even only enable video by default if MediaStreamTrackProcessor is available and use jpeg otherwise.
If it wasn't for libva support, I would be tempted to drop the enc_ffmpeg codec completely!

[TijZwa]

Some more information about the subject:

Draft: https://www.w3.org/TR/webcodecs/
Best practice: https://www.w3.org/TR/webcodecs/#best-practices-developers (demo: https://devnook.github.io/OffscreenCanvasDemo/keep-ui-responsive.html)
Google I/O talk: https://www.youtube.com/watch?v=wkDd-x0EkFU

It seems to be a good idea to use the OffscreenCanvas API inside a web-worker and feed it the VideoFrame.
But this also goes for RGB/Jpeg/WebP frames.

[TijZwa]

Implemented some more code:

• OffscreenDecodeWorker
• VideoDecoder
• ImageDecoder

This works great, but sometimes I see server logs like this:
2021-11-21 18:47:57,779 Warning: delayed region timeout 2021-11-21 18:47:57,779 region is 15 seconds old, will retry - bad connection? 2021-11-21 18:47:57,779 5 late responses: 2021-11-21 18:47:57,779 1488 h264 : 32s 2021-11-21 18:47:57,779 1489 h264 : 32s 2021-11-21 18:47:57,779 1496 h264 : 31s 2021-11-21 18:47:57,779 1497 h264 : 31s 2021-11-21 18:47:57,779 1501 h264 : 30s
Which is strange because I send the ack (damage) and can confirm this on the console.
Does the order of damage matters? Do I need to send damage for 1488 before 1489? Or can 1489 arrive earlier?

[totaam]

Which is strange because I send the ack (damage) and can confirm this on the console.

This one (15) must have gone missing somehow.
Perhaps the video decoder doesn't always produce frames?
Use --debug xpra.server.source.source_stats,compress to see what screen updates get sent and when the echo comes back. Perhaps there's a pattern there. B frames or delayed frames?

Does the order of damage matters?

No.

Do I need to send damage for 1488 before 1489? Or can 1489 arrive earlier?

They can arrive in any order.
The server sometimes specifies the order that should be used to group screen updates together using the flush option: any non-zero flush value indicates that more screen updates are coming and that the actual front buffer should not be flipped yet.
But the client is free to do whatever it wants with this information, as long as it eventually sends the ack.
Late acks are a sign that something isn't quite right, so this will slow down the rate of screen updates.
And at some point, the server may be forced to continue without the acks, this is what logs:

late responses: (..) 1488 h264 : 32s (..) 1489 h264 : 32s (..) 1496 h264 : 31s (..) 1497 h264 : 31s (..) 1501 h264 : 30s

You already have gaps, so 1490 to 1495 and 1498 to 1500 have been acked.

[totaam]

You already have gaps, so 1490 to 1495 and 1498 to 1500 have been acked.

Perhaps I was wrong about this.
A recent fix changed the way we assign sequence numbers, it was previously possible for them to increase by 2, skipping one value.

[TijZwa]

We did some internal testing and the speed of the native Video- and AudioDecoders is very good. Combined with offscreen painting ther performance is excellent.
Needs some work; I tuned ACK_DELAY and ACK_JITTER with env. variables. This needs to be reported reliable from the HTML5 client. Maybe using SetInterval and check the queue for old packages may be the way to go.

[totaam]

Should we close this ticket in favour of #122 and the offscreen video decoder?

[TijZwa]

Yes, let's do that.