How to extract video frame as imageData using webcodecs?

[lincolnneu]

Given a url to a mp4 video, can we extract its frames using webcodecs? Is there any real code examples (e.g. jsfiddle) on how to do this?

[tguilbert-google]

If you have a URL to an mp4 and you want frames, you first need to demux in into chunks (using a JS mp4 demuxing library) and then feed the chunk into a VideoDecoder.

This example doesn't show you the demuxing portion, but it showcases a working WC example. It reads from a canvas and encodes the frames into chunks, then decodes the chunks into frames and draws them into another canvas:
https://webcodecs-blogpost-demo.glitch.me/

As far as MP4 demuxers go, I don't have a strong recommendation, but some people have used mp4box.js in demos.

[lincolnneu]

I get "(index):96 Encoding error: Unexpected frame format." in your demo. My chrome is 86.0.4240.75 is it a bug?

[tguilbert-google]

It works for me (also 86.0.4240.75) on Linux. What platform are you using?

[lincolnneu]

Mac 10.15.6

[tguilbert-google]

We have been able to repro it a few times on our end, especially after clearing browsing data and reloading. Refreshing the page seems to fix the issue most of the time though.

We already planned some work that would prevent this issue (automatic frame conversion to I420 for the encoder), but I have opened crbug.com/1137037 in the mean time to see if there is anything to be done.

The issue is probably with the VideoTrackReader portion. For your use case, you should only need the VideoDecoder.

[tguilbert-google]

Actually, you could try:

let video = document.createElement('video');
video.src = // your mp4's URL
await video.play();

let vtr = new VideoTrackReader(video.captureStream().getVideoTracks()[0]);

vtr.start(frame => {
  // Read Frame here.
  frame.destroy();
})

I have no idea what performance or guarantees you might get from this, and I don't know this area well at all. This just a way to get some frames out of a video with little code.

VideoDecoder will offer you better reliability and controls.

EDIT 23/02/2021: VideoTrackReader is now deprecated. Use the MediaStreamProcessor API instead (demo here)

[chcunningham]

Looks like this is resolved. Please re-open if needed.

[tguilbert-google]

Here is a code example for extracting the frames from an mp4:
https://tguilbert-google.github.io/webcodecs/mp4/index.html

[lincolnneu]

This is so cool! I notice at the bottom of the frame it's somehow extended vertically. Is it the same as the raw video?

Looks like the raw video in https://tguilbert-google.github.io/webcodecs/mp4/bbb.mp4 does not have the extended bottom.

[lincolnneu]

Oh, this is because the video height should be 360 but you set it to be 368 lol

[tguilbert-google]

Ah, thanks! That is a bug. I have opened crbug.com/1145244

[Lincolnhuj]

Do I need to turn on
chrome://flags/#enable-experimental-web-platform-features
to use this feature?

[lincolnneu]

Some frames are green ( tested in windows 10 surface pro 7 chrome 86.0.4240.198 ) visiting https://tguilbert-google.github.io/webcodecs/mp4/index.html ( same result both turning on/off chrome://flags/#enable-experimental-web-platform-features ) :
screen recording:
https://drive.google.com/file/d/1JBOZTfpqQbuUkLZAkPNY5VVymZ3yhxT7/view?usp=sharing

[chcunningham]

Thanks for reporting. I think this may be a known issue fixed in Chrome 87. Can you try that version (beta channel)?

[lincolnneu]

Chrome 87 works! But the fix to crbug.com/1145244 seems to haven't been applied to 87 yet.

[chcunningham]

Awesome, thanks for trying it!

But the fix to crbug.com/1145244 seems to haven't been applied to 87 yet.

Sadly that fix didn't make the branch cut for 87, but it is available Canary build now and will push out to Dev in a few days (any build >= 88.0.4316.0).

[lincolnneu]

Tested on mid 2014 mac, V10.13.5, chrome 87.0.4280.67 (Official Build) (x86_64)
2.5 GHz Intel Core i7
NVIDIA GeForce GT 750M 2048 MB
Intel Iris Pro 1536 MB

DOMException: Failed to execute 'createImageBitmap' on 'VideoFrame':
onFrame (index.html:32)

( same result both turning on/off chrome://flags/#enable-experimental-web-platform-features )

[tguilbert-google]

Could you try on Chrome Canary and let us know if it repros there?

[lincolnneu]

Chrome Canary (89.0.4334.0) works properly on this old mac! But the stable version of 89 will be available after 99 days :(

[tguilbert-google]

I'm sorry, but there's not much we can do about that... If it works on Dev/Beta (M88), that will be out sooner!

[fnandoz21]

Actually, you could try:

let video = document.createElement('video');
video.src = // your mp4's URL
await video.play();

let vtr = new VideoTrackReader(video.captureStream().getVideoTracks()[0]);

vtr.start(frame => {
  // Read Frame here.
  frame.destroy();
})

I have no idea what performance or guarantees you might get from this, and I don't know this area well at all. This just a way to get some frames out of a video with little code.

VideoDecoder will offer you better reliability and controls.

Can you provide sample code for using the VideoDecoder after getting the frames from the mp4? Such as something similar to what is given with https://webcodecs-blogpost-demo.glitch.me/ but for the mp4 sample. @tguilbert-google

[tguilbert-google]

Here's a demo that directly uses a VideoDecoder to read an MP4 file (it might take a while to load because the file is somewhat big):
https://tguilbert-google.github.io/webcodecs/mp4/index.html

[tguilbert-google]

Actually, I'm not sure that answers your question @fnandoz21, as I had posted that link in an earlier comment. Was that what you were looking for? The key thing in that demo is that it demuxes the files itself.

[fnandoz21]

@tguilbert-google it doesn't really resolve it, I'm pretty much trying to do what's at https://webcodecs-blogpost-demo.glitch.me/ but with either an mp4 or frames derived from an mp4 instead of the rotating text.

[tguilbert-google]

If you mean to have a video element, grab the frames from it, and display them in a canvas below something like this should work:

This demo uses MediaStreamTrackProcessor (which replaces VideoTrackReader on the latest Canary) to read video frames from a webcam, but you can repurpose it to read from a video element. This, combined with the mp4 demo's display code would look like:

var canvas = document.querySelector("canvas");
var ctx = canvas.getContext('2d');

var video = document.querySelector("video");

var track = video.captureStream().getVideoTracks()[0];

var processor = new MediaStreamTrackProcessor(track);

const frameReader = processor.readable.getReader();

// Change this value from a "Stop button";
var stopped = false;

frameReader.read().then(function processFrame({done, value}) {
    if(done) {
      console.log("Stream is done");
      return;
    }

    if (stopped) {
      frameReader.releaseLock();
      processor.readable.cancel();
      value.close();
      console.log("Stopped");
      return;
    }

    var img = await value.createImageBitmap();
    ctx.drawImage(img, 0, 0, canvas.width, canvas.height);
    img.close();
    value.close();

    frameReader.read().then(processFrame);
})

The MediaStreamTrackProcessor's spec is here, but the important thing to note is that MediaStreamProcessor.readable is a ReadableStream of VideoFrames.

Otherwise:

Can you provide sample code for using the VideoDecoder after getting the frames from the mp4?

Do you mean using the VideoEncoder after getting VideoFrames from the mp4? In the sense that you want to grab the frames from an mp4, encode them, decode them, display them, like in the https://webcodecs-blogpost-demo.glitch.me/ demo?

[fnandoz21]

Thanks for the reply! This code should be helpful. And to answer your question, yes, I want to grab the frames from an mp4, encode them, decode them, and display them.

[fnandoz21]

let video = document.createElement('video');
video.src = // your mp4's URL
await video.play();

let vtr = new VideoTrackReader(video.captureStream().getVideoTracks()[0]);

vtr.start(frame => {
// Read Frame here.
frame.destroy();
})

Also, is this what you mean by the video display code? Because this does not really work when I run it. I just get a blank screen. This also seems quite different from the webcam code.

[tguilbert-google]

By display code, I mean this part of the mp4 demo:

async function onFrame(frame) {
  var img = await frame.createImageBitmap();
  ctx.drawImage(img, 0, 0, canvas.width, canvas.height);
  img.close();
  frame.destroy();

  ++frameCount;
  document.getElementById("frameCount").innerText = frameCount;
}

let decoder = new VideoDecoder({
  output: onFrame,
  error: e => console.error(e),
});

The MediaStreamTrackProcessor webcam demo directly displays the MediaStreamTrack in the video element, and doesn't actually render anything.

[fnandoz21]

Ok that makes sense. Could you provide a way to take those frames and encode them to ImageBitmaps? I’m not sure if you showed that and the example from WebCodecs only shows how to do this with a drawing on a canvas, not with something that started as an MP4. I’m currently doing something based on the example from your website with the bbb.mp4 but it doesn’t appear that the bitmaps get populated with data.

[ayanchax]

@tguilbert-google Hi the code snippet shared by you flawlessly worked for me in both Mac and Windows with Chrome Version 105.0.5195.127 (Official Build) (64-bit). Thank you for this solution. This looks promising and performant. I will test this more with large videos and let you know.

Here is how modified the code to get it working, seems like value.createImageBitmap is outdated, I used createImageBitmap(value) by passing the frame value as a parameter of the same.
Refer docs: https://developer.mozilla.org/en-US/docs/Web/API/createImageBitmap

Here is the implementation I did following your approach
https://github.com/ayanchax/video-frame-reader

I will work more closely on this and update this thread.

However, just letting you know, before I came across this, I have tried extracting media frames reliably using native javascript as well as using opencvjs, but both approaches were relatively slower and had issues which I posted on forums. Also lack of concrete documentation in this area is still a question mark for such implementations.

OpenCV Implementation: https://forum.opencv.org/t/capture-all-frames-of-a-video-in-open-cv-js/10155
Javascript Implementation approach: https://stackoverflow.com/a/52357595/3127178

In both of these implementations there are bottlenecks such as large videos are not processed accurately for frames, however in a holistic view, the native javascript works accurately than the opencvjs implementation, but is slower than opencv js.

[dalecurtis]

If you need pixel data then using VideoFrame::copyTo() is the most efficient way to get it. You may also try prefer-software when configuring the codec to ensure pixel data is accessible without GPU readback -- depending on the stream complexity readback may be cheaper though.

[ayanchax]

Here is my implementation using the webcodecs API that perfectly works
https://github.com/ayanchax/video-frame-reader/blob/master/index.html

[dalecurtis]

You can drawImage() the returned VideoFrame object directly without going through createImageBitmap(). You can also use OffscreenCanvas instead of if you want. toBlob() will compress to png, but if that's what you want instead of the raw data that seems fine.

[ayanchax]

@dalecurtis I will try removing the createImageBitmap and just rely on the canvas drawImage and update this thread again. Also, I might just end up using the offScreenCanvas instead of toBlob in the long run as I enhance this implementation further and delegate this entire process to a web worker for better performance and minimizing the risk of UI blocking.

I will update the implementation further and keep you posted. This is a kind of POC I just completed in an hour being inspired from @tguilbert-google solution snippet above. I will enhance and shorten the code further and check in. And l also request you to review the implementation by cloning the project in your local, it would not take you more than 5 minutes. Thank you so much for your reply and recommendation. Means a lot🚀😃

[RealAlphabet]

Hello, I hovered over but does window.createImageBitmap(frame) instead of frame.createImageBitmap() works?
According to the spec a VideoFrame is a CanvasImageSource.
whatwg/html#6589

[dalecurtis]

Yes you can use window.createImageBitmap(frame) if you need an ImageBitmap, however if you're just trying to draw to WebGL or canvas you generally want to use drawImage(frame, ...), texImage2d(frame, ...), etc. The createImageBitmap path will end up making a copy of the frame.

[RealAlphabet]

Yes you can use window.createImageBitmap(frame) if you need an ImageBitmap, however if you're just trying to draw to WebGL or canvas you generally want to use drawImage(frame, ...), texImage2d(frame, ...), etc. The createImageBitmap path will end up making a copy of the frame.

Thanks, is it possible to retrieve the VideoFrame's internal pixel array without having to duplicate it (read-only)?
PS: For example, to do machine learning operations on it. Detect similar frames etc.

[dalecurtis]

No, unfortunately JavaScript doesn't offer read-only arrays, so it wasn't possible to implement. The only way to get the data is via VideoFrame.copyTo() or read-back drawing to canvas if copyTo() isn't supported (in Chrome we're working on increasing the cases where it works).

[RealAlphabet]

Thanks for the clarifications!

[jenkins-pipeline390]

I faced that obstacle and discovered a viable solution https://hypertexter.space/questions/having-trouble-capturing-a-photo-from-webcam-using-html5-and-getusermedia-in-google-chrome-upon-initial-page-load