VTT Subtitles broken on iOS #3468

james-cohen · 2024-01-10T16:48:01Z

Bug

I am trying to show captions on my videos using the textTracks and selectedTextTrack properties. I am using VTT as per guidance in the documentation as it is the only type supported by iOS and Android.

On Android, I have no issues and it works perfectly. On iOS, there are several serious issues if textTracks and selectedTextTrack are provided - they resolve if removed:

Videos freeze and have to be reloaded several times
When they do play, they don't have any sound
Only the first entry of the VTT file shows. When its timeframe has ended, sometimes it disappears, sometimes it stays permanently.

The VTT file is being served from a signed S3 URL. By the fact that it works on Android and occasionally shows on iOS, I am assuming that the URL itself is not the problem.

Platform

iOS

Environment info

Library version: 6.0.0-beta.3
Device: iPhone 11, iOS 17.2

Steps To Reproduce

Watch streamed MP4 video with streamed VTT subtitles

Expected behaviour

Video plays with sound
Subtitles update at the correct timestamps

Reproducible sample code

        <Video
          style={videoStyle}
          ref={ref}
          source={{ uri: src }}
          resizeMode={resizeMode}
          posterResizeMode={posterResizeMode}
          automaticallyWaitsToMinimizeStalling
          disableFocus
          useTextureView={false}
          playWhenInactive
          paused={isPaused || isSeeking}
          ignoreSilentSwitch={IgnoreSilentSwitchType.IGNORE}
          controls={!!showControls}
          repeat={!!shouldRepeat}
          muted={!!isMuted}
          poster={loadingImage}
          onReadyForDisplay={onReady}
          onLoad={handleLoad}
          onBuffer={(ev) => setBuffering(ev.isBuffering)}
          onError={(err) => handleError(err.error)}
          onEnd={handleFinish}
          progressUpdateInterval={50}
          onProgress={handleProgress}
          selectedTextTrack={
            showCaptions && transcriptUri
              ? { type: SelectedTrackType.INDEX, value: 0 }
              : { type: SelectedTrackType.DISABLED }
          }
          textTracks={
            transcriptUri
              ? [
                  {
                    title: 'Closed captions',
                    uri: transcriptUri,
                    language: 'en',
                    type: TextTracksType.VTT,
                  },
                ]
              : undefined
          }
          subtitleStyle={subtitleStyle}
          bufferConfig={{
            minBufferMs: 2500,
            maxBufferMs: 60000,
            bufferForPlaybackMs: 2500,
            bufferForPlaybackAfterRebufferMs: 2500,
            maxHeapAllocationPercent: 0.5,
          }}
        />

I have tried with 2 versions of the same VTT data, one with optional style and index properties set and one without. Both result in the same issues:

With style and order properties set:

WEBVTT

1
00:00:00.000 --> 00:00:05.760 align:middle line:50% position:50% size:80%
This is a test recording that I'm going to make at least 10 to 15 seconds long because I want it to

2
00:00:05.760 --> 00:00:10.880 align:middle line:50% position:50% size:80%
pick up on my words and do a full transcription of everything that I'm saying. I want to see how

3
00:00:10.880 --> 00:00:17.520 align:middle line:50% position:50% size:80%
well that it works and to see that the subtitles are actually prepared to show at the right time.

Without:

WEBVTT

1
00:00:00.000 --> 00:00:05.760
This is a test recording that I'm going to make at least 10 to 15 seconds long because I want it to

2
00:00:05.760 --> 00:00:10.880
pick up on my words and do a full transcription of everything that I'm saying. I want to see how

3
00:00:10.880 --> 00:00:17.520
well that it works and to see that the subtitles are actually prepared to show at the right time.

The text was updated successfully, but these errors were encountered:

brycnguyen · 2024-01-10T18:09:16Z

@james-cohen I had similar issues on iOS. Could you try #3461 and see if that resolves ur issues?

james-cohen · 2024-01-10T20:38:22Z

Hi @brycnguyen, unfortunately not. It's now playing without freezing, but the sound is still missing and the text tracks aren't loading at all. This is the response body from onLoad:

{
  "textTracks": [],
  "canStepForward": true,
  "canPlayFastForward": true,
  "canPlaySlowReverse": true,
  "audioTracks": [],
  "currentTime": 0,
  "canPlayReverse": true,
  "duration": 17.753999710083008,
  "canPlaySlowForward": true,
  "target": 1719,
  "canStepBackward": true,
  "naturalSize": {
    "height": 1920,
    "width": 1080,
    "orientation": "landscape"
  }
}

I've confirmed in a separate console log that the transcriptionUri is definitely being supplied

castdrian · 2024-02-07T08:49:15Z

for me passing texttracks just breaks playback entirely

manyaagarwal · 2024-02-15T17:35:53Z

Hey, I am facing a similar issue where the video doesn't work at all with the following props:

 selectedTextTrack={{
          type: 'language',
          value: 'en',
        }}
        textTracks={[
          {
            title: 'English',
            language: 'en',
            type: TextTracksType.VTT,
            uri: 'https://bitdash-a.akamaihd.net/content/sintel/subtitles/subtitles_en.vtt',
          },
          {
            title: 'Deutsch',
            language: 'de',
            type: TextTracksType.VTT,
            uri: 'https://bitdash-a.akamaihd.net/content/sintel/subtitles/subtitles_en.vtt',
          },
        ]}

paulrinaldi · 2024-02-28T23:34:18Z

With a known .VTT file that works where the values on the

...
textTracks={ [{"language": "en", "title": "English", "type": "text/vtt", "uri": file:///Users/paulrinaldi/Library/Developer/CoreSimulator/Devices/<ID>/data/Containers/Data/Application/<ANOTHER_ID>/Documents/captions.vtt" }] }
selectedTextTrack={{"type": "title", "value": "English"}}
...

when I enable captions (i.e. set the selectedTextTrack)
on iOS in 6.0.0-beta.5 the video becomes a black screen and the video and audio stop.
on android in 6.0.0-beta.5 the video keeps going but no captions are shown.

KrzysztofMoch · 2024-03-04T09:46:51Z

Cloud someone test this PR #3557 ?

sntg-p · 2024-05-02T22:25:47Z

I'm also having freezes when using a VTT.

In my case, the video uses a signed S3 URL but VTT is public. It seems that if both are public URLs, the video plays properly but subtitles are currently not appearing on v6.0.0-rc.0.

Feeeliperesende · 2024-05-17T11:29:16Z

Same problem here, any solution?
On Android everything works ok, but on iOS the VTT subtitles do not appear at all

KrzysztofMoch · 2024-05-17T12:13:59Z

Are you able to reproduce this in sample
If you are using hls streams it will not work

Feeeliperesende · 2024-05-17T19:49:14Z

IOS

React native video - v 5.2

On Android everything works ok

stuttskl · 2024-05-24T16:25:17Z

Any updates? Facing the same problems

freeboub · 2024-05-24T16:28:09Z

Any updates? Facing the same problems

As we identify this is a limitation from avplayer, but maybe we are wrong, there is no chance for a quick fix ...

stuttskl · 2024-05-24T16:33:42Z

Any updates? Facing the same problems

As we identify this is a limitation from avplayer, but maybe we are wrong, there is no chance for a quick fix ...

Could you please elaborate? What is the limitation from AvPlayer?

freeboub · 2024-05-24T16:40:46Z

I forgot to put the link ...
Here is a thread explaining the complexity to implement the feature:
https://stackoverflow.com/questions/43161124/how-to-add-external-webvtt-subtitles-into-http-live-stream-on-ios-client

stuttskl · 2024-05-24T16:50:41Z

Huh, I see. My use case might be a bit different. I'm scratching my head as to why for some .vtt files, captions and video playback works, but others do not. For example, here is my code:

 <Video
                source={{ uri: "https://res.cloudinary.com/dqw5s3atw/video/upload/mm-depression-1" }}
                ref={videoRef}
                onBuffer={onBuffer}
                onError={onError}
                onLoad={e => onLoad(e)}
                onLoadStart={e => onLoadStart(e)}
                style={styles.video}
                controls={true}
                fullscreen={true}
                paused={paused}
                onSeek={handleSeekForward}
                poster={thumbnail || fallbackThumbnail}
                playInBackground={false}
                textTracks={[
                    {
                        title: "English CC",
                        language: "en",
                        type: TextTrackType.VTT,
                        uri: "https://rsysdemo-b7be9.web.app/MIB2-subtitles-pt-BR.vtt", // THIS WORKS
                      //  uri: "https://res.cloudinary.com/dqw5s3atw/raw/upload/v1712945521/mm-depression-1.vtt" // THIS DOES NOT
                    }
                ]}
                selectedTextTrack={{
                    type: SelectedTrackType.TITLE,
                    value: "English CC"
                }}
            />
            ```
            
            Both should be unsigned, public URLs. In my implementation, I don't have `uri` passed twice, just showing for example here.

freeboub · 2024-05-24T16:57:08Z

Your issue seems different, here the issue is that you cannot mix m3u and vtt.
But your issue is that some vtt files don't work. Btw we don't process the file we just give it to avplayer...

stuttskl · 2024-05-24T16:59:08Z

m3u and vtt.

Are you saying the video url is m3u? Or the vtt file that I am trying to use is m3u?

stuttskl · 2024-05-24T17:05:14Z

Nevermind, I see the issue. Thanks!

lucasBessaBairesDev · 2024-05-29T21:29:08Z

Just reinforcing that this is still an issue and also lemme share my observations:

passing an array of URL subtitles to "textTracks" prop will simply kill Video, most likely because the URLs of my project have HMAC authentication
I'm downloading the subtitles and using the local files, which makes it works a little better
the selectedTextTrack property cannot be of type "DISABLE", it must be "INDEX" and if you want no subtitles, set an invalid index, I use -1 (don't know if this is expected)
selecting different subtitles causes different effects. Eg: let's say the video has 2 subtitles, "En" and "Es". If there's none selected, the video image will get stuck even tho it's playing. If I select "En", the video will play normally with no audio. If I pick "Es" the audio comes back on but the image gets black.
sometimes hot-reloading the project will make it work perfectly!
99% of the time, if I let the video play till the end, It will be fixed(only once it didn't fix the issue).
once in a full moon it works perfectly. I think It happened twice.

react-native-video@6.1.2
Testing with simulators

sntg-p · 2024-06-07T06:09:29Z

In case somebody wants an easy solution, I'm now using another package to handle showing subtitles.

paulrinaldi · 2024-06-07T14:37:54Z

Should the title of this issue be altered to be more accurate?

stuttskl · 2024-06-14T11:47:46Z

Just reinforcing that this is still an issue and also lemme share my observations:

passing an array of URL subtitles to "textTracks" prop will simply kill Video, most likely because the URLs of my project have HMAC authentication

I'm downloading the subtitles and using the local files, which makes it works a little better

the selectedTextTrack property cannot be of type "DISABLE", it must be "INDEX" and if you want no subtitles, set an invalid index, I use -1 (don't know if this is expected)

selecting different subtitles causes different effects. Eg: let's say the video has 2 subtitles, "En" and "Es". If there's none selected, the video image will get stuck even tho it's playing. If I select "En", the video will play normally with no audio. If I pick "Es" the audio comes back on but the image gets black.

sometimes hot-reloading the project will make it work perfectly!

99% of the time, if I let the video play till the end, It will be fixed(only once it didn't fix the issue).

once in a full moon it works perfectly. I think It happened twice.

react-native-video@6.1.2

Testing with simulators

How has this solution been working for you? I am going to try your suggestions but am also searching for other options to add subtitles support for my app where subtitle support is a requirement for accessibility compliance

stuttskl · 2024-07-15T16:02:47Z

This is still an issue. Is there a proposed fix or work around? How are others getting subtitle support for videos?

Feeeliperesende · 2024-07-15T17:02:21Z

@stuttskl - I used react-native-subtitles to solve this problem

levantsivadze · 2024-07-30T09:54:31Z

Any fix for this one? Subtitles work on Android but IOS causes video player to crash. Don't want to use a bit outdated react-native-subtitles

brycnguyen · 2024-09-27T15:59:15Z

@levantsivadze, the library is outdated, but relatively a pretty small library. I've also moved to this as the native subtitles just don't seem to work for me. My video on iOS would just stop producing audio after 3 minutes or so. The video would still progress, but no audio/video would be visible to the user, it would just freeze at where-ever the user left off.

I'm assuming this library is trying to fetch the subtitles under the hood and failing, because I do see a native error
for this function RCTVideoUtils.getValidTextTracks sometimes.

It's also a lot easier to use with the react-native-subtitles library since it uses axios under the hood to fetch the subtitles. With iOS, I no longer have to download the subtitles onto the device via react-native-fs.

If you do use react-native-subtitles, there was a few things I had to patch.

diff --git a/utils/subtitle-parser.js b/utils/subtitle-parser.js
index eb35d95bc2cf8c1a66ebc161055feb54c917459d..f037f6f1794f4cf7705783edb65a5cb376263abf 100644
--- a/utils/subtitle-parser.js
+++ b/utils/subtitle-parser.js
@@ -4,7 +4,8 @@ const { default: srtParser2 } = require('srt-parser-2');
 import timeToSeconds from './time-to-seconds';
 const subtitleParser = async (subitleUrl) => {
     const { data: subtitleData } = await axios.get(subitleUrl);
-    const subtitleType = subitleUrl.split('.')[subitleUrl.split('.').length - 1];
+    const urlWithoutParams = subitleUrl.split('?')[0];
+    const subtitleType = urlWithoutParams.split('.').pop();
     const result = [];
     if (subtitleType === 'srt') {
         const parser = new srtParser2();
@@ -24,7 +25,12 @@ const subtitleParser = async (subitleUrl) => {
             result.push({
                 start: start / 1000,
                 end: end / 1000,
-                part: part.slice(0, part.length - part.split(' ')[part.split(' ').length - 1].length),
+                part,
             });
         });
     }

I had to patch the .vtt detection for the library since my url has query params in it. (i.e. .vtt?expires=123456)
There's a PR open on the library but never merged: nriccar/react-native-subtitles#9

Lastly, my subtitles were getting the last word of each sentence chopped off. I fixed this by removing the substring logic that was done here: https://github.com/nriccar/react-native-subtitles/blob/main/lib/utils/subtitle-parser.ts#L51C9-L54C11

lucasBessaBairesDev · 2024-10-11T12:56:59Z

@stuttskl sorry it took so long to reply.
In my project we ended up creating our own subtitle display component.
I tried using react-native-subtitle but it had some issue that prevented it from being used. I don't remember what it was.
But it turns out it's really simple to create your own. Check react-native-subtitle's code for reference.

That said, I do like this video player, this subtitle issue is it's only weakspot, at least from what I needed.

freeboub · 2024-10-11T20:45:43Z

If someone want to integrate code of react native subtitles inside react native video, I would be have to review and merge it !
Unfortunatelly, it is not supported on AVPlayer, no chance to fix the issue with an AVPlayer integration fix.

freeboub · 2024-10-16T16:06:39Z

linked to : #4238
Let's continue in the other ticket which is clearer

KrzysztofMoch added bug Platform: iOS labels Jan 11, 2024

lucasBessaBairesDev mentioned this issue May 31, 2024

[BUG]: Cannot select sideLoaded text tracks by index #3736

Closed

freeboub added the Accepted Issue is confirmed and accepted by maintainers team label Oct 11, 2024

freeboub closed this as completed Oct 16, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

VTT Subtitles broken on iOS #3468

VTT Subtitles broken on iOS #3468

james-cohen commented Jan 10, 2024

brycnguyen commented Jan 10, 2024

james-cohen commented Jan 10, 2024

castdrian commented Feb 7, 2024

manyaagarwal commented Feb 15, 2024 •

edited

Loading

paulrinaldi commented Feb 28, 2024

KrzysztofMoch commented Mar 4, 2024

sntg-p commented May 2, 2024 •

edited

Loading

Feeeliperesende commented May 17, 2024

KrzysztofMoch commented May 17, 2024

Feeeliperesende commented May 17, 2024

stuttskl commented May 24, 2024

freeboub commented May 24, 2024

stuttskl commented May 24, 2024

freeboub commented May 24, 2024

stuttskl commented May 24, 2024

freeboub commented May 24, 2024

stuttskl commented May 24, 2024

stuttskl commented May 24, 2024

lucasBessaBairesDev commented May 29, 2024

sntg-p commented Jun 7, 2024

paulrinaldi commented Jun 7, 2024

stuttskl commented Jun 14, 2024

stuttskl commented Jul 15, 2024

Feeeliperesende commented Jul 15, 2024

levantsivadze commented Jul 30, 2024

brycnguyen commented Sep 27, 2024 •

edited

Loading

lucasBessaBairesDev commented Oct 11, 2024

freeboub commented Oct 11, 2024

freeboub commented Oct 16, 2024

VTT Subtitles broken on iOS #3468

VTT Subtitles broken on iOS #3468

Comments

james-cohen commented Jan 10, 2024

Bug

Platform

Environment info

Steps To Reproduce

Expected behaviour

Reproducible sample code

brycnguyen commented Jan 10, 2024

james-cohen commented Jan 10, 2024

castdrian commented Feb 7, 2024

manyaagarwal commented Feb 15, 2024 • edited Loading

paulrinaldi commented Feb 28, 2024

KrzysztofMoch commented Mar 4, 2024

sntg-p commented May 2, 2024 • edited Loading

Feeeliperesende commented May 17, 2024

KrzysztofMoch commented May 17, 2024

Feeeliperesende commented May 17, 2024

IOS

React native video - v 5.2

On Android everything works ok

stuttskl commented May 24, 2024

freeboub commented May 24, 2024

stuttskl commented May 24, 2024

freeboub commented May 24, 2024

stuttskl commented May 24, 2024

freeboub commented May 24, 2024

stuttskl commented May 24, 2024

stuttskl commented May 24, 2024

lucasBessaBairesDev commented May 29, 2024

sntg-p commented Jun 7, 2024

paulrinaldi commented Jun 7, 2024

stuttskl commented Jun 14, 2024

stuttskl commented Jul 15, 2024

Feeeliperesende commented Jul 15, 2024

levantsivadze commented Jul 30, 2024

brycnguyen commented Sep 27, 2024 • edited Loading

lucasBessaBairesDev commented Oct 11, 2024

freeboub commented Oct 11, 2024

freeboub commented Oct 16, 2024

manyaagarwal commented Feb 15, 2024 •

edited

Loading

sntg-p commented May 2, 2024 •

edited

Loading

brycnguyen commented Sep 27, 2024 •

edited

Loading