Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Merging audio and video together using MergingMediaSource #9876

Closed
louiewh opened this issue Jan 19, 2022 · 4 comments
Closed

Merging audio and video together using MergingMediaSource #9876

louiewh opened this issue Jan 19, 2022 · 4 comments
Assignees
@tonihei
Labels
question

Comments

@louiewh
Copy link

louiewh commented Jan 19, 2022

Unfortunately we can't answer all questions. Unclear questions or questions with
insufficient information may not get attention.

Before filing a question:

When filing a question:

Describe your question in detail.

I have 3 media url . Video Url1 Audio Url3 Audio Url3. And Video Url1 also 1 audio track.
I want play Video and audio Url at the same time. like one media container have 3 audio track.

so I create MediaSource like this:

public static MediaSource createMediaSource(Uri videoUri, Uri audioUriOri, Uri audioUrlAcc) {
    // Build the video MediaSource.
    MediaSource videoSource = new ProgressiveMediaSource.Factory(
            new CryptoDataSourceFactory("user-agent"))
            .createMediaSource(videoUri);

    // Build the audio MediaSource.
    MediaSource audioSourceOri = new ProgressiveMediaSource.Factory(
            new CryptoDataSourceFactory("user-agent"))
            .createMediaSource(audioUriOri);

    MediaSource audioSourceAcc = new ProgressiveMediaSource.Factory(
            new CryptoDataSourceFactory("user-agent"))
            .createMediaSource(audioUrlAcc);

    MergingMediaSource mergedSource =
            new MergingMediaSource(videoSource, audioSourceOri, audioSourceAcc);

    return mergedSource;
}

And StyledPlayerView corrent show exoplayer have 3 audio track。 But only the audio Track of VideoUrl can play. Another alone Audio Url play slient.
@louiewh
Copy link
Author

louiewh commented Jan 19, 2022 •
edited
Loading

EventLogger:
show Audio switch succes
2022-01-19 19:59:42.931 21074-21074/com.example.karaoke D/EventLogger: Group:1, adaptive_supported=N/A [
2022-01-19 19:59:42.931 21074-21074/com.example.karaoke D/EventLogger: [X] Track:0, id=1, mimeType=audio/mp4a-latm, codecs=mp4a.40.2, channels=2, sample_rate=44100, language=und, supported=YES
2022-01-19 19:59:42.931 21074-21074/com.example.karaoke D/EventLogger: ]

2022-01-19 19:58:46.292 21074-21074/com.example.karaoke D/EventLogger: surfaceSize [eventTime=0.05, mediaPos=0.00, window=0, 2560, 1438]
2022-01-19 19:58:46.630 21074-21074/com.example.karaoke D/EventLogger: timeline [eventTime=0.38, mediaPos=0.00, window=0, periodCount=1, windowCount=1, reason=PLAYLIST_CHANGED
2022-01-19 19:58:46.630 21074-21074/com.example.karaoke D/EventLogger: period [?]
2022-01-19 19:58:46.630 21074-21074/com.example.karaoke D/EventLogger: window [?, seekable=false, dynamic=true]
2022-01-19 19:58:46.630 21074-21074/com.example.karaoke D/EventLogger: ]
2022-01-19 19:58:46.630 21074-21074/com.example.karaoke D/EventLogger: mediaItem [eventTime=0.39, mediaPos=0.00, window=0, reason=PLAYLIST_CHANGED]
2022-01-19 19:58:46.632 21074-21074/com.example.karaoke D/EventLogger: state [eventTime=0.39, mediaPos=0.00, window=0, BUFFERING]
2022-01-19 19:58:46.636 21074-21074/com.example.karaoke D/EventLogger: playWhenReady [eventTime=0.39, mediaPos=0.00, window=0, true, USER_REQUEST]
2022-01-19 19:58:46.657 21074-21074/com.example.karaoke D/EventLogger: timeline [eventTime=0.41, mediaPos=0.00, window=0, period=0, periodCount=1, windowCount=1, reason=SOURCE_UPDATE
2022-01-19 19:58:46.657 21074-21074/com.example.karaoke D/EventLogger: period [?]
2022-01-19 19:58:46.657 21074-21074/com.example.karaoke D/EventLogger: window [?, seekable=false, dynamic=false]
2022-01-19 19:58:46.657 21074-21074/com.example.karaoke D/EventLogger: ]
2022-01-19 19:58:46.657 21074-21074/com.example.karaoke D/EventLogger: loading [eventTime=0.41, mediaPos=0.00, window=0, period=0, true]
2022-01-19 19:58:46.816 21074-21074/com.example.karaoke D/EventLogger: timeline [eventTime=0.57, mediaPos=0.00, window=0, period=0, periodCount=1, windowCount=1, reason=SOURCE_UPDATE
2022-01-19 19:58:46.816 21074-21074/com.example.karaoke D/EventLogger: period [230.48]
2022-01-19 19:58:46.816 21074-21074/com.example.karaoke D/EventLogger: window [230.48, seekable=true, dynamic=false]
2022-01-19 19:58:46.816 21074-21074/com.example.karaoke D/EventLogger: ]
2022-01-19 19:58:46.868 21074-21074/com.example.karaoke D/EventLogger: videoEnabled [eventTime=0.62, mediaPos=0.00, window=0, period=0]
2022-01-19 19:58:46.868 21074-21074/com.example.karaoke D/EventLogger: audioEnabled [eventTime=0.62, mediaPos=0.00, window=0, period=0]
2022-01-19 19:58:46.869 21074-21074/com.example.karaoke D/EventLogger: tracks [eventTime=0.62, mediaPos=0.00, window=0, period=0
2022-01-19 19:58:46.869 21074-21074/com.example.karaoke D/EventLogger: MediaCodecVideoRenderer [
2022-01-19 19:58:46.869 21074-21074/com.example.karaoke D/EventLogger: Group:0, adaptive_supported=N/A [
2022-01-19 19:58:46.869 21074-21074/com.example.karaoke D/EventLogger: [X] Track:0, id=1, mimeType=video/avc, codecs=avc1.64001E, res=848x476, fps=25.0, supported=YES
2022-01-19 19:58:46.869 21074-21074/com.example.karaoke D/EventLogger: ]
2022-01-19 19:58:46.869 21074-21074/com.example.karaoke D/EventLogger: ]
2022-01-19 19:58:46.869 21074-21074/com.example.karaoke D/EventLogger: MediaCodecAudioRenderer [
2022-01-19 19:58:46.869 21074-21074/com.example.karaoke D/EventLogger: Group:0, adaptive_supported=N/A [
2022-01-19 19:58:46.869 21074-21074/com.example.karaoke D/EventLogger: [X] Track:0, id=2, mimeType=audio/mp4a-latm, codecs=mp4a.40.2, channels=2, sample_rate=44100, language=und, supported=YES
2022-01-19 19:58:46.869 21074-21074/com.example.karaoke D/EventLogger: ]
2022-01-19 19:58:46.870 21074-21074/com.example.karaoke D/EventLogger: Group:1, adaptive_supported=N/A [
2022-01-19 19:58:46.870 21074-21074/com.example.karaoke D/EventLogger: [ ] Track:0, id=1, mimeType=audio/mp4a-latm, codecs=mp4a.40.2, channels=2, sample_rate=44100, language=und, supported=YES
2022-01-19 19:58:46.870 21074-21074/com.example.karaoke D/EventLogger: ]
2022-01-19 19:58:46.870 21074-21074/com.example.karaoke D/EventLogger: Group:2, adaptive_supported=N/A [
2022-01-19 19:58:46.870 21074-21074/com.example.karaoke D/EventLogger: [ ] Track:0, id=1, mimeType=audio/mp4a-latm, codecs=mp4a.40.2, channels=2, sample_rate=44100, language=und, supported=YES
2022-01-19 19:58:46.870 21074-21074/com.example.karaoke D/EventLogger: ]
2022-01-19 19:58:46.870 21074-21074/com.example.karaoke D/EventLogger: Metadata [
2022-01-19 19:58:46.870 21074-21074/com.example.karaoke D/EventLogger: TSSE: description=null: value=Lavf58.12.100
2022-01-19 19:58:46.870 21074-21074/com.example.karaoke D/EventLogger: ]
2022-01-19 19:58:46.870 21074-21074/com.example.karaoke D/EventLogger: ]
2022-01-19 19:58:46.870 21074-21074/com.example.karaoke D/EventLogger: TextRenderer []
2022-01-19 19:58:46.870 21074-21074/com.example.karaoke D/EventLogger: MetadataRenderer []
2022-01-19 19:58:46.870 21074-21074/com.example.karaoke D/EventLogger: CameraMotionRenderer []
2022-01-19 19:58:46.870 21074-21074/com.example.karaoke D/EventLogger: ]
2022-01-19 19:58:46.871 21074-21074/com.example.karaoke D/EventLogger: downstreamFormat [eventTime=0.63, mediaPos=0.00, window=0, period=0, id=1, mimeType=video/avc, codecs=avc1.64001E, res=848x476, fps=25.0]
2022-01-19 19:58:46.920 21074-21074/com.example.karaoke D/EventLogger: videoDecoderInitialized [eventTime=0.68, mediaPos=0.00, window=0, period=0, OMX.qcom.video.decoder.avc]
2022-01-19 19:58:46.920 21074-21074/com.example.karaoke D/EventLogger: videoInputFormat [eventTime=0.68, mediaPos=0.00, window=0, period=0, id=1, mimeType=video/avc, codecs=avc1.64001E, res=848x476, fps=25.0]
2022-01-19 19:58:46.924 21074-21074/com.example.karaoke D/EventLogger: downstreamFormat [eventTime=0.68, mediaPos=0.00, window=0, period=0, id=2, mimeType=audio/mp4a-latm, codecs=mp4a.40.2, channels=2, sample_rate=44100, language=und]
2022-01-19 19:58:46.953 21074-21074/com.example.karaoke D/EventLogger: audioDecoderInitialized [eventTime=0.71, mediaPos=0.00, window=0, period=0, c2.android.aac.decoder]
2022-01-19 19:58:46.954 21074-21074/com.example.karaoke D/EventLogger: audioInputFormat [eventTime=0.71, mediaPos=0.00, window=0, period=0, id=2, mimeType=audio/mp4a-latm, codecs=mp4a.40.2, channels=2, sample_rate=44100, language=und]
2022-01-19 19:58:46.968 21074-21074/com.example.karaoke D/EventLogger: videoSize [eventTime=0.72, mediaPos=0.00, window=0, period=0, 848, 476]
2022-01-19 19:58:46.982 21074-21074/com.example.karaoke D/EventLogger: renderedFirstFrame [eventTime=0.74, mediaPos=0.00, window=0, period=0, Surface(name=null)/@0x73ba9d5]
2022-01-19 19:58:46.999 21074-21074/com.example.karaoke D/EventLogger: state [eventTime=0.75, mediaPos=0.00, window=0, period=0, READY]
2022-01-19 19:58:47.000 21074-21074/com.example.karaoke D/EventLogger: isPlaying [eventTime=0.76, mediaPos=0.00, window=0, period=0, true]
2022-01-19 19:58:50.612 21074-21074/com.example.karaoke D/EventLogger: loading [eventTime=4.37, mediaPos=3.41, window=0, period=0, false]
2022-01-19 19:58:51.584 21074-21074/com.example.karaoke D/EventLogger: loading [eventTime=5.34, mediaPos=4.38, window=0, period=0, true]
2022-01-19 19:58:51.957 21074-21074/com.example.karaoke D/EventLogger: loading [eventTime=5.71, mediaPos=4.76, window=0, period=0, false]
2022-01-19 19:58:57.799 21074-21074/com.example.karaoke D/EventLogger: loading [eventTime=11.55, mediaPos=10.60, window=0, period=0, true]
2022-01-19 19:58:58.178 21074-21074/com.example.karaoke D/EventLogger: loading [eventTime=11.93, mediaPos=10.98, window=0, period=0, false]
2022-01-19 19:59:03.834 21074-21074/com.example.karaoke D/EventLogger: loading [eventTime=17.59, mediaPos=16.64, window=0, period=0, true]
2022-01-19 19:59:04.217 21074-21074/com.example.karaoke D/EventLogger: loading [eventTime=17.97, mediaPos=17.01, window=0, period=0, false]
2022-01-19 19:59:07.571 21074-21074/com.example.karaoke D/EventLogger: loading [eventTime=21.33, mediaPos=20.37, window=0, period=0, true]
2022-01-19 19:59:07.954 21074-21074/com.example.karaoke D/EventLogger: loading [eventTime=21.71, mediaPos=20.75, window=0, period=0, false]
2022-01-19 19:59:12.563 21074-21074/com.example.karaoke D/EventLogger: loading [eventTime=26.32, mediaPos=25.36, window=0, period=0, true]
2022-01-19 19:59:12.942 21074-21074/com.example.karaoke D/EventLogger: loading [eventTime=26.70, mediaPos=25.74, window=0, period=0, false]
2022-01-19 19:59:18.444 21074-21074/com.example.karaoke D/EventLogger: loading [eventTime=32.20, mediaPos=31.24, window=0, period=0, true]
2022-01-19 19:59:18.830 21074-21074/com.example.karaoke D/EventLogger: loading [eventTime=32.58, mediaPos=31.63, window=0, period=0, false]
2022-01-19 19:59:24.000 21074-21074/com.example.karaoke D/EventLogger: loading [eventTime=37.76, mediaPos=36.80, window=0, period=0, true]
2022-01-19 19:59:24.389 21074-21074/com.example.karaoke D/EventLogger: loading [eventTime=38.14, mediaPos=37.19, window=0, period=0, false]
2022-01-19 19:59:28.896 21074-21074/com.example.karaoke D/EventLogger: loading [eventTime=42.65, mediaPos=41.69, window=0, period=0, true]
2022-01-19 19:59:29.274 21074-21074/com.example.karaoke D/EventLogger: loading [eventTime=43.03, mediaPos=42.07, window=0, period=0, false]
2022-01-19 19:59:33.648 21074-21074/com.example.karaoke D/EventLogger: loading [eventTime=47.40, mediaPos=46.45, window=0, period=0, true]
2022-01-19 19:59:34.025 21074-21074/com.example.karaoke D/EventLogger: loading [eventTime=47.78, mediaPos=46.83, window=0, period=0, false]
2022-01-19 19:59:41.860 21074-21074/com.example.karaoke D/EventLogger: loading [eventTime=55.62, mediaPos=54.66, window=0, period=0, true]
2022-01-19 19:59:42.230 21074-21074/com.example.karaoke D/EventLogger: loading [eventTime=55.99, mediaPos=55.03, window=0, period=0, false]
2022-01-19 19:59:42.929 21074-21074/com.example.karaoke D/EventLogger: audioDisabled [eventTime=56.68, mediaPos=55.71, window=0, period=0]
2022-01-19 19:59:42.929 21074-21074/com.example.karaoke D/EventLogger: audioEnabled [eventTime=56.69, mediaPos=55.71, window=0, period=0]
2022-01-19 19:59:42.930 21074-21074/com.example.karaoke D/EventLogger: tracks [eventTime=56.69, mediaPos=55.73, window=0, period=0
2022-01-19 19:59:42.930 21074-21074/com.example.karaoke D/EventLogger: MediaCodecVideoRenderer [
2022-01-19 19:59:42.930 21074-21074/com.example.karaoke D/EventLogger: Group:0, adaptive_supported=N/A [
2022-01-19 19:59:42.931 21074-21074/com.example.karaoke D/EventLogger: [X] Track:0, id=1, mimeType=video/avc, codecs=avc1.64001E, res=848x476, fps=25.0, supported=YES
2022-01-19 19:59:42.931 21074-21074/com.example.karaoke D/EventLogger: ]
2022-01-19 19:59:42.931 21074-21074/com.example.karaoke D/EventLogger: ]
2022-01-19 19:59:42.931 21074-21074/com.example.karaoke D/EventLogger: MediaCodecAudioRenderer [
2022-01-19 19:59:42.931 21074-21074/com.example.karaoke D/EventLogger: Group:0, adaptive_supported=N/A [
2022-01-19 19:59:42.931 21074-21074/com.example.karaoke D/EventLogger: [ ] Track:0, id=2, mimeType=audio/mp4a-latm, codecs=mp4a.40.2, channels=2, sample_rate=44100, language=und, supported=YES
2022-01-19 19:59:42.931 21074-21074/com.example.karaoke D/EventLogger: ]
2022-01-19 19:59:42.931 21074-21074/com.example.karaoke D/EventLogger: Group:1, adaptive_supported=N/A [
2022-01-19 19:59:42.931 21074-21074/com.example.karaoke D/EventLogger: [X] Track:0, id=1, mimeType=audio/mp4a-latm, codecs=mp4a.40.2, channels=2, sample_rate=44100, language=und, supported=YES
2022-01-19 19:59:42.931 21074-21074/com.example.karaoke D/EventLogger: ]
2022-01-19 19:59:42.931 21074-21074/com.example.karaoke D/EventLogger: Group:2, adaptive_supported=N/A [
2022-01-19 19:59:42.931 21074-21074/com.example.karaoke D/EventLogger: [ ] Track:0, id=1, mimeType=audio/mp4a-latm, codecs=mp4a.40.2, channels=2, sample_rate=44100, language=und, supported=YES
2022-01-19 19:59:42.931 21074-21074/com.example.karaoke D/EventLogger: ]
2022-01-19 19:59:42.931 21074-21074/com.example.karaoke D/EventLogger: Metadata [
2022-01-19 19:59:42.931 21074-21074/com.example.karaoke D/EventLogger: TPE1: description=null: value=周杰伦
2022-01-19 19:59:42.931 21074-21074/com.example.karaoke D/EventLogger: ----: domain=com.apple.iTunes, description=cdec
2022-01-19 19:59:42.931 21074-21074/com.example.karaoke D/EventLogger: ----: domain=com.apple.iTunes, description=iTunSMPB
2022-01-19 19:59:42.931 21074-21074/com.example.karaoke D/EventLogger: TIT2: description=null: value=夜曲
2022-01-19 19:59:42.931 21074-21074/com.example.karaoke D/EventLogger: TSSE: description=null: value=Nero AAC codec / 1.5.4.0
2022-01-19 19:59:42.931 21074-21074/com.example.karaoke D/EventLogger: ]
2022-01-19 19:59:42.931 21074-21074/com.example.karaoke D/EventLogger: ]
2022-01-19 19:59:42.931 21074-21074/com.example.karaoke D/EventLogger: TextRenderer []
2022-01-19 19:59:42.931 21074-21074/com.example.karaoke D/EventLogger: MetadataRenderer []
2022-01-19 19:59:42.931 21074-21074/com.example.karaoke D/EventLogger: CameraMotionRenderer []
2022-01-19 19:59:42.931 21074-21074/com.example.karaoke D/EventLogger: ]
2022-01-19 19:59:42.933 21074-21074/com.example.karaoke D/EventLogger: loading [eventTime=56.69, mediaPos=55.73, window=0, period=0, true]
2022-01-19 19:59:42.935 21074-21074/com.example.karaoke D/EventLogger: audioDecoderReleased [eventTime=56.69, mediaPos=55.73, window=0, period=0, c2.android.aac.decoder]
2022-01-19 19:59:42.961 21074-21074/com.example.karaoke D/EventLogger: audioDecoderInitialized [eventTime=56.72, mediaPos=55.73, window=0, period=0, c2.android.aac.decoder]
2022-01-19 19:59:42.961 21074-21074/com.example.karaoke D/EventLogger: audioInputFormat [eventTime=56.72, mediaPos=55.73, window=0, period=0, id=1, mimeType=audio/mp4a-latm, codecs=mp4a.40.2, channels=2, sample_rate=44100, language=und]
2022-01-19 19:59:43.443 21074-21074/com.example.karaoke D/EventLogger: loading [eventTime=57.20, mediaPos=55.73, window=0, period=0, false]
2022-01-19 19:59:43.477 21074-21074/com.example.karaoke D/EventLogger: droppedFrames [eventTime=57.23, mediaPos=63.95, window=0, period=0, 156]
2022-01-19 19:59:43.505 21074-21074/com.example.karaoke D/EventLogger: loading [eventTime=57.26, mediaPos=63.95, window=0, period=0, true]
2022-01-19 19:59:43.882 21074-21074/com.example.karaoke D/EventLogger: loading [eventTime=57.64, mediaPos=64.23, window=0, period=0, false]
2022-01-19 19:59:45.224 21074-21074/com.example.karaoke D/EventLogger: loading [eventTime=58.98, mediaPos=65.49, window=0, period=0, true]
2022-01-19 19:59:45.611 21074-21074/com.example.karaoke D/EventLogger: loading [eventTime=59.37, mediaPos=65.88, window=0, period=0, false]
2022-01-19 19:59:47.518 21074-21074/com.example.karaoke D/EventLogger: droppedFrames [eventTime=61.27, mediaPos=67.96, window=0, period=0, 50]

@louiewh
Copy link
Author

louiewh commented Jan 19, 2022

I overrid
public class KaraokeRenderersFactory extends DefaultRenderersFactory {
public KaraokeRenderersFactory(Context context) {
super(context);
}

@Override
protected AudioSink buildAudioSink(
        Context context,
        boolean enableFloatOutput,
        boolean enableAudioTrackPlaybackParams,
        boolean enableOffload) {
    DefaultAudioSink audioSink =  new DefaultAudioSink(
            AudioCapabilities.getCapabilities(context),
            new DefaultAudioSink.DefaultAudioProcessorChain(),
            enableFloatOutput,
            enableAudioTrackPlaybackParams,
            enableOffload
                    ? DefaultAudioSink.OFFLOAD_MODE_ENABLED_GAPLESS_REQUIRED
                    : DefaultAudioSink.OFFLOAD_MODE_DISABLED);

    return new KaraokeAudioSink(audioSink);

// return audioSink;
}
}

KaraokeAudioSink

@Override
public boolean handleBuffer(ByteBuffer buffer, long presentationTimeUs, int encodedAccessUnitCount) throws InitializationException, WriteException {
    Log.e("KaraokeAudioSink", buffer.toString());
    boolean result =  mDefaultAudioSink.handleBuffer(buffer, presentationTimeUs, encodedAccessUnitCount);
    Log.d("KaraokeAudioSink", "result"+result);

    return result;
}

image

The log show that Audio data is send to AudioSlink

@louiewh
Copy link
Author

louiewh commented Jan 20, 2022 •
edited
Loading

AudioSlink.pcm.zip
pcm is 16bits 2 channel and 44100HZ

I save audio pcm at handbuffer function to AudioSlink.pcm
I switch audio Track at 20s position. And audio is silent after 20s

@tonihei
Copy link
Collaborator

tonihei commented Jan 20, 2022

And StyledPlayerView corrent show exoplayer have 3 audio track。 But only the audio Track of VideoUrl can play. Another alone Audio Url play slient.

By default, ExoPlayer only selects one track for playback. This means the other two tracks are not just silent, they are not processed at all.

If you want to let them play together, you need to make two changes:

  1. Modify the RenderersFactory set at the player to have multiple audio renderers.
  2. Modify the TrackSelector set at the player to assign each audio track to a different audio renderer.
  3. Make sure only one of the audio tracks is the primary time source, for example by customizing two of the audio renderers to return null from getMediaClock.
    There is a more detailed code example of how this could look like in Simplify/allow selection of multiple renderers of the same type in DefaultTrackSelector. #6589 (comment).

Note that the approach above doesn't perfectly synchronize the audio tracks, which is considerably more complicated and tracked by #4261.

@tonihei tonihei closed this as completed Jan 20, 2022
@google google locked and limited conversation to collaborators Mar 22, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@tonihei tonihei

Labels
question
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@louiewh @tonihei