Help with streaming from FFMpeg to browser

[swilcox3]

I'm trying to stream the graphical output of a VTK window through FFMpeg and libdatachannel to a web browser. I've successfully got it so I can write the graphical output to a MP4 file with H264 codec. I've adapted the libdatachannel streamer example into a single-threaded implementation with copy/paste SDP exchange with the browser for now. I'm taking the AVPackets and sending them to libdatachannel instead of the MP4 file. I can get the peer connection up and connected, and I can send frames across, but nothing shows up in the <video> element. Looking at chrome://webrtc-internals, I can see that we're receiving frames and bytes across the transport, but no frames get decoded in the RTP connection. What am I doing wrong?

graphics_stream_poc.tar.gz

[vulkan101]

I’ve only had a quick look on my phone but it looks like you’re encoding h264 with the preset ultrafast, which won’t be supported by most browsers (if any)

[swilcox3]

No dice. I changed it to constrained baseline and "veryfast" for the preset with "zerolatency" for the tune, but no change. Here's the results of RTCInboundRTPVideoStream from chrome://webrtc-internals:

So I can see that packets and bytes are going up, but there's never any frames decoded. I'm pretty sure the frames themselves are fine, I can write them to disk and VLC opens the movie just fine.

[paullouisageneau]

Indeed, you should always use constrained baseline as it's the only profile with guaranteed support.

How often do you send keyframes? If the browser starts playing after the beginning of the stream, it needs to wait for one in order to start decoding.

[swilcox3]

I'm showing my ignorance here, but what's the difference between a keyframe and a regular frame? I'm not sending keyframes on purpose at all, I thought all of that would be handled by ffmpeg.

[vulkan101]

GOP size and max_b_frames dictate how often key frames are sent. If you set b frames to zero or gop size to 1 you’re going to be sending key frames only

[swilcox3]

I have max_b_frames set to 2 and gop_size set to 12. Does that seem reasonable?

[swilcox3]

I'm still struggling with this. I've simplified my use case to eliminate variables by only trying to stream a prerecorded .mp4 file. I've confirmed that the video is playable by the browser by putting the file into a <video> src attribute. The browser loads the video just fine when the video is present in the src attribute without using webrtc. It still doesn't work when I try to stream the file through libdatachannel to the browser though. I'm opening the file with ffmpeg and storing the raw AVPackets into a vector, like this:

VideoStreamer::VideoStreamer(const std::string& filename)
{
   AVFormatContext* format_context = avformat_alloc_context();
   avformat_open_input(&format_context, filename.c_str(), NULL, NULL);
   int video_stream_index = -1;
   for(int i = 0; i < format_context->nb_streams; i++)
   {
    AVCodecParameters* local_codec_parameters = format_context->streams[i]->codecpar;

    if(local_codec_parameters->codec_type == AVMEDIA_TYPE_VIDEO)
    {
        if(video_stream_index == -1)
        {
            video_stream_index = i;
        }
    }
   }

    AVPacket* packet = av_packet_alloc();
    while(av_read_frame(format_context, packet) >= 0)
    {
        if(packet->stream_index == video_stream_index) 
        {
            std::vector<std::byte> frame;
            for(size_t i = 0; i < packet->size; i++) {
                frame.push_back(std::byte(packet->data[i]));
            }
            mFrames.push_back(frame);
        }
        av_packet_unref(packet);
    }
    avformat_close_input(&format_context);
    av_packet_free(&packet);
}

std::vector<std::byte> VideoStreamer::getFrame(size_t index)
{
    return mFrames[index];
}

I'm then indexing into that vector and sending the packets to libdatachannel with following sendFrame function, like this:

CubitWebRTC::CubitWebRTC(CubitWebRTCParams params) : mIsOpen(false)
{
  rtc::InitLogger(rtc::LogLevel::Debug);

  mPeerConnection = std::make_shared<rtc::PeerConnection>();

  // ONLY FOR POC TESTING
  mPeerConnection->onStateChange(
    [](rtc::PeerConnection::State state) { std::cout << "State: " << state << std::endl; });

  // ONLY FOR POC TESTING
  mPeerConnection->onGatheringStateChange([this](rtc::PeerConnection::GatheringState state) {
    std::cout << "Gathering State: " << state << std::endl;
    if (state == rtc::PeerConnection::GatheringState::Complete) {
      auto description = mPeerConnection->localDescription();
      util::JSONObject message;
      message.upsert("type", description->typeString());
      message.upsert("sdp", std::string(description.value()));
      std::cout << "Open index.html in your web browser." << std::endl;
      std::cout << "Copy/paste this text into the browser field: " << std::endl;
      message.print();
      std::cout << "Copy/paste the SDP text from your browser window: " << std::endl;
    }
  });

  // From https://github.com/paullouisageneau/libdatachannel/blob/master/examples/streamer/main.cpp
  rtc::Description::Video media(params.mCName);
  media.addH264Codec(params.mPayloadType);
  media.addSSRC(params.mSSRC, params.mCName);
  mTrack = mPeerConnection->addTrack(media);
  auto rtpConfig = std::make_shared<rtc::RtpPacketizationConfig>(params.mSSRC, params.mCName, params.mPayloadType, rtc::H264RtpPacketizer::defaultClockRate);
  auto packetizer = std::make_shared<rtc::H264RtpPacketizer>(rtc::H264RtpPacketizer::Separator::LongStartSequence, rtpConfig);
  // create H264 handler
  auto h264Handler = std::make_shared<rtc::H264PacketizationHandler>(packetizer);
  // add RTCP SR handler
  auto currentTime_s = currentTime();
  mRtcpSrReporter = std::make_shared<rtc::RtcpSrReporter>(rtpConfig);
  mRtcpSrReporter->rtpConfig->setStartTime(currentTime_s, rtc::RtpPacketizationConfig::EpochStart::T1970);
  mRtcpSrReporter->startRecording();
  h264Handler->addToChain(mRtcpSrReporter);
  // add RTCP NACK handler
  auto nackResponder = std::make_shared<rtc::RtcpNackResponder>();
  h264Handler->addToChain(nackResponder);
  // set handler
  mTrack->setMediaHandler(h264Handler);
  mPeerConnection->setLocalDescription(rtc::Description::Type::Offer);
  mTrack->onOpen([&]() {
    std::cout << "TRACK IS OPEN" << std::endl;
    mIsOpen = true;
  });
}

void CubitWebRTC::sendFrame(std::vector<std::byte>& frame_data)
{
  size_t i = 0;
  while (i < frame_data.size()) {
    assert(i + 4 < frame_data.size());
    auto lengthPtr = (uint32_t *) (frame_data.data() + i);
    uint32_t length = ntohl(*lengthPtr);
    auto naluStartIndex = i + 4;
    auto naluEndIndex = naluStartIndex + length;
    assert(naluEndIndex <= frame_data.size());
    auto header = reinterpret_cast<rtc::NalUnitHeader *>(frame_data.data() + naluStartIndex);
    auto type = header->unitType();
    switch (type) {
        case 7:
            previousUnitType7 = {frame_data.begin() + i, frame_data.begin() + naluEndIndex};
            break;
        case 8:
            previousUnitType8 = {frame_data.begin() + i, frame_data.begin() + naluEndIndex};;
            break;
        case 5:
            previousUnitType5 = {frame_data.begin() + i, frame_data.begin() + naluEndIndex};;
            break;
    }
    i = naluEndIndex;
  }
  if(this->mTrack->isOpen()) {
    std::cout << "TRACK IS OPEN, SENDING" << std::endl;
    // sample time is in us, we need to convert it to seconds
    auto rtpConfig = mRtcpSrReporter->rtpConfig;
    auto elapsedSeconds = currentTime();
    // get elapsed time in clock rate
    uint32_t elapsedTimestamp = rtpConfig->secondsToTimestamp(elapsedSeconds);

    // set new timestamp
    rtpConfig->timestamp = rtpConfig->startTimestamp + elapsedTimestamp;

    // get elapsed time in clock rate from last RTCP sender report
    auto reportElapsedTimestamp = rtpConfig->timestamp - mRtcpSrReporter->previousReportedTimestamp;
    // check if last report was at least 1 second ago
    if (rtpConfig->timestampToSeconds(reportElapsedTimestamp) > 1) {
        mRtcpSrReporter->setNeedsToReport();
    }

    std::cout << "Sending sample with size: " << std::to_string(frame_data.size()) << std::endl;
    try {
      // send sample
      mTrack->send(frame_data);
    } catch (const std::exception &e) {
      std::cerr << "Unable to send packet: " << e.what() << std::endl;
    }
  }
  else {
    std::cout << "TRACK IS CLOSED, NOT SENDING" << std::endl;
  }
}

I just seem to be missing something here. The handshake is happening, the frames are getting sent, but no video is ever shown. Am I doing the NALU wrong? Is there some kind of diagnostic log on the browser side above the console or chrome://webrtc-internals? I don't even know what the problem is.