Record audio and web received from web client

[Javierd]

Hi!
I'm working on a project where I would like being able to record the audio and video streams received from a web client.
Using the media-sfu example with a small modification, I'm able to receive both video and audio from a web client and forward it to multiple web clients.
As far as I know, If I save the messages received at the onMessage callbacks of the audio and video tracks, I should end with two files containing the raw h264 and opus files. However, that is not the case, as the files are not recognized nor playable by ffmpeg.
As the messages received on those callbacks are RTP messages, I have also tried extracting the payload of those messages, therefore getting the raw files, by using the following code:

auto rtp = reinterpret_cast<rtc::RtpHeader *>(message.data());

const char* body = rtp->getBody();
ulong size = message.size() - (body - reinterpret_cast<const char *>(message.data()));
const bool hasPadding = rtp->padding();
if (hasPadding) {
    // Read the last byte, which contains the number of padding bytes, including itself, and update the size
    const uchar paddingSize = body[size-1];
    size -= paddingSize;
}

audioFile.write(body, size);
audioFile.flush();

I have also discovered the function IsRtcp which, from my understanding, should be used to discard the Rtcp messaged from being saved to file too.

By checking the received and stored messages I have checked that the RTP headers are correctly removed, however, it seems strange to me that the received h264 file does not contains any NAL Unit separators, for example. I supose this is due to RTP being used to distinguish between the NALUs, so I tried ading the 0x000001 separator before each RTP payload, but it made no difference.

Anyway, neither of this approximations work, and I don't know why.
Does somebody have any cues? Is it possible that the received samples are encrypted or something similar?
Thanks a lot!

[paullouisageneau]

You understanding is correct in the sense that indeed, you can't just dump the packets in a file to generate a valid video file.

You can indeed extract raw video/audio data from the packets, however, the packetization payload is codec-dependent and the format way more complex than you guessed for h264. It is defined in RFC 6184. The main reason is that NAL units sizes can be wildly different from the MTU, therefore they may be grouped on a single packets, but also fragmented over multiple packets. For OPUS it is simpler, as defined in RFC 7587.

To properly generate proper audio/video files you would need to write the data in a container with timing information. The simplest way to record such a file is actually to input the RTP stream in ffmpeg or gstreamer and convert it.

I think there might be a larger issue though: if you only need to record video from a web client, you should not use RTP because you don't need the real-time aspect and you probably don't want the video to be videoconference-grade quality (RTP is lossy, meaning part of the information might be lost), instead, you should encode the video in the browser and upload video chunks as files.

Filtering with IsRtp is not necessary if you set an RtpcReceivingSession to handle RTCP.

[Javierd]

In this case I don't need any type of container arround the data, I just need the raw audio and video samples, so in theory FFMPEG should't be needed. However, I think it may be the best way to fetch data from the RTP stream, as it avoids me from having to implement that logic manually.
I have tried to do this by using ffplay ffplay -i rtp://127.0.0.1:7779 and opening an UDP socket which sends all the data received by the WebRTC stream to 127.0.0.1:7779. However, this doesn't seem to work, as FFMPEG needs an SDP file describing the payloads. I assume there is no other simpler way to do this, am I wrong?

I need real time communication, the recording is just another feature, but thanks a lot for your recommendations.

[paullouisageneau]

OK, it makes sense to use the RTP stream then. Here ffmpeg can't guess the codec and settings as there is no such information in the RTP stream, you need to pass it the track SDP description so it knows what to decode. A gstreamer pipeline is a bit easier to use, see the media-receiver example.