Add support for WebRTC MediaStreams for real-time voice chat

[Wavesonics]

Describe the project you are working on:
Realtime Voice chat for multiplayer using WebRTC

Describe the problem or limitation you are having in your project:
WebRTC is an obvious choice for implementing voice chat inside of Godot, but the current WebRTC implementation is missing one key feature: Media Steams

Without them doing a true real-time voice chat would not be possible with WebRTC

Describe the feature / enhancement and how it helps to overcome the problem or limitation:
From Faless over on the webrtc-native repository:

some form of WebRTCMediaStream/WebRTCMediaSteamTrack class will have to be added to core. Then appropriate implementation for JS and GDNative interface will have to be added in modules/webrtc, a couple of signals and method will have to be added to RTCPeerConnection and finally the GDNative plugin in this repository will have to be updated.

Describe how your proposal will work, with code, pseudocode, mockups, and/or diagrams:
Faless explains what would be required better than I can:
godotengine/webrtc-native#13 (comment)

If this enhancement will not be used often, can it be worked around with a few lines of script?:
Not possible.

Is there a reason why this should be core and not an add-on in the asset library?:
Not possible to do as a library as far as I can figure.

[goatchurchprime]

I'd note that this is a must-have in VR. Developers spend a lot of time showing off incomplete games to senior managers. The immersive nature of VR cuts them off from the developer being able to talk them through what to do and what's going on in the way that a game on the screen game does not.

In fact in many experiences game-play is taught from one person to the next, rather than from the instructional materials. This is a challenge for VR games where many of them resort to presenting the player with a wall of instructions, or a very extensive tutorial mode, or both. It's so much easier for a teacher to watch what the novice is doing wrong and correct them, than to try and do it with sets of boring instructions that attempt (and often fail) to cover all eventualities.

[bojidar-bg]

I managed to implement this on top of 3.2, though only for web at the moment: https://github.com/bojidar-bg/godot/tree/3.2-webrtc-mediastream.

[rKinson]

I have a Godot project that can't be progressed without sending/receiving realtime VoIP and would love to see this implemented. I don't know how to code this, so all I can do is add my support for this feature proposal. In the meantime, I'll just keep that project on the shelf.

[rKinson]

What are the minimum requirements that would have to be added to core to allow for WebRTC MediaStream VoIP to be developed as a plugin?

[bojidar-bg]

I would suggest checking my branch. I would not be developing it any further (priorities change), but it could be used as a foundation for bringing WebRTC to non-Web platforms. In it, I have implemented mediastreams only for the web platform; I suppose that implementation could be mirrored in the GDNative bindings and the GDNative library ( https://github.com/godotengine/webrtc-native ) in order to get things working on desktop.

Last time I was playing around with it, I got a bit stuck on the GDNative bindings (and specifically, properly versioning the API). I suppose some of the resident GDNative experts might help on this one.

[rKinson]

Thanks for the comment @bojidar-bg but I know nothing about C++, WebRTC, GDNative etc and so looking at your branch would be like a monkey (me) looking at a Rembrandt!

If I understand the core devs policy, they prioritise features that have the widest possible use case which makes sense. Supporting WebRTC fully is a bit niche so I get their reticence however my question wasn't so much a technical one. I was trying to understand the scope of the work that would be required by the core devs to implement just enough to allow other 3rd parties to develop WebRTC for web/desktop as a plugin.

This is because the core devs policy includes a preference for core functionality requests that add functionality "that can be implemented as plugins".

[realkotob]

@bojidar-bg Thanks for the great work on the webrtc-mediastream branch!

I'm not very familiar with WebRTC apis so I wanted to ask what would be the next step to use this in a web export? I'm assuming it's not as simple as calling add_track() and the audio would automatically arrive on the other side :b

[bojidar-bg]

@asheraryam I'm not sure now, but checking a test project from that time (godot-webrtc-test.zip, apparently based on the webrtc_signaling demo), it seems like the very few things are needed.

To send audio:

• Use peer.add_track(..) with an AudioEffectRecord on the correct bus (you might need an AudioStreamMicrophone playing on the same bus).

To receive audio:

• Pass "offerToReceiveAudio": true in the options to peer.initialize
• Connect to media_track_received in order to receive the audio track.
• Add the received audio track as an AudioStreamPlayer with autoplay = true

[mattkanwisher]

Seems like issue has been silent for a few years. With Gd 4 now I'm beta, would this be easier to build ?