Configurable helper threads for Streaming plugin RTP mountpoints

[lminiero]

In the Streaming plugin, RTP mountpoints are quite straightforward: a dedicated thread waits for incoming RTP packets, and when some arrive, it iterates on all the viewers calling relay_rtp on them, which results in the RTP header being (possibly) updated and the packet allocated and enqueued for delivery in the core for each of them. This normally works just fine, but in some performance assessment we did recently we found out that this can become a bottleneck when numbers increase: all the operations done for each viewer, even though quite lightweight, may take more time to complete than it takes for the next packet to arrive. This can result in the "read RTP from the network" part not to be able to catch up, packets start piling up, and the performance degrades considerably.

This patch revives an effort initially devised by @atoppi, by basically allowing you to configure a number of helper threads to spawn when you create an RTP mountpoint. When these helper threads are used, they become responsible for a subset of the viewers of the mountpoint, which means that any time a new viewer comes in it is assigned to one of the helper threads; as such, it's these threads that actually take care of the "update RTP header, allocate, enqueue" part, freeing the main thread of that burden. This means that, should helper threads be involved, when the mountpoint receives RTP packets it doesn't iterate on all viewers, but only on the limited number of threads, queueing the packets there: this allows it to be able to keep up with the incoming packets, not causing any bottleneck there.

The feature is very easy to configure: when creating a new RTP mountpoint, be it via config or via API, just pass a numeric thread property. By default it is 0, which means that if you don't pass anything (or pass 0) no helper thread is used, and everything works exactly as it did before. If you pass a valid value instead, that number of helper threads is created, and the mountpoint will start using them for new viewers. Notice that, at the moment, this is a value you can only configure at creation. You cannot update an existing mountpoint to, let's say, switch a non-threaded mountpoint to use threads or viceversa, or change the number of threads a mountpoint will use. As such, it's up to you to decide whether to use the feature or not, and how many threads dedicate for the purpose.

If you're asking whether you'll need this or not, it depends on how many viewers you expect on a specific mountpoint, and if you noticed performance issues when numbers increased there (assuming you're using the patched libnice library and not the stock one, which is known to have a bottleneck of its own). Typically, if you don't have hundreds of viewers, you won't need this, and can keep on using the old approach: if you have a wide audience, it might be worth a try, and you should notice improvements. We plan to do a new test campaign soon to compare the performance between non-helper-threaded and helper-threaded mountpoints.

In the meanwhile, feedback is very welcome as usual!

[lminiero]

Merging.

[tmatth]

@lminiero re: assuming you're using the patched libnice library and not the stock one, which is known to have a bottleneck of its own, which patch are you referring to? UPDATE: I'm guessing it's this:
https://gitlab.freedesktop.org/libnice/libnice/merge_requests/13

[atoppi]

@tmatth
latest progress about the patch: here

[rygo6]

Sorry to revive this old merge. But I am curious to know if any of you have any more information about this. Namely, how well do threads help the streaming plugin scale? If I put this on a 72 core machine, and then specify 72 threads, does that mean this is going to stream to tens of thousands of users?