Performance Testing

[jcdmacleod]

Are you able to share the test methodology used for https://www.ovenmediaengine.com/benchmark ?

I would like to run some similar tests but with differing hardware.

Thanks

[getroot]

We developed a separate testing client tool for that. However, it cannot be opened yet because the packaging completeness is very low. If we have a lot of time in the future, we plan to do it.

[getroot]

This issue has been closed since it has been inactive for quite some time. If you want to continue discussing this issue, please feel free to reopen it.

[jcdmacleod]

Could you provide some insight into what you used and how to load test? As you know load testing webrtc is difficult as it is not supported by any of the usual load test tools.

[getroot]

We developed and used a tool to test the performance of OvenMediaEngine. Since it was only a tool for us, it is not packaged and therefore not suitable for use by others. We can perhaps release performance testing tools this year. (But it depends on how busy we are because it is low priority for us.)

The idea of ​​the tool is simple. Command to start stream to OME WebRTC signaling server. Also, STUN PING is sent periodically so that transmission is not interrupted. Do multiple of these at the same time. The tool then collects RTP sequence numbers and arrival times to generate statistics. About 10 sessions are connected by a real person with a real browser and checked for playback. Also check the actual network bandwidth of the server where OME is running.

We created and tested about 500 sessions simultaneously on an 8 core machine on AWS EC2. And four such machines were used. As mentioned earlier, the test tools are not packaged and very rough.

We can open this right now. (It's in the Github's Priviate repository.) But someone needs to improve it so that everyone can use it. And a manual has to be made. But we don't have enough time to do it because we are doing other high-priority tasks. So we cannot open it.

If we open this, can you contribute to its improvement? Or is there another volunteer here?
(This tool used C ++ Boost.)

[alex-jhones]

It seems like a good idea. During the NAT process, if the network quality is poor, such as WIFI, there is a high probability of failure at the signaling stage. In order to deal with these coverage issues, you should be able to increase enough sessions and be measurable. Performance testing tools seem essential. If you open it, I would like to participate in the contribution.

For reference, we are currently doing various QAs as an OEM, and we are observing by making multiple clients on Azure servers or connecting to remote clients or checking LTE signals from multiple local locations.

However, Webrtc is being tested by connecting multiple Chrome browsers because it is difficult to configure clients easily due to signaling issues.

As a result of observation so far, once connected sessions are reliably connected. However, many connection failures are observed when accessing in a network environment where many sessions are not good. It is not particularly common, but it is the case when both the server and client are NAT environments.

NAT uses UDP. Therefore, it seems that these symptoms are better observed when the packet loss gets worse.
Of course, it is not a problem in a stable wired environment and is easy to observe in a mobile wireless environment.

For reference, in Samsung LTE / 5G environment, the signal quality can be checked by pressing * 123456 # on the keypad. (only for Samsung pads)

In System / Debug Screen
-RSRP> 90db, SINR> 5dbm, CQI> 5 seems to be stable now, and below this, it is observed that signaling frequently fails.

Note that you should test while streaming. Otherwise, the signal cannot be measured since the mobile network channel is connected in idle mode.

[getroot]

@alex-jhones

Thanks for the good comments.
In your comments I give my opinion on one thing.

OvenMediaEngine was developed with the assumption that it works in a public network environment like a normal streaming server.

Player connecting to OvenMediaEngine can receive data without TURN server even in Symmetric NAT environment. To do this, the OvenMediaEngine does not connect to the candidate that the Player sends in the signaling stage, but waits for the STUN ping packet from the player to arrive and transmits the data there. This is a possible scenario when OvenMediaEngine is running on a public network.

[getroot]

@alex-jhones
Thank you for applying to be a contributor. We'll open up our performance test tool soon and talk to you.

[getroot]

We have released a tool to test OvenMediaEngine's WebRTC streaming load and performance. This is currently a very low quality code. We suggest that we all work together to improve this. We ask for help. Thank you.

https://github.com/AirenSoft/OvenWebRTCTester

[getroot]

A performance testing tool has been launched.
#507