How 100ms Tests for Network Reliability

From virtual classrooms to business meetings, shopping to dating apps, video is quickly becoming the de-facto communication mode online.

Innovative developers and product thinkers are looking to create engaging live experiences in their applications. So naturally, it's critical that the audio-video SDK they build these experiences on top of provides a stable, extensible, and scalable bedrock.

Among the many factors to consider before purchasing an audio/video SDK, network reliability stands out. After all, nobody enjoys running a twenty-minute monolog on a video call only to realize your network was down the entire time…

Testing Network Reliability for Real-World Scenarios

In this article, we've downloaded, deployed, and tested the reliability of the 100ms React SDK. To do so, we designed a series of tests that simulate common scenarios in real-life. Of course, since that's not fun enough, we decided to unleash our “full crazy” by battle testing each round against extreme conditions.

The tests verify how the 100ms SDK fares across three parameters that define network reliability: low bandwidth, network blips & network switching.

Why Network Reliability Matters

In the real world, individuals often have to deal with unstable or less-than-ideal network conditions. This happens when:

• moving from one network area to another while traveling
• suddenly experiencing slow internet because of an expiring data pack
• suddenly experiencing call disconnection for a few seconds due to issues in the larger infrastructure

Network connectivity issues occur more often than we think. Video SDKs need to, at best, be resilient to these issues, and, at worst, provide developers with tools to deal with them gracefully.

Deployment

100ms has a sample React app (100ms 2.0 Sample React App) meant to facilitate the testing of its SDK. We deployed it on Heroku and exposed it to a few commonly occurring end-user scenarios.

https://github.com/100mslive/100ms-web

We had to generate some credentials from the 100ms console and then deployed this example React app on Heroku.

The SDK was deployed and tested on the Chrome browser running on macOS Monterey.

Conditions and cases

All these tests were 1:1 calls, performed with 2 people in the room. A few details about each test before we get into the results:

1. Low Bandwidth Test

   Network speed varies across devices. For instance, users operating on 4G mobile data often experience a volatile network, as it tends to vary in speed and stability. In this test, we checked how 100ms handles calls with varying connection speeds on low bandwidth.

2. Network Blip Test

   Network crises can happen in the middle of a call. In this test, we checked how 100ms handles the sudden loss of network connectivity followed by automatic reconnection.

3. Network Switching Test

   It is common for users to switch between networks inadvertently. For example, they might be on a call while moving between state lines or from a city to the countryside, which may affect network strength.

   Network switching usually occurs when you move away from the range of one network to another or when you switch between your available networks for a higher speed. In this test, we checked how 100ms handles a network switch.

1. Low Bandwidth Handling/Management Test

Audio/Video applications need to handle usage across varying network bandwidths. In this section, we monitor how 100ms handles calls for users with low bandwidth.

Testing Methodology for the Low Bandwidth Test

We used Network Link Conditioner to emulate different network conditions. We set the ideal resolution to 640x360, and tested the app on 4 different configurations: 300 Kbps, 500 Kbps, 800 Kbps, and 1 Mbps, switching from one to another in the middle of a call.

Test Results

The 100ms SDK handles the drop in bandwidth by prioritizing audio/video upload for other peers instead of audio/video download.

• If the network is adequate (800 Kbps), the video of active or recent speakers continues to be visible. The audio remains perfectly functional.
• If the network is poor, only peer audio is functional while their video degrades.
• On the device facing poor network conditions, the video is somewhat degraded but not entirely non-functional. At lower bandwidths (500Kbps and 300Kbps), audio quality remains functional for all other peers in the meeting and only sees a drop for the attendee experiencing bandwidth constraints.

LOW BANDWIDTH TEST	Observation 1	Observation 2	Observation 3
1 Mbps	Audio/video okay for both peers	Audio/video okay for both peers	Audio fine; video blurry Other peer: audio & video fine
800Kbps	Video freezes; audio bad Other peer: audio, video fine	Video freezes; audio drops in quality Other peer: audio, video fine	Video is frozen; audio quality drops slightly Other peer: audio, video fine
500Kbps	Video froze; audio breaking Other peer: audio fine; video slightly blurry	Video completely frozen; audio broken & distorted Other peer: audio fine; video blurry & lagging intermittently	Video frozen; audio distorted & breaking Other peer: audio fine; video blurry
300Kbps	Audio a bit distorted; video breaking & frozen Other peer: audio fine; video blurry	Audio distorted; video frozen Other peer: audio lagging; video blurry & lagging	Frozen video; bad audio Other peer: Audio fine; video blurry & lagging

2. Network Blip Test

In this section, we check how 100ms handles call connectivity when a user’s network connection gets switched off, or drops for several seconds.

Testing Methodology for the Network Blip Test

First, we check the call by switching off the internet connection for 10 seconds. This is done by toggling the connected wifi network from the menu bar and connecting back by re-toggling the same.

Then, we iteratively repeat the same test for 20, 30, 45, and 60 seconds. While doing so, we observe the state of the call connection and how the app behaves during disconnection.

Test Results

The 100ms SDK reconnects every time when internet is disabled for 10, 20 and 30 seconds. When switched off for 45 and 60 seconds, the app tries to reconnect for 35s before disconnecting entirely.

NETWORK BLIP TEST	Observation 1	Observation 2	Observation 3
Turned off for 10s	Audio/video reconnects after 2s	Audio/video reconnects after 2s	Audio/video reconnects after 2s
Turned off for 20s	Audio reconnects after 14s; video reconnects after 3s	Audio/video reconnects after 15s	Audio/video reconnects after 14s
Turned off for 30s	Audio/video reconnects after 6s	Audio/video reconnects after 6s	Audio reconnects after 6s; video reconnects after 7s
Turned off for 45s	Audio/video disconnects after 35s with error message Network Connection Lost	Audio/video disconnects after 35s with error message Network Connection Lost	Audio/video disconnects after 35s with error message Network Connection Lost
Turned off for 60s	Audio/video disconnects after 35s with error message Network Connection Lost	Audio/video disconnects after 35s with error message Network Connection Lost	Audio/video disconnects after 35s with error message Network Connection Lost

3. Network Switching Test

Apps are often exposed to different network conditions in the real world. In this case, we’ve tested how the 100ms SDK reacts when the app moves from one network strength to another.

Testing Methodology for the Network Switching Test

This test checks how 100ms handles connection when switching from one network to another. We tested the app in 3 Wi-Fi networks: 2.5G and 5G from the same router, and a mobile hotspot.

• To start the call, we connected to the Wifi 2.5G network.
• Then, we switched from Wifi 2.5G to Wifi 5G.
• Then, we switched back to Wifi 2.5G.
• Then, we repeated the same process, switching to and from Wifi 2.5G and the Mobile Hotspot.

We waited for the call to reconnect during every network switch and monitored the time (in seconds) it took for the reconnection to occur.

Some of the flawed behavior in the ‘Wifi 2.5G to Hotspot’ test section might be due to the unstable 4G network connection we experienced while testing.

Test Results

The 100ms SDK manages to reconnect after every network switch. Sometimes the video reconnects after the audio. The average reconnection time when switching within the same network is 9.1s for audio and 10s for video. The time for reconnection between 2 different networks is 19.2s for audio and 13.8s for video.

NETWORK SWITCH TEST	Observation 1	Observation 2	Observation 3
Wifi 2.5G to 5G	Audio/video reconnects after 12s	Audio reconnects after 6s; video reconnects after 10s	Audio reconnects after 10s; video reconnects after 11s, but quality is slightly lowered
Wifi 5G to 2.5G	Audio/video reconnects after 10s	Audio/video reconnects after 10s	Audio/video reconnects after 7s
Wifi 2.5G to Hotspot	Audio/video reconnects after 11s	Video reconnects after 19s; audio didn’t reconnect and the peer automatically dropped off after a few seconds	Audio reconnects after 27s; video reconnects after 7s
Hotspot to Wifi 2.5G	Audio/video reconnects after 26s	Audio/video reconnects after 15s	Audio reconnects after 17s; video reconnects after 5s

Closing Notes

Given the centrality of reliability when it comes to choosing an audio-video SDK, we decided to lay all our cards on the table and reveal exactly how we fare in diverse network, bandwidth and end-user circumstances. Across all tests 100ms fared well under regular usage conditions. In some cases, like bandwidth drops, the SDK allows for graceful handling of degradation issues.

Of course, as an SDK provider, we pride ourselves for making 100ms even more bullet-proof, so we can’t wait to elegantly solve across all these conditions and meet you again with even more aggressive scenarios.