Comparing 100ms' Call Quality to a Leading Video Conferencing Provider

Hello! If you’d like to read the condensed snippet of this, you can click on "the Tl;Dr" section on the left 👈

• Some Background
• About the Study
• Test Setup and Method
• Metrics Captured
• Results
  • Unlimited Conditions
  • Changing Bitrate
  • Packet Loss
• The Tl;Dr

Some Background

In ideal conditions, most platforms can run a successful call. It's under constraints that the platform has to make choices—like whether to reduce video FPS or resolution under reduced bandwidth to ensure the call still works for users. What differentiates platforms like 100ms and other leading video conferencing platforms is (a) the speed at which a platform detects and identifies the aspects of a video call that get affected during bad network conditions, and (b) the way it is designed to respond to these issues.

In this blog, we will compare the call quality of 100ms with one of the leading video conferencing platforms and SDK providers in the market, which we will refer to as Provider Z throughout this post.

Our north star metric at 100ms has always been clear: to be the best in call quality among live video APIs. This means ensuring that any call powered by 100ms works seamlessly, whether on a 6-year-old Android phone or under a 100 kbps network limit.

This clear goal has aligned numerous projects on the engineering and customer success sides over the last three years, encompassing tens of billions of minutes of call time. While we've been benchmarking our efforts internally against competing SDKs, we decided to bring in experts to conduct an independent test.

About the Study

We partnered with an independent lab with over 10 years of QA experience and a team of 500+ ISTQB-certified QA engineers. Their clients include Discord, Zoom, and Twilio. We trust their process and findings.

The lab ran a test comparing 100ms and Provider Z, the results of which we will cover in this blog.

Quick Notes on Test Setup and Method

• The study was to test the call quality for 1:1 calls. To simulate that, a 1:1 call was set up between 2 devices (both macOS M1) using each, Provider Z's pre-built app and 100ms' pre-built app
• An extra device was added as a silent participant to the call to ensure all calls went through a server-route rather than P2P. This is the architecture most commonly used in real-world implementations
• Capture scripts are initiated to monitor various performance metrics, including video and audio quality, delay, and network bitrate
• Receiver Video Capture is conducted using an Elgato Camlink (HDMI device) connected to a separate MacBook for precise screen recording
• Performance Data is collected from the test devices to monitor CPU, memory, and GPU usage
• Audio Capture is executed on a separate device to measure audio delay and quality using the Perceptual Objective Listening Quality Analysis (POLQA) standard
• The call is tested across 3 conditions:
  1. Unlimited: no restrictions on network
  2. Changing bitrate: Network bitrate was altered in predefined intervals: unlimited for 1 minute, 1 Mbit for 1 minute, 500 kbps for 1 minute, 200 kbps for 1 minute, and back to unlimited for 1 minute.
  3. Changing packet loss: Packet loss was introduced in increments: 5% for 1 minute, 10% for 1 minute, and 20% for 1 minute to simulate varying levels of network instability.

Metrics Measured

Video Metrics:

• Frames Per Second (FPS): Indicates the fluidity of the video stream. The more frames captured, the more smooth the end video will look

• Video Delay: Measures end-to-end latency between sent and received video frames

• Video Quality: Assessed using

  1. VQTDL (Video Quality with Temporal and Data Loss): Measures video quality with a focus on temporal aspects and data loss. A score >4 means clear picture.
  2. VMAF (Video Multimethod Assessment Fusion): Measures video quality, scored from 0 to 100, where higher meaning better video quality.

• PSNR (Peak Signal-to-Noise Ratio) Metrics: Measures the ratio between the maximum possible power of a signal and the power of corrupting noise. Higher the score, better the fluidity of the image

Audio Metrics:

• Audio Quality: Evaluated using POLQA, which provides a MOS (Mean Opinion Score) scale rating. 1 is the lowest and 5 is the highest
• Audio Delay: Measures end-to-end latency between sent and received audio signals

Network Metrics:

• Sender and Receiver Bitrate: Monitored to evaluate network consumption and efficiency
• Network Stability: Assessed during changing bitrate and packet loss conditions

Performance Metrics:

• CPU Usage: Monitored to assess the processing load on test devices
• Memory Usage: Evaluated to understand the memory footprint of each Video SDK
• GPU Usage: Measured to gauge the impact on graphics processing performance

Results

Unlimited network conditions

1. The quality of video is comparable between Provider Z and 100ms—VMAF score is comparable, 100ms has a marginally higher score in VQTDL.
2. 100ms has the lowest latency for both audio and video transmission. Provider Z’s audio lags behind almost twice as much as 100ms. 100ms audio quality is also better than Provider Z.

3. Provider Z is better optimized for overall transmission with lower bitrate consumption on both the sender and receiver sides, without compromising quality. But 100ms excels in performance, consuming very low CPU and GPU power compared to Provider Z.

Changing Bitrate

1. 100ms’ video quality drops immediately compared to Provider Z when the network is throttled. But as the network fluctuates, the quality of video is comparable between both platforms, with Provider Z performing marginally better than 100ms on both VMAF and VQTDL scores.

2. 100ms produces a more fluid video transmission compared to Provider Z, with higher frames captured per second and marginally higher PSNR scores.

3. Provider Z experiences alarmingly high delays in video transmission compared to 100ms. Provider Z video freezes up to 10 times as much as 100ms in poor network conditions.

4. 100ms also wins in audio quality in poor network conditions—its audio quality score is 3x higher than Provider Z. The audio delay is up to 10 times as much as 100ms.

5. Provider Z consumes less memory compared to 100ms, but 100ms wins out in overall performance with significantly less GPU consumption.

Packet Loss

1. Video quality is comparable, with similar VMAF and VQTDL scores.
2. Provider Z scores higher in PSNR and processes more frames per second, making Provider Z’s video more fluid during packet loss.

3. 100ms is stable at the start but as the packet loss increases, it experiences comparable video delay to Provider Z (albeit slightly lower). Provider Z freezes significantly less compared to 100ms.

4. While the audio quality is comparable between both platforms (100ms lagging slightly behind), Provider Z experiences significantly more latency in audio transmission as the packet loss increases.

5. 100ms is the clear winner in performance, beating out Provider Z in CPU, GPU, and memory consumption.

The Tl;Dr

Provider Z is a great consumer app, but the real question here is how well it serves as a platform for products to be built upon. The conclusion here is:

1. With network conditions changing, 100ms performs better. Provider Z video freezes up to 10x as much as 100ms. 100ms’ audio quality is much better. Both the video and audio latency are significantly higher for Provider Z compared to 100ms. This makes 100ms a more reliable live video platform even when the network is unreliable.
2. In high-traffic situations that cause network congestion, Provider Z can transmit more frames per second compared to 100ms. The video also freezes less than 100ms. However, it is important to note that this is achieved by compromising latency: 100ms audio and video have significantly lower latency compared to Provider Z. This means as more people join the call and packet loss occurs, the end users will not experience as much lag in audio and video with 100ms as they would with Provider Z.

If you have more questions about this study, or if you want to see 100ms in action, please get in touch with us.