A streaming services provider hired a global telecommunications company to provide the necessary infrastructure to be able to stream its content to a vast consumer audience in Asia. The telecommunications company needed to validate that the content delivery network (CDN) it chose as part of this infrastructure would be able to meet the performance expectations of the streaming service.
In order to do so, it was important to make sure it could achieve its desired level of performance, (which was initially determined to be 100 Gbps per second) as well as test how the CDN would respond, to ensure it would not become a bottleneck as thousands of viewers watched the video.
Solution: Large Scale Video Streaming Load Testing
To help the telecommunications provider with this project, BlazeRunner, the authorized reseller of Broadcom’s cloud performance testing platform, BlazeMeter, contracted Abstracta to devise and execute this large scale load test.
To begin, one of our senior performance engineers created a JMeter script to test the CDN behavior, distributing the load between the HLS and DASH protocols and also different levels of streaming quality for both.
This load scenario allowed for the simulation of the different types of users who would consume the video content while it went through the CDN.
The test script considered the list of specifications that the streaming provider desired to achieve in terms of bandwidth consumption, the pattern they wanted to follow (a certain percent of HLS and Dash), etc.
We first created the JMeter script and ran it locally, followed by some proofs of concept that we generated to calibrate how many engines were needed to achieve the desired scenario, (100 Gbps).
Our engineers and the CDN provider together determined that it was necessary to use 500 engines from multiple locations in order to simulate 85,000 concurrent virtual users (for the whole duration of the test), which gave us 8,200 requests per second (RPS) in terms of throughput.
Once the script was complete, the Abstracta engineer used the BlazeMeter cloud performance testing platform in order to generate a load streaming near 100 Gbps of video.
By the end of the test, we arrived at this load scenario:
- Max Throughput: 8,295 RPS
- Max Users: 85,000
- - Central Korea Central (Seoul, Azure)
- - South Korea (Busan, Azure)
- Singapore (Google, AWS, Azure)
- Duration: 10 minutes
- Ramp up: 5 minutes
- Load Distribution:
- - 90% DASH: 10% 200Kbps, 70% 1.5Mbps, 20% 7Mbps.
- - 10% HLS: 10% 400Kbps, 70% 1.8Mbps, 20% 8Mbps.
In order to achieve this configuration, we used Taurus (an open source test automation tool that extends and abstracts leading open source load testing tools), to define the test configurations and JMeter for distributing the load with the given pattern.
JMeter Video Streaming Plugin
In conjunction with using BlazeMeter and Taurus, to simulate the load, it was fundamental to utilize the HLS plugin (video streaming plugin) that a team of Abstracta’s in-house software engineers helped BlazeMeter to develop.
The plugin behaves in the same way that a typical video streaming client application works and enables you to download a file with a playlist and then load it in separate portions of video from the plugin.
The plugin has several useful features that make stress testing video streaming much simpler than before. It allows you to select subtitle and audio tracks, support both HLS and DASH, select the video variant to use, the amount of time, etc. all the while greatly simplifying testing such protocols and provides easy to use statistics and reports.
Using BlazeMeter and the video streaming plugin, Abstracta performance engineers were able to verify that the CDN would be able to handle the maximum expected load (85,000 users) without suffering any crashes or outages.
With these results and information about the robustness of the streaming service, the end client was able to move forward with its video streaming in South Korea, utilizing the infrastructure services it was provided by the telecommunications company and its CDN provider.