Streaming services like Netflix, Disney+, Amazon prime, etc. have seen a substantial rise in subscriptions in recent times. NBC has also unfurled its streaming site ‘Peacock’, making it available to Comcast subscribers. Nielson’s latest report reveals that the on-demand video streaming is up nearly 100%, which is significantly much more than the pre-pandemic time. The rise in the consumption and production of OTT video has brought about its challenges. Increased consumer choices lead to churn if the quality of viewing experience is poor. Therefore, it is critical to make Quality Control (QC) part of the end-to-end OTT process to ensure that high-quality content gets delivered across multiple platforms. This helps to avoid higher operational costs and delays in issue resolution, thereby increasing customer satisfaction.
How is Quality Control done?
Whenever we encounter stalling or buffering in a video, it can be due to various reasons. It can either be a single point of failure within one of the components or an integration failure between different components or low internet speed at the endpoint. Testing needs to be done at various points to identify the cause of the disruption. Every OTT provider adopts different practices for quality control.
Let us discuss the stages at which the providers generally do the testing with the help of the underneath diagram:
In a typical OTT system, the validation happens at three main points:
Test point#1 – Testing after encoding and ingestion
The ingested files are checked for their integrity and compliance to ensure that they are not corrupt and have been encoded to the standard that the downstream systems can consume without any issues. This quality assessment is done with the help of a perpetual quality assessment algorithm called VMAF (Video Multi-Method Assessment Fusion). VMAF is developed by Netflix and is made available as open-source. VMAF score varies between 0 to 100. Higher the VMAF score, better the quality.
Test point#2 – Testing after transcoding
During transcoding, every input stream or file results are converted into an array of outputs or profiles. Multiple renditions of the same stream or file are created at different quality or bitrates, also called profiles. Below are the tests performed at this stage:
- QoE (Quality of Experience) checks – Each stream or profile needs to be tested for its quality like rate, format, syntax, loudness, blockiness, etc.
- ABR (Adaptive Bit Rate) checks – Test is performed to ensure if the frames are time aligned across each of the profiles to enable seamless switching.
Test point #3 – Testing at CDN
This is the final point of testing. Below are the tests performed at this stage
- QoS (Quality of Service) checks – Here, the tests are performed to ensure the content is accessible over HTTP/HTTPS to catch any download delays/failures. All requests/responses(3xx,4xx,5xx) are passively monitored and logged. Content downloads are simulated in the network congestion environment to observe the behavior of the distribution server under stress conditions.
- QoE (Quality of Experience) checks – Basic audio and video quality checks done to check for blockiness, black frame, loudness, etc.
- Entitlement checks – Check if the file is accessible only by those entitled and if the unauthorized redistribution is prevented. The test is also done to decrypt each file and look at the audio and video available to subscribers (for every profile)
- ABR (Adaptive Bit Rate) checks – Test is performed to ensure if the frames are time aligned (between audio, video, and subtitle) across each of the profiles to enable a seamless switching
- Player control checks – This includes testing of play, pause, rewind and fast forward controls of players during content or ad playback, content bookmark validation for the played content and autoplay validation at the end of the video
- Video playback analytics validation- Validation is performed if the analytics calls are triggered on user actions like play, pause, full screen, etc.
Looking Ahead
OTT testing is still evolving, and so are the requirements for testing. Broadcasters and content owners are continuing to spend a significant portion of their revenue on acquiring content. However, this cannot be monetized until the content quality gets to attract and retain viewers.
The methods used need to be architecturally versatile and must allow content providers/broadcasters to figure out the most critical areas to focus on. An industry-driven by rapid change requires more than technical evolution. It needs what we call ‘revolutionizing.’ By deploying the complete QC automation and monitoring solution at the appropriate points from ‘Ingestion to Delivery’, broadcasters can deliver the best experience to all the viewers in the OTT world.
