JCCP: successfully refactoring the DASH-IF conformance tool
Overview of the tool
The DASH-IF Conformance Software is an indispensable tool to validate MPEG-DASH content against various specifications such as ISO/IEC 23009-1 and the DASH-IF Interoperability Guidelines. Early in its development, the Conformance Software was made available online to DASH-IF members and then to the whole industry.
Over the years the Conformance Software gained both in popularity and in compatibility, making it essential when deploying new content. Organizations such as DVB, CTA WAVE, Dolby, and HbbTV have extended the Conformance Software for testing content according to their specifications and profiles.
The Conformance Software is an integral part of the content creation progress and tightly integrated in the workflow of maintaining a database of test vectors. Moreover, it plays a crucial role in the maintenance of the DASH-IF open-source reference player dash.js. Issues and unexpected behavior reported by dash.js users are often related to wrong content authoring.
"The DASH-IF Conformance Software helps in identifying issues in the content without the need to dive deeper into the dash.js code." Daniel Silhavy, principal dash.js maintainer
The usage of the Conformance Software is straightforward, a user interface provides input fields for the required parameters and the various possible options and supported profiles:
Refactoring
By the end of 2020, the number of daily requests reached 200 and over 1000 unique visitors per year. As a result, the server started to become overloaded. With so many features having been stacked, the service started to become unreliable, regressions were spotted, and adding new features became increasingly costly. Welcome to technical debt.
Consequently, the DASH-IF Test and Conformance Working Group evaluated options to improve the current situation towards moving to a modern and more maintainable project base. As a result, a call for proposals was emitted and Motion Spell, which had decided to team with Fraunhofer FOKUS, won the call.
We’ve spent 2022 updating and re-arranging the code without changing the functional scope. A core team of 4 people was created (Romain Bouqueau and Erik Zandvliet for Motion Spell, and Daniel Silhavy and Alexander Futász, later joined by Louay Bassbouss and Fritz Heiden for Fraunhofer FOKUS). Here is what we put in place:
- We set up internal weekly calls to discuss and coordinated the tasks.
- We set up monthly meetings with the sponsors to discuss the progress and next steps as well as discuss potential questions.
- We communicated with members of the different sponsoring organizations through dedicated workshops.
Achievements
On the technical side we:
- Improved the documentation and moved all the documents to Github
- Added CI/CD scripts as Github actions for automatically running unit tests, verification tests, creating the API documentation and rolling out new versions of the software
- Added support for running the Conformance Tools in a Docker container
- Added a Command Line Interface for easy usage of the Conformance Software
- Added a REST interface for easy integration with a new UI and content creation workflows
- Refactored the existing code to move to a mono repository, moving all the tests to dedicated files and implementing a separate module file for each of the different submodules
- Developed a management dashboard to visualize verification and code coverage results. The management dashboard is tightly integrated with the new CI/CD scripts and allows us to identify issues in the code and tests that are not covered by any of the existing test vectors.
- Improved the existing segment validator in terms of stability and code design
- Implemented a new user interface that integrates tightly with the new REST interface
Screenshots of the improvements
Acknowledgments
This project wouldn’t have been possible without our sponsors (by alphabetical order): ATSC, DVB, CTA WAVE, DASH-IF, HbbTV. The level of transparency and trust throughout the project allowed us to explore unexpected (but still fruitful) territories. Special thanks to the persons who contributed to this journey: Thomas Stockhammer, Ian Medland, Iraj Sodagar, Rufael Mekuria, Mike Dolan, Jon Piesing, Mike Bergman, Madeleine Noland, Will Law.
Access to the online tool.
