Reference software, test-sets, and compliance checkers are important
Before the summer, we were invited to a private 4K demonstration from several vendors. The GPAC team had provided its Osmo player for HEVC playback with DASH on Android. There were also other demonstrations.
In particular we saw a booth with some HbbTV (the successor of Teletext). While it allows many neat stuff, HbbTV is mostly used to convey URL through AIT tables supported in GPAC. HbbTV is a deployment success. It's been deployed in Korea recently, in many other countries. The major TV manufacturers have integrated the specification.
Someone asked us a question: how would you test HbbTV?
Conformance
The standard is a common understanding for interoperability. How can a software editor check its implementation?
In the multimedia field, the standard describes the client's process. Therefore following the standard allow to make decoders or parsers (decoders which don't try to reconstruct the audio or video data they process) quite easily. The price for a decoder is 1-10$.
Encoders are hard to make. They need to generate a stream that is compliant with the standard. The number of encoders sold every year is much lower. The price for an encoder is 1000-10000$.
Encoders are easy to test. You need a parser which makes some checks. The checks are often written in the specification. Such a modified parser is called an analyzer.
Decoders are hard to test. You need to provide a set of decodable streams. These streams try to cover the whole specification. They are called conformance streams.
Such conformance streams are very convenient for hardware manufacturers: chip's production is expensive and shipped chips cannot be modified. Video standards starting at H264/AVC describes a syntax which allow decoded streams to be bit accurate (i.e. two decoders will output exactly the same image for the same input stream). Older codecs (such as MPEG2 video) had rounding issues which didn't allow such certification mechanisms.
Conformance streams for HbbTV
Let's come back to our primary subject: HbbTV. How do you test HbbTV?
HbbTV is a restriction over a set of standards. So checking it would require to be able to test each of the other specifications' subsets (MPEG2-TS, MPEG-DASH, etc.). This means that as technologies are made of an assembly of other technologies, existing actors are best positioned.
Another aspects is that encoders already output almost-HbbTV streams. But decoders may be more complex to upgrade. Therefore the testing should be made more intensely on the decoding side, e.g. TV manufacturers... which is the hardest side.
Our conclusion was:
- Encoders testing: it is easy to test encoders by extending existing analyzers.
- Decoders: it is difficult to test decoders, because it would require to generate many streams. Probably by hand at first.