Video Codecs: It’s All Alphabet Soup to Us

There are many things in life that one can never have enough of, like money, vacation days, good wine, music from your favorite band, and dessert. (Especially dessert!) Add bandwidth to that list. You depend on having “enough” bandwidth every time you send someone a photo from your smartphone, stream Hamilton to your TV, or upload a video to Facebook from your laptop.

Fact is, there’s a never-ending quest to use bandwidth more efficiently, one which the vast majority of us are blissfully unaware of. Groups of very smart people are constantly developing and refining video compression and decompression (codec) algorithms to ensure those selfies, Lin-Manuel Miranda’s music and lyrics, and cute kitten videos get to where they’re supposed to go with minimal signal degradation.

And codec developers can barely keep up. 27 years ago, we saw the first high-efficiency codec emerge to handle low-resolution video on optical disks – MPEG-1, produced for the short-lived compact disc interactive (CD-I) format. Video compressed with MPEG-1 looked okay on small CRT screens, but on larger screens, the quality was pretty bad. Although MPEG-1 and CD-I are fortunately distant memories now, the audio compression format they spawned – MPEG Audio Layer 3 or MP3 for short – lives on to this day.

MPEG-1 was followed by MPEG-2 in 1996 to encode digital video discs (DVDs) and to compress standard-definition and high-definition video for broadcast, cable, and satellite. It’s still in use today. MPEG-2 encoders have come a very long way in 20+ years and can pack two 720p HD programs plus a handful of SD programs into a single 6 MHz TV channel, all with acceptable visual quality.

Like rust, compression experts never sleep. 17 years ago, a newer and more efficient codec made its debut. MPEG-4 H.264 (aka Advanced Video Codec) promised 50% compression efficiency over MPEG-2…and delivered it! By that time, HD video was becoming widely adopted across a multitude of delivery platforms, but available bandwidth was slow to keep up. With the growth of video streaming a few years later, MPEG-4 H.264 became the codec of choice and is supported on everything from tablets and smartphones to laptops, smart TVs, camcorders, and DSLRs.

The only problem was that 4K video became the latest flavor by the turn of this past decade. And to make the job even more difficult, high dynamic range video with its associated wider color gamut was part of the 4K package. But bandwidth hadn’t kept up! Hence, the High-Efficiency Video Codec (HEVC) was rolled out in 2013, again promising 50% more compression efficiency over H.264. HEVC requires quite a bit of computing power to pull off that trick, but it works (and so does its close relative, Google’s VP9 codec). And it’s not cheap to license.

HEVC is used for the UHD Blu-ray optical disc format and for streaming 4K content from just about everywhere except YouTube, which being owned by Google employs the VP9 “open” codec. But the licensing costs spurred more tech types to come up with yet another codec, known as the Alliance for Open Media (AOM) AV-1 codec. This is a royalty-free codec that competes with HEVC but is intended solely for streaming video over Internet connections.

Since the people developing high-resolution video formats always seem to be a few steps ahead of the codec people, more codecs have been proposed in an attempt to catch up. Essential Video Coding (EVC, or MPEG-5) was developed as an alternative video codec for streaming and OTT, but with a streaming performance at least equivalent to HEVC. MPEG-5 Part 2, Low Complexity Enhancement Video Coding (LC-EVC), is yet another MPEG standard and is intended to provide enhanced compression efficiency for existing MPEG-based video codecs.

To top it off, the successor to HEVC H.265 is now preparing to take the stage. The Versatile Video Codec, or VVC, is designed for maximum compression efficiency across all compatible devices and platforms with a specific focus on applications like high-dynamic range, high frame rate video, the 360-degree video for virtual and augmented reality, and 8K UHD-2 video.

Using spatial-only image metrics for reference, VVC is about 40% more efficient than HEVC for UHD & HD compression. However, a VVC reference encoder has about ten times the complexity of an HEVC reference encoder, while a VVC reference decoder is about twice as complex as an HEVC reference decoder.

We should mention that HEVC, AV-1, EVC, LC-EVC, and VVC are software-intensive codecs, unlike the older MPEG-2 and H.264 AVC codecs that are still in widespread use. All five employ larger coding block sizes and need super-fast CPUs and plenty of memory to analyze and compress video streams.  By the way, H.264 AVC is no slouch – it’s had 26 updates since it was first rolled out in 2003.

Okay, so you’re getting a headache from all these abbreviations. (We are, too.) The takeaway here is that methods for enabling the transport of compressed, high-bit-rate video over everything from broadcast airwaves, Wi-Fi, 5G, and broadband are continuously being refined. You’ll go through life blissfully unaware of which particular codec is being used to stream The Marvelous Mrs. Maisel or let you watch Clemson’s and Alabama’s football teams slug it out on your iPhone. (You will, however, notice any impairments to video quality as a result of excessive compression and complain vigorously to your service provider about them!)

Will there ever be a unified codec? That’s the goal, but no one knows how, when, or even if it will happen. In the ongoing quest for efficiency, codec designers are now implementing artificial intelligence (AI) to perform the lightning-quick analysis of incoming video and decisions on which picture elements to compress, by how much, and for how long.

We already use a simple version of AI to dynamically adjust bit rates of multiple programs in a stream, based on constantly-changing available network bandwidth (dynamic stream shaping and adaptive bit rate encoding are two examples). It stands to reason that a unified codec – one based on advanced AI that can optimize the delivery of high-resolution video across any network or platform – should emerge at some point and rid us of the “alphabet soup” of codec formats.

Dream on…