ecodis
efficient coding of digital signals

  go back to the home page  Home       Audio       Video       About the Author 

 by C. Helmrich


Video coding projects

    ecodis :: Efficient Video Codecs


Video codecs which I worked on:

link to sub-page  MPEG-I VVC / ITU-T H.266
Versatile broadcast/streaming codec



Other state-of-the-art codecs:

link to sub-page  MPEG-5 Essential Video Coding
Alternative MPEG codec for 4K video

link to sub-page  MPEG-H HEVC / ITU-T H.265
Currently utilized codec for 4K video

link to sub-page  Alliance for Open Media AV1
General-purpose codec for streaming



Comments, further information:

link to sub-page  My comments on video coding
More coding gain hard to implement

link to sub-page  HEIF and AVIF for still images
Latest-generation still picture coding

link to sub-page  Further codecs and resources
Other video codecs, further reading



In signal processing, data compression, source coding, or bit-rate reduction involves encoding information using fewer bits than the original representation. Compression can be either lossy or lossless. [...] The process of reducing the size of a data file is often referred to as data compression. In the context of data transmission, it is called source coding (encoding done at the source of the data before it is stored or trans­mitted) in opposition to channel coding.

link to external web page Wikipedia page on data compression, 2017



Storing or transmitting contemporary ulta-high-definition (UHD) digital video content in uncompressed form is virtually impossible due to the extremely high data rates; only one second of link to external web page HDR video with 3840×2160 pixels at 50 frames per second would fit onto a CD. Therefore, efficient lossy coding with very good visual quality even at very low data rates is even more important than in link to sub-page audio applications. This implies that a maximum of redundant and irrelevant information must be removed during the coding.

   On this page, the four most efficient newest-generation video coding standards are introduced. The first one, whose specification I am involved in, has just been finalized.



MPEG-I Versatile Video Coding (VVC), also Standardized as ITU-T H.266
2016–2020

   VVC, also known as link to external web page H.266, is a flexible general-purpose video coding specification recently standardized by ISO/IEC and ITU-T [link to external web page source]. Developed by the Joint Video Experts Team (JVET), the final VVC version is intended to exceed all existing standards (most notably, the three mentioned link to sub-page below) in compression performance at the same subjective reconstruction quality, with only a moderate increase in decoding workload.

   Since the VVC standard is still new (it was completed in July 2020), I cannot present any comparisons between VVC and other state-of-the-art video codecs. However, I can report from the April 2018 JVET meeting in San Diego that some proposals submitted for standardization in February 2018 achieved notable compression efficiency gains near 40 % over VVC's predecessor (see link to sub-page below) at acceptable decoder complexities of 3–4 times that of the state of the art [link to external web page source]. Therefore, at that time I expected the finalized version of the standard to provide bit-rate savings of at least one third at a decoding workload even closer to that of current video coding standards (factor of 2).

Update June 2019 A formally performed link to document subjective test indicates that, for HD and UHD standard dynamic range content, even in its by then unfinished state, VVC already offered the same visual coding quality as its predecessor, link to sub-page HEVC/H.265, with 36–40 % lower bit-rate. The fundamental compression principle applied in Versatile Video Coding, block-based transform coding of spatiotemporal prediction residuals (also called hybrid block transform coding), is identical to that used in HEVC but improved in many details:

  • partitioning now allows for rectangular in addition to HEVC's square-shape blocks,

  • prediction includes affine, triangular, intra-matrix, and luma-to-chroma predictors,

  • residual transformation was extended to more types and a two-stage technique,

  • quantization is now link to external web page trellis-based and configurable over a wider bit-rate range,

  • entropy coding was improved for lossy, lossless, and partial-lossless applications,

  • in-loop filtering for deblocking was improved, adaptive Wiener filtering was added.

Moreover, decoder-side motion compensation refinement tools were included to further increase the coding efficiency especially for videos containing strongly moving objects.

   During the first stages of the VVC development, I proposed an encoder optimization algorithm improving the subjective video coding quality for some content and presented my work in link to external web page Macau, link to external web page Ljubljana, and link to external web page Marrakech. At the Ljubljana meeting, I also suggested adding support for a new 10- and 12-bit packed YUV/RGB image and video storage format to the VVC code base [link to external web page report], a description of which is given below. Later during the standardization, I contributed to better in-loop filtering and the joint transform coding of the chroma components in color images and videos [link to sub-page reports].


PYUV PDF thumbnail
link to document A Packed Planar RGB and YUV Format
for
Uncompressed Storage of HDR Images / Videos


   VTM, the VVC reference encoding/decoding software, is publicly accessible link to external web page here, and the draft specification text (currently version 10) is freely available via link to document this link. Since May 2019, VVC achieves objective efficiency gains (in terms of Bjøntegaard delta rate over its predecessor, the link to external web page HM reference software) of about 24 % for still pictures (1.8x decoder runtime of HM) and more than 34 % for random-access videos (1.7x decoder runtime of HM) [link to document table]. The stable version, due July 2020, will improve upon this only by another 2–3 percent [link to document source] but may encode and decode a bit faster. (Update June 2020 According to my link to document last contribution to the VVC standardization, a crosscheck report, the final version of the VVC standard provides around 39 % rate re­duction over HEVC in the random-access use case, thanks to a last-minute change.) I'll update this section once the results of the VVC verification test have become available.

Update Oct. 2020 The results of the SDR UHD verification test (3840×2160 pixels and 30 or 60 fps) have now been link to document published. Two VVC encoders, VTM and link to external web page VVenC, were evaluated against HM on 5 representative 10-second video sequences and at 5 similarly spaced quality points. The pooled quality scores (averaged per encoder and quality/rate point) are illustrated below. They confirm the VVC-vs-HEVC bit-rate savings observed in the 2019 visual test (see above) and show that excellent quality (MOS > 8) is reached

  • above 8.1 Mbit/s with HM or a bit less than that with a visually optimized encoder,

  • above 5.0 Mbit/s with VTM for average bit-rate savings of roughly 40 % over HM,

  • above 3.8 Mbit/s with VVenC for 53 % rate savings over HM and 24 % over VTM

on typical high-resolution video content. Especially with regard to VVenC, this is a great achievement since this encoder is also the fastest of the three. Besides contributing to VTM, I helped develop VVenC in 2020 and implemented perceptual optimizations based on the link to sub-page XPSNR model. The 43 % rate savings of VTM over HM (when averaging across the link to document whole quality scale) underline the successful completion of the VVC standard. A press release mentioning this part of the verification test is archived link to external web page here.

Figure 1. Results of the
VVC verification test for
SDR UHD content, JVET,
2020.  Blue/lower-right:
HEVC (HM enc. 16.22),
orange/inbetween: VVC
(VTM enc. 10.0), gray/
upper-left: VVenC 0.1.
MOS scale: 0-2 bad, 2-4
poor, 4-6 fair, 6-8 good,
8-10 excellent quality.
The thin gray line shows
the expected quality for
later versions of VVenC.
(Values pooled from the
per-sequence MOS data
as they are illustrated in
doc. JVET-T0097, which
are courtesy of M. Wien
)
VVC verif. test results for SDR UHD (click to view full-sized)


The latest stable, tested VVenC release, version 1.11.0 of early 2024 which, like VTM, is publicly available under a clear open license, can be accessed via the following links. As of version 0.2.1, VVenC features a decently performing single-pass or two-pass rate control mode. (Update In version 1.8, the single-pass rate control mode was improved further, and in version 1.11, a tentative rate capped constant quality factor (CQF) mode was added.) The VVC specification text has been published not only as ITU-T standard link to external web page H.266 (freely downloadable) but also as ISO/IEC international standard link to external web page 23090-3.


VVenC logo
link to external web page Download VVenC 1.11.0
,   link to external web page VVenC information page
open repository on GitHub, Fraunhofer's web page for VVenC


An optimized, fully standard compliant open-source decoding counterpart of VVenC, called VVdeC, is link to external web page also available. It's also worth noting that commercial availability of another optimized VVC software decoder implementation, offering real-time decoding of VVC streams with up to 8K resolution, was announced by link to external web page Sharp in December 2020.




MPEG-5 Essential Video Coding (EVC), an Alternative 4K Video Codec
2018–2020

   Around the April 2018 JVET meeting in San Diego, where the first draft of the VVC referencec software and specification text was agreed upon (see link to previous section above), the Motion Picture Experts Group (MPEG) decided to initiate work on a separate, more constrained (in terms of development duration and included technology) video coding solution, to be completed and standardized in mid 2020 under the name MPEG-5 Essential Video Coding (EVC). More details and the motivation behind this approach are given link to external web page here.

   In November 2018, two solutions were submitted to MPEG in response to its Call for Proposals (CfP) on new video coding technology with «simplified coding structure and an accelerated development time of 12 months» [link to external web page source]. Two months later, at its January 2019 meeting, MPEG evaluated both proposals [link to external web page report] and selected the one by Samsung, Huawei, and Qualcomm (a description of which is provided link to document here) as the starting point of the EVC standardization. The relevant EVC documents are provided on the following web pages. Note that an MPEG user name and password are required to access these pages, which implies that this standardization is essentially closed-source.


EVC logo
link to external web page EVC Test Model (ETM)
,   link to external web page Working Draft
EVC draft reference software, specification text


   In its first revision, MPEG-5 EVC roughly matches the joint MPEG-H/ITU-T link to sub-page HEVC in both objective (PSNR) and subjective (visual quality) performance when operated in its so-called Baseline profile setup, at least according to the CfP evaluation. Its Main profile configuration, however, was verified to already deliver 24 % better coding efficiency than HEVC, which may increase by a few percent until the end of EVC's development. (Update Oct. 2019 ETM 3.0 provides about 26 % rate reduction over HEVC [link to document source], which should be very close to the final performance that this coding standard will offer.) Note that this value remains roughly 13 % short of the latest results for MPEG-I link to previous section VVC. The next few years will show which of these two codecs will achieve a wider market adoption. I will update this section once the EVC codec software has been published. By then, the specification text will have become available as ISO/IEC standard link to external web page 23094-1.



MPEG-H High Efficiency Video Coding, also Standardized as ITU-T H.265
2010–2015

   Employing the finalized High Efficiency Video Coding (HEVC) specification in ISO/IEC   link to external web page 23008-2 and ITU-T link to external web page H.265 currently is the most efficient way to compress moving pictures, especially high-resolution HD and UHD video. Developed mainly between 2010 and 2013, with some screen content and 3D coding extensions added after 2013, HEVC achieves an increase of about 50 % in perceptual compression efficiency over previous coding standards like H.264 [link to document source]. In other words, averaged across several coded video sequences, HEVC provides roughly the same subjective video quality as the older coding formats, and it does so using encoded bit-streams which are only half as large. This performance boost is what allowed, for the first time, the delivery of high-quality UHD video to consumers via link to external web page broadcasting, link to external web page streaming, and link to external web page UHD Blu-Ray disc.

   As of 2018, hardware-based HEVC decoding is supported by most TVs, set-top boxes, video players, computers, tablets, and even smartphones. The best freely and publicly usable HEVC encoder is maintained by the x265 project team and is located here:


x265 logo
link to external web page x265 web site
, link to external web page x265 software project
project sites with link to the x265 source code


Of course, the HEVC reference encoding/decoding software is also publicly link to external web page available. HEVC, as described link to previous section above, is the predecessor of the VVC standard, and most of its underlying technology can still be found in the current draft of the VVC link to document specification. In fact, all visual codecs discussed on this page use the exact same algorithmic building blocks which define a modern hybrid block transform video codec like HEVC. These are

  • a partitioner segmenting each component of the input into nonoverlapping blocks,

  • a prediction stage attempting intra- or inter-picture prediction of each input block,

  • a residual transform converting the prediction error into a spectral representation,

  • a quantizer mapping the residual transform values to a smaller set of coefficients,

  • an entropy coder applying lossless compression to the quantized coefficients, and

  • a few postfilters reducing blocking, denoising and ringing artifacts upon decoding.

Some codecs also add encoder-side prefilters to complement the decoder's postfilters. Note that modern link to sub-page audio codecs employ the same elements and that, in both audio and video coding, a second prediction stage may be used before or after the quantizer.



AOMedia AV1: A General-Purpose Codec for Internet Video Streaming
2015–2020

   The AV1 video codec, jointly developed by the Alliance for Open Media (link to external web page AOMedia) between 2015 and 2018, is a general-purpose format for Internet streaming based on well known technology. The video compression capability of AV1 is realized primarily with coding techniques derived from VP9/VP10, Daala, and Thor [link to external web page source]. Inside a WebM container, audio compression support is added through the link to sub-page OPUS codec [link to external web page source].

   The IETF is expected to adopt AV1 as the Internet Video Coding (NetVC) standard in late 2018 [link to document source] alongside the OPUS codec, which has already been standardized in link to external web page RfC 6716 in 2012. I anticipate broad hardware decoding support for AV1 in late 2020. Note that software decoding is already provided, even on Windows [link to external web page source], and work on a BSD-licensed optimized decoder called link to external web page dav1d has progressed as well. The current versions of the AV1 specification and software are available at these pages:


AV1 logo
link to external web page AV1 specification page
link to external web page AOMedia page
AV1 bit-stream & decoding process specification


   Surprisingly, the subjective performance of the AV1 codec in comparison to its latest competitors, link to previous section VVC and link to previous section HEVC, is relatively inconsistent. In some independent tests, AV1 matched the coding efficiency of HEVC [link to external web page source], while in others, the codec was objectively and subjectively inferior to HEVC [link to document source]. This can be attributed to the use of different encoder speed presets in the evaluations: for HEVC-like performance, a very slow AV1 encoder preset must be used [link to document source]. Moreover, the default encoder configuration for random-access scenarios is a bit different from that of other codecs, making direct codec comparisons difficult. However, since there is clear evidence that the precursor to VVC, known as Joint Exploration Model (link to external web page JEM), outperforms both AV1 and HEVC and also encodes faster than AV1 [link to external web page source 1, source 2], it is obvious that VVC will, indeed, become the most efficient video codec during the next few years. Some online documentation of AV1's most interesting and innovative coding tools can be found link to external web page here and link to external web page here. An overview of all coding tools is given in link to document this paper.

Update Nov. 2018 As link to document this image indicates, the speed of the AV1 reference encoder has recently been increased by at least a factor of 60 without a significant reduction in coding efficiency [1.6 %, link to external web page source], so it seems that a runtime and coding gain similar to that of the HEVC reference encoder will, indeed, become possible with AV1 soon. This observation is also supported by a May 2019 test by the BBC, summarized link to external web page here. Still, the efficiency of the VVC reference software clearly remains out of reach for AV1. This shortcoming will be addressed by link to external web page AV2, whose standardization will begin in a few years when support for AV1 has been widely deployed to consumer devices [link to external web page source].



Summary: More Coding Gain Possible but Hard to Implement Efficiently
Nov. 2018

   The current VVC standardization (see link to previous section above) indicates that further gains in image and video coding performance are still possible. However, given the order of magnitude increases in encoding runtime of both VVC and AV1, I feel that we are rapidly leaving the path of reasonable gain-efficiency tradeoff followed for so long: with next-generation standards, coding of a single 4K image on one processor takes up to half an hour, and the hardware requirements especially for moving-picture coding are substantial. For this reason, most of the often promising but experimental coding tools in JEM (like link to external web page FRUC, for example) won't make it into VVC: their algorithmic complexity and/or fast memory de­mands are just too high for usable implementations in both hardware and software. It's true that encoding can now be performed highly parallelized in the link to external web page cloud, but spending a year of aggregate computing power on one hour of UHD video is clearly not efficient and, so far, not environmentally friendly [link to external web page article]. Don't forget the countless coding-decoding iterations performed by the various participating companies during experiments towards a codec's standardization itself (more and more of which need to be run due to the vanishing potential for further coding gain)! And remember that the final HEVC/H.265 reference encoder was only three times slower than its AVC/H.264 ancestor [link to external web page source]!

   Therefore, aside from working on speeding up the new-generation video encoders by at least an order of magnitude, I believe it's time to reconsider the current approach in video coding development. Some experts at, e.g., link to external web page Netflix share this vision and call for innovation «beyond block-based hybrid video coding» as outlined link to previous section above. If that means using more extreme measures like large 4D-DCTs or CNNs, I disagree since, in my view, the computational burden for a competitive level of performance will likely be even more problematic than in today's codecs. If, however, the idea is to refrain from squeezing 9% more coding gain (i.e., statistical redundancy) out of existing block-based schemes, and instead to further exploit the inaccuracies of human vision in the design of image and video codecs (using parametric tools as in link to sub-page audio, which we still hardly do), then I fully support that approach. In fact, with my link to sub-page current work I already do. I hope you do too.



HEIF/MIAF and AVIF: The Two Latest-Generation Still-Image Codecs
2015–2019

   Still-image coding, like video coding, has come a very long way since the early days of link to external web page T.81/JPEG and link to external web page H.262/MPEG-2 about a quarter of a century ago. Recently, two additions to the long list of image coding specifications have emerged, namely, single-picture constrained variants of HEVC, called High-Efficiency Image Format (link to external web page HEIF), and of AV1, named AV1 Still Image File Format (link to external web page AVIF). An extension of the HEIF standard known as Multi-Image Application Format (link to document MIAF) is currently being finalized as well, and as if that weren't enough, the JPEG committee is also working on a novel still-picture coding standard, referred to as link to external web page JPEG XL (the L means longterm), to be finalized in late 2019 [link to external web page source]. (Update Oct. 2019 This milestone has been moved to April 2020 [link to document source]) All these contenders have in common that they provide efficient support for high-resolution, HDR, and wide color gamut (WCG) as well as (partially) transparent image content. JPEG XL also provides means for lossless transcoding to and from legacy JPEG, PNG, and GIF compressed images, which is a very useful feature in my opinion. I will update this page with further details on each coding specification and comparative demos once evaluation software for all of these coding standards has become publicly available. For now, I can recommend link to external web page this interactive, recently updated picture coding demo by Thomas Daede. See also link to external web page here and link to document here.

Figure 2. Basic
evolution of still
image coding in
the age of the
Internet. Top left:
JPEG (1992), top
right: JPEG 2000
(2000), bottom
left: HEIF (2015)
and bottom right:
original picture
(almost matched
in visual quality
by modern image
codecs like AVIF,
JPEG XL or VVC).
70:1 compression
(768:11 kilobyte)
was chosen for all
illustrated coders.
Notice how block-
ing and blurriness
vanishes from top
left to lower right.

(Fig. Lena image)
JPEG coded (Gimp)  JPEG 2000 coded (OpenJPEG)  HEIF coded (Gimp)  original Lena image 



Further, Lesser Known Video Codecs and Links to Additional Resources
2019–2020


  •  AVS logo
    AVS3, China's latest-generation Audio Video System standard. Tech-
    nically, its almost finalized phase 2 version of the video coding part
    seems to be quite similar to link to previous section VVC, with compression performance
    (efficiency, runtime) comparable to that of link to previous section EVC. See link to external web page this page.

  • A link to document PDF with presentations about (in that order) AVS3, AV1/AV2, EVC and VVC,
    given during the panel discussion at the 2019 Picture Coding Symposium (PCS) in
    Ningbo. Has some interesting technical and statistical details about these codecs.
    An even more link to external web page detailed presentation of VVC was held at the ICIP in late 2020.

page last modified in Feb. 2024, updated VVenC & JVET links

Copyright © 2008–2024 Christian Helmrich :: Privacy