JVET-U0100 Compression efficiency methods beyond VVC [Y.-J. Chang, C.-C. Chen, J. Chen, J. Dong, H. E. Egilmez, N. Hu, H. Huang, M. Karczewicz, J. Li, B. Ray, K. Reuze, V. Seregin, N. Shlyakhov, L. Pham Van, H. Wang, Y. Zhang, Z. Zhang (Qualcomm)]

This contribution presents changes and tools in the areas of intra prediction, inter prediction, transform and coefficient coding, in-loop filtering, and entropy coding added on top of the VVC standard. The implementation was done based on VTM-10.0 with added fixes related to MCTF, and it is reported to demonstrate −11.50%, −12.91%, −13.19% BD rate impacts for the luma and chroma components, respectively, with 243% encoder and 392% decoder run time in RA configuration using the VTM CTC.

Proposed features include the following:

• Max. BT/TT size is 256x256
• Intra prediction:
  • Multi-model LM (JVET-D0110)
    • Neighbouring samples are classified into 2 classes
    • Linear model is derived per class
    • Linear least square method is used
  • Gradient PDPC (JVET-Q0391)
    • Gradient PDPC method of horizontal/vertical modes is extended to other modes
  • Secondary MPM (JVET-D0114)
    • Primary MPM consists of 5 entries
    • Secondary MPM list of 16 entries is introduced
    • Shape dependent order of neighbouring blocks to construct the list
  • Reference sample interpolation (JVET-D0119)
    • 4-tap cubic interpolation is replaced by 6-tap filter
    • 6-tap Gaussian filter is used for larger blocks
    • 4-tap filtering is used instead of the nearest neighbour to derive extended reference
  • Decoder side intra mode derivation (JVET-O0449)
    • Intra prediction is derived as a weighted average between Planar and two derived directions
    • The first DIMD mode is stored with a block and is included into the primary MPM list of the neighbouring blocks
• Inter prediction:
  • Local illumination compensation (JVET-O0066)
  • Non-adjacent spatial candidate (JVET-L0399)
  • Template matching for MV refinement (JVET-J0021)
  • Multi-pass MV refinement
  • OBMC (JVET-L0101)
    • Top and left CU boundaries are refined using neighbour block motion
    • Top, left, bottom, and right subblock boundaries are refined using neighbour subblock motion
  • Sample-based BDOF
    • Instead of on a block basis, a BDOF refinement is derived per sample
    • 5x5 window around each sample is used to derive Vx and Vy refinement
    • On 8x8 subblock basis, whether to apply BDOF or not is determined by checking the SAD between the two reference subblocks against a threshold
  • Multi-hypothesis prediction (JVET-M0425)
    • Applied to bi-prediction in AMVP mode (for non equal BCW weights), affine AMVP, and regular merge mode
    • Up to 2 additional predictors are signalled or inherited for merge mode
  • 12-tap interpolation for luma
• Transform and coeff. coding:
  • CTU and max transform size is extended to 256 (for class A)
  • The number of LFNST sets is extended to 35 with 3 candidates
  • Full 64x64 LFNST matrix is used (no zeroing out)
  • 8-state DQ is utilized (JVET-Q0243)
  • Sign prediction is applied for up to 6 transform coefficients (JVET-D0031, JVET-J0021)
• Entropy coding:
  • The precisions for two probability states are both increased to 15 bits, in comparison to 10 bits and 14 bits in VVC
  • LPS range update process is modified
  • CABAC context window is defined for each slice type
• ALF:
  • Luma and chroma filter shapes are extended to 9x9
  • Classification is performed on 2x2 basis
  • Luma filtering is applied using 1 signalled F2 9x9 filter and 2 fixed F0, F1 13x13 filters
  • Weighting coefficients for F0 and F1 fixed filters are signalled
  • Different 2x2 based classifiers are used for F0, F1, and F2 filters, with directionality derived by comparing the ratio of horizontal, vertical, and diagonal gradients with a set of thresholds, which supports more edge strengths

Selection of tools was done based on complexity vs. performance assessment. The PowerPoint deck includes a table of tool-on test for some of the individual tools.

It was asked why a bilateral filter was not included, and the proponent responded that a version was tested, but was not performing well in combination with the modified ALF.

LDB results were not available for class B, but it was suggested they should be in a similar range of gain as for classes C and D.

Some encoder optimization was done.

More detailed analysis of all individual tool performances would be desirable, also in terms of memory bandwidth, etc.

It was suggested to establish an exploration experiment and an AHG, as an organized activity to also investigate also other compression technology than NN, while also being able to compare the benefit of both the NN and “conventional” tools approaches, or their combination.

It was agreed that this exploration should not be too restricted by complexity considerations.

Some experts suggested to primarily run tool-on tests against VVC in the first phase of such a study, and only based on such results to define a kind of “default package”. Others were also preferring tool-off testing relative to, e.g., the package proposed in this contribution..

A BoG (coordinated by V. Seregin) was established to define an EE for study of non-NN coding technologies beyond what is currently supported in VVC. See JVET-U1040 for the BoG report.