JVET-AP0024 EE2: Summary report of exploration experiment on enhanced compression beyond VVC capability [V. Seregin, D. Buğdayci Sansli, J. Chen, R. Chernyak, K. Naser, J. Ström, F. Wang, M. Winken, X. Xiu, K. Zhang (EE coordinators)]

List of tests

Intra prediction

Test 1.1: Improvement on chroma MPM (JVET-AP0133)

In the Test 1.1a, more chroma DIMD modes are added to the current MPM candidate list after the adjacent chroma modes and before the DBV modes.

In Test 1.1b, the costs of the second and third modes are considered during the pre-sorting process, when the cost of the first mode in the MPM candidate list is lower than that of the second mode, an additional cost comparison between the second mode and the third mode is performed. Only when the cost of the first mode is lower than that of the second mode and the cost of the second mode is lower than that of the third mode, the top 6 or top 7 modes in the MPM candidate list are selected to form the final MPM candidate list; otherwise, it is necessary to further execute the process of reordering all candidates in the MPM candidate list to construct the final MPM candidate list.

Test 1.1a: Add more chroma DIMD modes into the MPM list.

Test 1.1b: Modify the reordering strategy.

Test 1.1c: Test 1.1a + Test 1.1b

Test 1.2: Modification of EIP filter shapes (JVET-AP0145)

Two modifications of EIP filter shapes are tested.

In ECM, EIP filter shapes are shown in the next figure.

Test 1.2a modifies the horizontal and vertical filters: in the horizontal filter, one input sample is replaced by the above sample, and in the vertical filter, one input sample is replaced by the left sample.

Test 1.2b further modifies the square filter on top of Test 1.2a, in the square filter, one input sample is replaced by the nearest spatial input sample in the diagonal prediction order.

The corresponding filter shapes are shown in the figure below.

Test 1.2a: EIP with modified H-filter and V-filter

Test 1.2b: Test 1.2a + modified S-filter

Test 1.3: Clipping operation refinements in CCCM modes (JVET-AP0168)

This test modifies clipping operation in CCCM as follows:

• Apply a margin to the minimum and maximum values of the chroma samples from the reference area to derive the final clipping range for the chroma sample prediction.
• Introduction of a minimum gap between the minimum and maximum clipping values which are obtained from the minimum and maximum values of the chroma reference samples. Typically, a minimum gap corresponding of certain percentage of the maximum possible chroma range is applied.
• Apply the above two adaptations for small chroma PUs having less than 32 samples.
• Apply the above adaptations to all CCCM modes (called GLM, BVG, CFL) defined in the ECM.
• Apply the above minimum gap according to model type and whether the model is converted from CCP candidate (for Decoder-derived CCP and CCP merge mode).

Test 1.3: CCCM clipping operations with clipping range adjustment.

Results AI/RA

Test 1.1x: Some additional complexity by adding modes, no good tradeoff in terms of gain – no action.

Test 1.2x: No significant benefit from EE results. EE related document JVET-AP0143 was presented in this context, but the additional benefit is still too small for taking action.

Test 1.3: Some additional processing necessary to derive the range, but it gives some gain in chroma (in particular for intra, somewhat diverging results for RA). Several experts expressed support.

Decision: Adopt JVET-AP0168 Test 1.3.

Inter prediction

Test 2.1: Modified GPM partition mode (JVET-AP0086)

In ECM, GPM supports 32 angles, where a variable shiftHor defined below specifies the offset direction from the center of the CU.

shiftHor = (angleIdx % 16 = = 8 | | (angleIdx % 16 != 0 && nH >= nW ) ) ? 0 : 1

where nH and nW represent the height and width of the current block, respectively.

In the test, two modifications are evaluated. In the first aspect, the offset direction is derived by identifying whether the partitioning angle is in a proximity of the horizontal or vertical direction, which is done based on a comparison between the block aspect ratio with the aspect ratio of the GPM partition angle, as well as the relative dimensions of the block, when the partitioning angle is close to 45 degrees.

The GPM offset direction is controlled by a variable “shiftHor” as follows:

nearHor = g_angle2mask[angleIdx]==0 || g_angle2mask[angleIdx]==1|| g_angle2mask[angleIdx]==2

nearVer = g_angle2mask[angleIdx]==6 || g_angle2mask[angleIdx]==7||g_angle2mask[angleIdx]==8

blockRatio = blockWidth / blockHeight = nW / nH;

gpmRatio = angle_w / angle_h;

shiftHor = nearHor ? 1 : (nearVer ? 0 : ((gpmRatio > blockRatio || (gpmRatio = = blockRatio&&nH>=nW)) ? 0 : 1))

In the second aspect, block-shaped adaptive angle selection proposed by JVET-AJ0107 is enabled in the LB configuration.

Test 2.1a: Modified GPM partition mode with refined offset direction derivation.

Test 2.1b: Modified GPM partition mode with enabling block-shaped adaptive angle selection in LDB.

Test 2.1c: Test 2.1a + Test 2.1b.

Reults AI/RA/LB

Cross-checkers confirm results and support adoption of 2.1a. Also some other expert expressed support.

Decision: Adopt JVET-AP0086 Test 2.1a.

Transforms and coefficient coding

Test 3.1: Dead-zone adjustments and unbiasing of scalar quantizers (JVET-AP0065)

In ECM, in uniform quantization (RDOQ off, DQ off), an offset β is added at encoder side, the value of β is inherited from HEVC as β=171/512≈1/3 for Independent Random-Access Point (IRAP) pictures, and β=85/512≈1/6 for the rest of the pictures.

Consequently, the reconstruction points are not in the middle of their quantization intervals, resulting in a bias controlled by β.

In ECM, the quantization center shifting method (QCS) further shifts the reconstruction points in the dequantization process, where the dequantized coefficient t’ is computed from the quantization level and a lookup table T of QCS offsets.

The tests are targeting the removal of the reconstruction bias.

Two offsets β^* (applied in tests 3.1a and 3.1b) are introduced replacing the current β encoder offsets, which are set to 161/512 in AI configuration, 191/512 for IRAP and 120/512 for non-IRAP pictures in RA and LB configurations.

In Test 3.1c, two offsets at encoder are introduced with enabled RDOQ, they are set as 16/512 for AI, 0 for IRAP and 32/512 for non-IRAP pictures in RA and LB configurations.

In Test 3.1a, those new encoder offsets are applied.

In Test 3.1b, a flag is signalled indicating the QCS offset T is replaced with the signalled offset O for all quantization levels. In the implementation, the offset table T is replaced with the signalled offset O as follows T’[0]=0 and T’[i]=O for i>0. This test is performed with disabled RDOQ and DQ.

In Test 3.1c, the signalled offset O is added to the existing QCS offset T, and this test is performed for enabled RDOQ.

Test 3.1a: Modified encoder offsets (encoder only) (RDOQ off, DQ off).

Test 3.1b: Test 3.1a with signalled offset for inverse quantization replacing QCS (RDOQ off, DQ off).

Test 3.1c: Modified encoder offsets with signalled offset for inverse quantization in addition to QCS (RDOQ on, DQ off).

Test 3.2: On the dead-zone in quantization (JVET-AP0059)

In ECM, there is no offset in quantization, but an offset inversely proportional to quantization index is applied in inverse quantization. This offset is kept in a lookup table shown by in ECM. In this test, a new QP adaptive quantization offset is introduced. This new offset is a first order linear function of QP as in following equation.

The coefficients of this linear function can be different for slice and channel types. The optimal parameters are set as SPS level syntax elements for each encoding mode. This syntax elements are signalled if inverse quantization applies this offset in the test.

In the tests, the encoder quantization offsets and the offset applied for the inverse quantization at decoder are summarized in the following table for the tested configurations.

Test 3.2a: Modified encoder offset with signalled inverse quantization offset in addition to QCS

Test 3.2b: Modified encoder offset (encoder only)

Test 3.2c: Modified encoder and signalled inverse quantization offsets, both in addition to QCS

Test 3.2d: Using QCS offset at encoder (encoder only)

Test 3.2e: Test 3.2a (RDOQ on, DQ off)

Test 3.2f: Test 3.2b (RDOQ on, DQ off) (encoder only)

Test 3.2e: Test 3.2c (RDOQ on, DQ off)

Test 3.2h: Test 3.2d (RDOQ on, DQ off) (encoder only)

Test 3.3: Intra LfnstIdx prediction (JVET-AP0107)

In the test, the boundary discontinuity used in sign prediction is applied to signal LFNST indices. The dequantized coefficients are tested using all possible inverse transform kernels as shown in the next figure. Then, the boundary continuity cost is calculated for each possible inverse transform kernel, and the inverse transform kernel index with the smallest cost is used as the LFNST index prediction. A flag is signalled whether the LFNST index is equal to the prediction, otherwise the LFNST index remainder is signalled.

This method is not applied if DIMD, TIMD, MIP, IntraTMP, SGPM, or EIP is used for the prediction and is used when transform coefficient sign prediction is not utilized.

Test 3.3a: Intra LFNST index prediction.

Test 3.3b: Test 3.3a with sign prediction disabled (non-CTC).

Results AI/RA/LB

3.1/3.2/3.5: Only 3.2a gives gain (most is -0.1% in RA and -0.13% LB) compared to CTC. Some other tests demonstrate that benefit by signalling inverse quantization offset becomes larger under non-CTC (e.g. disabling DQ).

Some concern is raised whether the degree of interaction between encoder and decoder optimization is sufficiently understood, and would justify the normative change, also considering that the gain is not large. No support by other experts.

Test 3.3: Sign prediction is replaced for cases of regular intra mode in 3.1a. Small gain over CTC is shown in AI in test 3.1a (no gain in RA). This would not justify introduction/switching of an additional tool. Test 3.3b completely replaces sign prediction (which may be some complexity advantage), but does not show gain in CTC. Not relevant enough for taking action.

In-loop filtering

Test 4.1: Simplification on TALF (JVET-AP0087)

In ECM, Temporal adaptive loop filter (TALF) uses the reconstructed pixels in the reference pictures, and the motion-compensation padded samples located in the padded area of the reference. TALF has four uni-filtering modes and two bi-filtering modes. The uni-filtering process is given by the following equation,

and the bi-filtering process is given by the following equation,

In the above equation,

• is the offset generated by TALF filter and added to the luma ALF output;
• is one of the positions within the filter window;
• r0 represents the reference samples from the reference picture 0
• r1 represents the reference samples from the reference picture 1
• (x,y) is the current filtering position in the current picture;
• (x',y') is the collocated position or the MV guided position in the reference picture;
• K(d,b) is a clipping function which restricts the value d within the range from -b to b;
• saoLuma represents the output samples after the SAO filtering.

Two filter shapes are employed in TALF.

In the test, the filtering process is modified by replacing the SAO value with the sample value from the collocated position. The uni-filtering process is modified as follows,

,

where ,

and the bi-filtering process is modified as follows,

,

where .

Here, the proposed changes are highlighted in yellow.

With the above modifications, the TALF offset is generated only from the reconstructed samples from the reference pictures. The dependency on the SAO output samples is removed. In addition, the number of filter coefficients is reduced from 13 to 12.

Results RA/LB:

Small gain, and further simplification of ECM (which has many more complicated elements) is not of high importance at this stage. No action at this moment.

Entropy coding

Test 5.1: Counter-based temporal probability initialization (JVET-AP0082)

In ECM, temporal CABAC is used for the context initialization by using the probability states of the previously coded slice with the same slice type, slice QP, and temporalID, if available.

In the test, counter based method for the context initialization is introduced. Two counters and are introduced to each context. The counter tracks the total number of bins coded, and the counter tracks the total number of “1” coded. The quotient of the two counters (i.e., is used to adjust initialization probability as follows

where is the probability from the temporal CABAC initialization.

The counters are accumulated across slices with the same slice type, slice QP, and temporalID. To prevent overflow, the counters are not accumulated if a certain threshold value is reached.

The probability is adjusted if the total counter is greater than a predefined threshold, which is set to 32 for the low delay conditions, and to 48, otherwise.

Test 5.1a: ECM-19.1 CABAC context retraining.

Test 5.1b: Counter-based temporal probability initialization.

Test 5.1c: Test 5.1b + Test 5.1a.

Test 5.1d: Using quotient C₁/C_N as initialization probability.

Results AI/RA/LB:

It was commented that it is remarkable to further improve temporal probability initialization by a modification. However, some additional memory might be necessary to implement, and it might be questionable if it would be practical in a real implementation. Not of high importance at this stage. No action at this moment.

EE2 contributions: Enhanced compression beyond VVC capability (9)

There was no presentation or discussion about specific proposals in this category – contributions were discussed in the context of the EE summary report JVET-AP0024. For actions decided to be taken, see section 5.2.1, unless otherwise noted.