JVET-E0052 EE3: Decoder-Side Motion Vector Refinement Based on Bilateral Template Matching [X. Chen, J. An, J. Zheng (HiSilicon)]

This contribution reports the results of Exploration Experiment (EE) 3 “Decoder-Side Motion Vector Refinement Based on Bilateral Template Matching”. 4 cases that based on bilateral template matching are tested on top of JEM 4.0. the first case is DMVR with half pixel precision motion estimation on and tools (EMT/NSST/RSAF/PDPC/FRUC/BIO/OBMC/IlluCompEnable/AFFINE/ATMVP/IMV) off, The second case is DMVR with half pixel precision motion estimation on, the third case is DMVR with half pixel precision motion estimation off and tools (EMT/NSST/RSAF/PDPC/FRUC/BIO/OBMC/ IlluCompEnable/AFFINE/ATMVP/IMV) off, and the fourth case is DMVR with half pixel precision motion estimation off. The BD-rate luma gains for random access (RA) configurations are reported as follows:

[EE5.1 Half pixel precision ME on and tools off]: RA: −2.71%, EncT: 114%, DecT: 138%

[EE5.2 Half pixel precision ME on]: RA: −0.42%, EncT: 102%, DecT: 102% (this is CTC)

[EE5.3 Half pixel precision ME off and tools off]: RA: −1.86%, EncT: 103%, DecT: 111%

[EE5.4 Half pixel precision ME off]: RA: −0.32%, EncT: 100%, DecT: 100%

(Presentation deck missing from uploaded contribution file.)

From the EE summary report JVET-E0010:

Decoder-Side Motion Vector Refinement Based on Bilateral Template Matching: “The third” decoder side MV derivation technique for JEM. Applied under conditions:

• Not OBMC
• L0 and L1 reference are from opposite time directions
• Merge
• Not Local Illumination Compensation
• Not Affine MC
• Not FRUC

Similarly to FRUC it operates at sub-PU level and BIO is applied on top.

First motion compensation for luminance is done with an MV signalled in the bit-stream for a block extended by one row and one column on each side. So the memory access required for this method is (W+2+7) (H+2+7) instead (W+7) (H+7) in HEVC motion compensation. This is 28% higher for HEVC the worst case (88 bi-predicted PU).

Then decoder searches for MV refinement in L0 and L1. In the first round of search, up to 8 MV candidates (+/−1 int-pel displacement in vertical and horizontal directions) are checked. So up to 9 calculations for SAD and up to 8 comparisons are needed. The second round of MV search at the decoder side refines the MV with 1/2-pel precision. Additionally, up to 8 calculations for SAD and up to 8 comparisons are needed. The additional gain from this 1/2-pel refinement is 0.1%, but this second round doesn’t require additional memory access.

After MV refinement is performed, the MC for all three colour components is performed with new MV.

Questions recommended to be answered during EE tests:

[Q]: Test performance and complexity on JEM4.0 platform.

[A]: 0.4% gain is observed in RA case with ~2% encoder and decoder run-time increment.

[Q]: How does performance depend on number of MV0′ and MV1′candidates checked on decoder side?

[A]: Proponent provided 2 sets of test data: with int-pel and with 1/2-pel MV refinement precision. The latler one estimates roughly a twice smaller number of MV candidates and provides the largest part of the gain (0.3%).

Summary: 0.4%(RA) gain is observed ~2% encoder and decoder run-time increment. The major source of the gain is MV refinement with int-pel precision (0.3%).

From the discussion in JVET:

Two cases are considered: Half-pel search (16 positions) and integer search (8 positions).

“Integer precision” means that the additional search is with integer precision, the final position could be sub-pel, depending on the starting vector.

Several experts supported adoption of the proposal, because it can cover cases which are not supported by FRUC and gives some gain in RA, particularly for class A (some sequences of class A).

The application at the sub-CU level is not giving gain; therefore the usage is restricted to cases where affine and sub-CU are not used.

Decision: Adopt JVET-E0052, with integer step search (8 positions around the start position).

Also implement a high-level flag (in SPS) to disable the tool.

It is further clarified that the template is generated before loop filtering.

JVET-E0052 EE3: Decoder-Side Motion Vector Refinement Based on Bilateral Template Matching [X. Chen, J. An, J. Zheng (HiSilicon)]

Adopt JVET-E0052, with integer step search (8 positions around the start position). Also implement a high-level flag (in SPS) to disable the tool.