JVET-M0029 CE9: Summary report on decoder side motion vector derivation [X. Xiu, S. Esenlik]
The core experiment summary report is organized into 2 sub-tests as follows:
- CE9.1: BDOF design (3 tests)
- CE9.2: DMVR design (24 tests)
CE9.1: BDOF design
# | Description | Tester | Cross-checker |
9.1.1.a | 1. use 8-tap DCTIF filters to generate the prediction samples in extended region and inside CU 2. padding for reference samples outside [w+7, h+7] for final MC (DCTIF) | Xiu, Xiaoyu | H. Liu |
9.1.1.b | 1. use integer positions to generate the prediction samples in extended region 2. use 8-tap DCTIF filters to generate the prediction samples inside CU | Xiu, Xiaoyu | H. Liu |
9.1.1.c | 1. apply different gradient calculation method for prediction samples on the CU boundaries and inside CU 2. use 8-tap DCTIF filters to generate the prediction samples inside CU | Xiu, Xiaoyu | H. Liu |
VTM | Cross-Check | ||||||||
Test | Document | Crosschecker | Y | U | V | EncT | DecT | EncT | DecT |
9.1.1 a | H. Liu | -0.01% | 0.03% | 0.02% | 103% | 102% | 105% | 104% | |
9.1.1 b | H. Liu | 0.05% | 0.02% | 0.02% | 99% | 98% | 99% | 100% | |
9.1.1 c | H. Liu | 0.10% | 0.03% | 0.01% | 99% | 98% | 100% | 100% | |
9.1.1.a does not simplify, replaces bilinear filters in extended region by DCTIF
9.1.1.b simplifies by using no interpolation in extended region (just integer positions)
9.1.1.c changes the gradient calculation at boundaries and does not need extended region any more, it simplifies, but the design becomes less unified
Several experts supported 9.1.1.b as the best simplified design approach.
Decision: Adopt JVET-M0487 (solution 9.1.1.b) which uses integer positions to generate the prediction samples in extended region, and uses 8-tap DCTIF filters to generate the prediction samples inside the CU.
Specification text was made available, to be confirmed to be consistent with software by cross-checkers, and reviewed by spec editors.
CE9.2: DMVR design
# | Description | Document |
9.2.1 a DMVR base s/w | 1. padding for reference samples outside [w+7, h+7] for final MC (DCTIF) 2. use bi-linear filter to generate prediction samples [w+4, h+4] for motion refinement 3a. MRSAD is used as metric for motion refinement 3b. MRSAD calculation on every other row 4. error surface based sub-pixel refinement 5a. DMVR early termination based on MV distance 5b. DMVR early termination based on sample difference Block size criteria: CUs with height >= 8 and size > 64 and CUs with size <= 1024 Refined MV: MC, DBF, TMVP, spatial MV prediction from top and top-left CTU | |
9.2.1 b0 | CE9.2.1a + Integer DMVR, SAD, and refined MV used only for MC | |
9.2.1 b | CE9.2.1b0 + Enable 32x32 forced split + enable DMVR for w*h > 1024 + Replace CU level early termination to Sub-PU level | |
9.2.1 c | CE9.2.1b + Replace SAD with MR-SAD | |
9.2.1 d | CE9.2.1c + Replace Integer DMVR with Bilinear MC | |
9.2.1 e | CE9.2.1d + use of refined MV for deblocking, TMVP and for spatial MV prediction from top/top-left CTU neighbours | |
9.2.1 f | CE9.2.1e + use of refined MV also within CTU row from VPDUs that were processed 2 VPDUs behind current VPDU | |
9.2.1 f1 | CE9.2.1f + MC restricted to use samples within (64+7)*(64+7) for VPDU of CUs larger than 64x64 | |
9.2.1 g | CE9.2.1e + replace sub-PU size to 16x16 from 32x32 | |
9.2.2 c | CE9.2.1g + replace MR-SAD with PR-MR-SAD | |
9.2.2 d | CE9.2.2c + Change sub-CU level early termination threshold (3 times) to know DecT impact | |
*9.2.2 f | Results for CE9.2.2c when BDOF is disabled in both VTM3.0 and with DMVR + DMVR is enabled for ATMVP and MMVD | |
9.2.2 f1 | Disabling refined MV usage for anything other than MC in CE9.2.2c | |
9.2.2 f2 | Replacing PR-MR-SAD with SAD in CE9.2.2c | |
9.2.3 a | No diagonal direction checking when all cross points' cost value not less than center point cost value | |
9.2.3 b | Down sampling for MRSAD's mean value calculation | |
9.2.3 c | Replacing MRSAD with SAD in CE9.2.1a | |
9.2.4 a | CE9.2.1a + allow inter reconstruction to be used as intra reference | JVET-M0447 (M. Xu) |
9.2.4 b | CE9.2.1a + don't allow inter reconstruction to be used as intra reference | |
9.2.4 c | CE9.2.1a + don't allow DMVR reconstruction to be used as intra reference | |
9.2.5 a | CE9.2.1a is changed as follows: minCost equal to zero replaces minCost less than (w*h*(1<<(Max(2, 14 – Bit Depth) + (Bit Depth-8)+0)) | JVET-M0062 (T. Chujoh ) |
9.2.5 b | CE9.2.1a is changed as follows: minCost equal to zero replaces minCost less than (w*h*(1<<(Max(2, 14 – Bit Depth) + (Bit Depth-8)-1)) | |
9.2.6 | CE9.2.1a is changed as follows: Disable DMVR for CUs with size > 4096 | JVET-M0076 (K. Unno) |
9.2.7 | CE9.2.1a is changed as follows: Bilinear filter is replaced by no interpolation (rounding to nearest integer sample) | JVET-M0287 (S. Esenlik) |
* Below, CE9.2.2.f result is reported compared to BDOF off anchor.
Test | Description | Y | U | V | EncT | DecT |
9.2.1 a DMVR base s/w | 1. padding for reference samples outside [w+7, h+7] for final MC (DCTIF) 2. use bi-linear filter to generate prediction samples [w+4, h+4] for motion refinement 3a. MRSAD is used as metric for motion refinement 3b. MRSAD calculation on every other row 4. error surface based sub-pixel refinement 5a. DMVR early termination based on MV distance 5b. DMVR early termination based on sample difference Block size criteria: CUs with height >= 8 and size > 64 and CUs with size <= 1024 Refined MV: MC, DBF, TMVP, spatial MV prediction from top and top-left CTU DMVR is enabled for ATMVP and MMVD (disabled for all other tests exceptz 9.2.3ff.) | -0.71% | -0.81% | -0.87% | 110% | 118% |
9.2.1 b0 | CE9.2.1a + Integer DMVR, SAD, and refined MV used only for MC | -0.30% | -0.39% | -0.40% | 101% | 103% |
9.2.1 b | CE9.2.1b0 + Enable 32x32 forced split + enable DMVR for w*h > 1024 + Replace CU level early termination to Sub-PU level | -0.58% | -0.86% | -0.86% | 102% | 106% |
9.2.1 c | CE9.2.1b + Replace SAD with MR-SAD | -0.66% | -0.88% | -0.92% | 102% | 108% |
9.2.1 d | CE9.2.1c + Replace Integer DMVR with Bilinear MC | -0.74% | -1.02% | -1.07% | 102% | 111% |
9.2.1 e | CE9.2.1d + use of refined MV for deblocking, TMVP and for spatial MV prediction from top/top-left CTU neighbours | -1.12% | -1.25% | -1.30% | 103% | 114% |
9.2.1 f | CE9.2.1e + use of refined MV also within CTU row from VPDUs that were processed 2 VPDUs before current VPDU | -1.25% | -1.33% | -1.43% | 103% | 114% |
9.2.1 f1 | CE9.2.1f + MC restricted to use samples within (64+7)*(64+7) for VPDU of CUs larger than 64x64 | -1.25% | -1.33% | -1.42% | 103% | 114% |
9.2.1 g | CE9.2.1e + replace sub-PU size to 16x16 from 32x32 | -1.13% | -1.33% | -1.44% | 103% | 118% |
9.2.2 c | CE9.2.1g + replace MR-SAD with PR-MR-SAD | -1.07% | -1.27% | -1.36% | 103% | 118% |
9.2.2 d | CE9.2.2c + Change sub-CU level early termination threshold (3 times) to know DecT impact | -1.04% | -1.27% | -1.36% | 103% | 117% |
*9.2.2 f | Results for CE9.2.2c when BDOF is disabled in both VTM3.0 and with DMVR + DMVR is enabled for ATMVP and MMVD | -1.51% | -1.58% | -1.59% | 116% | 126% |
9.2.2 f1 | Disabling refined MV usage for anything other than MC in CE9.2.2c | -0.72% | -1.05% | -1.11% | 102% | 115% |
9.2.2 f2 | Replacing PR-MR-SAD with SAD in CE9.2.2c | -1.02% | -1.29% | -1.35% | 103% | 115% |
9.2.3 a | CE9.2.1a, No diagonal direction checking when all cross points' cost value not less than center point cost value | -0.71% | -0.84% | -0.90% | 108% | 116% |
9.2.3 b | CE9.2.1a, Down sampling for MRSAD's mean value calculation | -0.68% | -0.84% | -0.89% | 108% | 115% |
9.2.3 c | CE9.2.1a, Replacing MRSAD with SAD | -0.60% | -0.78% | -0.84% | 107% | 115% |
9.2.4 a | CE9.2.1a + allow inter reconstruction to be used as intra reference | -0.71% | -0.81% | -0.87% | 108% | 115% |
9.2.4 b | CE9.2.1a + don't allow inter reconstruction to be used as intra reference | 3.87% | 8.82% | 9.62% | 103% | 113% |
9.2.4 c | CE9.2.1a + don't allow DMVR reconstruction to be used as intra reference | -0.46% | -0.34% | -0.36% | 108% | 115% |
9.2.5 a | CE9.2.1a is changed as follows: minCost equal to zero replaces minCost less than (w*h*(1<<(Max(2, 14 – Bit Depth) + (Bit Depth-8)+0)) | -0.67% | -0.80% | -0.86% | 109% | 118% |
9.2.5 b | CE9.2.1a is changed as follows: minCost equal to zero replaces minCost less than (w*h*(1<<(Max(2, 14 – Bit Depth) + (Bit Depth-8)-1)) | -0.70% | -0.81% | -0.87% | 109% | 119% |
9.2.6 | CE9.2.1a is changed as follows: Disable DMVR for CUs with size > 4096 | -0.89% | -0.99% | -1.08% | 111% | 119% |
9.2.7 | CE9.2.1a is changed as follows: Bilinear filter is replaced by no interpolation (rounding to nearest integer sample) | -0.58% | -0.69% | -0.74% | 109% | 115% |
Additional results in JVET-M0147 which show results on CE9.2.1g but without using refined MV for spatial MV prediction and deblocking: Overall -1.01%. This could be a reasonable approach avoiding some of the dependency problems that were observed in DMVR before. It was asked for some cross-check and specification text to be made available.
Cross check became available as JVET-M0887. Has two results: For MR-SAD and SAD. MR-SAD -1.01%, SAD -0.92%. Results provided above are confirmed. Specification text was provided in JVET-M0147v5, has been investigated by cross-checker and spec editor as being appropriate.
Decoding time increase is reported as 15%.
It was confirmed that the following criteria are fulfilled by the SAD version, which are agreed to be a tradeoff between software and hardware implementation aspects:
- Early termination w/ (0,0) position SAD between list0 and list1
- Block sizes for DMVR W*H>=64 && H>=8
- Split the CU into multiple of 16x16 sub-blocks for DMVR of CU size > 16*16
- Reference block size (W+7)*(H+7) (for luma)
- 25 points SAD-based integer-pel search (i.e. (+/−) 2 refinement search range, single stage)
- Bilinear-interpolation based DMVR
- MVD mirroring between list0 and list1 to allow bilateral matching
- “Parametric error surface equation” based sub-pel refinement
- Luma/chroma MC w/ reference block padding (if needed)
- Refined MVs used for MC and TMVPs only
It is further noted that the proposal contains an additional element which checks the MV difference with regard to previous MVs in the merge list to make an early termination in DMVR. This has hardly any impact on the BD rate, but reduces decoding time by 3%. This has however some impact on hardware implementation, and decoding time is not overly critical.
Further reducing decoding time for software implementation (and possible optimization of code) is desirable.
Decision: Adopt JVET-M0147 with SAD cost function, and without the MVD based early termination check.
Note that the MVD based early termination should also not be used in upcoming CEs.
It is noted that disallowing DMVR blocks for intra prediction may not be that relevant in practical pipeline implementations. Typically, within a CTU first all inter blocks would be reconstructed, and finally the intra coded blocks,
Early termination approaches do not seem to be very effective in terms of runtime reduction, and also are not beneficial for hardware.
Generally, the investigation on DMVR has led to a point where it might be manageable implementation-wise (not low complex, but still giving around 1% gain)
A problem of the methods investigated in the CE could still be that DMVR and BDOF could be applied sequentially, before finally motion comp and reconstruction can be done. JVET-M0223 considers this issue.
The next step was review of CE related contributions (there were not too many on DMVR) and the assess whether they provide further improvement or complexity reduction.
A BoG was established (coordinated by S. Esenlik) to review the CE9 related contributions and suggest aspects to be studied in a CE (for BIO and DMVR). See the notes for the BoG report JVET-M0858.
The subsequent notes in this section only contain abstracts copied from the documents. Actions taken are noted above under JVET-M0029.