JVET-O0029 CE9: Summary report on decoder motion vector derivation [S. Esenlik, X. Xiu]
The tools in the scope of this CE include bi-directional optical flow (BDOF) and decoder motion vector refinement (DMVR).
The core experiment summary report is organized into 2 sub-tests as follows:
- CE9-1: BDOF early termination (5 tests)
- CE9-2: DMVR/BDOF enabling conditions (4 tests)
This report summarises the status of each experiment. Crosscheck results are integrated in the report.
Test | Description | Y | U | V | EncT | DecT | Doc# |
CE9-1.1 | Use integer-distance DMVR cost to disable BDOF with threshold equal to 4x4 level TH/32 (only keep BDOF early termination when DMVR is disabled) | -0.02% | 0.00% | -0.02% | 100% | 98% | |
CE9-1.2 | Use integer-distance DMVR cost to disable BDOF with threshold equal to 4x4 level TH/32 and disable BDOF early termination in all cases. | -0.02% | 0.00% | -0.02% | 100% | 100% | |
CE9-1.3 | withdrawn | ||||||
CE9-1.4 | Difference from CE9-1.2 is only threshold value. BDOF early termination by using the min SAD in DMVR first SAD stage or in integer search stage | 0.01% | -0.05% | -0.03% | 100% | 98% | |
CE9-1.5a | Disabling application of BDOF based on best cost computed or derived in DMVR, when both DMVR and BDOF are applied. Use only best SAD cost obtained from integer distance refinement positions. When only BDOF is applied, VTM5 16x16 early termination is applied. Threshold used is same as that will be used in VTM5 for 16x16 4x4 early termination check is disabled for all cases | -0.01% | 0.01% | -0.02% | 100% | 100% | |
CE9-1.5b | Use a derived best cost from parametric error surface when applicable. Otherwise, same as (CE9-1.5a). | -0.02% | 0.02% | -0.02% | 100% | 100% |
The SAD calculated in DMVR used to disable BDOF | Threshold | SAD comp. for Early Termination of BDOF, when DMVR is enabled | Early Termination of BDOF, when DMVR is disabled | |
CE9-1.1 | integer-distance SAD | BDOF 4x4Level TH | None / reuse DMVR | Subblock level, 4x4 level |
CE9-1.2 | integer-distance SAD | BDOF 4x4Level TH | None / reuse DMVR | Disable |
CE9-1.3 | integer-distance SAD | 4*W*(H>>1) for center position, otherwise 8*W*(H>>1) (=4x4 block level TH for BDOF) | None / reuse DMVR | Disable |
CE9-1.4 | integer-distance SAD | 16*W*(H>>1) | None / reuse DMVR | Disable |
CE9-1.5a | integer-distance SAD | BDOF Subblock-level TH | None / reuse DMVR | Subblock level |
CE9-1.5b | fractional-distance SAD; integer-distance SAD, if no fractional-distance SAD | BDOF Subblock-level TH | None / reuse DMVR | Subblock level |
No significant impact on compression. In terms of simplification, CE9-1.2 and CE9-1.4 would be the preferable solutions by completely removing the separate BDOF SAD computation. On the other hand, these would give up any early termination of BDOF in case where DMVR is disabled, and the runtime of BDOF would decrease (which is not the case in CTC, as the combination of DMVR and BDOF is used frequently). The case of disabling DMVR is not the worst complexity case of decoder, therefore the increase of BDOF runtime should not be a big issue.
CE9-1.2 is agreed to be the better choice, also with slightly better results.
Decision: Adopt JVET-O0055, version CE9-1.2. The alignment of text and SW implementation was confirmed by cross-checker.
Test | Description | Y | U | V | EncT | DecT | Doc# |
CE9-2.1a | Harmonization of DMVR and MMVD with MMVD distance threshold (for reference). | -0.10% | -0.10% | -0.15% | 104% | 101% | |
CE9-2.1b | CE9-2.1.a + encoder speed up | -0.10% | -0.10% | -0.13% | 102% | 100% | |
CE9-2.2 | The CIIP with disabled DMVR and BDOF. | -0.01% | -0.02% | -0.01% | 99% | 98% | |
CE9-2.3 | The CIIP with disabled BDOF | 0.00% | -0.01% | -0.02% | 99% | 98% |
CE9-2.1 uses DMVR in combination with MMVD (this is where the encoder runtime increase comes in place). Compared to the encoder runtime vs compression benefit of DMVR itself (which was about 1% versus 0.8% gain, the encoder time increase of 2% for another 0.1% appears low. On the other hand, the decoder was more significantly increased by DMVR. There is some logic that here DMVR improves MMVD, where for the larger range the precision of the MV is worse. It is noted that the benefit of MMVD has recently decreased to approx. 0.6% (was >1% originally).
There is some support from experts on adopting this, whereas others expressed concern about the tradeoff. This is obviously meant for coding efficiency improvement, not for unification. Not possible to reach consensus on adoption.
CE9-2.2 and CE9-2.3 are disabling DMVR and/or BDOF in case of CIIP. This comes without change of compression performance, indicating that the combination of these tools is not beneficial. It slightly decreases SW run time. For hardware, does not have much impact in terms of reducing complexity. The text change is straighforward.
(it is noted that JVET-O0103 covers a subset of this, by only disabling BDOF)
Decision: Adopt JVET-O0108, disabling DMVR and BDOF when CIIP is used.
A BoG (S. Esenlik) was arranged for reviewing CE9 related proposals and suggest actions from these.