JVET-N0815 BoG report on CE9 related contributions [S. Esenlik]

This BoG report was reviewed on Monday 25 March 2019 1720-2225, chaired by Y. Ye.

The BoG met in 3 sessions to discuss CE9 (decoder motion vector derivation) related contributions. The first session was held on 21 Mar 2019 from 20:00 to 22:45. The second session was held on 22 Mar 2019 from 18:15 to 22:30. The third session was held on 23 Mar 2019 from 10:45 to 12:00.

The CE9-related BoG recommend the following adoptions:

• JVET-N0146 Non-CE9: On conditions for DMVR and BDOF

This contribution proposes when any of BCW and weighted prediction are applied, DMVR and BDOF are both disabled.

On average no loss is observed, with 1% decoding time reduction.

On sequence basis, the maximum luma coding loss is 0.07% (daylightroad sequence). Slideshow sequence shows 0.34% luma coding gain.

For the BDOF part, the contribution proposes to align the specification draft text with the VTM software. It was confirmed during the track B discussion that the VTM 4.0 software is already disabling the BDOF when either BCW or weighted prediction is on. Decision (cleanup): change the spec to match that of the VTM software.

For the DMVR part, the motivation is alignment with BDOF. In the DMVR part both software and the spec text is recommended to be changed. In track B discussion, it was agreed that DMVR should be aligned with BDOF. Decision (cleanup): change the spec and the software as proposed.

The contributions proposing the same modification are: JVET-N0153, JVET-N0442, JVET-N0162, JVET-N0262, JVET-N0440 and JVET-N0086.

It was asked if all these contributions used the same software implementation. It was commented that all software implemention seem to produce very similar results, and that the suggested changes are simple.

Decision (SW): use the JVET-N0146 software implementation.

• JVET-N0325 CE9-related: Improvements on bi-directional optical flow (BDOF)

The contribution proposes to use fixed precision for BDOF calculations (gradient and flow), irrespective of internal bit-depth (8-12 bits). The fixed precision is the one used for 8-bit sequences.

In the 12-bit coding case 0.17% luma coding gain without increasing encoding and decoding times.

It is asserted that 32-bit precision condition is still kept.

For 10-bit coding (CTC): {Y, U, V} BD-rate of {-0.01%, -0.01%, -0.03%}, Enc: 102%, Dec: 101%;

For 8-bit: coding: {Y, U, V} BD-rate of { 0.00%, 0.00%, 0.00%}, Enc: 100%, Dec: 100%;

For 12-bit coding: {Y, U, V} BD-rate of {-0.17%, -0.10%, -0.09%}, Enc: 100%, Dec: 100%.

A crosschecker confirmed the software and the spec., and the crosscheck document had been uploaded as JVET-N0618.

In track B discussion, it was asked why this proposal has coding performance improvement. The proponent commented that the gain comes from redistributing the precision between gradient and sample values such that the sample values get more precision for higher bit-depth video.

The proposal changes some shift values in BDOF calculation from bit-depth dependent values to fixed values. It was asked whether this might produce an overflow situation for other bit-depth video, and it was commented that during BoG discussion, this aspect was checked and it was confirmed that for 8- to 12-bit video, there is no such overflow.

However, upon reviewing the proposed spec text change, it was commented that having the shift operations use fixed values instead of bit-depth-dependent values might “break” the text for bit-depths other than 8- to 12-bit.

The proponent was asked to upload revised specification text changes to only apply the proposed changes to 8- to 12-bit video.

This was discussed again just before the closing of the meeting, while the group was being requested to leave the room. The meeting had run past the pre-arranged hours of the meeting facility. The requested text had been provided and had been checked and confirmed to be correct by an interested leading non-proponent. However, an author of a different approach objected to proceeding with adoption, saying that something was wrong with this approach, although no clear indication of a problem was evident to others. The group considered extending the meeting further into the evening by having everyone move to another meeting room in a different building, but the overwhelming view of the group was that the meeting needed to be closed and any concerns about the adoption should be deferred for further study.

Decision (cleanup): Adopted the JVET-N0325 approach.

• JVET-N0444 Non-CE9: Mismatch beween test specification and reference software on BDOF and DMVR

This contribution is about text and software cleanup, to achieve alignment of BDOF conditions with contribution JVET-N0266 (disabling 4x8,8x4 bi-prediction), which is adopted recently.

Software: the block size condition for BDOF need to be modified. This change should be taken into consideration during integration of JVET-N0266.

Text: the modification in the text suggests to remove the “cbWidth x cbHeight>=64” condition, since this is not necessary anymore after adoption of JVET-N0266. This also needs to be taken into consideration during integration of JVET-N0266.

During track discussion, it was commented that the proposal produces bit-exact results as JVET-N0266. Due to JVET-N0266 adoption, all bi-predicted blocks must be at least 8x8. In VTM 4.0, after checking bi-prediction condition, block size is further checked using two conditions to determine if BDOF is applied: 1) block height is not equal to 4; 2) block size is not equal to 4x8. Due to the adoption of JVET-N0266, the second block size check (i.e. block size is not equal to 4x8) could be removed from the next VTM software version. The suggested software change will be delegated to the software coordinators.

It was confirmed the VVC draft 4 currently does not have block size condition checks when invoking BDOF, it is suggested to add the condition check of block height is not equal to 4 to the spec text to reflect the design intention and match the VTM software.

Decision (bug fix): add the suggested condition check into the draft text

The BoG recommends the following for further study in CE:

• JVET-N0097 CE9-related: Simplification of cascading DMVR and BDOF processes

2-stage early termination of BDOF is replaced by one-stage, where SAD is calculated 16x16 blocks. Same as DMVR.

Additionally the BDOF reuses the SAD cost calculated in DMVR.

No change (-0.01% luma gain) in BD-rate performance.

• JVET-N0148 CE9-related: Early termination for BDOF

Second test in contribution:

BDOF reuses SAD cost function of DMVR for early termination. Disable early termination of BDOF (completely). The SAD based early termination is disabled even if DMVR is not applied.

0.02% luma coding gain is reported with same encoding and decoding time.

Recommendation to study the second test in CE.

• JVET-N0158 CE9-related: Alternative method of SAD based early termination for BDOF

This contribution is similar to second test of JVET-N0148. The difference is only the threshold of early termination that is used.

It is reported to have: 0.01%luma coding loss with 2% reduction in decoding time.

• JVET-N0296 CE9-related: DMVR costs based early termination for BDOF in CE9-2.6

If DMVR and BDOF are applied together, SAD cost of DMVR is used for BDOF early termination.

A derived cost is calculated which is different from other proposals.

The results are shown on top of CE9-2.6.

2 stage early termination is removed.

• JVET-N0507 CE9-related: Early termination of BDOF with DMVR cost

Proposes to reuse the SAD cost calculated in DMVR for early termination of BDOF. There is a scaling (by a factor of 16) for threshold comparison.

Test 1: if DMVR is disabled original early termination of BDOF if used. If DMVR on, BDOF uses SAD cost of DMVR. 0.02% degradation in luma BD-rate.

Test2: in addition to test 1, if DMVR is disabled, no early termination for BDOF is applied. No BD-rate impact, 1% decoding time reduction.

Recommended to test both test 1 and test 2 in CE.

During discussion in track B about JVET-N0097, JVET-N0148, JVET-N0158, JVET-N0296, and JVET-N0507, it was noted that all these proposals are about using the SAD calculated in DMVR stage to early terminate BDOF.

It was commented that the early termination topic was discussed when CE9-2.4 (JVET-N0178) was discussed earlier in track B. It was questioned if such a CE was necessary. It was mentioned that JVET-N0148 might be a favorable choice for adoption because it uses the DMVR SAD for BDOF early termination if BDOF is applied on top of DMVR; otherwise, if BDOF is applied by itself, the early termination is simply removed.

It was commented that JVET-N0158 is the same as JVET-N0148 except that the threshold used to early terminate the BDOF process after DMVR is slightly different.

In terms of performance,

JVET-N0148: -0.02%, same encoding/decoding time

JVET-N0158: 0.01%, same encoding time, 2% reduction in decoding time

It was commented that other proposals are similar in spirit, and it might be the best to test all methods in a CE.

Further study in CE.

• JVET-N0145 Non-CE9: Harmonization of DMVR and MMVD

-0.10 % coding gain in RA under CTC. 6% increase in encoding time.

DMVR is applied on MMVD with selection threshold (set to 2).

In track B discussion, it was commented that the gain was somewhat small, and the increase in encoding time and potentially increase in decoding time (since DMVR may be applied on MMVD conditionally) may not be desirable.

It was reported that the class A1/A2 gains are -0.09% and -0.27% respectively, showing higher potential for UHD sequences.

Further study in CE.

The BoG requests to further discuss in track B the following proposal:

• JVET-N0162 CE9-related: Modified enabling condition for DMVR

Second part of this proposal is related to POC based enabling condition of BDOF. It proposes to only invoke the BDOF if the two reference pictures are at equal distance to the current picture. The same condition (of equal POC distance) is applied to DMVR.

0.01% luma coding loss for RA configuration. Reported decoding time might not be reliable. 1% decoding times increase is reported.

Proponents are asked to report number of samples processed by BDOF before and after this modification (similar to AHG 13). The crosscheck results to be updated by that time to measure decoding time.

In track B discussion, the cross check document was confirmed to have been uploaded as JVET-N0692. The cross check reported 1% decoding time reduction.

The proponent reported that samples processed by BDOF is 3% lower than that of anchor. In AHG13 report, BDOF sample usage was reported to be 44%.

The performance difference does not seem to justify action.

No action was taken on this.

Track B discussed a specific division operation in the spec text (section 8.5.3.5. Parametric motion vector refinement process). In the spec text, there is a note describing that this division can be implemented as shifts and bit masks. The software is currently using shifts and bit masks. It was suggested to change the text to match the software. It was suggested that this change is mathematically equivalent.

It was requested by some experts to see the suggested spec change. S. Sethuraman uploaded a late contribution JVET-N0868 and attached the suggested text change.

Editor action item: Consider this for potential editorial improvement.

JVET-N0815 BoG report on CE9 related contributions [S. Esenlik]

See the notes in section 6.9.