JVET-P0986 Report of BoG on CE4 inter prediction related contributions [H. Yang, X. Xiu]

Discussed Wed 9 Oct 1215 (GJS).

Four BoG sessions have been organized for the BoG meeting, 104 contributions had been reviewed.

• Oct. 4, 1430 to 2130, on the category of affine motion compensation and merge mode modifications.
• Oct. 5, 1000 to 1230, on the category of decoder side motion vector derivation.
• Oct. 5, 1400 to 2200, on the remaining contributions in the category of decoder side motion vector derivation and on the category of switchable interpolation filter.
• Oct. 6, 0900 to 1300, on the category of weighted prediction and others.

Contributions on geometric partition and CIIP with triangle partition have not been reviewed.

Actions recommended by BoG:

• Normative changes
  • JVET-P0275: Conditional signalling of TPM syntax max_num_merge_cand_minus_max_num_triangle_cand in SH (not signalling it for B slices). This is a cleanup of unused syntax in B slice header. This is a minor HLS issue and was planned for further study.
  • JVET-P0530: Derive the TPM blending matrix for chroma component by subsampling the matrix for the luma component according to the chroma format. An identical weighting matrix can be derived for 4:4:4 content using the proposed method, which could remove potential visual artefact introduced by applying different weighting matrix for luma and chroma in the current TPM blending. Decision: Agreed.
  • JVET-P0325: Change the checking order of the first two spatial merge candidates, A1->B1 to B1->A1. Coding gain of 0.05% for RA and 0.21% for LD is achieved by the proposed change. This is a very small optimization, but was considered minor cleanup. It was commented that the previous order had been rather arbitrarily chosen and this is just coming "for free". The cross-checker said this made logical sense as a likely consequence of the way HMVP works, an aspect that was added later. Decision (minor borderline cleanup agreed as an exception): Agreed.
  • JVET-P0091: BDOF and PROF unification by moving 1-bit right-shift of calculating the BDOF sample offset to the derivation of motion refinement. It was commented that part of JVET-P0201 is also found in this method. The proposed method aligns the sample offset calculation of the BDOF and PROF while reducing the number of right-shifts by moving one bitwise right-shift to the 4x4 subblock level (removing a shift at the per-sample level). Decision (minor cleanup): Agreed.
  • JVET-P0154 method 1/JVET-P0094/JVET-P0172 method 1/JVET-P0143/JVET-P0518 method 1/JVET-P0281 3: Clip the PROF sample offset to 14-bit. The prediction samples after the PROF refinement is guaranteed within 16-bit value, and an overflow of the 16-bit multiplication logic in the weighted prediction is avoided. (An alternative proposal was to disable PROF when BCW with unequal weights or higher-level weighted prediction is used.) Decision (bug fix): Agreed.
  • JVET-P0653/JVET-P0281 method 1: remove the dependency between internal bit-depth and the calculation of intermediate parameters, three shifts and one clipping threshold in BDOF. It was commented that parts of JVET-P0200 and JVET-P0238 are also found in this method. The proposed method unifies the shift parameters for deriving the BDOF parameters for different internal bit-depths. This was said to be partly because of a simplification adopted at the previous meeting that reduced the necessary dynamic range, such that some right shifting is no longer needed to stay within 32 bits. Decision (cleanup): Agreed.
• Decision (SW): Software optimizations (for which the software coordinator has the final call):
  • JVET-P0092: Encoder speed-up for SMVD. The first aspect is to reduce the maximum number of initial MV candidates to 5 from 18. The second aspect is to align the distortion calculation in motion search to the regular bi-prediction case. For about 50 lines of code changed, this provides 6% encoder speedup!
  • JVET-P0445: Encoder optimization for subblock-based merge candidate search. It is proposed to remove unnecessary chroma motion compensation for luma SATD-based merge candidate selection for affine merge and sbTMVP modes. No change in bitstream content and 2% encoder speedup!
  • JVET-P0512: SIMD implementation for motion compensated prediction when the internal bit-depth is larger than 10-bit. The proposal provides SIMD-based software optimization for motion compensation module in VTM for high internal bit-depth. For no change in bitstream or decoded output, for 12 bit video, this provides about 20% encoder speedup and 15% decoder speedup!
• Decision (Ed.): Per below
  • JVET-P0214: Change the chroma intra mode from DM to planar in CIIP (this is merely editorial, since the DM mode is always planar in this case).
• Decision (BF): Per below
  • JVET-P0595: Align spec with SW on the derivation of weighting factor for CIIP mode. Derive the weighting factor for luma component and apply the same weighting factor to chroma components as well.
  • JVET-P0856: Align spec to SW on the two mismatches in SIF and BCW index. Do not use hpelIfIdx and bcw_idx in the derivation process for history-based merging candidates and in the updating process for the history-based motion vector predictor candidate list. Also store the hpelIfIdx in the HMVP table.
  • JVET-P0280: Align the spec to SW on interaction of BCW and (explicit) WP.
  • JVET-P0519: Align the spec to the SW on the SAD threshold for the early termination in BDOF.
• Requested items for further discussion in track:
  • JVET-P0151/JVET-P0260/JVET-P0461/JVET-P0542/ (one approach in JVET-P0490): the derivation of hpelIfIdx for pairwise merge candidate. Two options,

1. Keep the current design (which has an "and" operation)
2. Set hpelIfIdx to 0, as proposed in JVET-P0151/JVET-P0260/JVET-P0461/JVET-P0542/JVET-P0490.

No action seemed necessary on this, and the design philosophy arguments did not converge.

• • JVET-P0236/JVET-P0617: Enable WP in combination with TPM. Currently, these cannot be used together. It was agreed that this is desirable. JVET-P0236 basically cascades the two operations (WP first, then TPM, with clipping after each – this also changes the non-WP case because of the clipping). JVET-P0617 combines the two operations and clips the final result. The results were cross-checked and had used the prior CE test conditions for WP. JVET-P0236 has a latency issue since it adds an additional stage; JVET-P0617 needs an additional multiplier for the blending. One comment was that if TPM doesn't work well together with WP, the encoder could just not choose TPM (or the combination could be prohibited in some way). Some participants commented that we should not add this extra complexity for a benefit that is only found on special content such as fading content. In the fading sequences, a benefit of about 0.9% was reported. It was commented that BCW (and BDOF and DMVR) are also mutually exclusive with WP. Further study was suggested, particularly to consider disallowing the combination and to consider whether the other prohibited combinations should be allowed and to consider subjective effects.
  • JVET-P0191/JVET-P0311/JVET-P0415/JVET-P0546/JVET-P0598: BDOF and DMVR enabling/disabling condition on POC distance and reference picture type

1. No change to the existing application conditions of BDOF and DMVR.
2. Apply an equal POC distance constraint to both BDOF and DMVR (in the current design, such constraint is applied to DMVR). JVET-P1023 reports that this constraint would have almost no impact on coding efficiency.
3. No equal POC distance constraint to both BDOF and DMVR (i.e. applying the current BDOF condition to DMVR).
4. Disable the two tools when any long-term picture is referenced.

JVET-P1023 the constraints 2 and 4. Decision (cleanup): Adopt.

• • JVET-P0314/JVET-P0544/JVET-P0314/JVET-P0434/JVET-P0524/JVET-P0601: slice-level (picture-level) on/off control of BDOF, DMVR and PROF

1. Use the same slice-level (picture-level) BDOF and DMVR control flag to control the PROF at the slice level.
2. Use one additional flag for PROF at slice level.

In Draft 6, there is a shared flag for BDOF and DMVR in the SH. These do not have block-level control. Decision (cleanup): Three distinct flags in the PH (presence conditioned on SPS), not in SH.

• • JVET-P0320: Add B slice condition for the syntax element slice_disable_bdof_dmvr_flag flag at slice header. Conditioning syntax element presence on picture types is a minor syntax cleanup issue and was left for further study.
  • JVET-P0491: Change the clipped range of BDOF and PROF motion refinement to [-31, 31] instead of [-32, 31]. This would save 8 clippings. In terms of spec impact, it just changes one character. Decision (cleanup): Adopt.
  • JVET-P0090: Discuss if the limitation of abs_mvd_min2 binarization within 32-bit is needed. Approach #2 is recommended if the limitation agreed. Decision (cleanup): Adopt.
  • JVET-P0385: Modify the rounding of offset vector for sbTMVP from “rounded towards negative” to “rounded to zero”. It was said that everywhere else where we have rounding of motion data, we round toward zero. Decision (cleanup): Adopt.
• Further study recommendation:
  • JVET-P0444: DMVR without padding and the search range for 8x16 and 16x8 DMVR CUs is reduced from 2 to 1. A substantial decoder runtime reduction is asserted with 0.05% loss in coding efficiency. Some participants said this would have an average memory bandwidth increase and a local buffering increase and suggested that the decoder runtime reduction might just be a matter of code optimization. It was said that this had been studied previously. Some suggested study in an AHG rather than a CE. Further study was recommended, but not in a CE.

JVET-P0986 Report of BoG on CE4 inter prediction related contributions [H. Yang, X. Xiu]

See section 6.4.