JVET-L0691 BoG report on CE4 related contributions [H. Yang]

Reviewed 1500- Monday (GJS)

Three sessions were held, 1600 ~ 2100 on Oct. 5, 0900 ~ 1400 on Oct. 6, and 1400 ~ 2400 on Oct. 7, for discussing 65 technical contributions in six categories,

• Affine motion compensation (17)
  • Line buffer reduction & CPMV unification
  • Harmonization with other tools, e.g., AMVR, MMVD, merge offset
  • Miscellaneous
• Merge mode enhancement (29)
  • ATMVP modifications
  • HMVP modifications
  • Miscellaneous
• Motion vector coding (5)
• Weighted prediction (3)
• Local illumination compensation (3)
• Memory bandwidth reduction (8)

Recommended adoptions to VTM:

• Normative changes
  • Unification of affine CPMV, choose JVET-L0047 method 1 or JVET-L0047 method 2 (the same as JVET-L0373)
    • Method 1 control point MVs are stored and used only for model inheritance, other places (ATMVP storage, deblocking, motion comp, spatial neighbours for merge list, AMVP list derivation) use subblock MVs calculated from the control point MVs. This method has some extra memory (~768 bytes for hardware implementations) and a small benefit (0.05% average). It was commented that a new contribution JVET-L0666 reported that the peak loss for method 2 for some non-CTC affine-friendly sequences was substantially bigger. Method 2 is reportedly always (a little) worse in coding efficiency. Another late contribution reports a way to reduce the extra memory. If the subblock size is made bigger, method 2 would have some inconsistency in the motion vector field relative to the model. Decision (design cleanup): Adopt method 1 as the more consistent and “clean” design (roughly neutral on coding efficiency 0.01%). Further study of other schemes is anticipated.
    • Method 2 uses CPMVs to calculate sub-block MVs for non-corner subblocks within the CU, and some corner subblocks have MVs overwritten by CPMVs. The set of the resulting subblock MVs are used for all operations.
  • ATMVP modification: use fixed subblock size 8x8 for ATMVP (JVET-L0198, JVET-L0468, JVET-L0104, possibly some others). Currently we’re adaptively using 4x4 or 8x8 subblock size, but this has no benefit. Decision: Agreed (approx. no coding efficiency impact).
  • ATMVP modification: restrict ATMVP mode to CUs of which both the width and height are larger than or equal to 8 (L0055), note that this is already a part of 4.2.8 which had been adopted. Decision: Agreed (approx. no coding efficiency impact).
  • ATMVP modification: check the first spatial neighbouring motion vector and use this as the reference motion vector for the collocated position for motion vector derivation (L0198). Decision (complexity reduction): Agreed (approx. no coding efficiency impact).
  • Reset the FIFO table in each CTU row for HMVP (JVET-L0106, JVET-L0158 method 1). Decision (complexity reduction): Agreed (approx. no coding efficiency impact).
  • Merge index coding: use one context for the first bin of the full-block merge index and bypass coding of other bins (L0194). After discussion, it was noted that this had not been tested for the subblock merge list and we don’t want different syntax for the two cases. Further study in a CE was planned to test doing the same thing for the subblock merge list. Decision (complexity reduction): Agreed (for the full-block merge index only at this time, approximately no coding efficiency impact).
  • Generalized bi-prediction (L0646). About 0.66% gain on RA, about 6% increase in encoding runtime. There was discussion of the alternative of using weighted prediction with multiple weights per picture. Weighted prediction would probably work better than this for fade-in, fade-out, and cross-fades (e.g., since it doesn’t need block-level weight selection and since it can extrapolate as well as interpolate), so this proposed method is not a complete replacement for weighted prediction. It was commented that there had been previous contributions describing a benefit for using weighted prediction with multiple weights, and there is some support in the JM for this sort of usage, but not in the HM. AVC had an extra, implicit mode of weighted prediction that was not adopted into HEVC, but it used POC weighting rather than signalling to establish the weights. A proponent noted that the syntax of weighted prediction does not have a shortcut for biprediction with two weights that add up to 1, so the proposed signalling of what is proposed as “generalized biprediction” for VVC would be more efficient when that constraint is intended to apply (e.g., the equivalent of selecting among 5 pairs of weights that add up to 1 would require selection among many more possibilities). Decision (coding efficiency): Adopt JVET-L0646 (0.66% coding efficiency; weighted prediction should also be put in the draft, but this and weighted prediction would be mutually exclusive at the picture level, when used with OBMC the weights of the neigbours would apply for the neighbour predictors, which is how the BMS software already does it, no consideration in deblocking filter). Further study of alternative approaches is expected and encouraged.
  • Prohibit 4x4 bi-prediction for inter CU (L0104). Decision (complexity reduction): Agreed (negligible effect on coding efficiency). Further study is planned for other related aspects.
• Bugfix of VTM software
  • Align the software with the draft text regarding ATMVP motion vector clipping (L0257). Decision (change software to match text): Agreed.
  • Rounding motion vectors toward zero rather than toward minus infinity for AMVR (L0377). Decision (change software to match text): Agreed.
• Encoder optimization
  • Encoder optimization for affine motion estimation (L0260). Decision (software): Adopt (0.3% coding gain, 3% encoding time increase).

Further CE4 tests were recommended based on the following contributions

• Affine motion compensation: L0046, L0281, L0273, L0330, L0259, L0332, L0305, L0320, L0389, L0193, L0522
• Merge mode enhancement: L0092, L0105, L0119, L0302, L0309, L0401, L0091, L0144, L0171, L0207, L0214, L0216, L0319, L0470
• Motion vector coding: L0300, L0301, L0355, L0408
• Local illumination compensation: L0120
• Memory bandwidth reduction: L0122, L0396

Open issues identified by BoG were discussed Tuesday 1100 (GJS)

• Requested review in track
  • JVET-L0093 align VTM with draft text regarding the pruning of regular merge list (the same as JVET-L0282). The draft text does not do full pruning for the spatial and TMVP candidates in the merge list. The software does full pruning. It was reported that there is no loss for not doing full pruning. Decision (bug fix): Align software with text.
  • Whether various LIC (local illumination compensation) techniques can be further studied in CE. LIC schemes can provide about 0.7% coding gain. These schemes have a pipeline dependency problem. Some other techniques do too (e.g., diffusion filter, combined intra-inter coding, Hadamard filter, and bilateral filter). It was commented that we need to control the number of tools of this sort that we would use together, and perhaps make them mutually exclusive. There are some common elements between LIC and CCLM. It was suggested to group together such proposed tools that operate in the post-reconstruction stage of the processing pipeline and to test ways to use these without cascading. CE study was planned.
  • JVET-L0265 to set the chroma subblock size to 4x4 instead of 2x2 for affine motion compensation by averaging the MVs of the 4x4 luma subblocks. It was commented that SIMD implementation is feasible for 4x4 but not 2x2. This has a negligible coding efficiency effect. It was noted that we also have 4x4 ordinary CUs, so this doesn’t entirely solve the 2x2 problem, but would leave that as the only case where this occurs. This would apply to both uni an bi-prediction. Decision (complexity reduction): Adopt.
  • JVET-L0317 sub-block MV clipping in affine prediction. This proposes that the maximum MV difference within an 8x8 area is constrained to +/−1 full-pel difference. This reportedly has a negligible loss (~0.02%). This would apply to both uni- and bi-prediction. There is also another proposal JVET-L0396 and also a third proposal JVET-L0122 and a fourth proposal to restrict the minimum subblock size to 8x4/4x8 for biprediction. The proponent of JVET-L0317 said the JVET-L0396 proposal has higher memory bandwidth and higher coding loss (although the loss is very small in both cases). Details of the comparisons and calculations used at the subblock level in JVET-L0317 and versus the block level for JVET-L0396 were discussed. It was planned to further study these techniques and test them in a CE, since multiple approaches could solve the problem.
  • JVET-L0168 on 16-bit motion vector constraint. MVs currently require 18 bits (due to having 1/16^th pel precision). This proposes several ways to reduce the storage (for temporal MV storage or the local line buffer or both). One of these is to not store that full range, but rather clip the stored MVs to a 16 bit range. Another approach has an adaptive precision, and a third approach removes the two LSBs of fractional precision. For very large picture sizes (e.g., for 360° video) this would not allow the whole reference picture to be addressed. This was suggested to be unnecessary for action at this stage of the work, so this was deferred for further study at a later time.
• JVET-L0048, JVET-L0390, JVET-L0425, JVET-L0187
  • JVET-L0048 had two proposed elements, part of it was the same as JVET-L0046. These will be tested in a CE.
  • A new document JVET-L0694 was submitted that was related to JVET-L0048, testing in combination with other actions taken at the meeting.
  • JVET-L0425 was agreed to be studied in a CE since there are competing proposals
  • JVET-L0390 had two schemes in it – the simplification aspect was agreed to be studied in a CE
  • JVET-L0187 was missing some test results but was preliminarily showing some coding gain. Further study in a CE was planned.
• Contribution not reviewed in BoG.
• JVET-L0201 on weighted prediction – see notes for that topic.

JVET-L0691 BoG report on CE4 related contributions [H. Yang]

See section 7.4.