JVET-M0858 BoG report on CE9 related decoder-side motion vector derivation contributions [S. Esenlik]

This report reviewed in Track B Wednesday 16 January 1115-1320 (chaired by JRO).

One session was held between 15:00 ~ 20:15 on January 12, 2019, for discussing 12 technical contributions in two categories, BDOF (7 contributions) and DMVR (5 contributions).

Recommended for adoption by BoG:

• JVET-M0063 Non-CE9: An improvement of BDOF
  • Generalization of BDOF bit-depth restriction for internal bit-depths other than 10 bit.
  • No impact on CTC.
  • 8-bit coding scenario: -0.32/-0.32/-0.31% change in BD-rate (Y/CB/CR)
  • 12-bit coding scenario: -0.46/-0.03/0.08% change in BD-rate (Y/CB/CR)

From discussion in Track B: A possible reason for this behaviour might be the wrong interpretation of the gradient in case of bit depths other than 10.

It was asked if the change would still be supporting the 16 bit SIMD design of software? The proponent confirms that this is the case; this may need further checking by SW coordinators.

Decision (BF): Adopt JVET-M0063.

This was further discussed in plenary: What is the general support of different bit depths in VVC? There might be other tools that were added in recent meetings that have similar problems. Definitely, bit depths up to 12 bits should be supported consistently, whereas it is likely that for higher bit depths some more precision might be required. The editors were asked to consider this; it needs to be resolved in further work.

Recommended CE tests, BDOF related:

• JVET-M0073 Non-CE9: On early termination for BIO
• JVET-M0249 Non-CE9: Modifications on Bi-Directional Optical Flow
• JVET-M0284 CE9-related: BDOF Modifications to Enable 64x64 VPDU
  • Regarding hardware implementation of BDOF, at least, there are two or three problems.
  • One is a buffering latency of SAD calculation for early termination before BDOF.
  • Second is huge number of memory access with large CU.
  • Third is memory access limitation to enable VPDU.
  • Creating a new CE to solve above problems.
  • None of the methods has significant impact on compression performance.

It was discussed in Track B if it might be better to remove early termination totally to simplify the design and solve the buffer latency problem, as this does not have impact on worst case complexity. Document JVET-M0890 reports that by removing the early termination, the result improves by 0.04%, but decoder runtime increases by 3%. This should be one of the comparison tests in the CE. Furthermore, the maximum size where early termination is used should be blocks of size 256 luma samples.

Recommended CE tests, DMVR related:

• JVET-M0077 CE9-related: Relaxation of block size restriction for DMVR
  • -0.81/-0.90/-1.00% change in BD-rate (Y/U/V) with 111/119% enc/dec time.

From the discussion in Track B: This is estimated to have up to 0.2% loss compared to the adopted version, and does not show decoding time decrease (even though it performs subsampling in SAD calculation). Not worthwhile to consider in CE.

• JVET-M0078 CE9-related: Combination of JVET-M0077 and CE9.2.5
  • Study early termination aspect of JVET-M0078 (which is also described in JVET-M0062) separately if MRSAD version of DMVR is included.

From discussion in Track B: It is not necessary to study MRSAD, since SAD version of DMVR was adopted. There is another aspect that points out a similar problem as in JVET-M0063 where the early termination is dependent on bit depth and block size. As this would not affect CTC condition, there is no need to study in CE. Proponents are requested to study this aspect further with regard to the adopted version of DMVR.

• JVET-M0148 Non-CE9: Simplifications to DMVR search pattern and interpolation for refinement
  • First part proposes to replace 25 sample search space with 13 total points.
  • Approximately: 0.08/0.1/0.09% BD-rate change (Y/U/V) w.r.t. to the anchor that is used (CE9.2.1g).
  • Second part proposes simplification in complexity of bilinear interpolation of DMVR CE9.2.1g version
  • 0.04/0.03/0.04 BD-rate change (Y/U/V) w.r.t the anchor that is used (CE9.2.1g).

From discussion in Track B: This reduces decoder run time by approx. 1%. It also introduces some more specific processing at boundaries. The benefit was not obvious, as it introduces some compression loss. This was determined not worthwhile to consider.

• JVET-M0516 Non-CE9.2.1.e: Non-local-mean-based MRSAD and Row-subsampled Search Pattern for DMVR
  • Approach 1: proposes Non-local mean value calculation for MRSAD
    • 0.07% luma BD-rate increase with same enc/dec times. Benefit over SAD might be around 0.05% bit rate decrease.

From discussion in Track B: This still would require one more stage to first compute the mean before the SAD can be computed.

• • Approach 2: proposes 15 point row-subsampled search pattern.
    • 0.17% luma BD-rate increase with about 2% reduction in dec time

From discussion in Track B: This amount of loss is not a good tradeoff versus the complexity benefit.

• • Approach 3: proposes to disable DMVR for 4xN and 8x8 DMVR.
    • 0.06% luma BD-rate increase with about 3% reduction in dec time
  • Approach 4: proposes to disable DMVR for 4xN and 8x8 DMVR, and also remove padding, such that in this combination the worst-case memory BW would still be the same as 8x4 bi pred.
    • 0.04% luma BD-rate increase with about 3% reduction in dec time

It is further claimed that for the current DMVR (which only disables DMVR for Nx4 but not 4xN), the current worst-case memory bandwidth may still be exceeded for certain memory access patterns.

Study the aspects 3 and 4 in CE, including an analysis of memory access bandwidth of current scheme and proposed schemes.

The following contributions were further discussed in Track B:

• JVET-M0223 Non-CE9: Co-existence analysis for DMVR with BDOF

From discussion in Track B: In the current design, DMVR and BDOF may be processed sequentially (depending on condition that DMVR does not determine a modified MV, BDOF is computed). This contribution shows that parallel computation, and deciding by an additional cost criterion, does not provide benefit. No action was taken to change the current approach, but it would be desirable to find another solution on the cascade operation.

• JVET-M0316 CE9-related: simplification of BDOF

From discussion in Track B: This has the benefit of re-defining the BDOF refinement such that no multiplications are necessary. Rate increase 0.15% for luma, with 4-5% decoding time reduction.

As we don’t know the BDOF gain when operated together with DMVR, the loss may become lower in VTM4. Depending on the outcome, such a complexity reduction could be attractive. It was agreed to study this in a CE.

• JVET-M0517 Non-CE9: Methods for BDOF complexity reduction

From discussion in Track B: Two aspects are considered a) subsampling in OF computation b) reduction of precision of multipliers. There is no noticeable change in decoder runtime, the benefit would be more relevant for hardware (in particular aspect b). However, in BDOF the gradient computation is less of concern in terms of complexity. The overall benefit was not clear – so this was considered not relevant for a CE.

JVET-M0858 BoG report on CE9 related decoder-side motion vector derivation contributions [S. Esenlik]

See section 6.9.