Back to Search Document details
22nd Meeting: by teleconference, April 2021 2021-04-26 23:56
AHG11: Deep-learning based inter prediction blending
Authors: Franck Galpin InterDigital Philippe Bordes InterDigital T. Dumas InterDigital Antoine ROBERT InterDigital Pavel Nikitin InterDigital F. Le Leannec InterDigital
Abstract
This contribution proposes a Neural Network-based Inter prediction blending. The proposed NN replaces the regular blending of bi-prediction of motion compensated blocks. Several networks with different complexity are proposed. All networks use a number of MACs per pixel between 11 and 17 kMACs.
Meeting Notes
JVET-V0076 AHG11: Deep-learning based inter prediction blending [F. Galpin, P. Bordes, T. Dumas, A. Robert, P. Nikitin, F. Le Léannec (InterDigital)]

This contribution proposes a Neural Network-based Inter prediction blending. The proposed NN replaces the regular blending of bi-prediction of motion compensated blocks. Several networks with different complexity are proposed. All networks use a number of MACs per pixel between 11 and 17 kMACs.

It is reported that averages of 0.99%, 0.34% and 0.31% BD rate saving for RA for Y, Cb, and Cr components respectively, are achieved for the most complex network.

  • Networks with 5 or 6 layers. One single network used for all block sizes (8x8 … 128x128)
  • Concatenation: input and output of network are mixed for final prediction
  • “Fast version” does not apply to prediction from certain modes such as affine, CIIP, BCW
  • BDOF is not needed (as it intends to perform a similar refinement) and therefore disabled
  • SATD as cost function in training (gave 0.2%-0.3% better performance)
  • More gain for lower resolution classes such as C,D
  • Approx 17 kMAC/pixel
  • Network parameters quantized to 16 bit integer (important for avoiding mismatch in prediction loop)

It is pointed out that it might be useful to specifically optimize for different POC distances.

It was suggested to study this in an EE. The proponents later expressed that they would not be able to contribute to the EE.

References:
JVET-T2001 JVET-T2006
Cited:
JVET-V0011
PATENTS:
WO/2025/098769 0.34 WO/2022/058293 0.33 20240031611 0.32 20250267310 0.31 WO/2022/017848 0.30 WO/2021/239500 0.28 WO/2021/244884 0.27 20230298219 0.26
ARCHIVE v4 :
JVET-V0076-v3.docx JVET-V0076.pptx m7056_medium.xlsm m9360_ldb_ctc.XLSM m9360_medium.xlsm m9360_medium_fast.xlsm m9360_medium_slow.xlsm m9360_medium_vsbiooff.xlsm vtm11-nobio.xlsm
Decisions
It was suggested to study this in an EE. The proponents later expressed that they would not be able to contribute to the EE.
Citation