Back to Search Document details
39th Meeting: Daejeon, KR, March 2025 2025-04-03 07:30
AHG 11: Neural Network Coded Reference Frame for Intra Coding with Residual Coding and Intra Blocks
Authors: Fabian Brand Huawei Timofey Solovyev Huawei Elena Alshina Huawei
Abstract
This contribution presents a method to combine end-to-end trained image compression methods with long-proven methods from VTM. The method comprises using an end-to-end coded image as reference image for a modified I-Frame. The encoder can then decide whether and how to use the NN-reference frame. In this method data transfer from a GPU/NPU is only necessary on full frame level into the decoded picture buffer, thus solving a major problem in realizing neural-network-based components in hardware while using specialized hardware. Compared to the previous contribution JVET-AK0177 the method now allows full residual coding and standard intra blocks by VTM on top of the AI-coded frame. This method is implemented on top of the single-pass heuristic-based mode selection method from JVET-AK0177. Compared to the AI gains of -3.27%/-0.01%/+0.18% with a runtime of 53% (JVET-AK0177), this method can increase and stabilize the gain to -4.11%/-15.70%/-13.29% with a runtime of 92%. To counter the runtime increase, this contribution furthermore proposes to reduce the runtime by setting the MTT depth to zero if the NN-coded reference frame is used, which reduces the encoder runtime to 63%, with gains of -4.10%/-15.22%/-12.98%.
Meeting Notes
JVET-AL0196 AHG 11: Neural Network Coded Reference Frame for Intra Coding with Residual Coding and Intra Blocks [F. Brand, T. Solovyev, E. Alshina (Huawei)]

This contribution presents a method to combine end-to-end trained image compression methods with long-proven methods from VTM. The method comprises using an end-to-end coded image as reference image for a modified I-Frame. The encoder can then decide whether and how to use the NN-reference frame. In this method data transfer from a GPU/NPU is only necessary on full frame level into the decoded picture buffer, thus solving a major problem in realizing neural-network-based components in hardware while using specialized hardware. Compared to the previous contribution JVET-AK0177 the method now allows full residual coding and standard intra blocks by VTM on top of the AI-coded frame. This method is implemented on top of the single-pass heuristic-based mode selection method from JVET-AK0177. Compared to the AI gains of -3.27%/-0.01%/+0.18% with a runtime of 53% (JVET-AK0177), this method can increase and stabilize the gain to -4.11%/-15.70%/-13.29% with a runtime of 92%. To counter the runtime increase, this contribution furthermore proposes to reduce the runtime by setting the MTT depth to zero if the NN-coded reference frame is used, which reduces the encoder runtime to 63%, with gains of -4.10%/-15.22%/-12.98%.

Switched off for QP22, where loss would occur.

The old approach had some local losses, which is now resolved by allowing to switch to conventional intra coding. Only quadtree is used in fast mode for intra pictures. For RA, normal partitioning is used

Decoding time (CPU) is around 80x compared to VTM in AI. Encoder runtime is also measured on CPU

RA gain around 1.1% on average.

It was asked if any information is available about percentage of usage? Around 50%, by tendency more at high rate, but often the usage is highly dependent on the rate matching. It would be desirable to make such information available in a new version.

It was commented that a combined RDO of NN and conventional branches might improve compression.

Basically every I frame is coded twice, however some early stopping criteria are applied in VTM also when MTT is on.

JPEG-AI would not be suitable for this, as it does not have guarantee of sample-accurate reconstruction.

Was it combined with ECM? No.

The bitstream conveying the NN coded part is separate from the VVC bitstream. Always the entire picture is sent, even when parts are not used. Further optimization could further improve results.

Proponents suggest to establish a VTM branch to enable further study of approaches of this.

ARCHIVE :
JVET-AL0196_v5.docx JVET-AL0196_v5_clean.docx JVET_AL0196.pptx JVET_AL0196_vs_VTM_NoResidual.xlsm JVET_AL0196_vs_VTM_Residual.xlsm JVET_AL0196_vs_VTM_Residual_MTT0.xlsm ~$ET-AL0196_v3_clean.docx ~$ET-AL0196_v4.docx ~$ET-AL0196_v4_clean.docx ~$JVET_AL0196.pptx
Decisions
Proponents suggest to establish a VTM branch to enable further study of approaches of this.
Citation