Back to Search Document details
39th Meeting: Daejeon, KR, June 2025 2025-07-01 04:55
AHG 14: Work on NN-coded Base Layer in VTM / NNVC
Authors: Fabian Brand Huawei Timofey Solovyev Huawei Elena Alshina Huawei F. Urban InterDigital Franck Galpin InterDigital
Abstract
In the April meeting, JVET decided to start developing a framework for including NN-coded reference frames into VVC / NNVC. A first version was provided by InterDigital. This version required an additional YUV sequence containing the reconstructed NN-coded frames, which needs to be passed to both encoder and decoder. This strategy makes the system very versatile with respect to the used NN-coder, however it has the drawback that is not easy to verify that the reported results are correct, since it relies on external bit-stream counting. This contribution presents modifications made to the code to alleviate this problem. In particular this includes
Meeting Notes
JVET-AM0050 AHG 14: Work on NN-coded Base Layer in VTM / NNVC [F. Brand, T. Solovyev, E. Alshina Huawei), F. Urban, F. Galpin (InterDigital)]

In the April meeting, JVET decided to start developing a framework for including NN-coded reference frames into VVC / NNVC. A first version was provided by InterDigital. This version required an additional YUV sequence containing the reconstructed NN-coded frames, which needs to be passed to both encoder and decoder. This strategy makes the system very versatile with respect to the used NN-coder, however it has the drawback that is not easy to verify that the reported results are correct, since it relies on external bit-stream counting. This contribution presents modifications made to the code to alleviate this problem. In particular this includes

  • Providing hooks in form of external scripts which call the NN-based coder
  • Including the bitstream of the NN-coder into the main bitstream
  • Decoding the NN-coder bitstream during encoding time

The code is available under https://vcgit.hhi.fraunhofer.de/jvet-ahg-nnvc/VVCSoftware_VTM/-/tree/hybrid_codec_e2e_dev

NN intra stream is transported in a new type of NAL unit. Possible to use with any NN based intra codec by modifying script.

Still more development necessary (bug fixes, extension to more than one I picture, enable LMCS, integration in new NNVC based on VTM23) – planned for subsequent meeting cycle.

Software should also work without NN coded base layer (was tested in an earlier stage of development).

Some loss observed for class A1.

Currently, only limited possibility of joint optimization of NN based and VTM layers. Quality of NN codec can be controlled by QP. As a first step, a CTU based switching might be implemented. Joint RDO may be difficult.

No immediate action was take on this – further development was requested in AHG11/14.

It was later reported that the reason for the mismatch had been found and fixed during the meeting.

ARCHIVE :
JVET-AM0050_v3.docx JVET-AM0050_v3_clean.docx JVET_AM0050_vs_VTM239.xlsm Presentation.pptx
Decisions
It was later reported that the reason for the mismatch had been found and fixed during the meeting.
Citation