JVET-AN0225 [AHG11][AHG14]: Further Improvements on Hybrid Multilayer Framework for End-to-End Learned Intra Frame [M. Aderdor, T. Solovyev, E. Alshina (Huawei)]

This contribution reports the recent progress in the development of the hybrid codec exploration branch of NNVC, which allows usage of an end-to-end coded I-frame in order to improve compression performance. The first version of the multi-layer framework did not provide the expected compression benefits compared to the previous proposal of an end-to-end coded I-frame, which uses a single layer in the VTM. In this contribution, the difference is narrowed down only to high level syntax associated with multi-layer. This contribution describes the details of modifications and proposes harmonization between ‘multi-layer’ and ‘single layer’ frameworks. Further software changes are proposed to improve performance further.

The presented bug fixes and proposals increase overall PSNR BD rate of multilayer hybrid VTM DCVC-FM vs. VTM 23.11 from:

AI: -4.75% (Y) -11.24% (U) -10.60% (V)

RA: 5.69% (Y) 1.00% (U) 3.25% (V)

to:

AI: -5.31% (Y) -11.58% (U) -10.80% (V)

RA: -2.23% (Y) -7.06% (U) -4.43% (V)

Some bug fixes were implemented in the context of this contribution on top of NNVC 14.1, caused by inconsistencies in RPL, and problems occurring when GOP size was not identical with intra period.

It was asked to include software with the bug fixes in an update of the contribution.

Further changes proposed are enabling LMCS in E2E base layer (shifting some chroma gain to luma, which however may depend on the training of the E2E network), and enabling MCTF (which was not used before, but not disabled for technical reason). This provides a gain of approximately 2% (most comes from MCTF).

As an additional result, an average overhead of 0.4% is reported for the multi-layer approach in AI (on average over all classes, by tendency larger for smaller resolution).

Run time reported in the xls is unreliable.

The usage of LMCS is asserted to be useful for flexibility in experimentation with AI based codecs for intra pictures. Enabling MCTF is just an issue of config file.

Residual in enhancement layer was not skipped, but probably only used for high rate.

Decision (SW): Implement LMCS as a configurable option from JVET-AN0225 in the NNVC multi-layer interface with external base layer.

Complexity around 500 kMAC/pix.

JVET-AN0225 (non-CTC). LMCS as configurable option in NNVC multi-layer interface for external base layer.