JVET-AF0043 AhG11/EE1: Status of the joint EE1-0 (LOP.2) training [D. Rusanovskyy, Y. Li (Qualcomm), J. N. Shingala, A. Shyam, A. Suneja, S. P. Badya (Ittiam), T. Shao, P. Yin (Dolby)]

A Unified Filter Architecture for the Low-performance Operation Point (LOP) was proposed in JVET-AE0281 to meet LOP complexity constraints (LOP.2), following HOP based training process. This contribution reports on the progress of LOP.2 joint training and presents full set of LOP2.0, LOP2.3 results. Information presented in this document was partly presented during the AhG11/14 Telcos on 08/09/2023 and 08/30/2023.

After Stage 3 training, selected LOP 2.3 candidate model demonstrated BD-rate change of: {-4.7%, -9.8, -10.1%} and {-5.3%, -10.9%, -10.4%} vs VTM for AI and RA configurations, respectively. Comparing to NNVC anchor (NN-Intra ON), LOP2.3 with enabled NN-Intra demonstrated BD-rate change of {-0.2%, -4.6%, -4.6%} and {-0.3%, -4.3%, -4.3%} for AI and RA, respectively. In this test, filter LOP2.3 utilizes a single model with 0.05M parameters, whereas the Anchor uses 4 models, total size of 0.2M.

Additional BD-rate gain of {-0.0%, -0.5%, -0.2%} and {-0.0%, -1.3%, -0.3%} for AI and RA, respectively, is reported in the sub-test EE1-0.4, targeting improved filter usage (interface).

It is suggested to adopt filter of EE1-0 into NNVC common software and enable it by default for LOP. It is also suggested to make training strategy defined for EE1-0 a part of NNVC software. Some aspects of LOP model training, such as training batch size, complexity-performance trade-off optimization and fast Stage 3 training can be further studied in EE1.

This was presented in JVET Saturday 14 Oct., at 1125 (chaired by JRO).

It was asked if a better luma/chroma balance could be achieved (currently, higher gain in chroma). According to proponents likely yes, pointing to some contributions on this topic.

It is noted that the boundary padding (similar to EE1-1.2.4 for HOP) is already part of the package. It was asked if zero padding (instead of boundary sample values padding) might have subjective impact?

Influence of parameters such as batch size, learning rate? Some of these are selected different than for HOP.

Decision: Adopt JVET-AF0043 LOP 2.3 architecture, and inference interface. This will become part of NNVC7.0 anchor. Configuration files for training shall also become part of NNVC7.0.

Further study in an EE: Fast stage 3 training as proposed in JVET-AF0043, influence of parameters such as batch size, learning rate, improved luma/chroma balance (bits from C to L), continuation of previous subtest EE1-0.2.