JVET-AP0222 AHG18: Decoder side NN-based picture generation for lost picture substitution [K. Malyshev, M. Sychev, S. Ikonin, E. Alshina (Huawei)]

This document describes an experiment of applying NNFP functionality for generation of lost pictures unavailable due to network transmission issues in ultra-low latency scenario, and proposes modifications to the AHG18 software to enable this functionality based on existing NNPF SEI mechanism. The proposed implementation is controlled via the NNPF interface and utilizes the additional syntax proposed in JVET-AP0226. The generated pictures can be used both for output to display, and for lost reference frames substitution controlled by specific flag. Being tested according to ULL CTC JVET-AN2039, this picture generation technique allows to reduce video freezes significantly, and noticeable improves subjective quality for short interval freeze substitution. At the same time long distance extrapolation remains challenging. The contribution provides a starting point for further investigation and improvement in various aspects of missed picture generation functionality.

Complexity? Not precisely described, run on GPU.

It was asked if also other information related to the lost picture (beyond reconstructed pixels) would be generated? No

Same basic architecture as for the error-correcting loop filter proposed before, also presented in CfE. Kind of non-normative version of it.

Agreed to be included in AHG software (same branch as for loop filter).

JVET-AP0200 AHG18: Random Access GOP4 Configuration for Live-streaming Applications evaluated in ULL test conditions [S. Ikonin, I. Gribushin, M. Sychev, K. Malyshev, V. Khamidullin, B. Shevchenko, X. Ma, E. Alshina (Huawei), Z. Deng (Bytedance)]

This contribution reports evaluation results of random access GOP4 configuration suggested by contribution JVET-AM0238 in low latency scenario. Under ULL test conditions described in JVET-AN2039 with latency restriction modified to the range of 70 to 130 ms it was observed that temporal layer 0 is delivered in most of the cases confirming the benefits of temporal scalability in low latency scenario. At the same time temporal layers 1 and 2 are often missed due to network transmission issues leading to instant FPS drop clearly noticeable during video playback. The ULL CfE response published as contribution JVET-AN0080 was also evaluated in the same conditions, and at QP22 demonstrated 2.1 dB of average Y-PSNR improvement, and 28% to 16% of freeze ratio reduction for transmission priority channel, and 0.8 dB average Y-PSNR improvement and 31% to 27% freeze ratio reduction on a channel without transmission priorities, with 1.9% BD-Rate overhead of Y-PSNR. The modifications required for evaluation of RA configuration on top of AHG18 simulation software are also described.

It was agreed to include this as an option for further study into the AHG software.

It was commented that this is not “ultra” low latency, unlike the other scenarios/configurations investigated so far, but a higher variety of cases is probably welcome.

JVET-AP0226 AHG9/AHG18: Missed Picture Generation purpose for NNPF SEI message [K. Malyshev, M. Sychev, S. Ikonin, E. Alshina (Huawei)]

This contribution proposes further extension of NNPF SEI messages to support a new purpose for applying NNPF specifically for missed picture generation. The proposal introduces a new MissedGenerationFlag within the nnpfc_purpose syntax element, allowing to dynamically select missed picture handling method. This addition includes the following aspects:

1. A new bit in nnpfc_purpose is dedicated for missed picture generation.
2. When nnpfc_purpose indicates missed picture generation, nnpfc_missed_generation_mode specifies the method, with mode 0 relying on closest picture replacement and mode 1 choosing between interpolation (picture rate upscaling) or temporal extrapolation.
3. A new flag nnpfc_absent_generation_flag for picture rate upsampling NNPF, which removes restrictions to allow generating absent input pictures
4. A new signalling for temporal extrapolation NNPF, that clarifies temporal positions of the extrapolated pictures, eliminating uncertainty.

This contribution signals recommended post-processing that a decoder would apply when the decoder detects a missing picture, e.g. using NNPF for error concealment.

Previously NNPF operation is encoder driven via the NNPFA SEI message. This proposed a fundamental change makes the operation become decoder driven, based on missing pictures.

For the absent generation flag, the proposed language says “the NNPF shall generated at least one new picture…”, which is problematic language for an SEI message.

It...