JVET-AP0222 AHG18: Decoder side NN-based picture generation for lost picture substitution [K. Malyshev, M. Sychev, S. Ikonin, E. Alshina (Huawei)]

This document describes an experiment of applying NNFP functionality for generation of lost pictures unavailable due to network transmission issues in ultra-low latency scenario, and proposes modifications to the AHG18 software to enable this functionality based on existing NNPF SEI mechanism. The proposed implementation is controlled via the NNPF interface and utilizes the additional syntax proposed in JVET-AP0226. The generated pictures can be used both for output to display, and for lost reference frames substitution controlled by specific flag. Being tested according to ULL CTC JVET-AN2039, this picture generation technique allows to reduce video freezes significantly, and noticeable improves subjective quality for short interval freeze substitution. At the same time long distance extrapolation remains challenging. The contribution provides a starting point for further investigation and improvement in various aspects of missed picture generation functionality.

Complexity? Not precisely described, run on GPU.

It was asked if also other information related to the lost picture (beyond reconstructed pixels) would be generated? No

Same basic architecture as for the error-correcting loop filter proposed before, also presented in CfE. Kind of non-normative version of it.

Agreed to be included in AHG software (same branch as for loop filter).