JVET-AP0222 AHG18: Decoder side NN-based picture generation for lost picture substitution [K. Malyshev, M. Sychev, S. Ikonin, E. Alshina (Huawei)]

This document describes an experiment of applying NNFP functionality for generation of lost pictures unavailable due to network transmission issues in ultra-low latency scenario, and proposes modifications to the AHG18 software to enable this functionality based on existing NNPF SEI mechanism. The proposed implementation is controlled via the NNPF interface and utilizes the additional syntax proposed in JVET-AP0226. The generated pictures can be used both for output to display, and for lost reference frames substitution controlled by specific flag. Being tested according to ULL CTC JVET-AN2039, this picture generation technique allows to reduce video freezes significantly, and noticeable improves subjective quality for short interval freeze substitution. At the same time long distance extrapolation remains challenging. The contribution provides a starting point for further investigation and improvement in various aspects of missed picture generation functionality.

Complexity? Not precisely described, run on GPU.

It was asked if also other information related to the lost picture (beyond reconstructed pixels) would be generated? No

Same basic architecture as for the error-correcting loop filter proposed before, also presented in CfE. Kind of non-normative version of it.

Agreed to be included in AHG software (same branch as for loop filter).

JVET-AP0200 AHG18: Random Access GOP4 Configuration for Live-streaming Applications evaluated in ULL test conditions [S. Ikonin, I. Gribushin, M. Sychev, K. Malyshev, V. Khamidullin, B. Shevchenko, X. Ma, E. Alshina (Huawei), Z. Deng (Bytedance)]

This contribution reports evaluation results of random access GOP4 configuration suggested by contribution JVET-AM0238 in low latency scenario. Under ULL test conditions described in JVET-AN2039 with latency restriction modified to the range of 70 to 130 ms it was observed that temporal layer 0 is delivered in most of the cases confirming the benefits of temporal scalability in low latency scenario. At the same time temporal layers 1 and 2 are often missed due to network transmission issues leading to instant FPS drop clearly noticeable during video playback. The ULL CfE response published as contribution JVET-AN0080 was also evaluated in the same conditions, and at QP22 demonstrated 2.1 dB of average Y-PSNR improvement, and 28% to 16% of freeze ratio reduction for transmission priority channel, and 0.8 dB average Y-PSNR improvement and 31% to 27% freeze ratio reduction on a channel without transmission priorities, with 1.9% BD-Rate overhead of Y-PSNR. The modifications required for evaluation of RA configuration on top of AHG18 simulation software are also described.

It was agreed to include this as an option for further study into the AHG software.

It was commented that this is not “ultra” low latency, unlike the other scenarios/configurations investigated so far, but a higher variety of cases is probably welcome.

JVET-AP0018 JVET AHG report: Ultra-low latency and packet loss resilience (AHG18) [S. Deshpande, S. Ikonin, V. Zakharchenko (co-chairs), S. Fößel, C. Kim, X. Ma, S. Puri, J. Ström, S. Wenger (vice-chairs)]

1. Software development

Adoptions of last JVET meeting (JVET-AN2039: ULL CTC implementation, JVET-AO0208: GDR LDB fixes) have been merged to ‘ull-master’ branch: https://vcgit.hhi.fraunhofer.de/jvet-ahg-ull/VVCSoftware_VTM/-/tree/ull-master.

1. Related contributions

A total of 3 contributions (not including this AHG report) are identified relating to the mandates of AHG18. They are listed below. One contribution also related to the work of AHG9.

1. Recommendations

The AHG recommended:

• Review all input contributions.
• Discuss test conditions and evaluation methodology.
• Collect test cases and specific requirements.
• Continue development of simulation software.
• Continue the study of ultra-low latency and packet loss resilience technologies in JVET.