JVET-AJ0059 AHG9: On NNPFC SEI message with multiple instances of temporal extrapolated pictures [E. François, C.-H. Demarty, Didier Doyen, P. de Lagrange (InterDigital)]

This contribution is a follow-up of contributions JVET-AI0296 and JVET-AI0322 on the possibility to generate multiple versions of a same temporal extrapolated picture for a same time instant. The contribution aims at explaining the latency issue in interactive applications such as cloud gaming or virtual reality (VR), and the way it may be solved thanks to producing, in advance of the rendering, several versions of a same extrapolated picture and selecting the proper version at the time of the rendering. The contribution proposes adaptations of the NNPFC SEI message design to support this feature when the purpose of the SEI message is temporal extrapolation.

In JVET-AI0322 the multiple extrapolations are provided via an additional dimension in the output tensor, while this contribution proposes using subpictures to contain each version of the temporal extrapolated picture.

Notes from review of JVET-AI0322 are as follows: Further study, especially on the impact on the NNPF system design and interfaces, and how it is determined which temporal extrapolation is selected for output.

This contribution and JVET-AI0322 suggest using external means to select which temporal extrapolation is used.

It was questioned whether this feature should be standardized or left to an application.

It was commented that the NNPFC in earlier version of VSEI it was required that all pictures output by an NNPF shall be in increasing output order and no two output pictures may have the same output time instance.

It was noted that a subpicture approach does not require having multiple output pictures at the same output time, but outputs a larger picture containing the results of multiple temporal extrapolations. The additional tensor dimension approach uses external means to determine what the output picture is.

It is noted that the current design does not support multiple versions of spatial extrapolation.

A question was raised about what is different about the temporal extrapolation purpose that benefits from multiple versions vs. for other purposes. It was suggested that extrapolation purposes are inherently more likely to benefit from multiple versions.

It was suggested that the decoder application could generate multiple versions of temporal extrapolated pictures.

Timing of issuance of the DIS ballot isn’t determined, but may be at this meeting or at the January meeting. Delaying to the January meeting might better align ISO/IEC and ITU-T approvals. If this capability of multiple temporal extrapolations will eventually be added, it would be preferable to be included in the DIS of VSEI v4, to avoid future extensibility issues.

If DIS is not issued this meeting, it was suggested to include both this contribution and JVET-AI0322 in a TuC and make the decision at the next meeting if something should go into the DIS.

Further discussion requested after it is decided if DIS will be issued at this meeting.

Further discussed 1415 on 7 Nov 2024.

Decision: Adopt into TuC. Also adopt JVET-AI0322 into TuC.