JVET-AG0089 AHG9: Temporal extrapolation purpose for the neural-network post-filter characteristics SEI message [M. M. Hannuksela, F. Cricri, H. Zhang (Nokia)]
The neural-network post-filter characteristics (NNPFC) SEI message includes the nnpfc_purpose syntax element, which enables to signal any number of purposes to which the defined neural-network post-filter (NNPF) is intended. For example, the purposes include picture rate upsampling through interpolating pictures between one or more pairs of input pictures.
It is asserted that there are use cases for temporal extrapolation, i.e., generating one or more pictures following all the input pictures in output order. Hence, this contribution proposes the addition of temporal extrapolation as one of the purposes of nnpfc_purpose. This addition includes the following aspects:
- A new bit in nnpfc_purpose is dedicated for temporal extrapolation.
- When nnpfc_purpose indicates temporal extrapolation, nnpfc_extrapolated_pics_minus1 plus 1 indicates the number of extrapolated pictures generated by the NNPF.
- The constraints to avoid conflicting or ambiguous output order are updated to concern extrapolated pictures in addition to interpolated pictures.
The main difference relative to frame rate upsampling would be that the extrapolated picture would be after the input picture (not allowed by the current syntax).
It was asked if a network designed for interpolation could also be used for extrapolation? Potentially, but it is more likely that another dedicated network would be designed.
It was pointed out that in picture extrapolation applications (navigation, cloud gaming) it is also common that multiple hypotheses are generated for the same temporal position. This would not be supported by the proposed syntax, but could be added and would make another differentiation from the current picture rate upsampling.
From the comments made during the review, the idea of defining a separate purpose was supported and assessed to be important for various application.
Decision: Include JVET-AG0089 in preliminary WD JVET-AG2034. The option of a network generating several instances at same temporal position shall be further studied, as there may be some general conflict with constraints in the existing NNPF architecture (disallowing several outputs at same time instance).