JVET-AF0061 AHG9: On grouping of post-processing filters [Y.-K. Wang, W. Jia, J. Xu, L. Zhang (Bytedance)]

At the 30^th JVET meeting in April 2023, the BoG on NNPF and SEI processing order SEI messages agreed and the JVET endorsed that it is desirable to support all the following NNPF related capabilities in v4 of VSEI and VVC:

1. Multiple activated NNPFs for a picture in a cascading manner.
2. Multiple activated NNPFs (with the same or different purposes) for a picture in an alternative manner (e.g. choose to apply none, filter a only, or filter b only).
3. Multiple activated NNPFs (with the same or different purposes) for a picture in a cascading plus alternative manner (e.g. choose to apply none, filter a only, filter b only, or both filter a and b in a cascading manner in the indicated order).
4. Multiple activated NNPFs for a picture while the receiver may process the bitstream multiple times to generate multiple different results (e.g. for each time, to choose to apply none, filter a only, or filter b only).
5. Multiple activated NNPF cascades in an alternative manner (e.g. choose to apply none, both filter a and b in a cascading manner in the indicated order, or both filter c and d in a cascading manner in the indicated order).

JVET-AE2032 (Technologies under consideration for future extensions of VSEI (draft 1)) includes a NNPF grouping design resulted from proposals motivated by the above agreement and endorsement. The design involves addition of two new SEI messages, the neural-network post-filter group characteristics (NNPFGC) SEI message and the neural-network post-filter group activation (NNPFGA) SEI message, among other changes.

This contribution asserts that the NNPF grouping design is complicated and hard to understand. The design is also limited to only support NN-based post-processing filters (PPFs).

To address the agreed/endorsed NNPF related capabilities in v4 of VSEI and VVC, and extend the scope to both NN-based and non-NN-based PPFs, this contribution proposes the following changes:

1. Add a ue(v)-coded syntax element po_id to the SEI processing order (SPO) SEI message, replace reserved_alignment_6bits in the SPO SEI message with byte alignment checking followed by po_alignment_zero_bit, and specify the following PPF grouping mechanism (either as part of the semantics of the SPO SEI message or as part of the general post-processing filtering process):
   1. For each picture, there can be multiple PPFs activated and belonging to one or more PPF groups. PPF groups are alternative to each other, i.e., at most one group can be chosen to be applied.
   2. For backward compatibitiy with v3 of VVC/VSEI, a special PPF cascading case is defined, as the case when such two PPFs are both activated for a picture: the two PPFs are both NNPFs, one of the two NNPFs has nnpfc_purpose equal to 4 and the other has multiple input pictures, and neither of the two NNPFs is associated with an SPO SEI message. In this case, the two NNPFs are implicitly considered as belonging to one PPF group, and the NNPF with nnpfc_purpose equal to 4 is applied first.
   3. Except for the special PPF cascading case, each PPF group containing multiple PPFs is associated with an SPO SEI message with a particular value of po_id.
   4. Except for the special PPF cascading case, any PPF not associated with an SPO SEI message is in its own PPF group.
   5. One or more PPFs in the chosen PPF group can be applied. When multiple PPFs (in the chosen PPF group) are applied, they are applied in the cascading manner where they are applied in the order indicated by the SPO SEI message associated with the chosen PPF group, and for each applied PPF that is not the last applied PPF, the output is used as the input of the next applied PPF.
2. Update the filtering process to apply to both NNPFs and non-NN-based PPFs and to enable the use of multiple activated PPFs from one PPF group in the cascading manner. Accordingly, both the NNPFC semantics in VSEI and the interface text in VVC are updated, such that when the PPF is not the first PPF that is applied for the current picture, the input to the PPF is the output of the previously applied PPF.

About item 5 in the first item list, it was asked if a v3 legacy decoder might operate in non-desirable manner. It was commented that wrapping concepts could resolve that, i.e. make the NNPF SEI within a group invisible if its standalone usage is undesirable.

From the concepts of JVET-AF0050, “alternating” and “parallel” are missing according to opinion of its proponents. It is explained that one application of “parallel” would be to generate two outputs, e.g. one for human vision, one for machine vision. The view of “alternating” would be more from encoder perspective, compared to “alternative” where a decoder chooses which one to use.

It was commented that similar could be achieved by an encoder activating both of the “parallel” or “alternating” NNPFs, and let the decoder decide which to use.

It is understood that the concept of JVET-AF0061 is, that if it is possible to activate (and choose at the decoder to run) more than one group, such intended behaviour could be accommodated.

It was commented that a case where number of inputs and outputs in the cascade is not matching (e.g., a temporal upsampler expecting two separately post-filtered frames) might not be supported.

Follow-up presentation on Sun. 15 Oct. on remaining aspects/proposed changes 1)/2) in the second item list:

1) refers to text changes in the SPO SEI.

2) refers to text changes in VVC (particularly specifying input/output interfaces of sequential postprocessing elements, including aspects of cropping and size changes), and changes in VSEI, particularly NNPFC

The aspects of po id are to be clarified in the text development for the agreed overall concept, for changes in the SPO SEI text. It was agreed that fixed length would be preferable.

The other suggested changes are largely agreed to be beneficial, but many changes are proposed, and more in-depth investigation is necessary to sort out which changes are necessary in the SPO SEI itself, in VVC/VSEI body text, and in VSEI NNPF. Therefore, it was agreed to move the include the suggested text changes (except for po_id) in the TuC document for further study, and add notes in the SPO SEI text where specifications might be missing (e.g. for input/output interfaces of sequential filters), and that a possible text can be found in the TuC.

It was suggested to move the paragraphs below semantics of po_id to the section on more general post processing filtering (as per yellow highlight above).