JVET-AF2032 Technologies under consideration for future extensions of VSEI (version 2) [M. M. Hannuksela, J. Chen, S. Deshpande, Hendry, S. McCarthy] [WG 5 N 242)] (2023-12-01)

The subsequent list was provided after the meeting by Y.-K. Wang, after consultation with relevant contributors which had been agreed for inclusion.

1. Changes in JVET-AF0061 other than the addition of po_id [Bytedance]
   1. The post-processing filter (PPF) grouping concept based on po_id in the SPO SEI message, taking into account backward compatibitiy with v3 of VVC/VSEI for the a special PPF cascading case
   2. Updates to the filtering process to apply to both NNPFs and non-NN-based PPFs and to enable the use of multiple activated PPFs from one PPF group in the cascading manner
   3. Updates to the NNPFC semantics in VSEI
   4. Updates to the NNPFC/NNPFA interface text in VVC
2. 2 items from JVET-AF0050 [Nokia]
   1. Addition of nnpfga_no_prev_clvs_flag and nnpfga_no_foll_clvs_flag
   2. Removal of the possibility to reference NNPF group identifiers from the NNPFA SEI message
3. The approach of separate SEI (applicable for any PF) from JVET-AF0051, for signalling of the gain provided by post-processing filter(s). Remove the URI, define PSNR in a bit-depth neutral. [Nokia]
4. Items 1, 2 and 3 in JVET-AF0091 [LGE]
   1. Adding the definitions of direct and indirect members of an NNPFGC
   2. Clarifying that the value of nnpfgc_purpose in NNPFGC, when present, includes all purposes of NNPFC or NNPFGC included in the NNPFGC
   3. When nnpfgc_grouping_type is equal to 1 (i.e., alternative grouping) or 3 (i.e., parallel grouping), purpose information is signalled for each member of the NNPFGC.
5. On intermediary output picture(s) from activation of an NNPFGC, from JVET-AF0093 [LGE]
6. Some text clarification of the intent on activation of an NNPFGC that contains another NNPFGC could be useful, as a result from the discussion of JVET-AF0094. [LGE]
7. Some of the items on the source picture timing information (SPTI) SEI message, from JVET-AF0055 [Sharp]
   1. Item 1: Add a NOTE.
   2. Item 2: Agreed that conditioning the syntax element “spti_max_sublayer_minus1” on persistence is appropriate.
   3. Item 3: Editorial action to clarify that the interval at highest layer is not necessarily 1.
   4. Item 5: Addition of text for “Use of source picture timing information” for the relevant standards.
   5. Item 6 (Syntax rearrangement is proposed such that the syntax element spti_source_picture_timing_type is signalled only when spti_source_timing_equals_output_timing_flag is equal to 0.): Agreed (in the context of discussing JVET-AF0069) to adopt item 6, however using the spti_source_type_present_flag for possible gating of the syntax element.
8. Items on the source picture timing information (SPTI) SEI message, from JVET-AF0069 [Dolby]
   1. Item 1: It was agreed that the name of the previous spti_source_timing_equals_output_timing_flag should not be modified into spti_source_timing_info_present_flag. If it is zero, the spti_source_type shall be sent (with its gating flag).
   2. Simplify syntax element name spti_source_picture_timing_type to spti_source_type.
   3. Change the precision of spti_source_type from u(8) to u(16).
   4. Simplify the syntax element name spti_num_units_in_elemental_source_picture_interval to spti_num_units_in_elemental_interval.
   5. Simplify the syntax element name spti_sublayer_source_picture_interval_scale_factor[ i ] to spti_sublayer_interval_scale_factor[ i ].
   6. Change the precision of spti_num_units_in_elemental_interval from u(32) to u(18)
   7. Add text to clarify the descriptions of the various source types, e.g., “slow motion”, “high-speed imaging”, etc.
   8. Add semantic constraints to prevent mutually exclusive timing relationships between source pictures and corresponding decoded output pictures. Specifically, prevent the combination of “high-speed imaging” and “time-lapse imaging”.
   9. Replace syntax element spti_source_timing_equals_output_timing_flag with spti_source_timing_info_present_flag and add corresponding semantics.
   10. Move specification of the variable temporalReversalFlag to the semantics following spti_sublayer_interval_scale_factor[ i ].
   11. Integrate the variable temporalReversalFlag in the equation for SourcePictureInterval[ i ] and remove the equation for SourcePictureTime[ i ] (i.e., the absolute source picture time).
9. Add the following text, resulted from JVET-AF0097 item 1 [LGE]:

The information provided by the SPTI SEI message pertains only for picture(s) starting from the picture in the current layer in the access unit that contains the SPTI SEI message and all subsequent pictures of the current layer in output order based on its persistence.

1. Some of the items on the encoder optimization information (EOI) SEI message, from JVET-AF0052 [Nokia]
   1. Item 1: Editorial updates to the phrasing of the cancellation and persistence.
   2. Item 2: A syntax element eoi_object_based_idc to indicate the type of object-based optimization, including blurring, quantization adjustment, and overwriting sample values of areas outside the detected objects.
   3. Item 4: Replacement of the optimization_ prefix in the syntax element names with eoi_ to obtain shorter syntax element names.
   4. Item 5: Sensibility constraints that when the persistence is for the current picture only, the temporal optimizations (eoi_temporal_subsampling_flag and eoi_temporal_quality_flag) are required to be off.
   5. Item 6: For temporal subsampling, addition of eoi_num_int_pics, which is indicative of the count of pictures that the encoding systems excluded between each pair of coded pictures in output order.
   6. Item 7: For temporal quality optimization, a clarification of the semantics and addition of a related NOTE.
2. Some of the items on the EOI SEI message, from JVET-AF0107 [LGE]
   1. On re-formulation spatial/temporal resampling from subsampling/downsampling (which would also apply to upsampling).
   2. Simplification of optimization_type table (editorial) in v2 of JVET-0107 in its section 2.1.
   3. It is further reported that in the combination with the adoption from JVET-AF0052 and JVET-AF0107, a discrepancy was found that is resolved in the green highlighted parts of v2 of JVET-0107. It was also agreed to include these changes in the updates of the EOI SEI in the TuC.
3. Items on the object mask information (OMI) SEI message, from JVET-AF0088 [Alibaba]
   1. Aspect 1: The text related to bounding box parameters was fixed and refined.
   2. Aspect 2: The binarization of bounding box parameters was changed from ue(v) to u(16)
   3. Aspect 3: A gating flag for bounding box parameters was added to give signalling flexibility to the encoder
   4. Aspect 4: The parsing dependency among different OMI SEI messages was removed by always signalling omi_mask_cancel[ i ][ j ][ k ]
4. SEI messages for image metadata formats EXIF, JFIF, and XMP, from JVET-AF0141, with an appropriate editor’s note on how these formats could be referenced in an ITU-T/ISO/IEC standard. [Tencent]
5. Signal picture modality type in VUI parameters, from JVET-AF0147. [Panasonic]
6. The lsei_message( ) syntax structure for carriage of information about an SEI payload and the SEI payload itself, from JVET-AF0148 [Tencent]
7. The multiplane image information (MPII) SEI message from JVET-AF0167 [Dolby]

JVET-AF2032 Technologies under consideration for future extensions of VSEI (version 2) [M. M. Hannuksela, J. Chen, S. Deshpande, Hendry, S. McCarthy] [WG 5 N 242)] (2023-12-01)

The subsequent list was provided after the meeting by Y.-K. Wang, after consultation with relevant contributors which had been agreed for inclusion.

Changes in JVET-AF0061 other than the addition of po_id [Bytedance]

The post-processing filter (PPF) grouping concept based on po_id in the SPO SEI message, taking into account backward compatibitiy with v3 of VVC/VSEI for the a special PPF cascading case

Updates to the filtering process to apply to both NNPFs and non-NN-based PPFs and to enable the use of multiple activated PPFs from one PPF group in the cascading manner

Updates to the NNPFC semantics in VSEI

Updates to the NNPFC/NNPFA interface text in VVC

2 items from JVET-AF0050 [Nokia]

Addition of nnpfga_no_prev_clvs_flag and nnpfga_no_foll_clvs_flag

Removal of the possibility to reference NNPF group identifiers from the NNPFA SEI message

The approach of separate SEI (applicable for any PF) from JVET-AF0051, for signalling of the gain provided by post-processing filter(s). Remove the URI, define PSNR in a bit-depth neutral. [Nokia]

Items 1, 2 and 3 in JVET-AF0091 [LGE]

Adding the definitions of direct and indirect members of an NNPFGC

Clarifying that the value of nnpfgc_purpose in NNPFGC, when present, includes all purposes of NNPFC or NNPFGC included in the NNPFGC

When nnpfgc_grouping_type is equal to 1 (i.e., alternative grouping) or 3 (i.e., parallel grouping), purpose information is signalled for each member of the NNPFGC.

On intermediary output picture(s) from activation of an NNPFGC, from JVET-AF0093 [LGE]

Some text clarification of the intent on activation of an NNPFGC that contains another NNPFGC could be useful, as a result from the discussion of JVET-AF0094. [LGE]

Some of the items on the source picture timing information (SPTI) SEI message, from JVET-AF0055 [Sharp]

Item 1: Add a NOTE.

Item 2: Agreed that conditioning the syntax element “spti_max_sublayer_minus1” on persistence is appropriate.

Item 3: Editorial action to clarify that the interval at highest layer is not necessarily 1.

Item 5: Addition of text for “Use of source picture timing information” for the relevant standards.

Item 6 (Syntax rearrangement is proposed such that the syntax element spti_source_picture_timing_type is signalled only when spti_source_timing_equals_output_timing_flag is equal to 0.): Agreed (in the context of discussing JVET-AF0069) to adopt item 6, however using the spti_source_type_present_flag for possible gating of the syntax element.

Items on the source picture timing information (SPTI) SEI message, from JVET-AF0069 [Dolby]

Item 1: It was agreed that the name of the previous spti_source_timing_equals_output_timing_flag should not be modified into spti_source_timing_info_present_flag. If it is zero, the spti_source_type shall be sent (with its gating flag).

Simplify syntax element name spti_source_picture_timing_type to spti_source_type.

Change the precision of spti_source_type from u(8) to u(16).

Simplify the syntax element name spti_num_units_in_elemental_source_picture_interval to spti_num_units_in_elemental_interval.

Simplify the syntax element name spti_sublayer_source_picture_interval_scale_factor[ i ] to spti_sublayer_interval_scale_factor[ i ].

Change the precision of spti_num_units_in_elemental_interval from u(32) to u(18)

Add text to clarify the descriptions of the various source types, e.g., “slow motion”, “high-speed imaging”, etc.

Add semantic constraints to prevent mutually exclusive timing relationships between source pictures and corresponding decoded output pictures. Specifically, prevent the combination of “high-speed imaging” and “time-lapse imaging”.

Replace syntax element spti_source_timing_equals_output_timing_flag with spti_source_timing_info_present_flag and add corresponding semantics.

Move specification of the variable temporalReversalFlag to the semantics following spti_sublayer_interval_scale_factor[ i ].

Integrate the variable temporalReversalFlag in the equation for SourcePictureInterval[ i ] and remove the equation for SourcePictureTime[ i ] (i.e., the absolute source picture time).

Add the following text, resulted from JVET-AF0097 item 1 [LGE]:

The information provided by the SPTI SEI message pertains only for picture(s) starting from the picture in the current layer in the access unit that contains the SPTI SEI message and all subsequent pictures of the current layer in output order based on its persistence.

Some of the items on the encoder optimization information (EOI) SEI message, from JVET-AF0052 [Nokia]

Item 1: Editorial updates to the phrasing of the cancellation and persistence.

Item 2: A syntax element eoi_object_based_idc to indicate the type of object-based optimization, including blurring, quantization adjustment, and overwriting sample values of areas outside the detected objects.

Item 4: Replacement of the optimization_ prefix in the syntax element names with eoi_ to obtain shorter syntax element names.

Item 5: Sensibility constraints that when the persistence is for the current picture only, the temporal optimizations (eoi_temporal_subsampling_flag and eoi_temporal_quality_flag) are required to be off.

Item 6: For temporal subsampling, addition of eoi_num_int_pics, which is indicative of the count of pictures that the encoding systems excluded between each pair of coded pictures in output order.

Item 7: For temporal quality optimization, a clarification of the semantics and addition of a related NOTE.

Some of the items on the EOI SEI message, from JVET-AF0107 [LGE]

On re-formulation spatial/temporal resampling from subsampling/downsampling (which would also apply to upsampling).

Simplification of optimization_type table (editorial) in v2 of JVET-0107 in its section 2.1.

It is further reported that in the combination with the adoption from JVET-AF0052 and JVET-AF0107, a discrepancy was found that is resolved in the green highlighted parts of v2 of JVET-0107. It was also agreed to include these changes in the updates of the EOI SEI in the TuC.

Items on the object mask information (OMI) SEI message, from JVET-AF0088 [Alibaba]

Aspect 1: The text related to bounding box parameters was fixed and refined.

Aspect 2: The binarization of bounding box parameters was changed from ue(v) to u(16)

Aspect 3: A gating flag for bounding box parameters was added to give signalling flexibility to the encoder

Aspect 4: The parsing dependency among different OMI SEI messages was removed by always signalling omi_mask_cancel[ i ][ j ][ k ]

SEI messages for image metadata formats EXIF, JFIF, and XMP, from JVET-AF0141, with an appropriate editor’s note on how these formats could be referenced in an ITU-T/ISO/IEC standard. [Tencent]

Signal picture modality type in VUI parameters, from JVET-AF0147. [Panasonic]

The lsei_message( ) syntax structure for carriage of information about an SEI payload and the SEI payload itself, from JVET-AF0148 [Tencent]

The multiplane image information (MPII) SEI message from JVET-AF0167 [Dolby]