JVET-S0142 AHG9: A summary of proposals on SPS, PPS, and APS cleanups [Hendry (LGE)]

This contribution was discussed in the HLS AHG meeting at 2330 UTC on Wednesday 27 May and in the HLS AHG pre-meeting at 2225 UTC on 19 June.

This contribution provides a summary of the 16 proposals on SPS, PPS, and APS cleanups (the agenda item 3.1.4 in JVET-S0137 and JVET-S0237).

It is suggested that this summary, in terms of a list of design questions, is used for the reviewing of these proposals, such that the discussions can be in a more structured and efficient manner. The following changes are proposed:

1. It is asserted that when extension_flag in DCI, VPS, SPS, PPS, or APS is equal to 1, it is allowed that no extension_data_flag syntax elements be present. The following options for change are proposed (JVET-S0042)
   1. Option 1: When an extension_flag in DCI, VPS, SPS, PPS, or APS is equal to 1, enforce the presence of at least one extension_data_flag syntax element in the syntax.

From the following design in the current spec:

To the following:

• 1. Option 2: Relax the semantics so that an extension_flag in DCI, VPS, SPS, PPS, or APS equal to 1 specifies that specifies extension_data_flag syntax elements may be present. (Currently, it says these flags are present.)
  2. Option 3: Do nothing

It was commented that option 2 seems like just a better editorial expression of the what the current syntax already specifies.

AHG Recommendation (editorial clarification): Option 2.

Note: This should be also considered as an errata report for HEVC semantics clarification.

1. Rearrange some syntax elements in SPS so that the syntax elements of similar coding tools are grouped together.
   1. All of the transform related SPS syntax are placed together (JVET-S0074 aspect 1)
   2. sps_lmcs_enabled_flag is signalled right after signalling of sps_ccalf_enabled_flag (JVET-S0074 aspect 1)
   3. Group the dual tree related syntax elements together in the SPS (JVET-S0132 aspect 1)
   4. Rearrange the partition constraint related syntax elements in the SPS to be consistent with the ordering of the corresponding syntax elements in the picture header (PH). More specifically, group such syntax in the SPS based on whether they are intra or inter related (JVET-S0132 aspect 2)

AHG Recommendation (minor cleanup of logical syntax order): Adopt. It is also desirable to have the constraint flag order match SPS order (to be considered by J. Boyce in further study).

1. Not signalling sps_max_sublayers_minus1 when sps_ptl_dpb_hrd_params_present_flag is equal to 0 (JVET-S0079)
   1. Move the signalling of sps_max_sublayers_minus1 and sps_reserved_zero_4bits immediately before the signalling of profile_tier_level( 1, sps_max_sublayers_minus1 )
   2. Condition the presence of the two syntax elements such that they are present only when sps_ptl_dpb_hrd_params_present_flag is equal to 1.

The proponent reported that the sps_max_sublayers_minus1 value is not actually used when sps_ptl_dpb_hrd_params_present_flag is equal to 0.

It was commented that there may be some interaction with other proposals or the extraction process.

It was commented that sps_max_sublayers_minus1 could be useful to know even if not necessary for parsing and the decoding process.

No clear need for action was evident, so no action was recommended.

1. Add a constraint such that when sps_video_parameter_set_id is greater than 0 and vps_all_layers_same_num_sublayers_flag is equal to 1, the value of sps_max_sublayers_minus1 shall be equal to the value of vps_max_sublayers_minus1 (JVET-S0115 aspect 2).

It was commented that this had been proposed in JVET-R0125 item 2. At that time a concern had been expressed about what would happen if some sublayers are removed and the SPS is rewritten.

It has been intended that it should be allowed to removed sublayers without rewriting the VPS.

AHG Recommendation (bug fix/expression of existing intent): Change the name and semantics of vps_all_layers_same_num_sublayers_flag to only control VPS syntax without requiring the number of sublayers to be the same for all layers.

No action on the proposed change, due to the agreement recorded above.

1. Add new flag sps_one_picture_only_constraint_flag and use the flag to skip signalling of some syntax elements in SPS. The skipped syntax elements include inter prediction, RPL, and POC related syntax elements in both SPS and PH (JVET-S0129 aspect 2).

It was commented that this would require PH rewriting when extracting a single picture from the bitstream (e.g., to remove RPL and POC).

Concern was expressed about POC, as we allow sending POC for IDR pictures.

A similar proposal was considered at the previous meeting, for intra-only rather than for single-picture-only. The basic issues seemed similar.

It was commented that it is an encoder choice how to set the values of non-relevant syntax elements if the bitstream will contain only intra pictures or only one picture.

No clear need for action was identified, so no action was recommended.

1. Constrain the value of pps_alf_info_in_ph_flag to be equal to 0 when the PH is in the SH (JVET-S0049 aspect 4)

It was asked why we hadn’t already done something like this, since we did something like this for five other similar syntax elements (see the next item for a list). There had been some argument of wanting the ALF info to be early in the header. However, it was commented that the ALF info can be in the SH already.

It was noted that another proposal has this as one of two options for action (see item 9a in the JVET-S0143 summary for JVET-S0120).

AHG Recommendation (consistency/cleanup): Adopt.

1. When pps_rect_slice_flag is equal to 1 and pps_num_slices_in_pic_minus1 is equal to 0, constrain the values of the 6 PPS flags (i.e., pps_alf_info_in_ph_flag, pps_rpl_info_in_ph_flag, pps_dbf_info_in_ph_flag, pps_sao_info_in_ph_flag, pps_wp_info_in_ph_flag, and pps_qp_delta_info_in_ph_flag) to be all equal to 0 (JVET-S0049 aspect 5).

The motivation / justification of the proposed changes is as follows:

• • To be better aligned with the inference of the value to 0 when pps_no_pic_partition_flag is equal to 1.
  • For more optimal syntax in the case of pps_rpl_info_in_ph_flag or pps_wp_info_in_ph_flag.
  • For the case of pps_alf_info_in_ph_flag, to enable the use of ALF for the same use case that applies independent parallel encoding of different subpictures as above.

It was commented that this could require SH rewriting if there is an original bitstream and one subpicture is extracted as one picture with one slice. These flags could be in the PPS and would need to be moved to the SH.

It was commented that there is a similar issue for when pps_no_pic_partition_flag is equal to 1. However, it was commented that this flag could be set to 0.

After offline further study, this proposal was withdrawn in the HLS AHG pre-meeting at 2225 UTC on 19 June.

1. Rearrange the syntax elements related to inter slices in the picture parameter set (PPS) in a similar manner as the grouping of the inter-slice related syntax elements in the PH. More specifically, group syntax elements in PPS based on whether they are intra or inter related (JVET-S0132 aspect 3).

It was commented that this change is pretty minor and logical, and this had been tested with software. See also item 2 above.

AHG Recommendation (consistency/cleanup): Adopt.

1. Currently due to existing constraints, only 3 bits out of 5 bits of APS Id are needed in current profiles. The following is proposed: split the u(5)-coded aps_adaptation_parameter_set_id into u(2)-coded aps_reserved_zero_2bits followed by u(3)-coded aps_adaptation_parameter_set_id (JVET-S0049 aspect 6).

Currently we signal the APS ID first, followed by the APS type, but the value spaces for different APS types are distinct.

It was commented that in some future profile we might want a larger range of values.

It was concluded that, as proposed, this would really be only an editorial change.

No clear need for action was evident, so no action was recommended.

1. If aps_chroma_present_flag is equal to 0, due to an existing constraint, alf_luma_filter_signal_flag must be equal to 1. Based on that, it is proposed that when aps_chroma_present_flag is equal to 0, alf_luma_filter_signal_flag is skipped and inferred to be equal to 1. Otherwise, when alf_chroma_filter_signal_flag and alf_cc_cb_filter_signal_flag are both equal to 0, alf_cc_cr_filter_signal_flag is skipped and inferred to be equal to 1 (JVET-S0049 aspect 7).

No clear need for action was evident, so no action was recommended.

1. It is asserted that the memory to store APSs may exceed the max memory used to store the max number of APSs in a PU due to the fact that in a PU, prefix and suffix APS NAL units with particular APS identifier and type can have different contents. The following two constraints are proposed (JVET-S0122)
   1. In a PU, for an APS type, a signalled suffix APS shall not have the same identifier as a referenced APS.

It was commented that this topic had been raised the previous meeting.

It was commented that this depends on whether the decoding process is applied after receiving the VCL NAL units or after also receiving the suffix APS NAL units. If the picture is decoded before parsing the suffix APS NAL units, the memory of the prior APS content can be re-used in the on-chip memory.

No clear need for action was evident, so no action was recommended.

• 1. Any two suffix (prefix resp.) APS NAL units with particular APS identifier and type signalled in a PU cannot have the same APS identifier

This would disallow repetition. It was commented that such repetition is not really a problem.

No clear need for action was evident, so no action was recommended.

Review of the above items in this category completed ~0120 UTC 28 May.

Discussion resumed here in the HLS AHG pre-meeting at 2225 UTC on 19 June.

1. Allow inferring some SPS syntax elements from SPS in the reference layer. It is asserted that in HEVC such mechanism is allowed (i.e., for scaling list data structure). If we allow this mechanism, the following syntax elements are proposed to be inferred from SPS in direct reference layer (JVET-S0166):
   1. RPLs of the current SPS from the SPS referred to by the direct reference layer.
   2. Subpicture information of the current SPS from the SPS referred to by the direct reference layer

Mechanism detail: for each category of syntax elements that can be inferred, add a new flag to specify whether the syntax elements are inferred or not. If inferred, signal additional syntax element to specify the index of the direct reference layer.

Concern was expressed about introducing such a prediction across layers at this late stage.

A benefit of coding efficiency could be argued, but did not seem like an important consideration.

The proponent said the proposal might help encoders avoid conformance problems.

It was noted that SPS sharing is allowed. This would be a form of partial SPS sharing.

No action was recommended on this, to be conservative about considering late changes.

1. It is asserted that the current semantics of sps_sublayer_dpb_params_flag and vps_sublayer_dpb_params_present_flag are inaccurate. It currently says that it control the presence of max_dec_pic_buffering_minus1[ i ], max_num_reorder_pics[ i ], and max_latency_increase_plus1[ i ] syntax elements. However, it actually controls the presence of those syntax elements for temporal sub-layers that are not the highest. It is proposed to change the semantics to be more accurate (JVET-S0169 aspect 1). (added or changed in bold)

sps_sublayer_dpb_params_flag is used to control the presence of max_dec_pic_buffering_minus1[ i ], max_num_reorder_pics[ i ], and max_latency_increase_plus1[ i ] syntax elements in the dpb_parameters( ) syntax strucure in the SPS for i in range from 0 to sps_max_sublayers_minus1 − 1, inclusive, when sps_max_sublayers_minus1 is larger than 0. When not present, the value of sps_sublayer_dpb_params_flag is inferred to be equal to 0.

vps_sublayer_dpb_params_present_flag is used to control the presence of max_dec_pic_buffering_minus1[ j ], max_num_reorder_pics[ j ], and max_latency_increase_plus1[ j ] syntax elements in the dpb_parameters( ) syntax strucure in the VPS for j in range from 0 to vps_dpb_max_tid[ i ] − 1, inclusive, when vps_dpb_max_tid[ i ] in VPS is larger than 0. When not present, vps_sub_dpb_params_info_present_flag is inferred to be equal to 0.

It is noted that j instead of i is used for specifying the range of syntax element array max_dec_pic_buffering_minus1[ ], max_num_reorder_pics[ ], and max_latency_increase_plus1[ ] in order to avoid confusion of the i in vps_dpb_max_tid[ i ].

It was commented that this is just editorial and should be considered by the editor.

Editor action item: The editor is asked to consider this potential clarification.

1. Move the two syntax elements (sps_chroma_format_idc and sps_log2_ctu_size_minus5) to take the place of sps_reserved_zero_4bits. (JVET-S0186 aspect 1)

The reserved bits were said to have been there only for byte alignment.

AHG Recommendation (cleanup): Adopt.

1. Reserve sps_seq_parameter_set_id value equal to 15 for future extensibility. Additionally, do the same for value 63 of pps_pic_parameter_set_id. (JVET-S0186 aspect 2)

No clear need for action on this seemed evident.

1. Move the location of decoded picture buffer (DPB) parameters syntax structure and hypothetical reference decoder (HRD) parameters syntax structure to directly follow sps_ptl_dpb_hrd_params_present_flag. (JVET-S0199).

Currently only the signalling of PTL syntax structure directly follows sps_ptl_dpb_hrd_params_present_flag (i.e., early in SPS), the signalling of DPB syntax structure is in the middle of SPS, and HRD syntax structure is near the end of SPS. The proposal would move DPB and HRD syntax structures to early position in the SPS.

It was commented that moving DBP and HRD syntax earlier could put it ahead of other syntax that may be more important to access from the system perspective and that saving some condition checks in the parsing is not so important, so no action was recommended on this.

1. Change the coding of syntax element sps_log2_transform_skip_max_size_minus2 from ue(v) to u(2). (JVET-S0204).

It was asked whether some future profile might want to allow a value greater than 3, and this seemed possible, although it was commented that such a future profile would probably have other, much more significant, changes in it.

In HEVC, RExt added larger TS block sizes without a syntax change for that, and used the same type of coding, although in the PPS rather than SPS.

It was noted that this would increase the number of bits used for coding the value 0.

Since no clear need for action was identified, no action was recommended on this.

1. Add additional conditions for the presence of the six PPS flags pps_alf_info_in_ph_flag, pps_rpl_info_in_ph_flag, pps_dbf_info_in_ph_flag, pps_sao_info_in_ph_flag, pps_wp_info_in_ph_flag, and pps_qp_delta_info_in_ph_flag, as follows (JVET-S0164) (additions in italic):

The contribution was said to be similar in spirit to item 7 above (JVET-S0049).

The desire is that when there is only one slice in the picture, to avoid signalling some information in the PPS.

It was commented that if an original bitstream has multiple slices and has this signalling in the PH, then extraction of a single slice could cause a rewriting requirement. The proponent indicated that there is a provision to avoid the rewriting based on the single slice per subpicture flag equal to 1 (at some expense of bits).

No clear need for action was identified, and thus no action was recommended.

1. Enable removal of parameter sets (i.e., PPS and APS) in non-output layer that is not used as reference for decoding of pictures in output layers during extraction process, as follows: (JVET-S0219 aspect 1)

• Modify the semantics of vps_max_tid_il_ref_pics_plus1[ i ][ j ] such that when it is greater than 0 it means for decoding pictures of the i-th layer, no parameter set and picture from the j-th layer with TemporalId greater than vps_max_tid_il_ref_pics_plus1[ i ][ j ] − 1 is used as reference.
• In extraction process, in addition to removing VCL NAL unit, also remove PPS and APS with TemporalId that is greater than or equal to NumSubLayersInLayerInOLS[ targetOlsIdx ][ GeneralLayerIdx[ nuh_layer_id ] ].

It was commented that this makes more sense for the APS than the PPS and that APSs are a bigger problem. This would treat APS more similar to PH for the extraction.

“Smart” extractors were discussed, and whether the extraction process model is sufficient.

Several participants saw value in being able to identify and remove useless APS from the bitstream based on this proposal. The change is also small.

AHG Recommendation (cleanup): Adopt APS aspect only.

1. Move the signalling of aps_params_type to an earlier position and change the signalling of aps_adaptation_parameter_set_id from u(5) to u(v). The length of the syntax element depends on the type of the APS. (JVET-S0219 aspect 2)

A participant said this (or swapping the order of the ID and type) may be useful for extensibility, as some future APS type might need more ID values. The fact that the type is before the ID seemed to just be for historical reasons – it was originally proposed that way and never modified.

It was commented that we took no action on something similar previously.

AHG Recommendation (cleanup): Just change the order to put the type before the ID.

1. Move the flag indicating the presence of sh_cabac_init_flag from PPS to SPS. (JVET-S0235)

The location is a carry-over from HEVC.

No action seemed clearly necessary, so no action was recommended on this.