JVET-S0147 AHG8: A summary of proposals on scalability and RPR [Y.-K. Wang (Bytedance)]

This contribution was first discussed in an AHG pre-meeting and then further discussed in the JVET meeting at 2030 on 22 June (chaired by GJS).

This contribution provides a summary of the 12 proposals on scalability and RPR (the agenda item 3.3 in JVET-S0137 and JVET-S0237).

It is suggested that this summary, in terms of a list of proposed items, is used for the reviewing of these proposals, such that the discussions can be in a more structured and efficient manner.

The following changes were proposed:

On scalability

1. When vps_max_layers_minus1 is equal to 0, vps_num_ptls_minus1 is skipped and inferred to be equal to 0. (JVET-S0063)

It was commented that this may not be strictly necessary but seems like a very small nice little cleanup and that since we are skipping nearby syntax elements under this condition.

AHG Recommendation (cleanup/consistency): Adopt.

1. Fix an asserted bug, that the specification allows to extract sublayers with TemporalId higher than the vps_ptl_max_tid[ 0 ] + 1 for an OLS, by setting NumSubLayersInLayerInOLS[ i ][ k ] to be equal to vps_ptl_max_tid[ i ] + 1, instead of vps_max_sublayers_minus1 + 1, for output layers in Equation 40. (JVET-S0100 aspect 1)

AHG Recommendation (bug fix/expression of existing intent): Adopt.

1. Add the following constraint to disallow certain indirect reference layers when the value of vps_max_tid_il_ref_pics_plus1[ i ][ j ] is 0 for all the direct reference layers: (JVET-S0100 aspect 2)

For any two layers k and j within an OLS i, with k > j, when both NumSubLayersInLayerInOLS[ i ][ j ] and NumSubLayersInLayerInOLS[ i ][ k ] are equal to 0, j is a (direct or indirect) reference layer of k and there is a picture in layer k that is an IRAP picture or a GDR picture with ph_recovery_poc_cnt equal to 0 in an AU, the picture in layer j shall also be an IRAP picture or a GDR picture with ph_recovery_poc_cnt equal to 0, respectively.

It was commented that a similar constraint is already enforced by the requirement that the extracted bitstream shall be a conforming bitstream. The proposed constraint is a bit tighter than the indirect existing constraint.

Editor action item: The editor is requested to consider adding some NOTE/explanation for this aspect.

1. Do one of the following, with option b being preferred: (JVET-S0100 aspect 3)
   1. The values of vps_ptl_max_tid[ i ], vps_dpb_max_tid[ i ] and vps_hrd_max_tid[ i ] shall be the same.
   2. (later modified) The values of vps_dpb_max_tid[ vps_ols_dpb_params_idx[ j ] ] and vps_hrd_max_tid[ vps_ols_hrd_idx[ j ] ] shall be greater than or equal to vps_ptl_max_tid[ vps_ols_ptl_idx[ i ] ] for the j-th multi-layer OLS with j in the range from 0 to NumMultiLayerOlss − 1 and i as the index of the j-th multi-layer OLS among all OLSs.
   3. Infer the DPB and HRD parameters for sublayers from vps_ols_dpb_max_tid[ dpbIdx ] and vps_hrd_max_tid[ hrdIdx ] up to vps_ptl_max_tid[ ptlIdx ].

The idea is to have a relationship of these three values for any particular OLS.

It was said that that JVET-S0097 aspect 3 is related (see section 6.1.8).

A participant commented that approach “a” is too restrictive, and indeed that approach “b” is preferred. Another participant considered approach “a” to be better.

Approach “b” would allow to send less unnecessary syntax for HRD and DPB parameters for the various OLSs than approach “a”.

It was asked whether JVET-S0097 aspect 3 would take care of the issue by itself.

The AHG did not reach a conclusion on this and JVET-S0097 aspect 3. After offline study, this was further discussed in the JVET meeting at 2030 on 22 June (chaired by GJS), and approach b above, as modified, was suggested.

Decision (bug fix / cleanup to avoid a need for inference): Adopt option b as expressed above.

1. When not present, the value of vps_max_tid_ref_present_flag[ i ] is inferred to be equal to 0. (JVET-S0129 aspect 1)

The contributor said this is not needed after review of the text and withdrew this aspect.

1. When vps_all_independent_layers_flag is equal to 1 and vps_each_layer_is_an_ols_flag is equal to 1, the value of vps_ols_mode_idc is inferred to be equal to 0. (also in JVET-S0129 aspect 1)

It was noted that this is purely editorial, as the value that is inferred has no effect; this is just a matter of editorial completeness – wanting to always have some value if something is checked in a condition.

It was commented that inferring the value 0 might be confusing or incorrect, since 0 ordinarily has a semantic interpretation that is not intended here. Inferring the value 3 was suggested. Alternatively, adding an extra “if” with a comment seemed better.

Editor action item: The editor is asked to rephrase the text to avoid the problem. The suggested method is to add an extra “if” condition with a comment to explain why it is there.

On RPR

1. Change the value ranges of the scaling window offsets such that the scaling window width and height can be up to 16 times the picture width and height, respectively. (JVET-S0048 aspect 1)

The purpose of the change is to allow extraction of a small subpicture from a large picture while keeping the scaling window unchanged, and in the multilayer case to avoid incorrect decoding results.

It was asked whether this might have a memory bandwidth impact and concluded that it does not.

However, it was commented that there could be a bit width impact on some variables.

This is closely related to item 8 below.

The HLS AHG did not reach a conclusion on this topic. It was further discussed in the JVET meeting at 2045 on 22 June (chaired by GJS).

A revision resulting from further study had been uploaded to further constrain the range of values of the four scaling window offsets.

Decision (bug fix): Adopt as modified.

1. Change the conformance specification for subpicture sequences for the case where higher layers of an OLS contain multiple subpictures while lower layers of OLS do not use subpicture partitioning, as follows: (JVET-S0098)
   1. Support extraction of subpictures that have reference layers with only one subpicture.

This is closely related to item 7 above.

The HLS AHG did not reach a conclusion on this topic. After further study, the issue was resolved as noted above for item 7.

• 1. Change the derivation of RprConstraintsActive[ i ][ j ] to incorporate the number of subpictures in reference picture and current picture, and set RprConstraintsActive[ i ][ j ] to 1 if the two values differ.

This is to fix a clear problem in the text.

AHG Recommendation (bug fix/expression of existing intent): Adopt this aspect.

• 1. Signal a new syntax element non_subpic_layers_fraction[ i ] that specifies the fraction of the bitstream level limits associated with layers in the bitstream that have sps_num_subpics_minus1 equal to 0, for each i.

The current derivation is asserted to not be sensible for cases where different layers have different subpicture layouts.

AHG Recommendation (bug fix): Adopt this aspect.

• 1. Handle accumulated levels for subpicture sets by using one of the following options:
     1. Option 1: Remove the accumulation and adjust equation D.11 to use a single subpicture.
     2. Option 2: Modify equation D.10 (and D.11).

AHG Recommendation (bug fix): Adopt option 1.

1. Change the semantics of the sps_ref_pic_resampling_enabled_flag
2. to be aligned with the derivation of the RprConstraintsActive variable, which would be set equal to 1 when one or more of the following 6 parameters differ between the current picture and the reference picture: 1) picture width, 2) picture height, 3) scaling window left offset, 4) scaling window right offset, 5) scaling window top offset, and 6) scaling window bottom offset. (JVET-S0048 aspect 2)
3. such that when sps_ref_pic_resampling_enabled_flag equal to 1, a slice may refer to a reference picture with a different spatial resolution or a different scaling window. (JVET-S0057 aspect 1)

This is editorial – a matter of accurately describing the semantics.

Approach “a” refers to PPS content in SPS semantics, which was suggested to not be appropriate. However, it is more explicit regarding what is involved, and this was suggested to be helpful.

The phrase “spatial resolution” in approach “b” was questioned.

Editor action item: The editor is requested to clarify the semantics as appropriate.

1. Add pps_res_change_in_clvs_allowed_flag, and when pps_res_change_in_clvs_allowed_flag is equal to 0, skip pps_pic_width_in_luma_samples and pps_pic_height_in_luma_samples and infer their values to be equal to sps_pic_width_max_in_luma_samples and sps_pic_height_max_in_luma_samples, respectively. (JVET-S0057 aspect 2)

It was commented that this could introduce a parsing dependency problem for the tile configuration in the PPS. The proposal had not been tested with an implementation. Thus, no action was recommended for this.

1. Add either of the following constraints (JVET-S0126):
2. When sps_res_change_in_clvs_allowed_flag is equal to 0, all pictures in the CLVS shall have the same values of pps_scaling_win_left_offset, pps_scaling_win_right_offset, pps_scaling_win_top_offset, and pps_scaling_win_bottom_offset, respectively.
3. When sps_res_change_in_clvs_allowed_flag is equal to 0, a current picture in the CLVS and a reference picture having the same nuh_layer_id shall have the same values of pps_scaling_win_left_offset, pps_scaling_win_right_offset, pps_scaling_win_top_offset, and pps_scaling_win_bottom_offset, respectively.

It was commented that this constraint would limit some use cases and that a similar constraint had previously been considered and agreed to be undesirable (see JVET-R0058 versions 1-3 aspect 2.c).

No clear problem was agreed to exist, so no action was recommended on this.

New submissions (for the June 10th submission deadline)

Discussion continued here in the JVET meeting at 2055 on 22 June (chaired by GJS).

1. Change Equation 37 as follows, considering that a reference layer can only be a lower layer: (JVET-S0169 aspect 2) (additions in bold and bigger fonts, removals in italic and smaller fonts)

for( i = 0; i <= vps_max_layers_minus1; i++ ) {
for( j = 0; j < i <= vps_max_layers_minus1; j++ ) {
dependencyFlag[ i ][ j ] = vps_direct_ref_layer_flag[ i ][ j ]
for( k = j+1 0 ; k < i; k++ )
if( vps_direct_ref_layer_flag[ i ][ k ] && dependencyFlag[ k ][ j ] )
dependencyFlag[ i ][ j ] = 1
}
LayerUsedAsRefLayerFlag[ i ] = 0
}
for( i = 0; i <= vps_max_layers_minus1; i++ ) {
for( j = 0, d = 0, r = 0; j < i <= vps_max_layers_minus1; j++ ) { (37)
if( vps_direct_ref_layer_flag[ i ][ j ] ) {
DirectRefLayerIdx[ i ][ d++ ] = j
LayerUsedAsRefLayerFlag[ j ] = 1
}
if( dependencyFlag[ i ][ j ] )
RefLayerIdx[ i ][ r++ ] = j
}
NumDirectRefLayers[ i ] = d
NumRefLayers[ i ] = r
}

This is intended to be purely editorial.

Some fear was expressed in whether this modification might have some bug in it unless it is well tested; it had not been tested.

The current expression is from HEVC; we’re pretty confident that it works.

No action was taken on this.

1. Do either of the following, considering that assertedly vps_num_output_layer_sets_minus1 is signalled when vps_max_layers_minus1 is greater than 0 (and when vps_ols_mode_idc is equal to 2) and in this case there are at least two OLSs: (JVET-S0183 aspect 1)
   1. Change vps_num_output_layer_sets_minus1 to vps_num_output_layer_sets_minus2.
   2. Add a constraint: The value of vps_num_output_layer_sets_minus1 shall be greater than 0.

Decision (cleanup to specify a coding that is sensible for the value range): Adopt approach “a”.

1. Add a flag for each multi-layer OLS (except the first one) to indicate whether the OLS DPB picture width, height, chroma format, and bit depth values are the same as the values for the previous multi-layer OLS, and if the flag is equal to 1, these fields are skipped and inferred. (JVET-S0183 aspect 2)

This is a syntax shortcut for the VPS. Multiview is an example case. Item 16 is an alternative approach.

This is not a bug fix, just a proposed avoidance of unnecessary repetition.

No action seemed necessary on this, so no action was taken.

1. Change to code the vps_ols_dpb_pic_width[ i ] and vps_ols_dpb_pic_height[ i ] in units of 8 luma samples, considering that assertedly the decoded picture width and height in a CLVS are constrained to be an integer multiple of Max( 8, MinCbSizeY ). (JVET-S0212 aspect 1)

It was commented that we should also do this for SPS and PPS picture width and height, as the way it is currently done is simply wasteful.

It was asked whether we might be able to (and interested in) create some future profile in which the multiple-of-8 constraint is relaxed.

It was commented that we should at least limit the scope of the discussion to just the VPS topic, as we have used the existing coding in HEVC. There was some reluctance to apply the idea to one but not the other.

In AVC it was in units of macroblocks (16×16).

No action seemed necessary on this, so no action was taken.

1. Instead of signalling a set of width, height, chroma format, and bit depth of a DPB picture storage buffer for each multi-layer OLS in the VPS, signal these DPB parameters in the dpb_parameters( ) syntax strucures, each of which (per the existing VVC text) can be shared by two or more OLSs. (JVET-S0160 aspect 10 late)

Item 14 above is similar in spirit. This moves syntax into an existing mechanism rather than adding a new mechanism. It was discussed whether the max picture format is more or less naturally shared as max reordering, max latency and max DPB size.

No action seemed necessary on this, so no action was taken.

1. Specify that the ue(v)-coded vps_ols_dpb_bitdepth_minus8[ i ] shall be in the range of 0 to 8, inclusive, considering that the value range is currently missing. (JVET-S0212 aspect 2)

This is purely editorial, just to try to have a specified value range for every syntax element (except picture width and height, apparently).

Editor action item: It is suggested to specify a value range of 0 to 2 for bitdepth_minus8.

1. In Equation 41, replace vps_ptl_max_tid[ 0 ] with vps_ptl_max_tid[ vps_ols_ptl_idx[ 0 ] ] and vps_ptl_max_tid[ i ] with vps_ptl_max_tid[ vps_ols_ptl_idx[ i ] ] (for all the instances). (JVET-S0248 aspect 1 late)

Decision (editorial bug fix/expression of existing intent): Adopt.

Scalability cleanups (8)