JVET-R0344 AHG9: A Summary of Proposals Related to Scalability Information Signalling [S. Deshpande (Sharp)]

Discussed in AHG Session 1.3 Monday 7 April at 2100 (chaired by GJS, YKW & JRO).

This contribution intends to provide a summary of proposals on scalability information signalling.

Seventeen proposals from the category “4.3.1.2 Scalability information signalling and related” listed in a revision of JVET-R0339-v4 are included in this summary. Thus in v3, summary is added for JVET-R0158, JVET-R0199, and JVET-R0222 aspect 1.

It is suggested that this summary be used for the reviewing of these proposals, such that the discussions may be done in a more structured and efficient manner.

List of design questions

Related to PTL information signalling

1. Omit signalling of index to the list of PTL structures for output layer sets when number of signalled PTL structures is equal to total number of output layer sets and instead infer its value? (JVET-R0161 PTL part of proposal 2, JVET-R0185 proposal 4, JVET-R0204, JVET-R0275 aspect 4)

Note: There may not be an OLS that contains all layers (regardless).

Something similar is in the draft for HRD.

Is the number of OLSs always less than or equal to the number of PTL structures? There is a constraint like that already (regardless of this proposal).

It was commented that this usage seems like it would be common. This is a syntax shortcut for that case.

AHG Recommendation (cleanup): Adopt. Text and software were provided by JVET-R0161 proponent (J. Chen).

Related to DPB information signalling

1. Modify the upper range of vps_num_dpb_params to allow signalling of DPB parameters for all OLSs from current fixed upper value of 16:
   1. upper limit is equal to total number of OLSs minus the number of single-layer OLSs (JVET-R0099 Proposal 1, JVET-R0191 aspect 3). AHG Recommendation (expression of existing intent): Adopt. Text was provided by Hendry and he is also to supply software.
   2. upper limit is equal to total number of OLSs − 1 (JVET-R0196)
2. Update the range value for num_ols_hrd_params_minus1
   1. similarly as in (a) in previous item? (JVET-R0191 aspect 3) AHG Recommendation (expression of existing intent): Adopt. Text was provided by Hendry and he is also to supply software.
   2. to total number of OLSs − 2 (JVET-R0204)
3. Don't signal and instead infer the index of the dpb_parameters( ) syntax structure that applies to the i-th OLS when a condition is met (JVET-R0099 Proposal 2, JVET-R0204, JVET-R0275 aspect 4)
   1. The condition is total number of output layer sets minus number of single layer output layer sets is equal to number of signalled dpb parameters (JVET-R0099 proposal 2). AHG Recommendation (expression of existing intent): Adopt, and also apply to HRD parameters. Text is provided in JVET-R0099-v2 by S. Deshpande and he is also to supply software.
   2. The condition is total number of output layer sets is equal to number of signalled dpb parameters (JVET-R0161 proposal 2, JVET-R0275 aspect4)
   3. The condition is total number of output layer sets is equal to number of signalled dpb parameters + 1 (JVET-R0204)
4. Start the for loop which signals ols_dpb_pic_width[ i ], ols_dpb_pic_height[ i ], and ols_dpb_params_idx[ i ] to start at 1 instead of at 0, since 0-th OLS is single layer? (JVET-R0099 Proposal 3, JVET-R0196). AHG Recommendation (expression of existing intent): Adopt (unless affected by proposals to redefine the 0-th OLS).
5. Replace if( !vps_all_independent_layers_flag ) condition on vps_num_dpb_params syntax element with if(!each_layer_is_an_ols_flag) (JVET-R0185 proposal 1, JVET-R0196, JVET-R0275 aspect 3). AHG Recommendation (bug fix): Adopt.
   1. If above main item is agreed, additionally change vps_num_dpb_params to vps_num_dpb_params_minus1? (JVET-R0185 proposal 2, JVET-R0196, JVET-R0275 aspect 3). It was commented the semantics of each_layer_is_an_ols_flag is a “one-way” constraint. It was asked why we would want to allow the flag to be 0 and still have each layer be an OLS – all of this is in the VPS, so the encoder should know what it is doing when it writes the VPS. AHG Recommendation (bug fix): Adopt and change the semantics to a “two-way” constraint (so if the flag is zero, there must be at least one multilayer OLS specified by the VPS).
   2. Additionally signal DPB parameters for OLS in this case only if(!each_layer_is_an_ols_flag) (JVET-R0185 proposal 3). AHG Recommendation (cleanup): Adopt.

Text was provided by Hendry (to be modified for the “two-way” constraint) and he is also to supply software.

Related to HRD information signalling

1. Allow control separately if HRD parameters are signalled for an OLSs or not on individual basis? (JVET-R0195).

The proponent indicated that this was motivated by the syntax allowing some HRD parameters to be present and some not for single-layer OLSs. This flexibility is not provided for other cases.

It was commented that the situation for single-layer OLSs was just a consequence of where the data is sent, and noted that each single-layer OLS could be extracted and become a single-layer stand-alone bitstream. So no action was recommended by the AHG on this.

• 1. If want separate control then signal a separate new flag for each OLS to specify if index to HRD parameters structure is signalled or not? OR
  2. Designate 0-th index to mean HRD parameters are not specified for an OLS?

1. Change condition for omitting signalling of ols_hrd_idx[ i ] from “num_ols_hrd_params_minus1 + 1 != TotalNumOlss” to “num_ols_hrd_params_minus1 + 2 != TotalNumOlss”? (JVET-R0204). This item no longer needed to be considered due to the action taken on item 4.a.
2. Add a constraint that vps_general_hrd_params_present_flag shall be equal to 1 when more than one layer is included into any OLS? (JVET-R0275). It was commented that HRD parameters presence is optional in the single-layer case and has been optional in AVC and HEVC and their extensions and should be optional, so no action was taken on this.

Common or Combination aspects of PTL, DPB, HRD signalling:

1. Constrain that each DPB, HRD, parameter structure signalled in VPS shall be associated with at least one OLS (in the VPS) that contains more than one layer and each PTL structure that is signalled is associated with at least one OLS? (JVET-R0191 Aspect 3). These are just “sensibility” constraints. AHG Recommendation (expression of existing intent): Adopt. Text was provided by Hendry and he is also to supply software.
2. Define and use a common gating flag vps_dpb_hrd_params_present_flag and use this to condition presence of dpb_parameters() and ols_hrd_parameters()? (JVET-R0275 aspect 2). The proposal is motivated by a desire for a consistent approach in the VPS and SPS. However, it was commented that the circumstances in the VPS and SPS are different. This item no longer needed to be considered due to the action taken on item 6.a.
3. Include PTL signalling in VPS under a common gating flag along with DPB and HRD signalling in VPS? (JVET-R0275 aspect 3). This is similar in spirit to item 11, so no action was taken on this.

Related to max_tid and number of sublayers:

1. Signal the syntax elements max_tid_ref_present_flag[ i ], max_tid_il_ref_pics_plus1[ i ] only when ols_mode_idc is not equal to 1 and each_layer_is_an_ols_flag is not equal to 1? (JVET-R0107 Proposal 1). This is a “sensibility” issue – avoiding sending information that is not used. A participant questioned the aspect about ols_mode_idc, and it was discussed whether this information is intended to be metadata or only for sub-bitstream extraction. The proponent said that the syntax in the VPS is intended only to be non-metadata syntax. It was commented that the two syntax elements are already gated by a !vps_independent_layer_flag[ i ] condition, and each_layer_is_an_ols_flag can only be true if all layers are independent. No action seemed needed unless offline study indicates otherwise.

Discussion stopped here for AHG Session 1.3 on Monday 6 April at 2300 UTC, and resumed here in AHG Session 1.7 on Tuesday 7 April at 2100 UTC.]

1. Assertedly simplify the condition checking for signalling ptl_max_temporal_id[ i ], dpb_max_temporal_id[ i ], and hrd_max_tid[ i ] to only use the flag vps_all_layers_same_num_sublayers_flag instead of using the flag vps_all_layers_same_num_sublayers_flag and vps_max_sublayers_minus1 syntax element. Also assertedly simplify the inference rules for ptl_max_temporal_id[ i ], dpb_max_temporal_id[ i ], and hrd_max_tid[ i ], when not present? (JVET-R0107 Proposal 2)

It was commented that this appears purely editorial – it is just removing checks that are unnecessary.

AHG Recommendation (editorial simplification): The editor is asked to confirm this and remove checks that are unnecessary.

1. Change the inferred value of max_tid_il_ref_pics_plus1[] when not present from 7 to vps_max_sublayers_minus1 + 1, to avoid an asserted wrong derivation case for the value of the variable NumSubLayersInLayerInOLS? (JVET-R0119 item 1)

AHG Recommendation (cleanup): Adopt this item. Text was provided by B. Choi, and he is also to supply the software.

1. Don't derive the NumSubLayersInLayerInOLS[] and layerIncludedInOlsFlag[][] values, when vps_all_independent_layers_flag is equal to 1? (JVET-R0119 item 2).

It was commented that this appears purely editorial – it is just removing an unnecessary derivation.

AHG Recommendation (editorial simplification): The editor is asked to confirm this and remove the derivation if confirmed editorially undesirable.

1. Fix an asserted bug in the iteration loop in eq. (40)? (JVET-R0119 item 3)

This was further discussed in the closing plenary of Friday 24 April (chaired by GJS & JRO). It was initially agreed to make this change to correct the error. However, this item is no longer needed after the action taken on JVET-R0193.

1. Signal max_tid_il_ref_pics_plus1 value separately for each direct reference layer of a layer, i.e. max_tid_il_ref_pics_plus1[ i ][ j ] for each direct reference layer j less than i, instead of single max_tid_il_ref_pics_plus1[ i ] as currently? (JVET-R0193)

It was commented that HEVC has a two-dimensional array similar to what is proposed. With the one-dimensional approach, in some cases there may be unnecessary sublayers present after operation of the specified extraction process. The issue is whether the maximum number of sublayers used for interlayer prediction could be different for different layers.

It was commented that if some kind of hypothetical extra metadata is available (e.g. in a system environment or some SEI message), it could provide a more highly optimized extraction capability.

It was commented that the one-dimensional approach was chosen at the previous meeting (see the notes for JVET-Q0398), with an understanding that it involved some loss of generality, although there had not been much careful consideration of the question at the time. The amount of complication needed for supporting the greater generality did not seem substantial.

In HEVC, the generality is present in the syntax, and this functionality is used for reference picture list construction but it is not used in the extraction process.

It was commented that the HRD parameters in the bitstream are for the “thin” bitstream – i.e., the bitstream from which all pictures not needed for an OLS have been removed.

AHG Recommendation (cleanup): Adopt. Text was provided in the contribution, and the authors are to supply the software.

1. Signal a flag in VPS to indicate that all dependent layers share the same value of max_tid_il_ref_pics_plus1. If the flag is set, signal a common vps_max_tid_il_ref_pics_plus1 for all layers. Otherwise conditionally signal separate values for max_tid_il_ref_pics_plus1[ i ]? (JVET-R0261/ aspect 2)

This is a proposed signalling shortcut in the VPS. We have shortcuts for “vps_all_layers_same_num_sublayers_flag” and “vps_all_independent_layers_flag”. This proposes an additional shortcut “vps_all_layers_same_tid_il_flag” to save repetition of values of max_tid_ref_present_flag[ i ] and max_tid_il_ref_pics_plus1[ i ]. With the adoption of JVET-R0193, this would save some more max_tid_il_ref_pics_plus1 values, since that becomes two-dimensional.

It was commented that we should have conformance bitstreams to test the shortcuts.

The proponent said this does address a common case. Others thought this was unnecessary complication, and the most common case would not use this part of the syntax at all. No action was taken on this.

1. Fix an asserted bug for semantics of max_tid_il_ref_pics_plus1[ i ] for special value 0? (JVET-R0107 proposal 3, JVET-R0296 aspect 1)? AHG Recommendation (editorial text bug): Adopt.

Additionally define the semantics for special value 0 to include GDR pictures with recovery_poc_cnt equal to 0 (JVET-R0107 Proposal 3)? It was said that such a GDR picture is functionally equivalent to an IRAP picture. Another participant commented that RPL constraints are different for such a GDR picture, and there was discussion of whether difference is appropriate or not. AHG Recommendation (editorial text bug): Adopt (assuming we don’t disallow GDR pictures with recovery_poc_cnt equal to 0).

1. Modify the sub-bitstream extraction process to account for GDR pictures with recovery_poc_cnt equal to 0? (JVET-R0107 Proposal 3) AHG Recommendation (editorial text bug): Adopt (assuming we don’t disallow GDR pictures with recovery_poc_cnt equal to 0).
2. Fix an asserted bug in the derivation of NumSubLayersInLayerInOLS by separating the cases for each_layer_is_an_ols_flag is equal to 1 and ols_mode_idc is equal to 0? (JVET-R0296 aspect2). AHG Recommendation (bug fix): Adopt.

Related to output layer sets and layer dependency:

1. Re-define 0-th OLS to include all independent layers when present and every included layer is output? (JVET-R0261 aspect 3). Currently, the 0-th OLS is conceptually a base layer, and there did not seem to be a strong need to change that, so no action was recommended on this by the AHG.

Discussion stopped here in AHG Session 1.7 on Tuesday 7 April at 2300 UTC.

Discussion began here for JVET Track A on 19 April at 0715 UTC (chaired by GJS & YKW).

1. Keep the design that the 0-th OLS contains only the lowest layer when each_layer_is_an_ols_flag is equal to 1, the output layer set mode is equal to 0 or the output layer set mode is equal to 1, but relax this when output layer set mode equal to 2 and if so modify the loop and derivation? (JVET-R0306)

The proponent discussed the case with all-independent layers (e.g., two layers). The contributor said that currently, it is required to have two OLSs. This was confirmed, but it was also remarked that the 0-th OLS is implicit in this case.

The proposal is to redefine the 0-th OLS when all layers are independent or the output layer set mode is 2.

V-PCC was mentioned as an example, where 3 independent layers are commonly used together (texture, depth and occupancy).

There is virtually no signalling for the 0-th OLS. The current scheme requires adding signalling for the all-encompassing OLS, whereas the proposal would make that be the 0-th OLS.

It was commented that the all_independent_layers_flag already skips a lot of signalling.

There was discussion of whether the current scheme is really a burden.

It was commented that V-PCC is codec-agnostic, such that the different components can even be encoded with different video encoders.

Another mentioned use case was multi-party videoconferencing, where each independent layer is a separate camera view.

The proponent said that basically it seems strange to require having more than one OLS for such bitstreams that contain only independent layers and that the requirement to have multiple OLSs in the bitstream is not especially easy to notice and seems artificial in the use cases.

The current design was somewhat motivated by traditional scalability or multiview.

Some concern was expressed about whether there could be some bug in redefining the 0-th OLS.

It was commented that in such a traditional scalability case there would become a need for explicit signalling, so this proposal would be burdening that usage in order to facilitate the all-inclusive case.

It was commented that we have an inference of the VPS if it is not present, and nesting requirements and interpretations that might be changed by the proposal.

The contributor discussed the OLS modes, saying this would only affect OLS mode 2.

It was commented that making the definition of what is the 0-th OLS become different in different OLS modes seems confusing. One participant said that currently modes 0 and 1 are just shortcuts for signalling with mode 2 for traditional scalability and multiview operation.

No action was taken.

1. Change vps_all_independent_layers_flag to 2-bit vps_layer_dependency_idc to indicate common layer dependency to align with VPS OLS mode signalling (0 means all layers independently coded, 1 means all non-base layers use ILP, with immediate lower layer as direct reference layer, 2 means general referencing, 3 is reserved)? (JVET-R0261 aspect 1)

This proposes shortcuts for particular layer dependency relationships that would be inferred by the value of this idc.

It was pointed out that the proposal was missing some necessary changes to the text. Another participant noticed a different problem.

The proposal did not seem mature or necessary, so no action was taken on it.

1. Add a constraint that for each independent layer (i.e., vps_independent_layer_flag[ GeneralLayerIdx[ nuh_layer_id ] ] is equal to 1), there shall be an OLS that contains that layer only? (JVET-R0191 item 2).

The proponent said this did not need to be considered, especially considering the action taken on JVET-R0343 item 1.g.

It was also commented that this proposal would not be friendly to the V-PCC use case.

Other VPS clean-ups:

1. Change the coding of ols_ptl_idx[ i ] from u(8) and possibly vps_num_ptl_minus1? (JVET-R0161 proposal 1)
   1. Option 1: Change to u(v) with length equal to Ceil(Log2(vps_num_ptls_minus1+1))
   2. Option 2: Change to ue(v) and also change vps_num_ptl_minus1 from u(8) to ue(v)

It was commented that there is a reason that FLC coding was used, which was to make those fields accessible to systems with fixed length. Variable-length codes are only used for syntax elements that are at later positions in the syntax structure, and PTL information can be carried separately by a system. Thus, no action was taken on this.

1. Change the coding of num_output_layer_sets_minus1 from u(8) to u(v) with length eqaul to min( 8, vps_max_layers_minus1 + 1 ) (JVET-R0161 proposal 3)

This has the same issue as item 27 above, so no action was taken on it.

1. Infer vps_layer_id[ 0 ] to be equal to nuh_layer_id of the first VCL NAL unit in a bitstream when vps_layer_id[ 0 ] is not signalled? (JVET-R0158 aspect 1)

The proponent indicated that they were trying to address a case when the VPS is not present.

Decision (bug fix for existing intent): For the AUD, the value of nuh_layer_id should not be constrained (as with DCI, VPS and EOB).

1. When VPS is not present:
   1. Require sps_max_sublayers_minus1 to be in the range of 0 to 6, inclusive, when sps_video_parameter_set_id is equal to 0? (JVET-R0158 aspect 2)

See the notes for item “b” below.

• 1. Infer vps_max_sublayers_minus1 to be equal to 6 when sps_video_parameter_set_id is equal to 0 (i.e. VPS is not present). (JVET-R0222 aspect 1)

The difference between approaches “a” and “b” is purely editorial. A participant commented that this inference would fix another case in the spec as well.

It was commented that JVET-R0125 item 1 is also the same proposal.

Decision (bug fix for existing intent): Adopt this inference. (The exact editorial expression is at the discretion of the editor.)

• 1. If DCI is present, infer vps_max_sublayers_minus1 to be dci_max_sublayers_minus1 or 6 otherwise. (JVET-R0199 aspect 2)

Because of another action taken at the meeting that removed the syntax element dci_max_sublayers_minus1, this item is no longer relevant.

1. Constrain the maximum value of vps_max_sublayers_minus1 to be less than or equal to dci_max_sublayers_minus1? (JVET-R0199 aspect 1)

Because of another action taken at the meeting that removed the syntax element dci_max_sublayers_minus1, this item is no longer relevant.

Discussion stopped here for JVET Track A on 19 April at 0900 UTC.