JVET-R0415 AHG12: A summary of proposals on subpictures [Hendry (LGE)]

Discussion began here for JVET on 16 April at 1315 UTC.

1. Condition sps_independent_subpics_flag on "sps_num_subpics_minus1 > 0". (JVET-R0071 #1, JVET-R0156 #4, JVET-R0284 #1)

It was asked whether, in the case of extraction, there would be value in knowing the original value of the loop_filter_across_subpic_enabled_flag. In this case the parameters wouldn’t be sent anyway in the current syntax.

This would save only one bit in the SPS.

It was commented that (at least after other actions of the meeting), subpic_treated_as_pic_flag and loop_filter_across_subpic_enabled_flag have no effect.

We would need to establish inference if these flags are used for anything.

This relates to #2 and #3 below.

The motivation is just cleanup to make it more clear what the logical relationships are.

Decision (cleanup): Adopt this aspect.

1. When sps_independent_subpics_flag is not present, it is inferred to be equal to 1 (JVET-R0071 #1, JVET-R0156 #4, JVET-R0136 #1)

After the action on item #1, this is editorial; see the notes for item #3 below.

1. Change the inference of subpic_treated_as_pic_flag[ i ] when not present.
   1. Infer it to be equal to 1 (JVET-R0071 #2)
   2. Keep the current inference, which is to infer it to 0 (JVET-R0284 #1)

Previously the inference would affect the ability to use wrap-around, but that dependence was agreed to be removed earlier in the meeting. Inference to 1 seems more logical, although it does not make a functional difference. At this point, it seems to be a purely editorial matter.

Decision (Ed.): It is suggested for the editor to specify inference of the value 1 for sps_independent_subpics_flag and the value 1 for subpic_treated_as_pic_flag[ i ] and the value 0 for loop_filter_across_subpic_enabled_pic_flag[ i ] when not present.

1. Change the inference of loop_filter_across_subpic_enabled_pic_flag[ i ] when not present.
   1. Infer it to be equal to 0 (JVET-R0071 #3)

This is just editorial, as was the case for item #3 above; see notes for item #3.

1. Infer single_slice_per_subpic_flag to be equal to 1 when no_pic_partition_flag is equal to 1 (JVET-R0071 #4).

This is just editorial, but the suggested change seems logical.

Decision (Ed.): It is suggested for the editor to specify inference of the value 1 for single_slice_per_subpic_flag when not present.

1. Condition the presence of sps_ref_wraparound_enabled_flag such that it is present only when sps_independent_subpics_flag is equal to 0. When not present, infer the value to be equal to 0. (JVET-R0284#2).

This item was no longer valid after an agreement reached earlier in the meeting.

1. Order of slices in PPS signalling and in picture. It is asserted that there is problem since the order of slices signalled in PPS may be different from the order of slices in decoding order.

Example 1

Example 2 (an example that has subpictures that contain only partial tiles, which would be disallowed by the constraint below)

Is it a problem? if yes, the following are proposed fixes:

• 1. Introduce constraints to ensure that the slice signalling order in the PPS and the slice coding order within the bitstream are the same. (JVET-R0091 #1)

The proposed constraint:

The signalling order of slices in the PPS shall follow the decoding order of slice NAL units. Let slice A be signalled by the syntax elements slice_width_in_tiles_minus1[ sA ], slice_height_in_tiles_minus1[ sA ], num_exp_slices_in_tile[ sA ] and exp_slice_height_in_ctus_minus1[ sA ] and let slice B be signalled by the syntax elements slice_width_in_tiles_minus1[ sB ], slice_height_in_tiles_minus1[ sB ], num_exp_slices_in_tile[ sB ] and exp_slice_height_in_ctus_minus1[ sB ]. If coded slice NAL unit A precedes coded slice NAL unit B in the bitstream, then sA shall be less than sB.

In order for item 7.a to work, the following constraints are also needed:

One or both of the following conditions shall be fulfilled for each subpicture and tile:

– All CTUs in a subpicture belong to the same tile

– All CTUs in a tile belong to the same subpicture

• 1. Introduce a mapping between the two indexing orders (JVET-R0091 #2, JVET-R0238). In addition, definition of subpicture-level slice index is updated in the spec text (JVET-R0238)

It was commented that the constraint approach could prohibit a hypothetical use encountered in one subpicture out of 96 in an example 360° use case.

We had previously agreed not to prohibit the hypothetical use unless we had a reason to prohibit it, but the potential need to introduce a mapping may be such a reason.

It was commented that there is some text in the draft currently about a subpicture-level slice index, and an equation expressing such an index is not currently specified clearly in the text.

The mapping proposed in JVET-R0238 is to specify a mapping from a subpicture-level slice index to a picture-level slice index.

It was commented that if the constraint approach is taken, some encoders might violate it. However, the constraint only requires slice order to follow the order in the header syntax, which would arguably be strange to violate.

In fact the difference between JVET-R0091 #2 and JVET-R0238 was only editorial.

The proponent of JVET-R0091 preferred the constraint approach.

Decision (cleanup): Adopt the constraint approach of JVET-R0091 option 1. (The editor has discretion over the manner of expression in the text.)

1. Alignment on subpic_treated_as_pic_flag value across layers (JVET-R0118 #2, JVET-R0186 #3)

Other aspects of subpictures are required to be aligned across layers.

This is only in regard to SNR scalability.

It was commented that this corresponds to item 1)b of document JVET-R0058 in section 6.1.1; see the notes for that topic.

1. Move the signalling of no_pic_partition_flag to be earlier than the signalling of pps_num_subpics_minus1. When the value of no_pic_partition_flag is equal to 1, pps_num_subpics_minus1 is not present and inferred to be equal to 0 (JVET-R0186 #1)

This is primarily motivated by a desire for logical structuring of the syntax.

It was commented that this is also a similar aspect in JVET-R0088.

Decision (cleanup): Adopt this aspect.

1. Constrain the value of single_slice_per_subpic_flag to be equal to 0 when no_pic_partition_flag is equal to 0, the number of tiles in picture is equal to 1, and the number of subpictures is equal to 1 (JVET-R0186 #2)

The intent was for this to only apply when the number of slices is equal to 1.

In the combination that is proposed to be prohibited, it would be possible to indicate the same behaviour using no_pic_partition_flag equal to 1.

This constraint is not strictly necessary, but the proponent suggests prohibiting it because it seems like a strange syntax combination. There was no clear need for action on this, so no action was taken on this.

1. When the maximum picture width and height are both less than or equal to one CTB size, sps_num_subpics_minus1 is not signalled and inferred to be 0 (JVET-R0239 #5)

Such a usage would seem extremely rare in practice, and no action was taken on this.

1. Signal a flag sps_raster_scan_order_subpics_flag in the SPS to specify whether subpictures are ordered in raster scan order in the bitstream. (JVET-R0257 #1)

Raster scan ordering of subpictures, which is a unique ordering of subpictures in the bitstream, is claimed to be useful for extraction and merging purposes and to provide a hook for, e.g., external use.

• 1. Use the above flag to skip the signalling of the top-left position of the subpictures in the SPS when sps_raster_scan_order_subpics_flag is equal to 1. (JVET-R0257 #2)

This would provide a shortcut for a mode to specify a raster scan order for the subpictures.

It was suggested not to provide the raster indication purely as metadata, without a syntax shortcut.

It was commented that specifying the shortcut would involve adding more text details to specify the special case and that this seems unnecessary, esp. due to our late stage in the development.

It was agreed that raster order would be common, and raster scan slices have a provision for this.

Using a VUI flag was suggested. However, it was noted that VUI is currently only being used for picture format interpretation purposes (colour interpretation and field indication).

Using a general constraint flag was suggested. These are, at least currently, being used as feature disabling indicators rather than as SEI-like metadata.

No action was taken on this.

1. Enable signalling of subpicture with filler / uncoded slices. (JVET-R0337, JVET-R0151)

It is asserted that such feature can be used for efficient coding when subpictures do not completely fill up a picture, by providing completely unused regions. The feature is asserted to be useful for V-PCC, 360° video, and layered coding applications.

If such support is agreed, the following changes to the text are proposed:

• 1. A flag sps_filler_slice_present_flag / (or sps_allow_uncoded_subpics_flag) is signalled when subpic_info_present_flag is equal to 1. When the flag is equal to 1, signal subpic_treated_as_filler_slice_flag[ i ] (or subpic_is_uncoded_flag[ i ]). (JVET-R0337, JVET-R0151)
  2. Decoding of filler slice in subpicture can be “normative” or “non-normative” (JVET-R0337)

There have been previous related contributions. In some variations this involves only a metadata indication. In previous discussion there had been a suggestion for some later development of metadata.

JVET-R0151 proposes SPS-level specification of subpictures that have no coded slices in the entire CLVS, and also an ability to have a PPS specification of areas with no coded slices in the picture.

Several example use cases are described in JVET-R0151. Viewport-dependent streaming was mentioned as another potential use.

The standard currently requires coding all regions of the picture (although this may involve coding regions as basically entirely skipped – e.g., planar prediction with no residual or inter prediction with no residual).

It was commented that the “normative” approach is basically a coding efficiency proposal, possibly with a complexity benefit for software decoders (depending somewhat on what is defined to be the normative output of the decoding process).

What JVET-R0337 refers to as “non-normative” is a metadata indicator that accompanies content that is coded in the ordinary manner.

These are proposing a significant added feature that, in some variations, would have a large impact on the standard and its concepts of normative behaviour for output. It was agreed that we are too late in the standard development process to add such a feature. The metadata approach could be developed as a later-standardized SEI message. No immediate action was taken on this. Later development of an SEI message approach is for further study.

1. Signalling pps_num_subpics_minus1 in PPS as mandatory, to avoid asserted parsing dependency on SPS (on single_slice_per_subpic_flag when pps_num_subpics_minus1 is not present). (JVET-R0117 #1)

The contributor said this did not need consideration, as there is no actual parsing dependency (just a constraint).

1. Change the signalling of subpicture layout in unit of integer multiples units of CtbSizeY (JVET-R0135)
   1. Option 1: signal subpic_unit_num_ctus_minus1 syntax element that is the number of CtbSizeY in the subpicture layout. All syntax elements of subpicture layout use the same in units.
   2. Option 2: signal subpic_unit_num_ctus_x_minus1 and subpic_unit_num_ctus_y_minus1 syntax elements that are the number of CtbSizeY in the subpicture layout. The first syntax element indicate the units for the subpic_ctu_top_left_x[ i ] and subpic_width_minus1[ i ], and the second for the subpic_ctu_top_left_y[ i ] and subpic_height_minus1[ i ]
   3. Option 3: combine Option 1 with Option 2 and two options can be selected.

This is proposing a shortcut method of signalling subpicture layout for bit savings in the SPS (as a generalization of the current syntax with width/height multiple). It was commented that we had previously considered something somewhat similar in spirit. The proponent said in the example shown in Fig. 7 of the draft standard, about 100 bits could be saved in SPS-level signalling.

It was asked whether the scheme had been implemented in software, and it had not. The bit savings estimate was based on calculation.

It was asked how this works if the picture width/height is not an exact multiple of the unit width/height.

Some participants commented that introducing such a concept at this stage would run a risk of introducing bugs, especially since this had not been tested. Saving bits at the SPS level is generally not considered very important. Some other participants noted that in this case the bit savings at the SPS level could be substantial in some uses and found it conceptually simple.

A further generalization was suggested in the discussion, which would be to use such a scaling factor in additional parts of the syntax.

This was further discussed on 23 April at 0800 UTC. Software had been provided in a -v4 of the contribution. The contribution had been revised to prohibit the use of the shortcut unless the picture is an exact multiple of the scaled number of CTUs in width/height. One participant said they had had cross-checked the scheme and said it seemed to work properly, but they thought most encoders would just not use the shortcut. They had some concern they had found a problem in the draft for the subpicture extraction process and that the scaling factor was missing in some other part of the text, and noted that it can only be used in the limited case with exactly uniform units. However, some other participants said it was straightforward and could save a lot of signalling in some example cases and volunteered to help with the editorial work. Several participants expressed support for approach “b”. The functionality is not changed – it is just a signalling shortcut. The opinions were rather mixed and the issue seemed minor. In the interest of stability, no action was taken.

1. Change the signalling of slice_subpic_id as follows: (JVET-R0087)
   • Add a flag called slice_subpic_info_present_flag
   • Replace subpic_info_present_flag to condition the presence of slice_subpic_id
   • The value of slice_subpic_info_present_flag is the same as subpic_info_present_flag

The motivation was said to be to remove a parsing dependency in the slice level on something in the SPS. It was noted that we have many such parsing dependencies; this is an ordinary part of our design. No action was taken on this.

1. Move the signalling of subpic_id_mapping_in_pps_flag, pps_num_subpics_minus1, pps_subpic_id_len_minus1, and pps_subpic_id[ i ] to be present only when (pps_)no_pic_partition_flag is equal to 0 (JVET-R0088)

This contribution was said to be related to JVET-R0186. Aspect 1 of the contribution belongs to this category. It was commented that even when there is no partitioning of the picture, the extraction case would require the possibility of signalling subpicture ID mapping when a single subpicture is extracted. Thus, no action was taken on this.

Discussion stopped here for JVET on 16 April at 1715 UTC (chaired by GJS, JRO, YKW).

Discussion began here for JVET Track A on 21 April at 0835 UTC (chaired by GJS & YKW).

1. Add a constraint that the value of subpic_treated_as_pic_flag[ ] shall be equal to 1 when the value of SubpicIdVal[ ] of the subpicture is changed from the previous picture (JVET-R0126)

It is intended to guarantee that only independently coded subpictures can be relocated by subpicture ID remapping in PPS.

It was commented that there is already an IRAP constraint for collocated subpicture ID changes.

It was asked whether there is some reason that makes it necessary to establish this proposed further constraint. It was not clear that this constraint would be necessary for the decoder, so no action was taken on it.

1. Add subpicture ID mapping signalling override mechanism (JVET-R0265)
   1. Remove subpic_id_mapping_explicitly_signalled_flag in SPS
   2. Repurpose subpicture ID mapping flag in PPS (i.e., change subpic_id_mapping_in_pps_flag to subpic_id_mapping_override_in_pps_flag). When it is equal to 1, subpicture ID is overridden in PPS.

The basic idea is to have subpicture IDs always in the SPS (either derived or explicitly signalled) and the ability to override some or all of them in the PPS. This is so that some of them (the ones earlier in the list) can be overridden while not overriding all of them. The basic motivation is to save PPS bits.

It was commented that an extra constraint is also missing from the proposal.

It was commented that the way this is proposed, with overriding at the beginning of the list (those at the left and top of the picture), the syntax/approach does not seem very “clean”, and this loses an ability to not send the mapping in the SPS. Thus no action was taken on this.

1. On subpicture Id and subpicture Idx in sub-bitstream extraction
   1. Use the subpicture index instead of the subpicture ID in the subpicture sub-bitstream extraction process (JVET-R0068 #5). This is because the subpicture index corresponds to a position in the picture, but the ID can change within a CLVS.

It was said that this could make the extraction process specification simpler.

It was also commented that this could avoid a potential problem with having some ID not appearing in the bitstream, whereas spatial positions should never be unexpectedly absent.

Decision (cleanup): Adopt.

• 1. Derive subpicIdx similar to CurrSubpicIdx for each slice, right after the definition of subpicId (JVET-R0294). This is proposed as a bug fix for the existing approach, and is no longer relevant after the action on subitem “a”.

Discussion stopped here for JVET Track A on 21 April at 0915 UTC.

Discussion began here for JVET Track A on 21 April at 1730 UTC (chaired by GJS & YKW).

1. On subpicture size and picture size rewriting for sub-bitstream extraction.

A bug is asserted exist in the current spec for rewriting of picture size during sub-bitstream extraction process. The root of the problem is when the subpicture is located at the bottom and/or right border of a picture that has a size that is not a multiple of the CTU size because subpicture size (i.e., width and height) is expressed in CtbSize, instead of luma samples.

• 1. Change the sub-bitstream extraction process with different calculation for picture size when the subpicture is the right most subpicture or the bottom subpicture in the original bitstream (JVET-R0092)
  2. Derive the subpicture width and height in luma samples and update the rewriting process of picture width and height (JVET-R0294)

Subpicture size is sent in CTU units.

The difference between the “a” and “b” approaches is only editorial.

Decision (spec bug fix / expression of existing intent): Adopt as proposed. The editorial difference can be worked out by the editor. No impact on the software.

1. Add a constraint such that no subpicture can be located completely outside of the conformance cropping window. (JVET-R0093 #1, JVET-R0294).

There are already constraints that when subpictures are used the picture size cannot change, and the conformance cropping window also cannot change.

This would require rewriting if an encoder wants to select a cropping window that doesn’t include anything from a subpicture.

It was commented that such a picture region might be usable for inter-layer reference.

A conforming bitstream needs to have a conformance window that is not empty.

If a subpicture is completely outside the conformance window, and if it is extractable (the subpic boundary treaing as picture boundary flag is equal to 1), but does it need to be included in the conformance test that use a subpicure sub-bitstream extraction process?

If a subpicture completely outside of the conformance window is extacted but in the extracted sub-bitstream there is a valid conformance window, would it be a problem? Seems not. However, this would be strange as this impose conformance requirement beyond what's was required to be conforming by the original encoder.

HEVC includes both the conformance window and the default display window. The latter can be used for the purpose of something inside the conformnce window but not intended to be displayed together with other stuff inside the conformance window but not in the default display window.

Decision (sensibility cleanup): Adopt.

1. Define rewriting process for conformance cropping window for sub-bitstream extranction process (JVET-R0093 #2, JVET-R0294)
   1. The conformance cropping window offsets of the full picture are kept or not depending on where the subpicture is located within the full picture. If the subpicture is located in the middle of the picture, the conformance cropping window offsets for the subpicture are set to zero (JVET-R0093 #2)
   2. Copy all offset values that cross the subpicture to be extracted. If a subpicture lies completely inside the conformance window, no conformance window shall be signalled (JVET-R0294)

The difference between 23.a and 23.b are basically only editorial.

Decision (expression of existing intent): Adopt. The editor was requested to figure out the exact wording based on the proposed changes.

1. Handling of decoded picture hash SEI msg (JVET-R0294, JVET-R0242):
   1. Option 1: The following applies: (JVET-R0294)
      • Decoded picture hash SEI messages are removed during extraction
      • Decoded picture hash SEI messages are allowed to be nested inside of scalable nesting SEI messages, if subpicture nesting is signalled in the scalable nesting SEI message
      • Decoded pictures hash SEI messages that are nested in a scalable nesting SEI message and associated with subpicId are extracted into the output bitstream
   2. Option 2: extended decoded pciture hash SEI msg with hashes for each subpicture (JVET-R0294)
   3. Option 3: add a separate standardalone SEI message. (JVET-R0242)

With a subpicture specific hash signalled, subpicture sub-bitstream extraction process would be easy. Otherwise the picture-level decoded picture hash SEI messages have to be discarded during the sub-bitstream extraction.

The identfication of which SEI message conveys the picture-level or subpicure-level information can be determined by the container of the SEI message (nested or non-nested) or by using a separate SEI message.

In production encoding and actually applications, people don't send the decoded picture hash SEI messages in the bitstream. The SEI message has really been included for debugging purposes during the development of the standard.

Option 1 is really proposing to change the extraction process to utilize the existing scalable nesting SEI message and the existing decoded picture hash SEI message in the extraction process.

Software is provided in JVET-R0242. The software coordinator took a look and said it was good.

Decision (cleanup): Adopt option 1.

1. Information contribution on successful experiments carried out for implementation of subpicture-based system. The experiment included the following steps (JVET-R0148):
   1. Encoding several bitstreams, each with one subpicture per picture, using the VTM encoder
   2. Merging selected encoded bitstreams into a bitstream with multiple subpictures, using a merger software developed by the authors
   3. Decoding the bitstream having multiple subpictures, using the VTM decoder

It was said that the experiments demonstrated that the subpictures design was mature.

If there is sufficient interest, the merger software could be considered to be donated, e.g., to be included as part of the VTM reference software.

Why to turn off ALF, LMCS, and SAO? The encoding used the VTM encoder, but indepednently for the "small" pictures.

Discussion stopped here for JVET Track A on 216 April at 1900 UTC (chaired by GJS & YKW).

General and misc. subpicture aspects (11)