JVET-Q0684 AHG9: A summary of HLS contributions on picture header, slice header, and access unit delimiter [Hendry (LGE)] [late]

This contribution was discussed on Thursday 9 January in Track A at 1500 (chaired by GJS).

On PH properties

1. Do we allow PH repetition?
   1. No. Keep the current design that does not allow PH repetition within a picture

JVET-Q0115 aspect 1 & 2: No PH repetition within a picture; consequently remove AUD.

• 1. Yes. Allow PH repetition (JVET-Q0177):
• Signal POC LSB in PH to identify it in temporal domain. – deferred to discussion below.
• Differentiate PH that is repeated. To do this, the following flags may be present: sps_ph_repetition_enabled_flag in SPS; ph_repetition_present_flag in PH.

AUD is currently optional, and PH is required. If there are multiple layers in an AU, detection of the AU boundary may involve checking the layer ID.

It was discussed whether we care about PH for the purpose of loss resilience.

It was commented that the PH was created for bit efficiency reasons, while the AUD is for AU boundary detection. They appear at different locations in the NAL unit stream – e.g., a common sequence of NAL units would be AUD, SPS, PPS, SEI, APS, PH, Slice, … Since AUD is optional anyway, we don’t have a clear need to remove it.

It was asked, if we allow PH repetition, why we wouldn’t just have a flag in the PH that identifies whether the current PH is a repeated one or is the first one?

PH repetition could also be inside an SEI message. Such an SEI message would not need to be defined in v1 of the standard.

The motivation for PH repetition would be partial-picture decoding when there are lost NAL units in a picture.

No action was taken on the repetition question, so repetition remains prohibited.

1. Where to signal PH. Currently PH is signalled in a NAL unit, 1 PH NAL unit is mandatory for a coded picture. Is it allowed that PH syntax structure is contained in a slice VCL NAL unit (JVET-Q0255, JVET-Q0419, JVET-Q0426)?
   1. If Yes, how the VCL NAL unit carry the PH syntax structure?

• Option 1 (JVET-Q0255): Specify a new NAL unit type Coded Picture. Coded Picture NAL unit contains a syntax element pic_type, a PH syntax structure, an SH syntax structure, and a slice data syntax.
• Option 2 (JVET-Q0419 aspect #1): In slice layer RBSP, signal ph_present_flag. If this flag is equal to 1, signal PH syntax structure in the slice VCL NAL unit.
• Option 3 (JVET-Q0426): The following is proposed:
• Signal sps_picture_header_enabled_flag. When this flag is equal to 0, PH is not in PH NAL unit; syntax elements in PH are present in SH (which can be put in separate common syntax structure).
• When sps_picture_header_enabled_flag is equal to 0, signal first_vcl_nal_unit_in_picture_flag in SH. It is recommended that sps_picture_header_enabled_flag is equal to 1 for bitstream that may be extracted / merged.

JVET-Q0426 was revised; in the new version, there would always be a flag in the SH for whether a PH NAL unit present for the picture or not. If not, that syntax is in the SH (of every slice in the picture). There might be a restriction that this could only happen when there is only one slice in the picture.

Option 1 would change the function of the NAL unit type, such that properties such as IRAP indication would not be indicated by the NAL unit type.

Options 2 and 3 are rather similar in concept.

It was commented that when we measure CTC results, the BD rates include start code overhead.

It was commented that for the LB case, with one of the 720p test sequences in the CTC (“Johnny”), this could have a BD impact of 1.4%.

We can’t remove the whole idea of allowing a PH to be a separate NAL unit, since that is needed for BEAM.

A suggested minimum approach:

• There was discussion of limiting the combined slice header to a special case for when there is only one slice per picture without subpicture support.
  • There could be an SPS flag indicating that the whole CLVS always has one slice per picture. This might or might not be necessary
• At a minimum, a flag would need to be added to the header of every slice (regardless of whether is being used or not) that would indicate whether the PH data are in the SH or not.
  • There should be a constraint that this flag must be 0 for the entire CLVS or 1 for the entire CLVS.
  • The flag would be required to be 0 unless there is only one slice per picture and no subpicutures.
• The POC LSBs would be in the PH, regardless of wether that is in a separate NAL unit or not.
• The presence of the four flags that control whether something is in the PH or SH could be conditioned on the that added flag. These flags could be grouped together.

See the notes for JVET-Q0775 and JVET-Q0819.

On PH / SH override mechanism

1. Change the current PH / SH override mechanism as follow (JVET-Q0200, JVET-Q0259):
   1. The flags that specify whether a syntax element of a related coding tool is present in PH or in SH (but not both) are proposed to be moved from the PH to the PPS.
   2. This mechanism may be applied for signalling of syntax element related to the following tools:

• reference picture lists, SAO, ALF (JVET-Q0200 & JVET-Q0259)
• deblocking (JVET-Q0200).

It was commented that since the PPS ID can change on every picture anyway, moving the control from the PH to the PPS doesn’t make much of a difference unless this is motivated by a desire for coding efficiency by saving bits in the PH. So this is a matter of saving bits in the PH.

It was commented that this makes particular sense if we allow the PH to be combined into the SH, because in that case we could require a particular location by requiring a value for these flags.

It was commented that it is a question of usage of these tools – whether it is envisioned that there should be a switching of the decision from picture to picture (with other aspects of the PPS staying constant) for whether the controlled syntax is in the PH or SH.

Decision (cleanup): Adopt. Text in JVET-Q0200 (v2), Hendry resp. for software. This text is also integrated into JVET-Q0819.

1. JVET-Q0270 aspect 3, 4, and 5:
   1. Replace pic_deblocking_filter_override_present_flag and pic_deblocking_filter_override_flag with a pic_deblocking_filter_override_idc syntax element specifying whether deblocking filter parameters are not overridden in picture header or slice header, deblocking filter parameters may be overridden in picture header or deblocking filter parameters may be overridden in slice header.
   2. Replace pic_sao_enabled_present_flag and pic_sao_luma_enabled_flag with a pic_sao_enabled_idc syntax element specifying whether SAO is disabled for the whole picture, SAO is enabled for the whole picture or SAO may be enabled per slice. Similar way is proposed for ALF aspect.

This is no longer relevant due to the action on JVET-Q0200.

On specific coding tool related signalling in PH / SH

1. JVET-Q0182: allow scaling list and LMCS to be signalled at slice level, instead of at picture level only. The asserted benefit is for the case of hybrid picture where a picture is a composition of different scenes (e.g., in a video conf showing multiple people and a PC screen).

Two options how to do this:

• Option 1: SL/LMCS syntax elements are signalled either at PH or slice header (SH) level. If SL/LMCS syntax elements are signalled in a PH, they cannot be signalled in any SH of the coded picture associated with the PH.
• Option 2: overriding the PH SL/LMCS settings at SH level is allowed. If overriding is enabled by an overriding flag, additional SL/LMCS information is signalled in the SH; otherwise, the PH SL/LMCS settings are used for the current slice.

This was addressed in a BoG; see JVET-Q0625.

1. On signalling of TMVP collocated ref pic.
2. Move signalling of TMVP collocated ref pic from SH to PH

• JVET-Q0207 option 1: when TMVP is enabled for picture associated with the PH, identify the collocated reference picture using its delta POC relative to the current picture’s POC.
• JVET-Q0259 aspect 5: indicate information about collocated picture in picture header when RPL information is signalled in the picture header.

Decision (cleanup): Adopt JVET-Q0259 aspect 5; proponent resp. for SW. This text is also integrated into JVET-Q0819.

1. Keep the signalling of TMVP reference picture in SH and change the signalling for the case when there is only one reference picture in the DPB with the same spatial resolution as the current picture (JVET-Q0207 option 2).

It was commented that this conditions the parsing on a complicated condition, and thus should not be done.

1. JVET-Q0207 aspect 1: Update the constraint for the value of pic_temporal_mvp_enabled_flag when there is no reference picture in the DPB with the same spatial resolution with the current picture. Rather than considering all reference pictures in the DPB, the constraint should be expressed more precisely by considering only the active reference picture(s) of the picture.

It was commented that this change is not necessary and that the constraint expression may not even be needed; no action unless not resolved offline.

1. JVET-Q0130: Modify semantics of collocated_ref_idx.

The contribution expressed a concern over the possibility that a non-conforming could be created that would actually be decodable. The contribution proposes that the decoder override the indicated use of temporal MVP if certain conditions are violated (instead of requiring the encoder to not indicate the use of temporal MVP under those conditions).

The group did not like that this would add extra processing to the decoder and would allow the encoder to provide a misleading indication of what the decoder would be doing. No action was thus taken on the proposal.

1. It is proposed to disallow weighted prediction with customized weights at the slice level and to move the signalling of the prediction weight table from SH to PH (JVET-Q0247). The proposed aspects include:
2. Combine pps_weighted_pred_flag (which) and pps_weighted_bipred_flag in the PPS into one flag to specify whether weighted prediction may be applied to pictures referring to the PPS.
3. When weighted prediction is enabled, a flag in picture header (i.e., pic_weighted_pred_present_flag) would be present to specify whether weighted prediction is applied to the picture or not. When pic_weighted_pred_present_flag is equal to 1, prediction weight table is present in the picture header.
4. When weighted prediction is enabled for a picture, if the weight table is in the picture header, it is proposed that all slices of the picture would be required to have the same reference picture lists and the same active/non-active status. (And it would be required to send the RPL at the picture level.)
5. In the prediction weight table signalling, explicitly signal the number of reference pictures to be weighted for L0 and L1. This is designed to remove dependency of the prediction weight table signalling to the number of active reference pictures that may be present in slice header.

This proposes the removal of a coding tool functionality and it was agreed we should not do that without careful study, which there probably isn’t time to do.

It does seem strange that the weight table cannot be shared by all slices of the same picture. Decision (cleanup): Make the prediction weight table a fifth type of data that can be signalled either in the PH or SH (like ALF, deblocking, RPL, and SAO). Text was included in the text being prepared for the other four in JVET-Q0200 (initially in the v2 version, refined as described below in the v3 version). Hendry resp. for SW.

There was further discussion of this aspect on Thursday 16 January in Track A at 0915 (chaired by GJS), with review of text from JVET-Q0819, and it was discussed what relationship there should be between the number of weights in the PH and the number of active entries in the slice level. It was discussed whether the number of active entries should be determined at the picture level or slice level in this case. It was agreed to keep the number of active entries established at the slice level, which is constrained to not be larger than the number of weights signalled in the picture header. Additional text drafting was needed to express the semantics of weighted prediction when the weights are sent at the picture header and the slices have a different number of active entries. Text is available in JVET-Q0200-v3.

On changes for signalling of syntax elements in PH / SH and additional features

1. Condition the presence of inter-/intra- related syntax elements.
   1. Using a flag to condition the presence of some syntax elements in PH

JVET-Q0116 aspect 1:

• A new syntax element ph_all_intra_slices_flag is proposed to be signalled in PH, and all the inter-related syntax elements in PH that are not needed for intra slices are conditioned on ph_all_intra_slices_flag not equal to 1.
• The slice_type in slice header (SH) is inferred to be equal to 2 (i.e., intra slice) when the ph_all_intra_slices_flag is equal to 1.
• This flag (ph_all_intra_slices_flag) is asserted to be useful not only for reducing PH size but also may be used by system for some features (e.g., trick mode).
  1. Using two flags (or 2 syntax elements) and use them to condition the presence of some syntax elements in PH
• JVET-Q0153 aspect 2 & JVET-Q0428 option 3: Have 2 flags such to specify whether intra related syntax element is present and whether inter related syntax elements are present.
• JVET-Q0245: In addition to JVET-Q0153, define some constraints for the values of the two flags and to anticipate bitstream extration and merging cases.
• JVET-Q0176 aspect 1: Signal new syntax element mixed_slice_types_in_pic_flag in the PPS. If this flag is equal to 0, signal slice type (i.e., B / P / I) in PH.
• JVET-Q0259 aspect 1: Signal two separate flags for indication of partition constraint override.
• JVET-Q0376 aspect 2: change override flag for partitioning parameters from 1 flag to 2 flags (one of inter and one for intra).
• JVET-Q0428 has 2 options for the definition the the 2 flags (the presence of the second flag is condition upon the value of the first flag):
  • Option 1: pic_single_coding_type_flag and pic_intra_picture_pred_only_flag.
  • Option 2: pic_inter_slice_only_flag and pic_intra_slice_only_flag
  1. Allowing dependent PH

JVET-Q0198: A dependent PH skips the signalling of some syntax elements which are inferred from the preceeding PH. A flag pps_dependent_pic_header_enabled_flag is proposed in the PPS and dependent_pic_header_flag is proposed in PH.

It was agreed to have a two-flag approach. One flag that indicates the presence of intra parameters and one that indicates the presence of inter parameters. When the first flag is 0, no inter slices can be present. When the second flag is 0, no intra slices can be present. If the first flag is 0, the second flag is not present and inferred to be equal to 1. When the first flag is 0, the SH would not contain a slice_type. This is very close to JVET-Q0428 option 2. Text will be developed and provided in a revision of JVET-Q0428. Text was later provided in JVET-Q0781, which was later integrated in to JVET-Q0819. Decision (cleanup): Adopt two-flag approach for structuring of PH syntax elements. Text in JVET-Q0819.

Hypothetically, a third flag could distinguish B-only and P-only from a mixture when I slices are indicated not to be present. Hypothetically, slice_type could become a flag or be absent under relevant conditions.

JVET-Q0176 has a somewhat different proposal that enables sending slice_type in the PH when the PPS indicates that there is only one slice type in the picture. (As originally proposed, it would not allow skipping when intra syntax elements when the picture contains a mixture of P and B slices.) Interest was not expressed by others in this sort of PPS special casing.

Track A stopped here Thursday 9 January evening.

Discussion resumed Friday 10 January at 1115 (chaired by GJS).

1. Rearranging the ordering of syntax elements in PH (also possibly other parameter set as well)
   1. JVET-Q0481: Reordering syntax elements in PH and SPS related to intra and inter coded picture. Also, group the partition constraint syntax elements by type (intra, inter, dual-tree chroma).

Resolved as noted by BoG and other Track A outcomes.

1. Allowing signalling of extra bits in PH.
   1. JVET-Q0400: Add reserved extra bits to the picture header similar to extra slice header bits in HEVC. It is asserted that the proposal, however, differs from the HEVC design in that the presence of the flags in the picture header is controlled by the sequence parameter set.

It was commented that it would be desirable to have the ability to control such extra bits for both the PH and SH. The contribution also has a proposed way to indicate the meaning of such extra bits (using presence flags) and some proposed specific meanings: global and layer-wise non-reference and sublayer non-reference.

The basic idea was supported (for both the PH and SH, each separately controllable). There was discussion of whether the proposed aspects that differ from HEVC’s method are desirable.

This was further discussed on Thursday 16 January in Track A at 1115 after offline discussion.

The presence of these bits is to be indicated in the SPS.

1. On the GDR picture indication in PH.
   1. JVET-Q0270 aspect 1 & JVET-Q0414 Option 1: Condition gdr_pic_flag in picture header on gdr_enabled_flag in SPS such that gdr_pic_flag is not present and inferred to 0 when gdr_enabled_flag is equal to 0.
   2. JVET-Q0414 Option 2: Imposing a constraint on the value of gdr_pic_flag
   3. JVET-Q0154 aspect 1: Have an IRAP or GDR picture indicator at the beginning of PH.

• Option 1: irap_or_gdr_pic_flag in the beginning of PH and use it to condition the presence of gdr_pic_flag in the PH.
• Option 2: irap_or_gdr_pic_idc in the beginning of PH. 1 means the pic associated with the PH is IDR pic, 2 means CRA, and 3 means GDR picture. Remove gdr_pic_flag and condition the presence of recovery_poc_cnt only when irap_gdr_idc is equal to 3.

JVET-Q0270 aspect 1 had a parsing problem and was withdrawn. It was also commented that it would be desirable to be able to use the flag without needing to identify and parse the applicable SPS, so no action on this method.

Decision (Ed.): Clarify as necessary per item b. The constraint is already expressed, but would benefit from clarification in the semantic of the gdr_pic_flag flag.

Decision (PH cleanup): Adopt JVET-Q0154 option 1. The semantics should be a “one way” indication, not a guarantee that the picture is not an IRAP or GDR. Proponent resp. for text and SW.

Ye-Kui Wang was requested to coordinate preparation of a syntax document reflecting the combined recorded agreements for review.

1. JVET-Q0154 aspect 3: Signalling slice layer NUTs as PH_NUTs at picture header. Slice layer NUTs are replaced by SLICE_NUT. For this piece, the following changes is proposed:
   1. If mixed_nalu_types_in_pic_flag is equal to 1, signal irap_gdr_idc in PH
   2. Semantics of irap_gdr_idc is as follows: 0 indicates IDR_W_RADL picture. 1 indicates IDR_N_LP picture. 2 indicates CRA picture. 3 indicates GDR picture.

Addressed by action recorded above.

1. JVET-Q0154 aspect 4: Recovery picture POC count (currently in the PH) signalling for mixed nal unit type pictures. Signal the recovery_poc_cnt at subpicture/slice level or change the definition of the gdr_pic_flag to include mixed nal unit type in picture that has GDR slices.

It was commented that we don’t know of a use case for mixing GDR with non-GDR.

Currently, GDR can only be mixed with IRAP. Other mixing is already prohibited. No clear need was also mentioned for the mixing of GDR with IRAP. Decision (BF): Disallow mixing of GDR and IRAP (Disallow mixing of GDR with any non-GDR).

1. On POC LSB signalling
   1. Move POC LSB signalling from SH to PH (JVET-Q0115 aspect 3, JVET-Q0153 aspect 1)
   2. Add POC LSB signalling in PH (JVET-Q0177 aspect 2). This is asserted to be useful to identify whether the PH is a repeated PH.

There was discussion of whether it is necessary to have POC LSB to enable detection of whether slices are from the same picture or not if a PH can be lost. It was agreed that relevant systems provide other indications, such as timestamps, to resolve such detection needs.

Decision (PH cleanup): Move POC LSB signalling from SH to PH.

1. JVET-Q0116 aspect 2: Change PH extension mechanism to be like the mechanism for extension of parameter sets. Note that currently it is like SH extension mechanism.

The rationale was said to be that slices have slice data that follows the SH data. Parameter sets do not – they only contain header data. Later, after determining that the PH can be combined in the same NAL unit as the SH, it was agreed that no action should be taken on this.

1. JVET-Q0155 aspect 1: move colour_plane_id to SH. Also specify that slice address values are unique within each colour plane (rather than unique within each picture) when separate colour plane coding is used. (Although all of this is somewhat hypothetical, per below.)

Decision (BF): Adopt.

It was asked whether separate colour plane mode is allowed in the 4:4:4 profile or not. In the current draft, it is not forbidden. This mode has been specified also in AVC and HEVC syntax but has been disallowed in all of their profile specifications. Decision (BF): Disallow separate colour plane mode in the 4:4:4 profile(s).

1. JVET-Q0270 aspect 2: signal an se(v) pic_qp_delta syntax element in the picture header and derive SliceQpY = 26 + init_qp_minus26 + pic_qp_delta + slice_qp_delta. The presence of pic_qp_delta is gated by a sps_pic_qp_delta_present_flag syntax element signalled in SPS.

Decision (PH cleanup): Add a PPS flag to determine whether qp delta is sent in the PH or SH, like other things (e.g., ALF, deblocking, SAO).

1. JVET-Q0358: add a TemporalId constraint between the ALF_APS NAL unit and the picture associated with PH.

Decision (BF): Adopt.

1. JVET-Q0376 aspect 1: parameters related to delta QP signalling for Inter and Intra are merged into single parameters.

It was commented that since inter slices may have very large regions relative to intra slices, it may be desirable to keep the ability for the level control of QP as different. No action was taken on this.

1. JVET-Q0379: move the syntax elements related to the APS ID at an early stage of the picture header and slice header. This involves APS ID of ALF, LMCS and scaling list. The suggested location is just after the poc_msb_val syntax element.

It was commented that it is unknown in general whether an APS is used in subsequent later NAL units or not (unless some special knowledge is available about how the data was encoded).

No action was initially planned to be taken unless offline discussion indicated otherwise. Further discussion was later requested. This was then agreed for action in the HLS BoG (see the notes for JVET-Q0625).

1. JVET-Q0419 aspect 2: the value of slice_type in slice_header( ) could be constrained by or inferred from syntax elements signalled in PH. Note that this may be paired with JVET-Q0428.

This was addressed by other actions taken at the meeting.

1. JVET-Q0426 aspect 5: Move mvd_l1_zero_flag from the picture header to the slice header since that is only relevant for the B slice type.

No action seemed necessary for this.

1. JVET-Q0273 editorial input on picture header

This contribution was only editorial – providing improvements for the text relating to the PH.

• It proposes to use “ph_” consistently for PH syntax elements.
• It corrects a mismatch of syntax element names for SAO
• It proposes to use a shared syntax structure for RPL syntax in the PH and SH. (There was a comment that this should be double-checked.)

These editorial improvements were appreciated.

Decision (Ed.): The editor should consider this input, which seems quite helpful.