JVET-R0404 AHG9: A summary of proposals on high level control of LMCS, Scaling list, ALF and SAO [L. Zhang, Y.-K. Wang (Bytedance)]

Discussed in AHG Session 1.16 (chaired by GJS).

This contribution intends to provide a summary of the 21 proposals on high level control of LMCS, scaling list, ALF and SAO submitted to this JVET meeting by the 3 April 2020 submission deadline.

It is suggested that this summary, in terms of a list of design questions, is used for the reviewing of these proposals, such that the discussions can be in a more structured and efficient manner.

For high-level control and semantics changes of LMCS, the following aspects are proposed:

1. Controlling of presence of the SH LMCS enabled flag slice_lmcs_enabled_flag
   1. Conditionally add a new SPS SE to indicate whether slice_lmcs_enabled_flag is present, and when not present, infer the value to be equal to the PH LMCS enabled flag. (JVET-R0051)

This is to save a flag at the slice header level if all slices have LMCS enabled if the PH has LMCS enabled.

It was commented that saving a bit at the SH level doesn’t seem especially important for LMCS, so no action was taken on this.

• 1. Replace the PH flag ph_lmcs_enabled_flag with a 2-bit ph_lmcs_mode_idc, with 3 modes specified: disabled (mode 0), used for all slices (mode 1), and enabled (mode 2); and only signal slice_lmcs_enabled_flag for mode 2. (JVET-R0063)

This is similar in spirit to item a, but using a bit at the PH level instead of at the SPS level to distinguish the cases.

It was commented that saving a bit at the SH level doesn’t seem especially important for LMCS, so no action was taken on this.

• 1. Skip the signalling of the SH LMCS enabled flag for the case when the PH is in the SH. (JVET-R0089, JVET-R0098, JVET-R0210, JVET-R0200, JVET-R0202)

AHG Recommendation (cleanup): Adopt. Text is in JVET-R0098 and software will be provided by that proponent.

• 1. Move the SH flag slice_lmcs_enabled_flag to be just after the ALF parameters (JVET-R0200) so that the header information for LMCS is grouped in a similar way as in the picture header. (It was commented that there may also be another parsing simplification from this.)

AHG Recommendation (cleanup): Adopt.

1. Slice-level control of chroma residual scaling (currently only controlled in the PH).
   1. Remove the PH control flag (ph_chroma_residual_scale_flag) and add one flag in SH. (method 1 of JVET-R0096, method 2 of JVET-R0171)
   2. Add a control flag in slice level under the condition "if( slice_lmcs_enabled_flag  &&  ph_chroma_residual_scale_flag )". (proposal 2 of JVET-R0089, method 2 of JVET-R0096, method 1 of JVET-R0171)

Currently, luma and chroma processing are switched on and off together at the SH level. This would enable controlling them separately.

The proposed chroma flag would only be sent if the luma flag is turned on.

It was commented that we generally have separate control of chroma at the same degree of local level as luma. Others commented that LMCS was brought in as a single tool and the design is somewhat unified (with the balance between luma and chroma somewhat maintained by changing chroma together with luma) and they thought currently logical.

In the absence of sufficient support, no action was recommended on this.

1. Add a constraint to disable chroma residual scaling of LMCS for pictures within a GDR period (JVET-R0393).

The proponent indicated that enabling the chroma scaling can cause a GDR “leak”.

It was asked whether there would be a leak if the virtual boundary is at a CTU boundary. The proponent responded that this would not cause a leak.

It was commented that another approach could be to just add a NOTE to caution the reader that enabling chroma residual scaling could cause a GDR problem if there is a virtual boundary that is not aligned with a CTU boundary. AHG Recommendation (Ed.): It is suggested to add such a NOTE. The editor is requested to consider this. (No normative effect.)

1. Revised semantics (italics for report emphasis only):
2. Change the semantics of sps_lmcs_enabled_flag equal to 1 to use the wording of "may be used" instead of "is used". (JVET-R0051, JVET-R0160)

sps_lmcs_enabled_flag equal to 1 specifies that luma mapping with chroma scaling may be used in the CLVS. sps_lmcs_enabled_flag equal to 0 specifies that luma mapping with chroma scaling is not used in the CLVS.

1. Revise the current semantics of ph_lmcs_enabled_flag to the following: (JVET-R0051, JVET-R0160, JVET-R0210)

ph_lmcs_enabled_flag equal to 1 specifies that luma mapping with chroma scaling may be enabled for slices associated with the PH. ph_lmcs_enabled_flag equal to 0 specifies that luma mapping with chroma scaling is disabled for all slices associated with the PH. When not present, the value of ph_lmcs_enabled_flag is inferred to be equal to 0.

1. Revise the current semantics of ph_chroma_residual_scale_flag to the following:

• ph_chroma_residual_scale_flag equal to 1
  • 1. specifies that chroma residual scaling may be enabled for slices associated with the PH. (JVET-R0051, JVET-R0160)
    2. specifies that chroma residual scaling is enabled for all slices associated with the PH and whether it is applied for each slice is further controlled by the slice_lmcs_used_flag signalled in the slice header. (JVET-R0063)
• ph_chroma_residual_scale_flag equal to 0
  • 1. ph_chroma_residual_scale_flag equal to 0 specifies that chroma residual scaling is disabled for all slices associated with the PH. (JVET-R0051, JVET-R0063, JVET-R0160)

1. Change the semantics of slice_lmcs_enabled_flag equal to 1 to use the wording of "luma mapping is enabled for the current slice and chroma scaling may be enabled for the current slice" instead of "luma mapping with chroma scaling is enabled for the current slice" (JVET-R0160):

slice_lmcs_enabled_flag equal to 1 specifies that luma mapping is enabled for the current slice and chroma scaling may be enabled for the current slice. slice_lmcs_enabled_flag equal to 0 specifies that luma mapping with chroma scaling is not enabled for the current slice. When slice_lmcs_enabled_flag is not present, it is inferred to be equal to 0.

AHG Recommendation (Ed. BF / expression of existing intent): Agreed as detailed above (editor has discretion over exact expression).

AHG Recommendation (Ed.): It is suggested to remove “one, more, or all” phrases in the text.

Discussion ended here for AHG Session 1.16 on 13 April at 1715.

Discussion began here for JVET on 15 April at 0600 UTC (chaired by GJS, JRO, YKW).

The following design questions on high level control of scaling lists were proposed:

1. Controlling of presence of the SH explicit scaling list enabled flag slice_explicit_scaling_list_used_flag
2. Conditionally add a new SPS SE to indicate whether slice_explicit_scaling_list_used_flag is present, and when not present, infer the value to be equal to the PH explicit scaling list enabled flag. (JVET-R0051)

It was commented that saving a bit at the SH level doesn’t seem especially important for explicit scaling lists, so no action was taken on this.

1. Replace the PH one-bit flag by a 2-bit ph_explicit_scaling_list_mode_idc, with 3 modes specified: disabled (mode 0), used for all slices (mode 1), and enabled (mode 2). and only signal slice_explicit_scaling_list_used_flag for mode 2. (JVET-R0064)

It was commented that saving a bit at the SH level doesn’t seem especially important for explicit scaling lists, so no action was taken on this.

1. Skip the signalling of the SH explicit scaling list enabled flag for when the PH is in the SH. (JVET-R0089, JVET-R0098, JVET-R0202)

Decision (cleanup): Adopt. Text is in JVET-R0098 and software will be provided by that proponent.

1. Move the SH flag slice_explicit_scaling_list_used_flag to be just after the ALF parameters (but after slice_lmcs_enabled_flag). (JVET-R0200) so that the header information for explicit scaling lists is grouped in a similar way as in the picture header. (It was commented that there may also be another parsing simplification from this.)

Decision (cleanup): Adopt.

For high level control and semantics changes of ALF/SAO, the following aspects are proposed:

1. Control ALF and SAO at SPS, PH (on/off control, ALF APS information for ALF) and SH level (on/off control) and remove the slice level ALF parameter adaptation. (JVET-R0160)

Currently, we allow the ALF APS ID to be either at the PH level or slice level (but not both). This is different from how LMCS and scaling lists are handled, which has the APS ID only at the PH level (with lower level on/off).

It was asked whether there was a reason that we got into this position where we have this sort of difference between the level of control of these different features.

Some test results were provided in JVET-R0149, showing that the ALF flexibility is useful for distributed encoding (e.g., 4% for 512×512 subpictures), assuming ALF would be disabled entirely for all subpictures if the parameters cannot change on a subpicture basis.

It was commented that coordinated encoding would probably not be feasible for real-time distributed encoding, although perhaps ALF could be used in just one subpicture and not the others.

It was commented that table size is more of a problem for LMCS, such that would make it more difficult to allow multiple parameters within a picture for LMCS than for ALF.

ALF allows indication of more than one APS in the SH or PH with selection between them at the CTU level. The CTC uses the CTU-level switching capability.

It was commented that from an implementation perspective it may not matter whether multiple ALF parameters are sent in the PH or the same number of them is used at the SH level, and sending them at the SH level seems more friendly to BEAM applications.

No clear need for action was identified for this, and the current flexibility seems useful for BEAMing, so no action was taken on this.

1. Indication of chroma ALF
   1. Use two separate flags (one for Cb, one for Cr) to replace ph_alf_chroma_idc in PH and slice_alf_chroma_idc in SH. (JVET-R0225)

The functionality is not proposed to be changed, just the signalling.

It was asked whether this proposed change is purely editorial or not. It seemed to be purely editorial, except for the order of the bits. We usually signal Cb first, then Cr, and that is what this is proposing.

Decision (cleanup): Adopt (as a non-editorial matter, this is just a swap of the bit order).

1. Indication of CC-ALF
   1. Use two separate SEs (alf_ctb_cc_cb_flag and alf_ctb_cc_cr_idx) to replace alf_ctb_cc_cb_idc in CTU level. (JVET-R0225)

This is different in concept from the previous item above.

The motivation is to make it more clear what is happening. It was commented that this is a low-level normative change, not really a matter of HLS, as it affects CABAC parsing. It takes one syntax element that is coded as ae(v) and makes it into two syntax elements that are coded differently. No test results were provided.

No action was taken on this.

1. In PH/SH, add a constraint such that if CCALF is disabled in SPS, an ALF_APS cannot contain any CCALF filters. (JVET-R0232 section 3.2)

It was discussed whether, editorially, the location of the constraint should be different from what is proposed.

Decision (cleanup): Adopt this aspect.

1. Revised semantics (italics for report emphasis only):
2. Change the semantics of ph_alf_enabled_flag equal to 0 to use the wording of "is disabled for all slices" instead of "may be disabled for one, or more, or all slices" (JVET-R0068, JVET-R0251)

ph_alf_enabled_flag equal to 0 specifies that adaptive loop filter is disabled for all slices associated with the PH. (JVET-R0068)

ph_alf_enabled_flag being present and equal to 0 specifies that adaptive loop filter is disabled for all colour components in all slices associated with the PH. When not present, ph_alf_enabled_flag is inferred to be equal to 0. (JVET-R0251)

(The phrase “being present” is noted to be necessary for proper expression of this.)

It was commented that it seems undesirable to infer the value 0 for ph_alf_enabled_flag if it is possible for slice_alf_enabled_flag to be equal to 0 in that case. It was suggested to rephrase the semantics to avoid this inference.

It was also commented, and agreed, that we should also avoid having a value of a syntax element that means something different when it is inferred versus what it would mean if it is present.

1. Change the semantics of sps_alf_enabled_flag equal to 1 to use the wording of "may be enabled" instead of "is enabled" (JVET-R0160):

sps_alf_enabled_flag equal to 0 specifies that the adaptive loop filter is disabled. sps_alf_enabled_flag equal to 1 specifies that the adaptive loop filter is enabled.

1. Change the semantics of sps_sao_enabled_flag equal to 1 to use the wording of "may be applied" instead of "is applied" (JVET-R0160):

sps_sao_enabled_flag equal to 1 specifies that the sample adaptive offset process may be applied to the reconstructed picture after the deblocking filter process. sps_sao_enabled_flag equal to 0 specifies that the sample adaptive offset process is not applied to the reconstructed picture after the deblocking filter process.

AHG Recommendation (Ed. BF / expression of existing intent): Agreed as detailed above (editor has discretion over exact expression).

For APS related aspects, the following are proposed:

1. Move scaling_matrix_for_lfnst_disabled_flag from the scaling_list_data( ) syntax to the SPS. (JVET-R0064)

The current location of the flag allows a scaling matrix with LFNST to be used in some pictures and not others.

Decision (cleanup): Adopt.

1. Parameter set updating, cross-layer sharing, and decoding order of APSs
   1. Update to the content of an ALF APS NAL unit within a PU is allowed. (JVET-R0070)

See the notes for the next item.

• 1. Allow update of the content of an ALF APS NAL unit between subpictures of a PU. (JVET-R0149)

The motivation for this and the previous item is basically to increase the number of ALFs that can be applied within a single picture (currently limited to 8).

An example use case is with 96 subpictures.

Each ALF APS takes about 512 bytes, so 8 of them take 4k bytes.

It was discussed that some decoders may either process the data in a different order from the parsing order or may perform ILF stages as a separate pass. All ALF parameters for the entire picture may need to be stored.

It was noted that re-using APSs across different pictures would be less feasible if encoders are forced to re-use the same indices within a picture.

It was commented that if the encoders are coordinated well, there may not really need to be entirely separate ALF parameters for each subpicture.

Given the substantial memory impact, at least for some decoder architectures, the methods proposed in these contributions were not supported.

It was suggested to consider a constraint on the total memory used (or the number of filters in the APSs – there are up to 25 luma and 8 chroma filters in one APS) rather than the number of APSs, since the amount of memory used by an APS depends on its content. JVET-R0480 is a late contribution containing such a proposal.

Discussion stopped here on Wednesday 15 April at 0915 UTC.

Discussion started here for JVET on 16 April at 0820 UTC (chaired by GJS, JRO, YKW).

• 1. Sharing of an APS NAL unit across layers is proposed to be disallowed. (JVET-R0070). The usefulness of the sharing was asserted to be questionable, and disallowing sharing could potentially simplify semantics and extraction and multilayer concepts.

It was commented that multiview might be a case where sharing may sometimes feasible, and noted that the number of APSs is limited (4 for LMCS, 8 for ALF and scaling lists, with all layers sharing the same value space). Updating of PSs is allowed between PUs but not within PUs.

It was commented that we already have sharing for SPSs and PPSs, so it should not be too difficult to express in semantics.

It was commented that JVET-R0194 discusses PS sharing issues.

Given the discussion and the limited number of APSs allowed, no action was taken on this.

• 1. Add the following constraints: When both one or more prefix APS NAL units and one or more suffix APS NAL units are present between two consecutive VCL NAL units in decoding order, the VCL NAL units shall belong to different subpictures and all the suffix APS NAL units shall precede, in decoding order, all the prefix APS NAL units. (JVET-R0149)

No action was needed on this due to the lack of action on items “a” and “b”.

• 1. To constrain suffix APS NAL units to be located after the last VCL NAL unit of the PU. (JVET-R0201)

It was discussed whether all constraints to enable random access functionality need to be in the VVC standard itself or some of them need to be specified somewhere else.

Discussion stopped here for JVET on 16 April at 0915 UTC. Subsequent discussions were from Thu 23 April 1525 in JVET (chaired by JRO)

• 1. To allow prefix and suffix APS NAL units with particular APS identifier and type to have different content. (JVET-R0201)
  2. To constrain prefix APS NAL unit to be located before the first VCL NAL unit of the PU. (JVET-R0201)

Was again discussed Thu 23 April after offline study. It was confirmed by independent experts that an issue exists, and that all three constraints are necessary to resolve the issue.

Decision (cleanup): Adopt JVET-R0201 (all three constraints)

1. Signalling APS information in PH/SH
   1. Add additional signalling of alf_data()/scaling_list_data()/lmcs_data() in SH or PH, and add a constraint such that all presence flags of ALF/Scaling list/LMCS in SH shall be 1 when no_aps_constraint_flag is equal to 1. (JVET-R0180)
   2. Add a mode of directly including the APS data structure inside a PH NAL. (JVET-R0273)

It is not clear what the benefit would be. The no_aps_constraint_flag introduced in the last meeting introduces the problem that an encoder uses it cannot use the related tools any more. It is therefore suggested to introduce another place of signalling the information carried in APS either in the picture header or in the slice. It is noted that eve when not using APS ALF could still be used at least with fixed filters.

The introduction of multiple mechanisms to signal the same information might overburden the specification for the benefit of very specific bit streams.

The opinion was also expressed that one intent seems to be coding efficiency. Several experts expressed concerns about introducing this. There were also concerns raised that aspects might be complicated. No support was expressed by non-proponents. No action was taken on this.

1. Constraint for APS types:
   1. Add a constraint for APS type based on the enabled tools in SPS. If a tool that uses an APS is disabled, then the APS with the corresponding APS type should not be present in a bitstream. (JVET-R0232 section 2)

There is no problem if an encoder sends an APS even if the related tool is disabled. On contrary, it might even inhibit cases where an APS which was sent in a stream before could re-used later. No action was taken on this.

1. Constraint for alf_data in ALF APS:
   1. Add a constraint to CC-ALF based on sps_ccalf_enabled_flag. When sps_ccalf_enabled_flag is equal to 0, an ALF_APS cannot contain any CCALF filters. (JVET-R0232 section 3.1)

Same comment as under 4) – if CCALF parameters are in an APS but never used there is no problem with the decoding. No action was taken on this.

1. Remove BitDepth constraint for lmcs_data in LMCS APS (JVET-R0232 section 7, it is noted that JVET-R0433 includes this – no need to discuss separately according to proponents):

lmcs_delta_cw_prec_minus1 plus 1 specifies the number of bits used for the representation of the syntax lmcs_delta_abs_cw[ i ]. The value of lmcs_delta_cw_prec_minus1 shall be in the range of 0 to 14, inclusive.

PH:

ph_lmcs_aps_id specifies the adaptation_parameter_set_id of the LMCS APS that the slices associated with the PH refers to. The TemporalId of the APS NAL unit having aps_params_type equal to LMCS_APS and adaptation_parameter_set_id equal to ph_lmcs_aps_id shall be less than or equal to the TemporalId of the picture associated with PH.

The value of lmcs_delta_cw_prec_minus1 of the APS NAL unit having aps_params_type equal to LMCS_APS and adaptation_parameter_set_id equal to ph_lmcs_aps_id shall be in the range of 0 to BitDepth – 2, inclusive.