JVET-K0206 AHG10: Improved perceptually optimized QP adaptation and associated distortion measure [C. Helmrich, H. Schwarz, D. Marpe, T. Wiegand (HHI)]

The notes and recommendation recorded for this contribution are from its review in the BoG JVET-K0562.

In JVET-H0047, the authors proposed a CTU-wise subjectively optimized QP adaptation (QPA) along with a correspondingly weighted PSNR (WPSNR) distortion measure. This QPA approach was submitted and evaluated for JVET’s 2018 Call for Proposals and is a part of the VTM/BMS 1 software.

This document proposes seven changes to the QPA and, if applicable, the associated distortion metric:

• disable the CTU size and beating reduction for base QPs higher than 37 or GOP sizes lower than 5
• use a frame-wise instead of CTU-wise QPA for base QPs higher than 37 or input smaller than 480p
• use a more isotropic high-pass filter kernel with 9 instead of 5 taps in the visual activity computation
• adapt the lower activity limit a_min to the modified high-pass filter and make it bit-depth dependent
• use β= (1/2) instead of β= (2/3) as the activity weighting exponent for more conservative bit-rate distribution
• re-adjust the mean activity value a_pic for correct rate matching after changed filtering, a_min, and β
• replace fixed chroma QP offsets with the transmission of picture-adaptive slice chroma QP offsets.

The subjective merit of these modifications is demonstrated via formal comparative subjective evaluation. It is suggested to adopt the proposed improvements to the QPA design in the next version of the VTM/BMS.

It was noted that slice chroma QP offsets should be available in VTM with the adoption of separate trees.

Note: package also includes a packed YUV format output mode.

Recommendation: Adopt thse changes into the VTM (were it remains disabled by default). The software AHG is suggested to decide whether to enable the macro (command line parameter would still be disabled by default).

Metrics and evaluation criteria (0)

See section 8 for contributions relating to memory bandwidth analysis and energy usage analysis.

Plenary meetings, joint Meetings, BoG Reports, and Summary of Actions Taken

Plenary meeting Friday 13 July 1400

The status of the work of the meeting up to this point was reviewed and discussed.

General discussions:

• The role of the BMS was discussed as follows:
  • The BMS can be used for holding extra potential features we aren’t so sure about yet
  • The BMS n eeds to have significant gain over the VTM
  • The testing in the BMS context sometimes unveils whether gains are independent or have diminished effects when tested with other candidate features.
  • The BMS can be a common basis for CE tests of modified versions of features
  • Features that don’t yet have a clear or properly worked out design can be put in the BMS for study.
  • We need to keep the runnable within some achievable limit on experiment encoding time.
• Having a separate tree for (intra) chroma was under consideration; see JVET-K0230, CE 1.5.1.1, CE1.5.2.5? This was not in the BMS yet. It was discussed whether to put JVET-K0230 in the BMS.
• The need for having spec text for any actions taken was emphasized.

WD / VTM actions:

• Increasing the upper bound on QP by 12 (no effect on CTC results) (JVET-K0251)
• PDPC (from JVET-K0063). AI 1.0%, RA 0.5%
• Intra 67 modes with 6 MPM and truncated binarization of non-MPM modes; otherwise per 3.2.3 (JVET-K0368), pending confirmation of mode coding after some experiment result (LGE / Huawei / Qualcomm were to test). AI 1.3%, RA 0.6%. It was remarked that we should consider non-normative speed-up for this.
• CCLM 1.2%/9.0%/8.0% for Y/Cb/Cr in AI, 0.8%/10%/9.2% for Y/Cb/Cr RA JVET-K0190
• AMT, both intra and inter, each controlled by an SPS flag, AI 3.3%, RA 2.0%, LB 1.3%). It was suggested to disable inter AMT for CTC (penalty 0.5%, only a non-normative issue – see section 6.6).

The impact of these aspects was estimated as 6.5% in AI for luma, 3.5% for RA for luma, and significantly more for chroma.

It was also suggested to consider increasing the QPs of the CTC or to add more QPs or spacing them 7 apart. This was further discussed in the closing plenary. No action was taken on this.

BMS actions:

• CPR as per HEVC SCC (It was suggested to consider adding more SCC into CTC Class F and making that class mandatory, see also JVET-K0294 – Jill Boyce was asked to to coordinate consideration of this.)
  • CTC: 1.3% for AI, 0.5% for RA
  • Class F: 21% for AI, 16% for RA
  • SCC TGM 1080p: 54% for AI, 39% for RA

Roughly what that seemed likely to provide, relative to the prior VTM was estimated as 7.8% for AI, 4.0% for RA.

Other further needs identified for Track A

• BoG on partitioning
• BoG on boundary handling
• BoG on 360° video
• Further discussion of division by block size in CE3 for DC intra prediction
• Further discussion of dependent quantization (a.k.a. trellis quantization) in CE7
• Further discussion of whether to keep secondary transforms in the BMS and which ones in CE6

Track B

• Moved AMVR from BMS into VTM (~1.8% for RA)
• Removing some partitioning restrictions
• BoG on ALF
• Deblocking viewing

Roughly what that seemed likely to provide, together with the Track A actions, relative to the prior VTM was estimated as 7.8% for AI, 5.8% for RA

Planned activities:

• BoG on boundary handling
• BoG on intra prediction and intra mode coding (Geert)
• BoG on inter prediction and motion vector coding (Haitao)
• Viewing on deblocking Saturday
• 360° viewing Friday pm
• BoG on 360° video

Plenary meeting Sunday 15 July 1430

General:

• QPs for CTC (add another or space apart by 7)? – this remained open at the time of this discussion
• CPR in CTC – yes, adopted (as part of the BMS).
• Text is to be reviewed during the meeting
• BMS status

Track A:

• Separate tree for luma and chroma in intra slices (which has a coding efficiency benefit)
• Implicit split to 64x64 for intra slices
• Prohibit ternary split of edges longer than 64
  • Intra and inter
  • About 32x128, see later contribution JVET-K0556 – this was to be further studied.
• Dependent quantization (which has a coding efficiency benefit)
• Sign data hiding when dependent quantization not being used
• Modification of DC prediction mode for rectangular blocks to avoid division
• QT+BTT status
• Chroma QP increase when separate tree (non-normative CTC or software)
• Intra prediction with 67 modes (see MPM note below)
• Open at the time of this plenary discussion:
  • BoG on picture boundary handling
  • Wide angles
  • 3 MPM vs. 6 MPM est. ~0.2% difference for BMS RA, about 0.5% for VTM AI when encoding search is equalized (there is also a difference due to using context modelling in the 6 MPM scheme). Decision: Use 3 MPM for now.
• Track B:
  • VTM
    • GALF (4x4 classification-based) without filter pred. (temporal/from default filters), 7x7 luma, 5x5 chroma (fixed), CTU level switching
    • ATMVP with 8x8 MV storage & simplifications, slice switching 4x4/8x8 subblock
    • AMVR
    • Deblocking “bug fix” for large TUs & 8x8 grid
    • Affine: New prediction & difference coding, fixed 4x4 sub-block, switchable 4/6 parameter model, bug fix affine merge (affine was agreed to be moved to the VTM in the JVET plenary Sunday afternoon)
  • BMS
    • BIO with limited WC complexity
    • DMVR modifications for latency problem solving
    • Generalized Bipred
  • Software:
    • Some speedups, e.g. affine
    • long-term reference mechanism, in combination with pic_output_flag = 0 (non CTC)
    • Cross-CTU optimization of SAO (non CTC)

Planning of the remaining reviews (Tracks A/B) was performed.

Closing plenary sessions

Additional plenary sessions of JVET were held on Monday 16 July at 1815–2030 and Tuesday 17 July at 1610–1800, and a final plenary was held on Wednesday 18 July at 1115–1515. These plenary sessions included BoG review, other review and finalization of meeting outcomes, CE and AHG planning, output document planning, assignment of editorships, future meeting plans, expressions of thanks, and closing of the meeting. The outcomes of these plenary discussions are recorded elsewhere in this report.

Relevent liaison communications

Liaison letters had been exchanged between MPEG and SG16, as follows:

• ISO/IEC JTC1/SC29/WG11 - N17672 (SC29/N17360), received as SG 16 TD 222/Gen
• SG16-LS82, received as WG 11 M 43857

Each of these liaison letters had expressed satisfaction regarding the strong outcome of the Joint Call for Proposals on Video Compression with Capability beyond HEVC and the state of work in JVET, authorized the Joint Video Experts Team (JVET) to proceed with the work to develop the new standard, and confirmed the selection of the informal nickname "Versatile Video Coding (VVC)" for the new standard.

It was noted that information about a Call for Proposals issued by JPEG (ISO/IEC JTC 1/SC 29/WG 1) for work on a potential new image coding standard nicknamed “JPEG XL” had been conveyed to the parent bodies in incoming liaison statements (received as TD207/Gen by SG16 and M 43861 by WG 11). These were handled at the parent level rather than within JVET.

Joint meetings

A joint discussion was held between Q6/16 (VCEG) and MPEG on Monday 16 July at 1645–1800 (chaired by GJS, JRO, and Joern Ostermann of the MPEG requirements subgroup) on VVC project development.

There had previously been various discussions on how to manage the project work and structure its results. Since the VVC project had just been formally launched at the previous meeting, it was suggested that the current meeting was a good opportunity to establish and refine working practices for the effort.

A multi-company contribution had been submitted to JVET as JVET-K0263 which advocated for the document text structure for VVC to be somewhat different from what was previously done for AVC and HEVC, by splitting the design of the standard into multiple documents that are developed together. In the joint discussion, one participant expressed some hesitation about the proposal, fearing that, in practice, it might fragment and complicate the situation. However, it was generally thought that using this approach at least to some extent would simplify the technical work, clarify the scope of the parts, and allow more flexible application, and a group of standards could be treated as a package where appropriate. An example of this approach that was particularly discussed was separating the specification of the "supplemental enhancement information" from the specification of the core coding technology. Following the discussion, splitting the design into multiple specifications was agreed in principle, with the specific document structuring to be determined in JVET.

Another multi-company contribution had been submitted as JVET-K0311, which advocated for an approach to interoperability point signalling for VVC with finer granularity than in past efforts, based on subsets of profiles & levels defined by JVET in the VVC specification and using a non-normative user-registered "sub-profile" indicator. Having relatively fine granularity switches for coding features (e.g., 64 flags) was suggested to be supported, not just as a local decision by an encoder over which features to use, but as a capability indication which could be used in system-level signalling for capability exchange. It was commented that it would be important to avoid a ridiculous number of configuration switches that could result in an untestable standard and interoperability problems in practical use. It was agreed in principle that having such switches (which might not necessarily be just flags) was desirable. It was agreed that it is valuable to be able to turn off particular features or simplify their operation and still have a usable package. The specifics of what is switchable were to be determined in JVET as the work proceeds, and it is important that the degree of variation should not become burdensome and that the design remain effectively usable with some such switchable features disabled. It was also agreed to have a sub-profiling indicator that would be signalled together in the syntax with the traditional profile/tier/level parameters. The potential establishment of a "registration authority" or some "filtering process" for evaluating allocations of sub-profiling identifiers was left to Q6/16 and MPEG to determine in further study. One possible registration method that was discussed was to use Rec. ITU-T T.35. As proposed, the intent was for the "sub-profiles" to be established by recognized organizations and to be documented.

On potential approaches to project management that had been discussed recently in the parent bodies and among the management, the following points were discussed:

• The groups are not opposed to externally specified profiling in principle.
• It was discussed whether to anticipate having multiple phases of profile development, with potentially more variations of the standard developed in such stages than has been common in past practice. this idea somewhat goes along with the concept of fine granularity of configurations, and somewhat doesn’t need to be considered until later when additional profiles are proposed. The idea of "sub-profiles" and switchable features, as discussed above, can also somewhat address the potential desire for this approach.
• The degree of priority to give to the original proponent of an adopted feature in the work on subsequent refinement of the feature design in the context of VVC development was discussed. In past practice for video coding standardization, we have generally tried to consult with the original proponent of features and understand their views on such matters. It was agreed, however, that the original proponent should not be considered to have any particular recognized special priority status in the work, and it was suggested that such a practice could possibly be counter-productive in producing the best resulting specification.
• It was also agreed to try to avoid unnecessary small technical changes as the standardization work progresses.
• It was agreed in principle to try to avoid simply "inheriting" technical elements from prior standards without considering their merits (e.g., by starting with a copy of HEVC as the draft text for VVC). Some elements of HEVC had already been excluded from the first draft of VVC, and further analysis was encouraged to avoid copying in unnecessary aspects.
• It had been suggested to encourage voluntary expressions of goodwill and timeliness for licensing practices in the informal patent licensing declarations submitted in proposal submissions (which shall indicate the category of anticipated licensing terms according to the ITU-T/ITU-R/ISO/IEC patent statement and licensing declaration form). This suggestion had arisen in contributions to Q6/16 and MPEG at recent meetings. It was agreed that this may be desirable, although no specific language suggestion had been reviewed. Since the JVET meeting was scheduled to end before those of its parent bodies and it seemed unlikely that a further joint discussion could be arranged to discuss the matter within the closing days of the current meetings, it was further agreed that if a parent body of JVET were to provide some additional example(s) of licensing declaration wording (compatible with the JVET terms of reference and the common patent rights policy of ITU-T/ITU-R/ISO/IEC), the additional example wording would be included as an example suggested by that parent body in the template distributed for preparation of JVET contributions for the next meeting.

BoGs (10)