JVET-W0133 Constrained RASL encoding for bitstream switching [R. Skupin, C. Bartnik, A. Wieckowski, K. Sühring, Y. Sanchez, B. Bross (HHI)] [late]

Document JVET-V0060 presented a constrained RASL encoding method that allows for resolution or quality switching in HTTP streaming with open GOP coding structures while avoiding the severe drift artefacts occurring when using unconstrained open GOP encoding for such switching.

The presented constraints entail disabling BDOF, PROF, DVMR and CCLM for RASL pictures as well as restricting the choice of the collocated reference pictures to pictures not preceding the associated CRA picture. It is reported that the constrained RASL encoding method retains most of the unconstrained open GOP coding gains with BD-rate gains of up to -8.57% (64 pictures RAP period) relative to closed GOP coding.

This document presents updated results based on the publicly available VVenC version 0.3.1 and proposes to add a GCI flag to VVC to clearly specify and indicate the necessary constraints and serve as a conformance point for related systems to allow for interoperability.

In response to comments received for JVET-V0060, a note was added to the proposal explaining the constraints intent.

Colour artefacts caused by CCLM were observed, which are resolved when the method is implemented in stream switching.

It was pointed out that an encoder can implement this method already in version 1. Why are additional constraints and a flag needed, and what would it help if only defined in a later version?

It was argued that the receiving system would act different in bitstream switching when knowing that the encoder has properly produced the bitstream.

It was pointed out by one expert that the proposal of the constraint flag is aligned with a specific implementation, but there may be other ways to resolve the problem. It is further pointed out that some restrictions may be too strong, e.g. if the RASL picture is an IDR picture it would not nbe necessary to disable CCLM.

An encoder might also have other mechanisms to limit the drift, without totally disabling certain tools.

Several experts expressed the opinion that other less restrictive ways might exist to resolve the problem. Further study of that is recommended.

The proponents announced to provide an update incorporating some of the suggestions that were made in the discussions. This was presented in session 23 at 0900 UTC on Thursday 15 July.

It was pointed out that PRO may not need to be restricted, as this just refines the affine prediction and does not rely on missing reference pictures.

Two options are included in the new version:

• Option 2 uses a gci flag, and adds the suggested restrictions as an informative note. It is commented that this stays somewhat vague and might not help.
• Option 1 also uses gci flag, but would somewhat specify more “normative encoder behaviour”.

Could not an encoder intended for a domain like stream switching prevent such behaviour without specific signalling? Answer by proponents: In an ideal word, yes, but in stream switching the source may be coming from different encoders, including some which do not take prevention mechanisms.

It is argued that the packager might have a chance not doing stream switching at RASL positions where it is not signalled that it is “safe”. It would be difficult to find out otherwise from the bitstream itself that this is the case. Actually, there are packagers which try avoiding switching in open GOP.

Could this alternatively be handled in the systems level, e.g. DASH?

It is commented that this would be a somewhat uncommon usage of GCI.

Looking at the intent that it is more meant for usage at systems level rather than decoder, signalling by an SEI message may be a better solution.

The SEI solution (based on option 1) appeared to be agreeable to other experts. This could even be an SEI message without syntax, just indicating by its presence that the encoder has observed certain constraints beneficial for open GOP stream switching.

The proponents are asked to propose a version based on an SEI signalling solution. It is to be checked if it is really necessary to restrict PROF.

The new version was presented in the plenary session 0500 UTC on Friday 16 July.

BDOF and PROF are not disabled (for further study if that might be necessary).

Decision: Adopt JVET-W0133v3 option 3 (for VVC V2). Further editorial refinements of the note appeared necessary and were delegated to the editor.

Plenary meetings, joint meetings, BoG reports, and liaison communications

JVET plenaries

Some of the discussions and actions at plenary sessions are noted in this section (especially those of Monday 12 July 0750–0920 and Wednesday 14 July 0820-0935).

Monday 12 July 0750–0920:

• BoG reporting
• Scheduling / further BoG sessions?
• Conformance V2 planning (BoG D. Rusanovskyy)
• Planning of output documents: Standard parts & DoCRs, CE & EE descriptions, verification test plan/report
  • Prepare integrated version texts of VVC V2 and VSEI V2, issue resolutions to ballot DIS of 2^nd edition rather than DAM texts.
  • Add Gary Sullivan as editor, nominate editors for new edition (an action item for a meeting recommendation)
• Whether to hold a hybrid meeting in October 2021: An informal poll gave 5% who would consider travelling to participate physically, 90% who would participate remotely, and 5% who would not want to participate.

Wed. 14 Jul. 0820-0935:

• Report from NN BoG:
  • Further work on summary analysis (Excel sheet attached to report)
  • Updates on future contributions’ reporting requirements
  • EE1 planning (JVET-W2023): proposals identified for next round
  • More emphasis on cross-checking, training should be cross-checked as well
  • No need to meet again; offline activities on preparing EE and CTC documents
  • Recommendations of BoG (as per v3 of JVET-W0182, further editing cleanups in v4) were approved
  • Plan liaison statement to JPEG on NN activities (E. Alshina and A. Segall were asked to prepare text for that)
• Report from BoG on VVC V2 conformance (JVET-W0188) discussed at 0850
  • Identified 65 desired bitstreams
  • Plan to issue draft 1 of VVC V2 conformance
  • Clarification and agreements: 9 new profiles, new tools not enabled for Main 12, Main 12 Still Picture, Main 12 Intra
  • Add chairs to AHG5
  • Make a request for ISO amendment? (this did not seem necessary/appropriate yet)
• EE2 (JVET-W2024):
  • It was agreed to also investigate the previous 1.5a in the next round, together with 1.5b
  • EE document to be prepared offline (by previous coordinators)
• CE on high bit depth to be discontinued
• New CE on film grain: Investigate the method from JVET-W0095 versus the RDD-5, in terms of complexity and quality; also investigate the impact of the block sizes, and whether larger block sizes than 8x8 would be necessary (JVET-W2022); it is commented that it would be desirable having a precise description of the algorithms, as useful for a TR later.

Information sharing meetings

In addition to the joint meetings listed below, information sharing sessions with other WGs of the MPEG community were held on Monday 12 July 0500–0730, Wednesday 14 July 0500–0630, and Friday 16 July 2100–2300. The status of the work in the MPEG WGs was reviewed at these information sharing sessions.

Joint meeting with Q6/16 (VCEG) and SC 29/WG 2 MPEG Requirements 0730–0845 Tuesday 13 July

The following topics were discussed in this joint session. See also the notes recorded on these topics in other sections of this document.

• VVC V2 Profiles (Tuesday 0730-0845 UTC with JVET and MPEG Requirements on VVC v2 profiles)
  • JVET-W0136 Suggested initial profile text for VVC operation range extension [T. Ikai (Sharp)] [late]
    • Monochrome profiles proposed
    • All-intra video profiles proposed (with loop filter support required)
    • 10, 12, and 16 b profiles proposed
    • Monochrome, 4:2:2, 4:4:4
    • Inter-predictive profiles proposed (unlike in HEVC)
    • Main 12 Still Picture, Main 12 4:4:4 Still Picture proposed (not in HEVC)
    • ~16 profiles were proposed in the contribution
    • Discussion:
      • Tiers and levels roughly per HEVC (OK for now – JVET-W0183?)
      • Do we need 4:2:2 profiles? (maybe not)
      • Do we need monochrome profiles? (maybe not)
      • Loop filter support requirement? (OK)
      • Palette mode and new coding tools proposed to be supported except in 10/12 b 4:2:0 & 10/12 b Mono profiles (OK)
        • (Some tools are already conditioned on b > 10)
• Film grain (optional mandatory, additional profile(s), or as an alternative VVC-based example)
  • JVET-W0095, proposed 8x8 blocks to be used for HD, 16x16 for 4K, and 32x32 for 8K
  • The possible publication of a technical report on the subject was discussed
  • Reference software can be provided
  • The SMPTE relationship was discussed, as SMPTE had previously published the RDD 5 scheme.
  • Discussion in VSEI vs VVC was discussed, as the proposed scheme uses VVC-based transforms.
  • The scheme was described as not necessarily for pre-filtering removal with post-process restoration of grain, but rather as a potential method for subjective improvement of low bit rate coded video
  • The current proposal is for inclusion in VSEI as an alternative example

Joint meeting with MPEG WG3 0630–0700 Wednesday 14 July on Systems metadata

Topics of discussion:

• Proposal m57457 is to use a single code for any system metadata rather than multiple SEI messages as proposed in JVET-W0169. The MPEG Systems WG did not have a strong opinion on that.
• Should the specification of such an SEI message go in VSEI or in a video coding standard directly? Tentatively directly in the video coding standard.
• What should it reference? Should it reference multiple systems standards (initially two)? The tentative answer is yes.
• Should it be extensible? Yes.
• Timeline? VDI work might go to CD at this meeting, with green metadata following shortly after.
• It was agreed we are not planning to put such SEI message(s) into the VVC v2 DAM text; it should follow a bit later.

Conclusions:

• Codepoints for Green Metadata and VDI: Better to define separate ones in VVC, and refer the corresponding systems specs directly
• SC 29/WG 3 would have the primary responsibility on what is going inside such an SEI message
• JVET should start a VVC amendment in October or January to align with the progression in WG3 (expecting this could be final at DAM stage)

Joint meeting of SC 29/AG 5 with WGs 4 (MPEG Video), 5 (JVET) and 7 (MPEG 3D Graphics Coding) 0720–0820 Wednesday 14 July

There was a presentation of JVET-W0185 (MPEG m57393) on remote testing coordination of proposals by J. Jung. See the summary of that document in section 4.4.

• Further work was expected to study and refine various aspects.
• It was commented that further information on the setup of testing sites and the level of training for test subjects should be further detailed.
• It was commented that information on suitable players for being used in the testing purposes would be very valuable. Initiatives from experts are solicited to contribute to the development of a suitable open source player to enable stable playout capabilities for these. It was commented that HDRtools provides means to pack raw video files and associated metadata into mp4 files which enlarges the number of players to handle this data.
• It was suggested to clarify the status of verifications tests in the standardization process in the guidelines document.
• The relationship with ITU subjective assessment Recommendations was discussed; nearly all aspects are well aligned – perhaps the forced-choice A-B comparison is less well established (5-grade comparison vs. 4-grade comparison)
• The scheme is currently not intended for formal subjective assessment, but rather for informal testing.
• It was asked how well verified it is that a remote viewing test can match the results of a lab test; this is a major question for study, e.g., in VQEG. However, since testing in labs is currently not feasible, such studies are not immediately planned.
• Also, expert viewing tests would not necessarily always match naïve subject viewing in any case.

There was also a presentation by M. Wien of draft verification test guidelines that had been submitted to MPEG as m57463.

• This document defines guidelines for MPEG verification tests to serve as a reference for ongoing and future verification test activities. It describes verification testing of standards which define visual media output. In order to assert the suitability of the standard for visual assessment by users, formal subjective evaluation of the visual media signal reconstructed or synthesized from the compressed bitstream is needed.
• MPEG verification tests has been since the beginning a fundamental step in the path to the release of a new standard. Nevertheless, the new organization of SC29 in Working Groups and Advisory Groups induces the need for a procedure defining the necessary communication and the exchange of test results between the relevant WGs and AG5. The document is supposed to reflect the common understanding of previous verification test activities in MPEG, and the transition of the best practice into the new organizational structure of MPEG.
• The description currently relies on the assumption of video sequences to be compressed and reconstructed. For visual media which is rendered (e.g. in case of 3D-video, or 3DoF, 3DoF+, and 6DoF), additional steps may be required and should be added in a revision of this document.
• There was discussion of types of encoder optimization, depending on what is intended to be tested and demonstrated.
  • Having similar types of optimization vs independent optimization of each encoder separately
  • Similarity of configurations - e.g., intra period and GOP structures – in some cases a different GOP structure is intended to be part of what is being tested, and in other cases it is not
• It was suggested that encoder implementations used in the verification test should include optimized rate allocation algorithm, potentially also rate control. It was commented that the level of optimization of encoder implementations of a new specification typically is lower than for a longer available, well-studied anchor scheme. At the same time, the test model implementations typically follow comparable encoder decision strategies, e.g. including fixed QP settings and rate-distortion based tool decisions. Further, these test model implementations and applicable configurations are typically well understood by the group such that a fair comparison is achievable. Nonetheless, additional implementations may be considered, such as the example of VVenC in the context of the VVC verification tests.
• It was further recommended to add a note to the reporting section on highlighting specific features or differences between the new scheme and the anchor in order to make readers of the public document aware of such aspects.
• A list was included of what prior verification tests have been conducted in recent years.

BoGs (3)

The following break-out groups were established at this meeting to conduct discussion and develop recommendations on specific topics.