JVET-L0110 On VVC HLS architecture and bitstream structure [S. Wenger (Tencent), Y.-K. Wang (Huawei), M. M. Hannuksela (Nokia), R. Sjöberg (Ericsson), S. Deshpande (Sharp)]

This contribution was reviewed in JVET plenary Wednesday 1830 (GJS & JRO)

This document proposes a VVC high-level syntax (HLS) architecture and design rationale. Additionally, a VCC bitstream structure is proposed. Some items are proposed for discussion.

Proposed VVC HLS architecture and design rationale:

1. (Proposal) That the NAL unit concept of AVC and HEVC should stay, as it has proven to be useful, and because at least some system specifications (to include certain file formats) rely on it.
2. (Proposal) The concept of CTU-based (independent, raster-scan-order, with terminating positions unknown after parsing the header data) slices is proposed to be removed, as a vestige of MTU size matching considerations, but tiles (rectangular regions of known size) are proposed to be supported.
   1. Tiles are generally expected to be independently parseable/decodable within the current picture.
3. (Proposal) Independent decoding of motion-constrained tile sets (MCTSs) sets is suggested to be useful for certain application scenarios. Encoding and signalling of MCTSs should be supported.
   1. Note: This could be just a matter of metadata, e.g., as in HEVC.
4. (Proposal) A picture header (which would carry data that applies to the entire picture, but without a picture header ID signalled in the picture header itself hence not referenceable by VCL NAL units) or header parameter set (HPS, which contains header parameters, contains an ID and hence referenceable by VCL NAL units), is proposed to be considered if it has a good impact on BD rate performance.
   1. This is a rate-distortion justified matter; see next item.
5. (Proposal) PPS and SPS are proposed to stay mainly as is, both in terms of syntax (individual NAL units) and functionality and persistence scope.
6. (Proposal) Decoder parameter set (DPS), required to stay constant for the lifetime of a video stream.
   1. This is a matter of maximum capability negotiation, subprofiling, decoder initialization.
7. Thoughts about profiling of tiles
8. Some of the co-authors think that tiling should perhaps be enabled based on profile used. Perhaps, a very basic tiling mechanism, to support straightforward parallelization, could be part of all profiles. More advanced techniques could be specified only for certain profiles. For example, a 360 profile using cube maps could allow motion constrained independent tiles tailored for that application (perhaps in addition to the basic tiles); e.g., 24 motion-constrained tiles as in 6 x 4 arrangement, or in a cross-style arrangement. Other profiles may be applicable to other projection formats.
9. Some of the co-authors think that these decisions are such that should happen later on. Generally, the fewer profiles the better for the success of VVC. Note that in HEVC, only "basic" tiles affected normative operation. Motion-constrained tiles (or tile sets) are constraints that an encoder could choose to use but which don't affect normative decoder behavior.
10. Using the same tool for different purposes is almost always problematic, as encoders need to weight between the needs of the purpose. Some of the co-authors think that this is also true for tiles. For example, if parallelization requires one tile layout, and 360-video related projection requires a different one, what should an encoder do? However, some of the co-authors think that whether different modes or different profiles are needed for tiles may also depend on how diverging is the difference of the desirable tile layouts for different purposes. At least for the ERP and CMP projections, which are most widely used today, aside from some special 360 video optimization scenarios, the flexibility allowed by the tile design in HEVC seems good for both purposes of parallel processing and viewport-dependent 360 video delivery optimization.
11. Thoughts about VPS
12. Some of the co-authors think that VVC first version should have Video Parameter Set (VPS) to tie together scalable layers; a VPS breaks at IDR across layers boundaries. It is preferred to have the VPS from the outset, and not to copy VPS data into the SPS.
13. Some of the co-authors think that maybe it'd OK to not have VPS in VVC version 1, unless multiple-layer is already enabled, which does not seem to be the case.

Based on the above discussions and proposed VVC HLS architecture and design rationale, the contribution proposes that the VVC bitstream structure should comprise of the following NAL units or data structures:

• (Proposal) Decoder parameter set (DPS), required to stay constant for the lifetime of a video stream
  • It was commented that something equivalent might be possible without a new syntax structure – e.g., with repeated elements in the SPS that are constrained to not change.
• (Proposal) Sequence parameter set (SPS) similar in functionality as in HEVC, scope is coded video sequence
• (Proposal) Picture Parameter Set (PPS) similar in functionality as in HEVC, scope is a coded picture. At the same semantic level and similar scope (covering full coded pictures, but can change from coded picture to coded picture)
• (Proposal) Picture Header (carries data that applies to entire picture and that can change from picture to picture, plus reference to PPS) or Header Parameter Set (HPS), if it has a good impact on BD rate performance
• (Proposal) Tile Group Header (TGH)
• (Proposal) VCL data of the VCL NAL unit (tile group) comprising a Tile Group Header (TGH) and Coding Tree Unit (CTU) data of an integer number of tiles
• (Proposal) EOS (end of sequence) and EOB (end of bitstream) with similar functionality to HEVC.
• (Proposal) Prefix and Suffix SEI messages
• (Thoughts) Video Parameter Set to tie together scalable layers; have the VPS from the outset, and don't copy VPS data into the SPS.

Decision on agreements in principle:

• Agreed: NAL units, SPS, tiles, not currently planning to have classical slices
• Do we need to be able to put multiple tiles in one VCL NAL unit? It is a coding efficiency matter whether we would just have one header per tile or also some header for a group of tiles.
• Do we need a maximum capability negotiation header level – something with a persistence scope beyond the CVS? Yes, either this or something like having some SPS syntax elements that are not allowed to change. This would not necessarily need to be carried within the bitstream. It could be repeated. It might or might not directly affect the decoding process (e.g., it could just establish constraints).
• Do we plan to have a PPS (referenceable by multiple pictures) or a (perhaps repeatable) picture header? Yes.

Other aspects were agreed to be for further study.

Interoperability and capability points definition and signalling (4)

Contributions in this area were presented Sunday 7 October 1400 (chaired by GJS).