JVET-S0213 AHG8/AHG9: Refinement of proposed positioning information SEI message of output independent layers with example bitstreams [E. Thomas (TNO)]

This was discussed at 2300 on 26 June (chaired by GJS).

There is a large amount of similarity in spirit with JVET-S0107 and follows up on prior contributions JVET-R0307 and JVET-R0308 (and updates JVET-S0108).

It is asserted that the contribution aims at supporting use cases where a multi-layer bitstream with independent layers can be used. Multi-party video conferencing is considered to be a scenario where multiple participant views can be encoded as independent layers of a single multi-layer bitstream.

This contribution proposed the definition of a SEI message to signal the positioning information of each output layer for a given output layer set into a final output picture. Based on the discussion related to JVET-R0307 at JVET #18, the following changes are introduced in this contribution:

• All the layers in an OLS are connected to the same 4-connected graph, i.e. no orphan layer.
• The process creating the final output picture clarifies that cropped decoded pictures are used and not decoded pictures.
• An informative algorithm describes how to calculate the position of each cropped decoded picture in the final output picture

JVET- JVET-S0214 provides the updated software implementation in VTM based on JVET-R0308 corresponding to the presented syntax and operations in the present contribution.

Lastly, example bitstreams with different arrangements are provided in the present contribution.

Like JVET-S0213, this indicates a master layout. It differs by defining the layout using neighbour indicators to the “north, south, east and west”. Left boundary and top boundary alignment rules are provided to build a connection map.

It does not include scaling relationships. All pictures have the same sample aspect ratio.

Each rectangle is a full cropping window output picture. Some forms of empty rectangles are supported, as are partial spanning relationships. All pictures are from the same AU of the same OLS.

For some discussed hypothetical layouts, the proponent said there could be a “layer” defined that has no picture in the AU. Or the encoder could, e.g., code flat pictures to fill the regions.

This proposes only one SEI message.

The proponent said this may not be intended to support fully rich application compositing layout, but rather to just identify where identified areas are, so that a later compositor could apply more general mappings.

There was a discussion of what to do if the frame rate is different in different layers (e.g. filling in by repeating prior pictures in output order or by filling in with blank regions).

It was commented that this assumes a compositor functionality that follows the decoding process, and that such a compositor would likely operate based on (x,y) coordinates. A translation of the coordinate systems would be needed. JVET-S0107 was said to be more of an (x,y) system, with a scaling of the coordinates by some constant factor.

In terms of features, this does not have the scaling or overlapped rectangle position support proposed in JVET-S0107.

Rotation was mentioned as a potential additional functionality that could be considered.

The hypothetical combination with region-wise packing was mentioned as a possibility.

This was further discussed for determining the future course of action. This contribution was discussed in joint meeting (see notes there). TuC output was planned.

This was further discussed on 30 June at 1730 (chaired by GJS & JRO).

Two approaches have been under consideration. As a TuC, both variations will be included. A software “Group” in Gitlab will be created for the software (as has been done in prior CE work).