JVET-AN0205 AHG9: Gaussian Splat Information SEI message [J. Ricard, G. Teniou, S. Wenger, A. Hinds (Tencent)]

This contribution proposes a Gaussian Splat Information (GSI) Supplemental Enhancement Information (SEI) message that standardizes how 3D Gaussian Splatting (3DGS) attributes are stored in one or multiple coded videos, enabling interoperable decoding and reconstruction. The main advantage of this approach is high degrees of flexibility in how 3DGS components are stored in video planes to accommodate different types of 3DGS data.

The GSI SEI is per video and may persist across AUs. Each instance declares:

• Enable/persistence flags
• General 3DGS parameters
  • Block size: size of the block used to pack the 3DGS components into video (Ex. 640x640)
  • Colour and spherical harmonics colour space: None, BT601, BT709 or BT 2020
  • 3DGS bit depths for the geometry and attribute coordinates ( Ex. {18,12}, {32,32}, …)
  • Most significant bits (MSB) and Least significant bits (LSB) bit depth allocate for geometry coordinates (Ex. {8,8}, {10,8}, …)
• Flag indicates if the geometry coordinates have been signed log transformed.
• Video parameters
  • Format of the video used to store the 3DGS data (Ex, YUV444, YUV420, YUV400)
  • Packing: way to pack the 3DGS components into the video frames:
    • PLANAR: components mapped to planes/channels/blocks or
    • TEMPORAL: components mapped to frames/time
  • Bit depth of the video
  • Component map: Ordered list of 3DGS attributes present in the associated video/layer:
    • Most significant bits of geometry components: x,y,z
    • Less significant bits of geometry components: x_add, y_add, z_add
    • Scale: scale_0, scale_1, scale_2
    • Rotation: rot_0, rot_1, rot_2, rot_3
    • Opacity: opacity
    • Colour: f_dc_0, f_dc_1, f_dc_2
    • Spherical harmonics: f_rest_0, f_rest_1, …; f_rest_44
    • Empty plane: zero
  • Quantization descriptors: mode (linear, gaussian, …), and per-component min/max (or offset/scale) needed to de-quantize.

Data may be packed in a planar and/or temporal manner.

They have a separate SEI message per video, that describes the attributes within the video.

The approach aims at having multiple separate video bitstreams, which a systems layer would be expected to synchronize.

It was suggested that multi-layer codecs are not widely supported in implementations. It is a design goal for implementability on current phones.

It was noted that 3GPP SA4 has defined media capabilities for augmented reality where they have limited the number of decoder instances and MV-HEVC to support only 2 independently coded layers.

In the showcase, results are shown using between 1 and 4 videos. It was questioned what the sample rate range is for the showcase.

Effective bit depth precision is used only for position. It is separated into MSB and LSB.

Two quantization methods may be used – linear and non-linear using Gaussian.

At a high level, this contribution is similar to the approach in JVET-AN0100, with some differences in features and signalling approaches.