JVET-AN0100 AHG9: Gaussian splatting information SEI message [Y. He, J. Jung, A. Akhtar, L. Kerofsky, G. van der Auwera, A. Akhtar, M. Karczewicz (Qualcomm), J. Xu, Y.-K. Wang, L. Zhang (Bytedance)]

This contribution proposes a Gaussian Splatting Information (GSI) SEI message to support video based Gaussian Splat Coding (GSC) currently being jointly explored by MPEG WG4 and WG7. The proposed GSI SEI message extends the previous work in m73127/JVET-AM0212 to achieve efficient and lightweight representation of 3DGS. Additional showcase and test results are provided for both static and dynamic scenes. This contribution is concurrently submitted to WG4 as m74340.

The proponent asserts that an SEI message approach would enable fast standardization and deployment.

The INRIA format is named after the first paper in the field, and is a term known in the larger research community, and not just within MPEG.

In the proposal, width and heights are the same for all attributes.

In the showcase, 61 attributes are used.

Bit depth is signalled for each attribute component.

The gsi_frame_packing_idc syntax element is unnecessary.

Attributes are rectangular regions packed into pictures in multiple layers.

The approach could be used for static or dynamic gaussian splats, using still picture or video, respectively.

For video coding, RPR hasn’t been considered, e.g. the coded picture size varies within a sequence.

The SEI message carries the information needed to reconstruct an INRIA format representation or any other format that have attributes in rectangular containers.

An asserted goal of the proposed approach is to be flexible to support future formats.

The Showcase uses HEVC HM 16.21+DCM8.8, but doesn’t use screen content coding tools. The approach hasn’t yet been tested with the screen content coding tools.

The showcase used separate 2 single layer bitstreams, although the syntax describes a multi-layer approach.

In the INRIA format, all attributes are floating point.

An encoder is able to select the bit precision for each attribute component, which may be scaled to be coded in a picture which is at a particular bit depth, which may differ from the attribute component bit precision.

In the showcase, the position had 16 bit precision, which was split into two 8 bit numbers – MSB and LSB and coded in a 10-bit 4:4:4 picture.

The showcase was tested using the CTC for Gaussian splats in document MDS25512, WG 07Explorations Draft CTC for Gaussian Splat Coding WG 07 MPEG 3D Graphics Coding and Haptics Coding.

At different rate points, the video codec QP was adapted, and the effective precision of each attribute component was adapted.

A question was raised if the coded picture could be downsampled to reduce pixel rate and bitrate. For some attributes that would not be appropriate.

Full reference metrics were shown that use as a reference a number of rendered views from the uncompressed INRIA format representation.

A comparison was made with G-PCC. There is some activity in WG4/WG7 for V-PCC, but it is a newer activity and no comparison was made with it.

A question was raised if V-PCC could be used with the same coded picture as in this proposal.

Experimental results were shown for both static and dynamic Gaussian splats.

In the showcase, each attribute was 754x754 resolution.

A comparison was made with GSCODEC-STUDIO_mpeg152, which is a video based approach that is being used as an anchor in the Gaussian splat CTC for 2 meetings.

Both the proposal and GSCODEC-STUDIO_mpeg152 has significant coding gain relative to G-PCC for dynamic content.

It was questioned if the proposed SEI message approach could have an pre-processor that configured to create the same video(s) as used in this GSCODEC-STUDIO_mpeg152 anchor, in which case only the signalling would differ. The answer to this question could help assess how flexible the proposed approach is, and its potential robustness to evolving research on best representations.

The proponents asserted that the number of bits of the SEI messages are negligible relative to the total bitrate for the showcase. It would be useful to have such a breakdown provided.

The software was not currently made available.