JVET-AN0174 AHG15: Compression of gaming content using auxiliary data [J. Sauer, Z. Li (Huawei)]

A method was presented for the compression of gaming content that uses depth maps from the auxiliary data provided for class G1 of the gaming CTC (JVET-AJ2027) in order to generate additional frames that are used for prediction. The depth map is encoded and included into the bit-stream. For some sequences we achieve gains of up to 10% Y-PSNR BD-Rate vs VTM were reported, where both anchor and test utilized the reduced runtime 3 configuration.

Multi-layer coding where intermediate “virtual” layers are used for the generated frames used for the prediction.

Average gain is -0.7%, due to large loss in darktree sequence. In that sequence, the amount of data for depth map is extremely large, and loss is 17%.

The camera motion is known from the available metadata of rendering and depth maps, which is then used to generate a globally compensated picture for prediction. Camera parameters are also transmitted per picture which cannot be done by SEI message as the decoding process depend on them.

It was commented that in principle this has some similarity with 3D-HEVC, but the proposed approach does not conduct picture compression dependent on depth locally.

It was suggested to make depth and auxiliary information available along with those CfE sequences that have them. They could also be used for encoder optimization. The information about the formats of auxiliary data is available in the CTC document JVET-AJ2027.

The depth maps were originally floating point and quantized to 10 bit using znear/zfar. Processing for the new reference frame is done in FP.

Further study was recommended.