JVET-G0024 InterDigital’s Response to the 360° Video Category in Joint Call for Evidence on Video Compression with Capability beyond HEVC [P. Hanhart, X. Xiu, F. Duanmu, Y. He, Y. Ye (InterDigital)]
Discussed Thursday 15:30 (GJS & JRO).
This proposal is the response to the joint Call for Evidence (CfE) on video compression with capability beyond HEVC by InterDigital in the 360° video category.
Four technologies are highlighted:
- “Hybrid cubemap projection” (similar in concept to ACP, with a generalization of the warping parameters) without “padding”, with a 3x2 layout slightly different than 360Lib default in order to facilitate prediction order. The encoder can select an appropriate set of projection format parameters to code the input video content.
- Intra prediction using face continuity: reference sample derivation for intra prediction is modified to consider the spherical nature of the 360º video;
- Inter prediction with geometry padding (which would involve limiting the MV range to prevent excessive padding unless implemented on the fly in the decoder – currently allowing half the face width/height of off-picture region): reference sample derivation for intra prediction is modified to consider the spherical nature of the 360º video;
- In-loop filtering with face continuity: spherical neighbours are used to perform in-loop filtering.
Parameters of HCP (one for horizontal and one for vertical correction, controlling a second order polynomial) are determined per face. The parameters are adapted for each IRAP and subsequent GOP; the criterion is minimization of squared sample error between the original and reconstructed ERP (an iterative algorithm with convergence after 3–4 iterations).
It was suggested to possibly show some video examples later during the week to get some understanding about variation of parameters.
Average bit rate saving was reported as 31% compared to HM anchors, 11.5% compared to JEM (further detailed below). Subjective results reportedly indicate less visibility of face boundaries.
For padding and loop filtering, the decoder requires knowledge about the projection. The padding extends each face by half of its size.
The table below reports the BD-rate of the response compared to the HM 360° anchors.
Comparison of the response with the HM 360° anchors in terms of coding performance.
Sequence | E2E WS-PSNR Y | E2E WS-PSNR U | E2E WS-PSNR V |
SkateboardInLot | −39.64% | −63.18% | −70.08% |
Chairlift | −45.95% | −66.27% | −60.73% |
KiteFlite | −23.02% | −56.60% | −63.16% |
Harbor | −25.92% | −56.11% | −56.50% |
Trolley | −21.56% | −44.27% | −50.52% |
Average | −31.22% | −57.29% | −60.20% |
Comparison with the JEM-based 360° anchors
The table below reports the BD-rate of the response compared to the JEM 360° anchors. As reported in the table, the response can effectively improve the coding performance of the current JEM by providing an average BD-rate savings of 11.5% based on E2E WS-PSNR Y. The maximum bit rate savings of the response can reach 22% (sequence Chairlift).
Comparison of the response with the JEM 360° anchors in terms of coding performance.
Sequence | E2E WS-PSNR Y | E2E WS-PSNR U | E2E WS-PSNR V |
SkateboardInLot | −15.38% | −25.27% | −24.29% |
Chairlift | −22.06% | −17.77% | −17.60% |
KiteFlite | −6.35% | −9.09% | −14.69% |
Harbor | −7.41% | −12.71% | −12.91% |
Trolley | −6.41% | −6.88% | −11.25% |
Average | −11.52% | −14.35% | −16.15% |
ACP is a special case of the HCP scheme.
The parameters of the warping were calculated from the first picture of each IRAP segment using an optimization to minimize the conversion error.
In the discussion, it was commented that the interaction with tiles should be considered, as the alignment of tile boundaries with the faces may be desirable.