JVET-J0085 BoG report on 360° video [J. Boyce]
This report was discussed Sunday 15 April 1010–1050.
The BoG met on Saturday April 14 1430–1630 with the primary goal of preparing a survey of the proposed technologies includes in responses to the Call for Proposals in the 360° video category. A spreadsheet containing the prepared summary is attached.
Some questions were raised during the BoG meeting regarding the plans for the 360Lib software. Currently, the 360Lib software is integrated with the HM and JEM. When a Test Model is defined, should the 360Lib software or a variant of it be integrated with it as well? For experiments involving 360°-specific coding tools such as those included in some CfP responses, some sort of integration of projection mapping and the codec is desirable.
Several new projection formats were proposed by proponents for inclusion in 360Lib, but BoG did not discuss yet, since it depends on the general plans for 360Lib. It was suggested to define a CE on projection formats.
On the status of the 360Lib software: It was suggested to not be aggressive about removing features from 360Lib. The test model will eventually depend on what is proposed in contributions. We could hypothetically split the documentation at some point or have different status identified in the provided documentation.
The report contained a list of features proposed in some proposals, in addition to the survey in the attached spreadsheet.
In addition to the summary information included in the spreadsheet table, some additional points were captured when discussing some of the contributions, which are noted below.
This proposes MCP projection format plus coding tools.
MCP Format:
- For top and bottom faces, use EAC
- For other four faces, use EAC in horizontal dimension, and other projection in vertical dimension
- Proposing MCP be added to 360Lib
Coding tools:
- For inter, derive projected MV from spherical neighbour block
- For other cases, consider neighbours unavailable if across dis. boundary
Coding gains (not included in CfP response individually):
- 0.35% gain for MCP vs. EAC with HM
- 1.08% for inter
- 0.45% for intra
- 0.2% for loop filter (asserts bigger impact on subjective quality)
Coding tools average 2.2% gain for Low Complexity
RSP with content-adaptive spatial adjustment, which is signalled per sequence. This should really be signalled per RAP. 0.2% gain was shown for the RSP change.
This has CU-adaptive QP, based on spatial position.
This proposes the PAU projection format. It is similar to JVET-J0019 in that one dimension’s projection function is changed, but the projection function differs. 0.2% coding gain was reported for PAU vs EAC. The proponent requested for PAU to be added to 360Lib.
No coding tool changes were included.
This proposes the HAC adaptive projection mapping function. It is a generalization of EAC.
Two parameters per face are signalled, so 12 total. The parameters can change for each RAP.
Conversion of reference frames between projection mapping formats is used (for open GOP.)
The proponent requested for HAC to be added to 360Lib.
From tool-on test, coding gains:
- 0.32% for HAC over EAC
- 0.54% for adaptive frame packing
- 0.33% for face discontinuity (but more subjective impact)
- 1.62% for geometry padding
Other discussion in the BoG:
- Question: Should there be some type of a “Test Model” for 360Lib type functionality? Should some 360Lib-like software have some new status?
- It was suggested to have a CE on projection formats, to study proposed new formats and existing formats.
- The CE would bring experimental results using the CTC. Will need to define CTC for projection formats and for 360° coding tools using the new test Model.
- Which codec to use for projection formats? A new test model would require integration work
- Consider removing unused formats from 360Lib.
- It was agreed to discuss 360Lib status in JVET, and hold an additional BoG session afterwards.
Additional summary of proposal properties from JVET plenary:
7 proposals would take effect on the coding loop, proposing specific coding tools
12 submission, 9 different projection formats (3 parties made 2 submissions)
- Several proposals use EAC derivatives (none of them the original one from 360lib)
- Others use ERP/PERP, or RSP
Action for BoG: Excel sheet to be updated to make more clear which elements of proposals would affect the coding loop, and which elements only require out-of-loop processing (In the Excel sheet, everything from column W is somehow like that)
Elements that would affect the coding loop:
- Decoder needs knowledge about positions of face boundaries, if they are continuous or discontinuous, and if they are discontinuous, where to find the spherical neighbour.
- Such information is then used for disabling/modifying coding tools to avoid using “wrong” neighbours, or are used to fetch the correct spherical neighbours for cross-boundary operations (e.g. intra prediction, loop filter, CABAC context, etc.)
- Further, several proposals used “geometry padding”, which is typically implemented by modification of the reference picture (which becomes larger than the picture to be coded); could be done on-the-fly at the decoder
Note: Any operation in the coding loop probably requires more precise description than provided in the proposals.
It was suggested to consider defining a CE on projection formats.
Further BoG developments
A follow-up report was presented Friday 20 April 1135 (chaired by GJS).
The BoG had met again on Apr 19 from 6:30 pm – 8:00 pm. A number of recommendations were made to the Common Test Conditions for 360 Video
Topics in the further BoG discussions:
- Updates to CTC
- Cosmetic changes to align to new SDR CTC
- Test sequences
- CE for projections
- Which projection formats to include?
- Limit to those in CfP responses or already in 360Lib
- 8 listed in the draft CE description
- Objective metrics
- Plan for expert viewing at next meeting
- CfP dynamic viewport paths?
- Static viewports?
- Evil viewports?
- Any scoring, or just informal viewing
- What viewing equipment requirements?
- Which projection formats to include?
The BoG recommended the following:
- Eliminate the requirement to have coding tools required to test with ERP.
- For coding tool proposals, provide gains relative to the same projection format without the coding tool.
- If the coding tool is tested using a variation of cube map such as EAC, it is encouraged to also provide data for the basic CMP format.
- Remove the 4K source sequences.
- Don’t add the moving camera versions of KiteFliteWalking and HarborBiking this meeting, but sequences are available on ftp site, and may be used to provide additional information. Consider at next meeting if should add to the CTC.
- It was suggested to have a face size that is a multiple of the CTU size, unless that would exceed the level limit for HEVC.
- Restrict coded sizes to be +/− 3% of ERP size.
- Switch to the viewport sizes from the CfP.
- Metrics: only require codec based metrics for coding tool based proposals and be optional for projection format based proposals.
- Provide anchors for PERP, CMP.
CE planning aspects reported by the BoG:
- For RSP:
- Basic format that was present in 360Lib before CfP
- Rotation
- Rotation + Inactive area filling + blending
- For MCP
- No padding
- With padding
- Add the EAC w/ padding, even though proponent was not asking for inclusion in the CE, and try to get someone to perform the test.
- For PAU
- No padding
- With padding
- For CMP
- No padding
- With padding
- Proponents can select the amount of padding and type of blending for their format, but need to specify by the CE document deadline.
- Objective metrics – follow CTC.
- Use evil viewports for informal subjective viewing at next meting, especially for padding vs. no padding.
In the JVET further review, all the BoG recommendations were agreed except as per the following items that had follow-up discussion:
- It was agreed to remove 4K sequences from the CTC (there were only two).
- It was suggested to have a face size that is a multiple of the CTU size, unless that would exceed the level limit for HEVC. However, the purpose of making comparison against HEVC is to investigate the gain in compression, which can be done provided that the HM software gives support for it. This was to be further discussed in CE finalization.
High Dynamic Range and Wide Colour Gamut CfP results
This was discussed Wednesday 18 April 1430–1600 (chaired by GJS & JRO).
Two of the top few performing contributions had no HDR customization of the decoding process (just encoder QP control) and did not use "reshaping". So to some degree it can be concluded that having a strong basic coding scheme for ordinary SDR was a large element of performing well on HDR video in this test.
The bit-rate overhead of the JEM QP adaptation for PQ video, versus decoder inferred adaptation, was suggested to be about 1%.
The CfP discouraged the use of colour volume transformations that varied spatially and temporally. None of the proposals used such techniques.
The CfP also discouraged QP adaptation other than for light level (which is what was done in the JEM reference). One of the top few performing contributions did use such a technique (and described the technique).
It was commented that much of the scoring difference between the group of proposals was due to one or two test sequences, so there is a high sequence dependence.
Further study of objective metrics was encouraged, including how the subjective test results correspond to objective measurements.
A participant said that a preliminary analysis indicated that the L100 and DeltaE measures seemed better correlated with the perceptual data than WPSNR for the chroma components. Another participant indicated that the anchor encoding is optimized for WPSNR, so if some other metric is better, it would be desirable to find an encoding method optimized for that.
Effective testing of HDR quality continues to require subjective testing thus far.
It was noted that none of the proponents used a specific scheme for HLG content.
Suggested CEs were considered as follows (some aspects could be in a more general AHG study):
- "Reshaping" [E. François]
- Anchor using adaptive QP versus alternative using an adaptive reshaper with the same sort of spatially varying adaptation (accounting for any signalling overhead bit rate)
- Anchor versus in-loop and out-of-loop reshaping
- Luma-chroma bit-rate allocation (and metric effects)
Testing conditions should be established for experiments. The CfP and/or CTC test conditions may suffice, perhaps along with QP settings of the prior CTC.
For HDR purposes, testing against the "BMS" may not be necessary.
The test model should support the anchor PQ adaptive QP scheme.
A BoG (coordinated by A. Segall) was asked to further discuss the issues and recommend next steps. See the notes for the BoG report JVET-J0097.