JVET-J0080 Preliminary Results of Subjective Testing of Responses to the Joint CfP on Video Compression Technology with Capability beyond HEVC [V. Baroncini]

Preliminary results were shown and discussed Thursday 12 April 1745–1900 (chaired by GJS and JRO).

This document reports the work to design, organize and conduct the Formal Subjective Test for assessing the quality of Submissions to the Joint Call for Proposals on Video Compression Technology with Capability beyond HEVC, issued by MPEG and VCEG.

After receiving expressions of interest, the Test Coordinator prepared a plan to accommodate the large number of submissions expected; more in detail, JVET received a total of 46 submissions in the following category groups:

• 22 SDR submissions with 36 test cases each,
• 12 HDR submissions with 32 test cases each,
• 12 360° submissions with 20 test cases.

In addition, HM and JEM anchors had to be tested in all three categories, which made a total of 52 category-specific tests to be performed.

Additional partners were added to some proposals later, bringing the total number of institutions to 32. All proponent groups properly completed the submission process.

A total of 7 test labs, which were experienced in their respective fields, had agreed to contribute to the test effort. Tests were partially overlapping (conducted in at least two labs) such that verification about the validity and comparability of the tests would be possible. In particular, the test labs were as follows:

• For SDR @ HD resolution: BBC (England), Queen Mary University of London (QMUL - England), University of West Scotland (UWS - Scotland);
• For SDR @UHD resolution: Centrum Wiskunde & Informatica (CWI - Netherlands)
• For 360°: University of Padova, GBTech (Italy);
• For HDR - PQ: RAI, Sisvel (Italy);
• For HDR – HLG: DBW and EVATech (Italy).

GBTech, EVATech and CWI collaborated in general coordination and taking care of contacts and administrative matters with the proponents; designing the test sessions and creating the scripts, as well as visually checking all files on the disks that were received. They also created the dynamic viewports for the 360° video test cases (not known to proponents beforehand), which had been selected with help from Jill Boyce (Intel) as an independent expert.

A wide number of human viewing subjects were hired and participated to the test in the above test sites; the number of subjects was high (around twice the theoretical number) to allow a low level of fatigue and maintain the high response and efficiency of the participants.

Following this approach, more than 2000 people were involved as test participant to perform a total of 58 test sessions, for a total testing time exceeding 500 hours of work including test execution and laboratory set-up time.

The subjective results for the 360° category were further investigated Thu 19 April 0900 (chaired by JRO).

Almost all proposals were superior in terms of quality compared to the HM (much superior) and JEM anchors, which indicates that:

• Projection formats different from ERP likely have advantage in terms of subjective quality at same bit rate
• Tools for better compression also give advantage for 360° video (as they give for any other video)

However, from the results it is difficult to interpret how large the benefit of 360°-specific coding tools would be. Whereas some proposals in the group of “best performers” used 360°-specific tools, other proposals achieved equally good results without doing so. Further study on these aspects is necessary. It was decided that a CE (P. Hanhart, J.L. Lin) on projection formats will be established, and that the aspects on 360°-specific tools will be studied in an AHG. Subjective evaluation will be needed both for the CE as well as the AHG study. The common test conditions also need to be revised.

It was agreed that, to provide comparison points that already have “normative” specification (in an SEI message of HEVC), both PERP and cubemap should be used. If a proposal for a cube-based new projection format uses elements that could be implemented in a non-normative way (e.g. guardbands, by using a combination of cubemap and region-wise packing, or blending as part of the viewport projection), it should be compared against a cubemap with the same approach. Otherwise, it would not be possible to identify the advantage of the new projection format.

BoG (J. Boyce) was asked to coordinate further discussion of CE and CTC.

It was clarified that our 360lib software package is an experimental software platform without a “status” in terms of standardization. It would, however, be desirable to extract the elements that are used in the HEVC/AVC SEI messages, making them part of the related reference software (i.e. the HM versions submitted to ITU-T and ISO/IEC).

There was no need to update the 360lib description at this meeting, as nothing in it was agreed to need to be modified.

Correlation analysis between the viewport PSNR and MOS scores was performed. Overall, the correlation seems to be low; however if one regression line per sequence is designed, correlation coefficients around 0.95 are achieved. It is, however, not possible to draw sufficiently certain conclusions from mapping PSNR to a MOS estimate.