Abstract not available in document
JVET-D0194 BoG report on test material [T. Suzuki]
The mandates of this BoG were as follows:
- Review input contributions, sort out sequences which are assessed to be inappropriate for codec testing (as per assessment of the testers)
- Prepare and perform visual assessment. One is HM vs JEM comparison to select test sequences and rate points for visual assessment. The other is to select 1080p test sequences
- Report the results of the visual assessment
Details of the BoG report are not included here, so it should be consulted by interested parties.
10 evaluations of test material were reviewed and summarized: 6 for 4K, and 4 for 1080p. Three other contributions of new test sequences were also reviewed, including:
- 4K Aerial video sequences from DJI (D0060)
- HDR and SDR corresponding graded sequences from Netflix (D0083)
- Virtual reality test sequences from GoPro (D0032)
Initial visual assessment work was done for testing the state of the JEM. The encoding condition was summarized in JVET-C1002.
Subjective viewing will be held at the next meeting, for assessing the subjective quality in terms of appropriateness for a formal subjective viewing and selecting appropriate rate points.
From the presentation on Wednesday afternoon:
Test with 4K sequences, the test with AB comparison does not allow to obtain MOS values, however this was not the goal.
A 5 sec duration was concluded to be too short for conducting reasonable visual comparison.
For 1080p, a survey about “suitability for testing” was conducted (see table in BoG report).
Expected output docs were identified as follows:
- A new test plan for 1080p and 4K with more rate points (as needed when issuing CfE by next meeting)
- A report about the test outcome (could also be in BoG report)
Results of the expert viewing with 4K sequences were presented during the plenary on Thursday afternoon. The results indicated that a visual benefit of JEM over HM can be obtained in many test cases, and additionally give hints about the suitability of certain sequences for visual testing.
Viewing equipment included a Samsung 4K 65KS9800 display (supporting 3840*2160 @ 60p) and a 4K player provided by Huawei. A computer provided by GBTech was used as the video server.
As suggested by JVET plenary meeting, an AB test was used. The test procedure was as follows. A and B is showed to viewer as follows.
Results of the preliminary 4K testing done during the meeting were as shown below.
In some cases, a difference between the results can be observed between HM and JEM for both high and low bit rate. Most of the cases, the difference was not so significant at high bit rate. At low bit rate, a difference can be observed in most of the cases.
The quality should be confirmed. For the cases where we did not see difference, the reason for this should be determined (e.g. if quality is high enough or if there was really no coding efficiency difference).
The characteristics of the test sequences were studied as well as coding efficiency analysis.
Further work must be done by the next meeting.
The BoG recommended
- To continue visual assessment to understand what sequences and what bit rates are appropriate for subjective test, targeting CfE/CfP.
- To make another visual assessment work plan (for 4K and 1080p), for which a 2 week editing period is necessary
- To allow use of a “floating point QP” to adjust bitrate (to be more close between HM and JEM)
- To upload new sequences for further study
The BoG appreciated the following companies supporting the HW and SW used in the evaluation of JEM vs. HM: Samsung for providing the display (TV set 75KS 9800), GBTech for providing the computer used as the video server, Huawei for providing the software player for 4K content
The BoG appreciated the experts for participating to the expert viewing assessment, and M. Sychev (Huawei) for designing the testing scripts for 4K video player, T. Solovyev (Huawei) for designing the 4K video player, A. Norkin (Netflix) for providing 4K test material, V. Baroncini (GBTech) for designing the test procedure and processing the results, G. Baroncini for acting as test administrator (GBTech).
The final report of the BoG was presented during the plenary on Friday. All recommendations of the BoG were approved. The big effort conducted within this BoG was highly appreciated (see also the MPEG meeting resolutions on expressing thanks to contributing individuals).
List of actions taken affecting the JEM4
The following is a summary, in the form of a brief list, of the actions taken at the meeting that affect the text of the JEM3 description for preparation of JEM4. Both technical and editorial issues are included. This list is provided only as a summary – details of specific actions are noted elsewhere in this report and the list provided here may not be complete and correct. The listing of a document number only indicates that the document is related, not that it was adopted in whole or in part.