JVET-R0004 JVET AHG report: Test material and visual assessment (AHG4) [V. Baroncini, T. Suzuki, M. Wien, R. Chernyak, A. Norkin]

The draft verification test plan JVET-Q2009 was prepared and was available at the JVET web site.

Due to the coronavirus situation, activities on test sequence identification and viewing for the VVC verification tests have been much lower than intended. So far, RWTH had scanned through the JVET ftp site for material outside of the CTC set which could be useful, and also looked elsewhere a bit. A summary of this can be found in document JVET-R0461.

The test sequences used for CfP/CTC are available on ftp://jvet@ftp.ient.rwth-aachen.de in directory “/jvet-cfp” (accredited members of JVET may contact the JVET chairs for login information).

Due to copyright restrictions, the JVET database of test sequences is only available to accredited members of JVET (i.e. members of ISO/IEC MPEG and ITU-T VCEG).

One particularly related contribution was noted

• JVET-R0461 AHG4: Candidate test sequences for verification tests [M. Wien (RWTH)]

The AHG recommended:

• To continue to study the coding performance comparison and to update the verification test plan.
• To collect volunteers to conduct the verification test, including volunteers to encode.
• To collect more variety of test sequences suitable for the verification test.
• To continue to collect new test sequences available for JVET with licensing statement.

It was commented that some preliminary viewing experiments had begun in Rome, including some experiments with remote test operation.

It was commented that blurring is an increased phenomenon in VVC, and that subjective tuning (vs. PSNR emphasis) would be beneficial for use in subjective testing.

It was commented that the use of a VMAF measure for the optimization may be helpful. However, it was also commented that VMAF comparisons may be vulnerable to problems as well (e.g. too much weight given to the quantity of high frequencies). MS-SSIM was also suggested to be considered. How encoder control could optimize subjective quality and pseudo-subjective measures was discussed. The final judgment is to be a matter for human eyes, of course.

Assistance with computing resources for encoding experiments (coordinated by M. Wien and V. Baroncini) was requested. Track B Sunday morning discussion was suggested.