JVET-AC0267 Training Methods in Visual Assessment: Potential Improvements for Expert Viewing Tests [M. Wien (RWTH), V. Baroncini (VABTech)] [late]

This document reports results of a subjective assessment which was carried out in addition to the tests reported in JVET-AB2029 at the Mainz meeting site with a small group of experts and a modified training procedure. The goal of this additional subjective test is to investigate the effectiveness of the modified training procedure and the feasibility of a smaller group of viewers for the expert viewing scenario.

The scatter plots seem to indicate a consistent behavior of the MOS scores resulting from the two tests. This applies to the resulting MOS values as well as to the corresponding confidence intervals. These results suggest that a comparable confidence may have been reached with half the number of expert viewers. Considering the effort of conducting expert viewing tests, this result is taken as an encouraging indication to further study and potentially refine the modified training method.

The MOS-rate plots resulting from the reported tests reveal a benefit of the VVenC bitstreams compared to the VTM bitstreams for about half of the test sequence. For the other half, the results do not indicate a clear trend towards the one or the other encoder. It is noted that the plots for the Campfire, CatRobot1, and DaylightRoad2 sequences show saturation effects for the highest rate point each. These effects were not observed in the results reported in JVET-AB2029.

This was a contribution for information.

The training consisted in demonstrating examples and explaining to the experts which range of MOS would be expected to be allocated with those examples, in order to better “calibrate” the votes of the individuals.

This was useful guidance to consider for future expert viewing tests.

Conformance test development (0)

This section is kept as a template for future use.

Software development (3)

Contributions in this area were discussed in session 26 at 1530–1630 UTC on Thursday 19 Jan. 2023 (chaired by Y. Ye).