Back to Search Document details
26th Meeting: by teleconference, April 2022 2022-04-13 19:46
Evaluation of 17 objective quality metrics on JVET contents
Abstract
Objective metrics are usually considered to correlate insufficiently with Mean Opinion Scores (MOS) to provide a reliable overall quality score. This contribution updates the contribution m58816. It evaluates 13 full-reference and 4 no-reference objective metrics using subjective scores obtained on JVET content, and during JVET remote expert viewing sessions. Instead of computing classical correlations (Pearson, Spearman, etc) with average MOS, and trying to assess how much the metric reflects the overall quality level, the contribution investigates the ability of the metrics to answer the question: “Is the proposal better than the anchor?”. A Receiver Operating Characteristics approach with a binary classifier is used. The confidence intervals and the bias resulting from the usage of a forced-choice scale are taken into account. It is reported that a group of metrics can be considered as reliable for most of the decisions, with correct decision rates above 85%.
Citation