JVET-AC0092 AHG15: Investigations on the common test conditions of Video Coding for Machines (VCM) [S. Wang, J. Chen, Y. Ye (Alibaba), S. Wang (CityU HK)]

This contribution illustrates the non-monotonic issue for machine task performance in the common test condition (CTC) of video coding for machines (VCM), which affects the ability for us to compare the performance of different proposals in an apple-to-apple manner. Specifically, the non-monotonic rate distortion behavior of various machine vision tasks are observed and illustrated in both some of the proposed technologies and the anchor. In this contribution, we investigate the CTC of VCM and propose three possible solutions to solve/alleviate this issue.

The proposal is to better focus on BD quality rather than BD rate, but several experts suggested that BD rate should also be considered.

Non-monotonic behaviour happens at low rates (high QP) where perhaps the encoding simplifies the video such that it is for the benefit of the machine vision task – avoiding high QP could potentially resolve this problem

The quality-over-rate graphs have the tendency of a “threshold-like” behaviour (i.e. breakdown below a certain rate point, constant performance above) – how useful are BD metrics in that case?

How useful is it to include rate points where the machine vision performance becomes unacceptable, e.g. mAP in the low range of 10-20?

Is a part of the problem, that in some cases the performance of the machine vision task is not good even for the uncoded video? It was reported that some of the algorithms were not really trained for the type of video that are used.

It was suggested to further discuss in a joint meeting with WG 4 how the evaluation methodology could be improved (see section 7.4).

General aspects of standards development and applications of standards (3)

Contributions in this area were discussed in session 25 at 1415–1440 UTC on Thursday 19 Jan. 2023 (chaired by JRO), unless noted otherwise.