JVET-AM0323 AHG17: Report on CfE dryrun testing [M. Wien, C. Lehmann, P. de Lagrange]

First presentation of some results in joint meeting with AG 5 Monday 30 June 1705-1810.

From the results shown

• DCR and ACR provide very similar results
• Rate ranges are OK in most cases
• Good percentage of non-overlapping confidence intervals ECM/VTM
• JVET-AM0200 at lowest rate usually equivalent to rt3

Specific observation on sequence basis

• Ducks take off has strong “knee” behaviour
• Beatriz: Lowest rate point may be too high
• Dota: Lowest rate point may be too high
• Camellia 2 highest rates saturating, lowest rate too high
• Nightlandscape 2 highest rates saturating
• Neptunfountain: R4 of ECM5 worse than R3

It was agreed to use JVET-AM0200 as reduced-runtime VTM anchor

For JVET-AM0045, report cases where one or the other (anchor with QP+1) was better. It was later reported that there were roughly equal number of cases where one or the other was better, or they were equal. There was only one case out of 120 where JVET-AM0045 was significantly better (non-overlapping confidence intervals). Not enough justification to change from the QP+1 method, which is also used in ECM and allowed/suggested for CfE proposals.

For JVET-AM0078, no conclusion is possible, as it was only tested in few cases which were not the same as for JVET-AM0045, such that it impossible to judge if improvement is due to rate matching or DMVR fix. Also some further viewing was conducted, which did not reveal any issue with the current CfE test cases.

Keep all sequences, but maybe not all will be used in visual testing in CfE (depending on number of submissions).

Offline activity (P. Nikitin, E. François, E. Alshina) to suggest rate point modifications

For lowest rate modifications, informal viewing should be conducted with closest fixed QP, to avoid that new rate becomes too low.

It was agreed to use ACR-HR (ref. ITU-T P.910) instead of DCR for the CfE. For ACR-HR, also originals are included for all test sequences (which was not the case for all sequences during the dry-run). This will help to identify which MOS value is the “saturation point” (which however may mean that an encoded version could be judged better and makes results better interpretable at high rate points.