JVET-J0094 Suggested process to select the Benchmark Set [B. Bross, A. Wieckowski, H. Schwarz, D. Marpe, T. Wiegand (HHI)]

This document suggests a process to select the benchmark set (BMS). The proposed method starts with a base configuration that includes HM + QTBT + TT. On top of that, additional coding tools are tested individually and ranked according to their BD-rate vs. encoder/decoder runtime slope. The contributors suggest that this ranking can provide guidance and an order for selecting tools for a test model for given coding efficiency and runtime expectations. As an example, the method is applied to JEM tools. Two different combinations of tools with a certain minimum BD-rate gain and a high slope are selected to generate two operation points. For example, the BD-rates and runtimes of SDR UHD sequences are as follows:

• MTT with −15% BD-rate over the HM anchor at 113% encoder and 87% decoder runtime.
• OP1 with −26% BD-rate over the HM anchor at 370% encoder and 109% decoder runtime.

The proposed method provides a way to compile a benchmark set from a set of tools for given BD-rate and complexity conditions.

The model calculates a slope of coding gain vs. the complexity metric. Models complexity by weighted average of 5* decoder run time plus encoder runtime, with the factor configurable. Some participants questioned if 5x weighting for decoder makes sense.

The contributors used QPs (27, 32, 37, 42) to closer match the CfP bit rates. There was a suggestion to use lower QPs.

A table was provided of coding efficiency gains and runtimes for encoder and decoder for short versions (49 frames) of test sequences.

This contribution was discussed in the BoG that reported in JVET-J0096.