JVET-K0013 JVET AHG report: Tool reporting procedure (AHG13) [W.-J. Chen, J. Boyce, E. Alshina, J. Chen, E. François, Y. He, Y. W. Huang]

This document summarizes the activity of AHG13: “Tool reporting procedure” between the 10th Meeting in San Diego, US (10–20 Apr 2018) and the 11th meeting in Ljubljana, SI (10–18 July 2018). Tool on/off experimental results vs. VTM and BMS anchors are provided for the tools specified in JVET-J1005, which include the BMS tools and the HEVC tools not included in the VTM.

The initial version of JVET-J1005 “Methodology and reporting template for tool testing” was provided on April 25, with updates provided on May 8 and May 29. The document contained a reporting template.

All tests described in JVET-J1005 were conducted. The tested tools, testers, and cross-checkers are listed in the tables below.

List of tools included in BMS but not included in VTM

List of tools included in HEVC but not included in the VTM or BMS

The results of the tests were included in the AHG report with an attached spreadsheet that provides additional data. Scatter plots were also provided for the tested tools in the random access configuration, comparing PSNR-Y based BD-rate on the Y axis vs. each of encoder runtime ratio, decoder runtime ratio, and a weighted average of encoder and decoder runtime ratio, (encoder + a * decoder) / (a+1), with a configurable weight variable a. The exemplary weighting was set to 6 and can be adjusted in the spreadsheet attached to the report.

Experimental results and configuration files can be found at the link below:

https://hevc.hhi.fraunhofer.de/svn/svn_VVCTestConfig/branches/VTM-1.0/

There was no bit rate or PSNR differences between the testers and cross-checkers.

Encoder and decoder runtime ratios provided by both the testers and cross-checkers are included in the reporting template, to identify if there were significant runtime differences. The largest runtime differences were found for TRM (AMT+4x4 NSST), where the tester used GCC 6.3.0 and SIMD=SSE42and the crosschecker used GCC 4.8.3 and SIMD=AVX.

Experiment test results were provided.

Sign data hiding had the largest gain among the things we had disabled from HEVC, which was about 0.7%.

The AHG recommended the following:

• Consider the reported tool test results during tool adoption decision making
• Review related contributions
• Refine list of tested tools and test methodology for the next meeting cycle
  • Consider the reported tool test results as a benchmark for CE tests
  • Consider including reporting of compute system information for testers and cross-checkers
  • Consider additional performance or complexity metrics

Among the BMS tools, 65 direction intra prediction modes, LM chroma, DMVR, and ALF had significant gains. Various techniques seemed to fall along the same line of complexity versus performance.

JVET-K0410 discusses the reliability of runtimes - e.g., on different encoders.

A participant remarked that running the decoder more than once was suggested as a way to make the runtimes more stable.

JVET-K0312 discusses the percentage of the video for which a tool is used.

It was remarked that the VTM has 4x4 biprediction and 2x2 chroma (intra and bipred) prediction, which are difficult from an implementation perspective, and it was suggested to measure the impact of these (which would require a software change).

Discussions were needed to determine what should be done in an AHG versus a CE.

Project development (13)

Text and general standard development (2)