JVET-M0013 JVET AHG report: Tool reporting procedure (AHG13) [W.-J. Chien, J. Boyce, R. Chernyak, R. Hashimoto, Y.-W. Huang, S. Liu, D. Luo]
This document summarized the activity of AHG13: “Tool reporting procedure” between the 12th Meeting in Macao, CN (3–12 Oct. 2018) and the 13th meeting in Marrakech, MA (9–18 Jan. 2019). Tool on/off experimental results vs. VTM anchor are provided for the tools specified in JVET-L1005.
The initial version of JVET-L1005 “Methodology and reporting template for tool testing” was provided on Oct 27th. The document contained a reporting template.
All tests described in JVET-L1005 were conducted, except 67IPM and PDPC. VTM tool tests were conducted on VTM-3.0 software with VTM configuration by switching off specific tool either in configuration files or macros. Tool tests of 67IPM and PDPC were not conducted because there was no associated configuration setting nor associated macros in VTM-3.0 in order to disable the coding tools.
The tested tools, testers, and cross-checkers are listed in the tables below.
Tool Name | Abbrev. Name | Document reference(s) | AI | RA | LD | Tester | Crosscheck |
Chroma dual tree | CST | X | X | X | Tzu-Der Chuang (peter.chuang@mediatek.com) | Wei-Jung Chien (wchien@qti.qualcomm.com) | |
Dependent quantization | DQ | X | X | X | Wei-Jung Chien (wchien@qti.qualcomm.com) | Yuwen He (yuwen.he@interdigital.com) | |
Cross-component linear model | CCLM | JVET-K0190, JVET-L0085, JVET-L0136, JVET-L0191, JVET-L0338, JVET-L0340 | X | X | X | Roman Chernyak (chernyak.roman@huawei.com) | Shan Liu (shanl; leolzhao@ tencent.com) |
multiple transform set | MTS | JVET-K0171, JVET-K0173, JVET-K0096, JVET-L0059, JVET-L0111, JVET-L0118, JVET-L0285 | X | X | X | Shan Liu (shanl; xinzzhao@ tencent.com) | Wei-Jung Chien (wchien@qti.qualcomm.com) |
67 intra prediction mode +6 MPM intra mode coding + Wide angle intra prediction (test not available) | 67IPM | X | X | X | Shan Liu (shanl; xinzzhao@ tencent.com) | Roman Chernyak (chernyak.roman@huawei.com) | |
Position dependent prediction combination (test not available) | PDPC | X | X | X | Shan Liu (shanl; leolzhao@ tencent.com) | Wei-Jung Chien (wchien@qti.qualcomm.com) | |
Adaptive loop filter | ALF | JVET-K0371, JVET-L0082, JVET-L0083, JVET-L0147, JVET-L0392, JVET-L0664 | X | X | X | Wei-Jung Chien (wchien@qti.qualcomm.com) | Yuwen He (yuwen.he@interdigital.com) |
Affine motion model | AFF | JVET-L0045, JVET-L0047, JVET-L0142, JVET-L0265, JVET-L0260, JVET-L0271, JVET-L0632, JVET-L0694 | X | X | Roman Chernyak (chernyak.roman@huawei.com) | Shan Liu (shanl; guichunli@ tencent.com) | |
subblock-based temporal merging candidates | SbTMVP | JVET-K0346, JVET-L0055, JVET-L0104, JVET-L0195, JVET-L0257, JVET-L0369, JVET-L0468 | X | X | Shan Liu (shanl; guichunli@ tencent.com) | Wei-Jung Chien (wchien@qti.qualcomm.com) | |
Adaptive motion vector resolution | AMVR | X | X | Shan Liu (shanl; guichunli@ tencent.com) | Wei-Jung Chien (wchien@qti.qualcomm.com) | ||
History based motion vector prediction | HMVP | X | X | Kiho Choi (kiho14.choi@samsung.com) | Wei-Jung Chien (wchien@qti.qualcomm.com) | ||
Pairwise merge candidate | PMC | X | X | Tzu-Der Chuang (peter.chuang@mediatek.com) | Roman Chernyak (chernyak.roman@huawei.com) | ||
Triangular partition | TAP | X | X | X | Kiho Choi (kiho14.choi@samsung.com) | Shan Liu (shanl; leolzhao@ tencent.com) | |
Bi-directional optical flow | BDOF | X | X | Kiho Choi (kiho14.choi@samsung.com) | Tzu-Der Chuang (peter.chuang@mediatek.com) | ||
Combined intra/inter prediction | CIIP | X | X | Kiho Choi (kiho14.choi@samsung.com) | Tzu-Der Chuang (peter.chuang@mediatek.com) Roman Chernyak (chernyak.roman@huawei.com) | ||
Merge with MVD | MMVD | X | X | Kiho Choi (kiho14.choi@samsung.com) | Roman Chernyak (chernyak.roman@huawei.com) | ||
Bi-predictive weighted averaging | BPWA | X | X | Yuwen He (yuwen.he@interdigital.com) | Wei-Jung Chien (wchien@qti.qualcomm.com) | ||
Multi-reference line prediction | MRLP | X | X | X | Shan Liu (shanl; xinzzhao@ tencent.com) | Roman Chernyak (chernyak.roman@huawei.com) | |
Current picture referencing* | CPR | X | X | X | Shan Liu (shanl; xiaozhongxu@ tencent.com) | Yuwen He (yuwen.he@interdigital.com) |
* indicates a tool-on test against the VTM-3.0 anchor.
The results of the tests are summarized in the tables below. The spreadsheet attached to the AHG report provides additional data. Scatter plots are also provided for the tested tools in random access configuration, comparing PSNR-Y based bd-rate on the Y axis vs. each of Enc runtime ratio, Dec runtime ratio, and a weighted average of Enc and Dec runtime ratio, (Enc + a*Dec)/(a+1), with a configurable weight, a. The exemplary weighting is set to 6 and can be adjusted in the spreadsheet attached to this report.
Full experimental results and configuration files can be found at the link below:
https://hevc.hhi.fraunhofer.de/svn/svn_VVCTestConfig/branches/VTM-3.0/
There were no bit rate or PSNR differences between testers and cross-checkers.
Encoder and Decoder runtime ratios provided by both the testers and cross-checkers were included in the reporting template, to identify whether there were significant runtime differences.
Simulation results in all-intra configuration (AI) of VTM tool “off” test. (VTM anchor)
AI | |||||||
Abbreviation | BDR-Y | BDR-U | BDR-V | Tester EncTime | Tester DecTime | XChecker EncTime | XChecker DecTime |
CST | 2.14% | -3.28% | -2.62% | 129% | 102% | 131% | 99% |
DQ | 1.91% | 1.15% | 0.86% | 82% | 101% | 84% | 101% |
CCLM | 2.07% | 18.76% | 18.66% | 99% | 100% | 99% | 100% |
MTS | 2.81% | 2.35% | 2.38% | 47% | 85% | 47% | 84% |
ALF | 2.25% | 3.05% | 3.18% | 99% | 88% | 100% | 90% |
MRLP | 0.54% | 0.26% | 0.28% | 95% | 98% | 95% | 103% |
CPR | -0.27% | -0.40% | -0.33% | 137% | 100% | 133% | 100% |
SAO | 0.30% | 0.41% | 0.72% | 100% | 101% | 100% | 98% |
Simulation results in random access configuration (RA) of VTM tool “off” test. (VTM anchor)
RA | |||||||
Abbreviation | BDR-Y | BDR-U | BDR-V | Tester EncTime | Tester DecTime | XChecker EncTime | XChecker DecTime |
CST | 0.30% | 2.44% | 2.58% | 101% | 101% | 100% | 100% |
DQ | 1.66% | 0.51% | 0.06% | 95% | 102% | 96% | 102% |
CCLM | 0.95% | 16.61% | 16.67% | 99% | 100% | 100% | 103% |
MTS | 1.26% | 1.14% | 1.28% | 90% | 97% | 89% | 98% |
ALF | 3.68% | 3.54% | 3.15% | 100% | 87% | 101% | 87% |
AFF | 2.57% | 1.85% | 1.79% | 89% | 98% | 89% | 97% |
SBTMC | 0.53% | 0.42% | 0.48% | 100% | 99% | 100% | 99% |
AMVR | 0.88% | 1.39% | 1.41% | 91% | 101% | 91% | 101% |
HMVP | 0.42% | 0.49% | 0.49% | 102% | 100% | 101% | 100% |
PMC | 0.13% | 0.07% | 0.05% | 100% | 100% | 100% | 100% |
TAP | 0.37% | 0.66% | 0.67% | 90% | 101% | 90% | 103% |
BDOF | 1.19% | 0.39% | 0.26% | 93% | 96% | 91% | 93% |
CIIP | 0.47% | 0.23% | 0.28% | 96% | 100% | 95% | 100% |
MMVD | 0.86% | 0.63% | 0.65% | 89% | 101% | 88% | 100% |
BPWA | 0.44% | 0.65% | 0.66% | 97% | 101% | 97% | 100% |
MRLP | 0.26% | 0.13% | 0.14% | 99% | 97% | 99% | 100% |
CPR | 0.09% | -0.01% | 0.05% | 100% | 100% | 99% | 99% |
SAO | 0.71% | 1.21% | 1.17% | 104% | 102% | 102% | 99% |
Simulation results in low-delay B configuration (LDB) of VTM tool “off” test. (VTM anchor)
RA | |||||||
Abbreviation | BDR-Y | BDR-U | BDR-V | Tester EncTime | Tester DecTime | XChecker EncTime | XChecker DecTime |
CST | 0.15% | -0.54% | -0.03% | 101% | 100% | 100% | 101% |
DQ | 1.48% | 0.86% | -0.31% | 95% | 103% | 96% | 101% |
CCLM | 0.08% | 4.25% | 4.67% | 100% | 100% | 100% | 105% |
MTS | 0.34% | 0.54% | 0.67% | 96% | 102% | 96% | 100% |
ALF | 2.59% | 3.38% | 3.40% | 101% | 89% | 101% | 89% |
AFF | 2.10% | 1.34% | 1.51% | 82% | 96% | 81% | 95% |
SBTMC | 0.63% | 0.73% | 0.57% | 100% | 97% | 101% | 99% |
AMVR | 0.54% | 0.94% | 0.97% | 87% | 101% | 89% | 101% |
HMVP | 0.25% | 0.21% | 0.28% | 100% | 98% | 102% | 102% |
PMC | 0.01% | -0.08% | -0.11% | 100% | 103% | 100% | 100% |
TAP | 0.89% | 1.19% | 1.18% | 88% | 104% | 87% | 104% |
CIIP | 0.47% | 0.54% | 0.59% | 96% | 102% | 95% | 100% |
MMVD | 0.21% | 0.09% | -0.02% | 95% | 99% | 95% | 100% |
BPWA | 0.31% | 0.29% | 0.25% | 96% | 100% | 97% | 101% |
MRLP | 0.12% | 0.16% | 0.15% | 100% | 98% | 100% | 100% |
CPR | 0.14% | 0.26% | 0.01% | 107% | 99% | 106% | 100% |
SAO | 1.41% | 3.25% | 4.20% | 101% | 97% | 100% | 98% |
Pixel usage and memory bandwidth results of VTM tool “off” test. (VTM anchor)
AI | RA | LDB | |||||
Abbreviation | Pixel usage | Pixel usage | Ave mem BW | Max mem BW | Pixel usage | Ave mem BW | Max mem BW |
CCLM | 51% | 3% |
|
| 1% |
|
|
ALF | 99% | 62% |
|
| 59% |
|
|
AFF |
| 24% |
|
| 28% |
|
|
SBTMC |
| 16% | 95% | 94% | 15% | 97% | 98% |
AMVR |
| 4% | 102% | 102% | 2% | 101% | 100% |
TAP |
| 2% | 100% |
| 5% | 98% |
|
BDOF |
| 42% | 98% |
|
|
| |
CIIP |
| 2% | 99% |
| 2% | 100% |
|
MMVD |
| 10% | 99% |
| 8% | 98% |
|
BPWA |
| 10% | 100% | 100% | 7% | 100% | 99% |
MRLP | 7% | 1% |
|
| 0% |
|
|
PSNR-Y vs encoding runtime ratio of VTM with VTM tool “off” test (VTM anchor) are shown in the figure below.
PSNR-Y vs decoding runtime ratio of VTM with VTM tool “off” test (VTM anchor) is shown in the figure below.
PSNR-Y vs weighted runtime ratio (a = 6) of VTM with VTM tool “off” test (VTM anchor) is shown in the figure below.
A related contribution was noted to be the following:
- JVET-M0111 AHG13: On bi-prediction with weighted averaging and weighted prediction [Y. Ye, J. Chen, M. Yang (Alibaba), P. Bordes, E. François (Technicolor)]
For two topics, testing was not performed because this part of the design could not be disabled in the software:
- 67 intra prediction mode +6 MPM intra mode coding + Wide angle intra prediction
- Position dependent prediction combination
The AHG recommended to:
- Consider the reported tool test results during tool adoption decision making
- Review related contributions
- Refine list of tested tools and test methodology for the next meeting cycle
- Consider the reported tool test results as a benchmark for CE tests
- Consider including reporting of compute system information for testers and cross-checkers
- Consider additional performance or complexity metrics
It was remarked that decisions need to be made regarding which features can be disabled, and it is difficult to measure the benefit for a feature if it cannot be turned off. See also the AHG15 report on this topic.