JVET-Q0013 JVET AHG report: Tool reporting procedure and testing (AHG13) [W.-J. Chien, J. Boyce, W. Chen, Y.-W. Chen, R. Chernyak, K. Choi, R. Hashimoto, Y.-W. Huang, H. Jang, R.-L. Liao, S. Liu]
This AHG report was discussed Wednesday 8 January 2020 at 1505 (chaired by GJS & JRO).
This document summarizes the activity of AHG13: “Tool reporting procedure” between the 16th meeting in Geneva, CH (1–11 Oct 2019) and the 17th Meeting in Brussels, BE (7–17 Jan. 2020). Tool on/off experimental results vs. VTM anchor are provided for the tools specified in JVET-P2005.
The initial version of JVET-P2005 “Methodology and reporting template for tool testing” was provided on November 28th.
All tests described in JVET-P2005 were conducted. VTM tool tests were conducted on VTM-7.0 (or VTM-7.1 for adaptive colour transform) software with VTM configuration by switching off or on specific tool either in configuration files or macros.
The tested tools, testers, and cross-checkers are listed in the tables below.
Tools included in VTM (Tool off test vs VTM Anchor)
Acronym | Document reference(s) | AI | RA | LD | Tester | Crosscheck | |
Chroma separate tree | CST | X | X | X | Tzu-Der Chuang (peter.chuang@mediatek.com) | Wei-Jung Chien (wchien@qti.qualcomm.com) | |
Dependent quantization* | DQ | X | X | X | Tzu-Der Chuang (peter.chuang@mediatek.com) | Wei Chen (wei.chen@interdigital.com) | |
Cross-component linear model | CCLM | X | X | X | Roman Chernyak (chernyak.roman@huawei.com) | Shan Liu (shanl; leolzhao@ tencent.com) | |
multiple transform set | MTS | X | X | X | Kiho Choi (kiho14.choi@samsung.com) | Shan Liu (shanl; xinzzhao@ tencent.com) | |
Adaptive loop filter | ALF | JVET-O0064 JVET-O0090 JVET-O0216 JVET-O0228 JVET-O0247 JVET-O0625 JVET-O0662 JVET-O0669 JVET-P0162 JVET-P0164 JVET-P0505 JVET-P0554 JVET-P0665 JVET-P1038 | X | X | X | Wei-Jung Chien (wchien@qti.qualcomm.com) | Wei Chen (wei.chen@interdigital.com) |
Affine motion model | AFF | X | X | Roman Chernyak (chernyak.roman@huawei.com) | Shan Liu (shanl; guichunli@ tencent.com) | ||
subblock-based temporal merging candidates | SbTMVP | X | X | Shan Liu (shanl; guichunli@ tencent.com) | Wei-Jung Chien (wchien@qti.qualcomm.com) | ||
Adaptive motion vector resolution | AMVR | X | X | Shan Liu (shanl; guichunli@ tencent.com) | Wei-Jung Chien (wchien@qti.qualcomm.com) | ||
Triangular partition mode | TPM | X | X | X | Kiho Choi (kiho14.choi@samsung.com) | Shan Liu (shanl; leolzhao@ tencent.com) | |
Bi-directional optical flow | BDOF | JVET-O0055 JVET-O0304 JVET-O0570 JVET-O0594 JVET-P0091 JVET-P0519 JVET-P1023 | X | Kiho Choi (kiho14.choi@samsung.com) | Tzu-Der Chuang (peter.chuang@mediatek.com) | ||
Combined intra/inter prediction | CIIP | X | X | Kiho Choi (kiho14.choi@samsung.com) | Tzu-Der Chuang (peter.chuang@mediatek.com) | ||
Merge with MVD | MMVD | JVET-N0127JVET-N0332 JVET-N0448 JVET-N0380 JVET-P1023 | X | X | Kiho Choi (kiho14.choi@samsung.com) | Hyeongmun Jang (hm.jang@lge.com) | |
Bi-predictive with CU weights | BCW | X | X | Wei Chen (wei.chen@interdigital.com) | Tzu-Der Chuang (peter.chuang@mediatek.com) | ||
Multi-reference line prediction | MRLP | X | X | X | Shan Liu (shanl; leolzhao@ tencent.com) | Hyeongmun Jang (hm.jang@lge.com) | |
Intra block copy mode | IBC | JVET-O0078 JVET-O0162 JVET-O0258 JVET-O0455 JVET-O1170 JVET-P0400 JVET-P0457 JVET-P1018 | X | X | X | Shan Liu (shanl; xiaozhongxu@ tencent.com) | Wei-Jung Chien (wchien@qti.qualcomm.com) |
Intra sub-partitioning | ISP | X | X | X | Roman Chernyak (chernyak.roman@huawei.com) | Hyeongmun Jang (hm.jang@lge.com) | |
Decoder motion vector refinement | DMVR | X | Wei Chen (wei.chen@interdigital.com) | Roman Chernyak | |||
SBT | X | X | Roman Chernyak (chernyak.roman@huawei.com) | Shan Liu (shanl; xinzzhao@ tencent.com) | |||
Luma mapping with chroma scaling | LMCS | X | X | X | Taoran Lu (tlu@dolby.com) | Hyeongmun Jang (hm.jang@lge.com) | |
Symmetric motion vector difference | SMVD | X | Yi-Wen Chen(yiwenchen@kwai.com) | Hyeongmun Jang (hm.jang@lge.com) | |||
Quantized residual DPCM | BDPCM | X | X | X | Ru-Ling Liao (ruling.lrl@alibaba-inc.com) | Yi-Wen Chen(yiwenchen@kwai.com) | |
Matrix based intra prediction | MIP | X | X | X | Ru-Ling Liao (ruling.lrl@alibaba-inc.com) | Yi-Wen Chen(yiwenchen@kwai.com) | |
Low frequency non-separable transform | LFNST | JVET-O0094 JVET-O0213 JVET-O0219 JVET-O0368 JVET-O0472 JVET-O0529 JVET-P1026 JVET-P0350 | X | X | X | Ru-Ling Liao (ruling.lrl@alibaba-inc.com) | Yi-Wen Chen(yiwenchen@kwai.com) |
Joint coding of chrominance residuals | JCCR | X | X | X | Ru-Ling Liao (ruling.lrl@alibaba-inc.com) | Yi-Wen Chen(yiwenchen@kwai.com) | |
Sample-adaptive offset | SAO | HEVC | X | X | X | Tzu-Der Chuang (peter.chuang@mediatek.com) | Wei Chen (wei.chen@interdigital.com) |
Prediction refinement using optical flow | PROF | JVET-O0070 JVET-P0409 JVET-P0057 JVET-P0154 JVET-P0491 JVET-P0653 | X | X | Wei Chen (wei.chen@interdigital.com) | Ru-Ling Liao (ruling.lrl@alibaba-inc.com) | |
Palette coding mode** | PLT | X | X | X | Yung-Hsuan Chao (yunghsua@qti.qualcomm.com) | Yi-Wen Chen(yiwenchen@kwai.com) | |
Adaptive colour transform*** | ACT | X | X | X | Xiaoyu Xiu (xiaoyuxiu@kwai.com) | Shan Liu (shanl; xinzzhao@ tencent.com) |
* Test was conducted by disabling DQ and enabling Sign Data Hiding.
** Test was conducted with test sequences and test condition defined in JVET-P2022.
*** Test was conducted Test sequences and test condition are defined in JVET-P0517.
Additional test results are provided in the tables below and spreadsheet attached to the AHG report. This includes tool test results across several VTM versions. The combined BD-Rate is computed based on (BD-Rate_Y*8+ BD-Rate_U+ BD-Rate_V)/10. Scatter plots are also provided for the tested tools in random access configuration, comparing PSNR-Y based bd-rate on the Y axis vs. each of Enc runtime ratio, Dec runtime ratio, and a weighted average of Enc and Dec runtime ratio, (Enc + a*Dec)/(a+1), with a configurable weight, a. The exemplary weighting is set to 6 and can be adjusted in the spreadsheet attached to this report.
Full experimental results and configuration files can be found at the link below:
https://hevc.hhi.fraunhofer.de/svn/svn_VVCTestConfig/branches/VTM-6.0/
There were no bitrate or PSNR differences between testers and cross-checkers.
Encoder and decoder runtime ratios provided by both the testers and cross-checkers are included in the reporting template, to identify if there were significant runtime differences.
Simulation results in all intra configuration (AI) of VTM tool tests. (VTM anchor)
AI | |||||||
Acronym | BDR-Y | BDR-U | BDR-V | Tester EncT | Tester DecT | XCheck EncT | XCheck DecT |
CST | 0.35% | 9.34% | 9.26% | 152% | 103% | 154% | 103% |
DQ | 1.99% | -0.66% | -0.73% | 99% | 101% | 96% | 104% |
CCLM | 1.61% | 14.73% | 15.86% | 100% | 100% | 100% | 98% |
MTS | 1.22% | 0.98% | 1.07% | 81% | 98% | 86% | 101% |
ALF | 2.37% | 2.91% | 3.62% | 98% | 91% | 96% | 92% |
MRLP | 0.32% | 0.10% | 0.14% | 98% | 101% | 98% | 101% |
IBC | 0.65% | 0.62% | 0.66% | 54% | 100% | 58% | 99% |
ISP | 0.52% | 0.29% | 0.26% | 85% | 98% | 85% | 98% |
LMCS | 1.09% | -1.11% | -0.71% | 99% | 99% | 98% | 98% |
BDPCM | 0.01% | 0.04% | -0.01% | 94% | 100% | 98% | 105% |
MIP | 0.61% | 0.16% | 0.17% | 89% | 101% | 86% | 96% |
LFNST | 1.20% | 0.72% | 1.03% | 111% | 101% | 107% | 98% |
JCCR | 0.59% | 0.28% | 0.41% | 98% | 100% | 96% | 98% |
SAO | 0.00% | 0.15% | 0.17% | 100% | 96% | 100% | 97% |
Simulation results in random access configuration (RA) of VTM tool tests. (VTM anchor)
RA | |||||||
Acronym | BDR-Y | BDR-U | BDR-V | Tester EncT | Tester DecT | XCheck EncT | XCheck DecT |
CST | 0.10% | 3.78% | 4.53% | 104% | 100% | 102% | 100% |
DQ | 1.76% | -0.25% | -0.52% | 104% | 98% | 99% | 102% |
CCLM | 1.02% | 11.85% | 13.71% | 99% | 100% | 99% | 100% |
MTS | 0.70% | 0.56% | 0.72% | 90% | 100% | 93% | 100% |
ALF | 4.56% | 4.93% | 4.94% | 98% | 89% | 96% | 90% |
AFF | 3.01% | 2.04% | 2.00% | 81% | 96% | 82% | 97% |
SbTMC | 0.46% | 0.32% | 0.36% | 101% | 101% | 101% | 100% |
AMVR | 1.42% | 2.17% | 2.27% | 84% | 101% | 85% | 102% |
TPM | 0.38% | 0.64% | 0.68% | 95% | 100% | 98% | 101% |
BDOF | 0.76% | 0.31% | 0.27% | 98% | 97% | 101% | 94% |
CIIP | 0.28% | 0.01% | 0.01% | 99% | 100% | 98% | 101% |
MMVD | 0.51% | 0.47% | 0.51% | 93% | 101% | 93% | 101% |
BCW | 0.40% | 0.42% | 0.45% | 94% | 100% | 98% | 99% |
MRLP | 0.16% | 0.07% | 0.10% | 100% | 100% | 100% | 100% |
IBC | -0.04% | 0.05% | 0.05% | 91% | 100% | 91% | 100% |
ISP | 0.32% | 0.24% | 0.31% | 95% | 100% | 96% | 100% |
DMVR | 0.83% | 1.08% | 1.10% | 100% | 97% | 100% | 97% |
SBT | 0.40% | -0.03% | -0.01% | 95% | 100% | 95% | 100% |
LMCS | 1.42% | 1.39% | 0.96% | 95% | 98% | 94% | 98% |
SMVD | 0.25% | 0.25% | 0.26% | 93% | 97% | 97% | 101% |
BDPCM | -0.01% | -0.04% | -0.05% | 99% | 100% | 103% | 103% |
MIP | 0.33% | 0.40% | 0.52% | 95% | 100% | 92% | 97% |
LFNST | 0.88% | 0.02% | 0.49% | 94% | 100% | 91% | 98% |
JCCR | 0.57% | 0.04% | -0.52% | 98% | 100% | 94% | 97% |
SAO | 0.08% | 0.20% | 0.33% | 100% | 98% | 100% | 98% |
PROF | 0.46% | 0.16% | 0.13% | 98% | 99% | 98% | 98% |
Simulation results in low delay B configuration (LDB) of VTM tool tests. (VTM anchor)
LDB | |||||||
Acronym | BDR-Y | BDR-U | BDR-V | Tester EncT | Tester DecT | XCheck EncT | XCheck DecT |
CST | 0.00% | 1.34% | 2.21% | 110% | 97% | 100% | 98% |
DQ | 1.56% | 0.29% | -0.05% | 109% | 99% | 100% | 102% |
CCLM | 0.01% | 3.39% | 3.69% | 100% | 100% | 100% | 99% |
MTS | 0.53% | 0.12% | 0.10% | 99% | 100% | 98% | 98% |
ALF | 4.26% | 5.11% | 4.67% | 96% | 90% | 94% | 90% |
AFF | 2.96% | 1.90% | 2.29% | 74% | 94% | 75% | 96% |
SbTMC | 0.78% | 0.86% | 0.78% | 101% | 97% | 101% | 97% |
AMVR | 0.60% | 0.83% | 0.67% | 86% | 100% | 87% | 101% |
TPM | 0.92% | 1.29% | 1.25% | 97% | 102% | 97% | 100% |
CIIP | 0.39% | 0.46% | 0.48% | 99% | 100% | 97% | 97% |
MMVD | 0.45% | 0.34% | 0.36% | 96% | 100% | 95% | 100% |
BCW | 0.28% | 0.17% | 0.10% | 100% | 103% | 97% | 99% |
MRLP | 0.05% | -0.36% | -0.04% | 100% | 100% | 100% | 100% |
IBC | -0.01% | -0.01% | -0.06% | 85% | 100% | 85% | 100% |
ISP | 0.07% | -0.05% | 0.13% | 99% | 100% | 99% | 99% |
SBT | 0.57% | -0.22% | -0.13% | 93% | 99% | 93% | 98% |
LMCS | 0.97% | -0.59% | -0.86% | 97% | 99% | 94% | 95% |
BDPCM | 0.03% | 0.28% | 0.01% | 99% | 100% | 102% | 103% |
MIP | 0.17% | 0.48% | 0.49% | 95% | 103% | 103% | 99% |
LFNST | 0.42% | 0.09% | -0.07% | 92% | 103% | 108% | 98% |
JCCR | 0.15% | 1.92% | 2.55% | 99% | 98% | 97% | 99% |
SAO | 0.09% | 0.36% | 0.93% | 101% | 99% | 100% | 96% |
PROF | 0.33% | -0.03% | 0.00% | 98% | 98% | 97% | 92% |
Simulation results for screen coding tools for ClassF and ClassTGM (VTM anchor)
AI | |||||||
Acronym | BDR-Y | BDR-U | BDR-V | Tester EncT | Tester DecT | XCheck EncT | XCheck DecT |
IBC Class F | 15.22% | 15.16% | 15.31% | 54% | 101% | 57% | 99% |
IBC Class TGM | 47.19% | 44.63% | 44.63% | 64% | 103% | 67% | 103% |
BDPCM ClassF | 0.92% | 0.81% | 0.95% | 98% | 100% | 96% | 93% |
BDPCM ClassTGM | 1.39% | 1.27% | 1.24% | 101% | 102% | ||
RA | |||||||
IBC Class F | 12.19% | 12.14% | 12.29% | 85% | 100% | 88% | 100% |
IBC Class TGM | 22.12% | 21.66% | 22.06% | 88% | 102% | 102% | 105% |
BDPCM ClassF | 0.68% | 0.66% | 0.70% | 99% | 100% | 94% | 95% |
BDPCM ClassTGM | 0.70% | 0.75% | 0.78% | 100% | 101% | ||
LD | |||||||
IBC Class F | 6.04% | 6.85% | 6.63% | 84% | 101% | 86% | 99% |
IBC Class TGM | 11.33% | 12.03% | 12.34% | 84% | 105% | 95% | 102% |
BDPCM ClassF | 0.45% | -0.18% | 0.90% | 99% | 101% | 97% | 97% |
BDPCM ClassTGM | 0.27% | 0.08% | 0.14% | 100% | 100% |
Simulation results of coding tools for colour space 4:4:4 (VTM anchor)
AI | |||||||
Acronym | BDR-Y | BDR-U | BDR-V | Tester EncT | Tester DecT | XCheck EncT | XCheck DecT |
PLT | 11.35% | 14.77% | 15.95% | 98% | 108% | 98% | 107% |
ACT, RGB | 10.61% | 2.74% | 3.57% | 104% | 100% | 98% | 102% |
RA | |||||||
PLT | 7.83% | 10.48% | 11.87% | 99% | 101% | 100% | 102% |
ACT, RGB | 19.12% | 6.66% | 8.27% | 104% | 100% | 97% | 101% |
LD | |||||||
PLT | 3.98% | 7.23% | 8.24% | 96% | 101% | 96% | 99% |
ACT, RGB | 28.47% | 9.24% | 11.13% | 103% | 100% | 98% | 101% |
Luma sample usage and memory bandwidth results of VTM tool “off” test. (VTM anchor)
AI | RA | LDB | |||||
Acronym | Sample usage | Sample usage | Ave mem BW | Max mem BW | Sample usage | Ave mem BW | Max mem BW |
CCLM | 48.72% | 3.71% | 0.80% | ||||
ALF | 99.00% | 54.96% | 51.76% | ||||
AFF | 19.05% | 28.61% | |||||
SBTMC | 11.54% | 14.55% | |||||
AMVR | 5.44% | 2.59% | |||||
TPM | 2.09% | 5.55% | |||||
BDOF | 44.56% | ||||||
CIIP | 0.86% | 1.46% | |||||
MMVD | 6.98% | 8.42% | |||||
BCW | 9.83% | 8.02% | |||||
MRLP | 6.41% | 0.59% | 0.24% | ||||
DMVR | 39.82% | ||||||
SBT | 2.50% | 3.91% | |||||
SMVD | 2.80% | ||||||
MIP | 23.73% | 5.12% | 2.44% | ||||
LFNST | 9.41% | 0.86% | 0.39% | ||||
JCCR | 10.81% | 0.52% | 0.12% | ||||
SAO | 31.33% | 7.10% | 7.88% |
Test results of VTM tool “off” test on various VTM versions
VTM RA | |||||
Abbreviation | VTM3 | VTM4 | VTM5 | VTM6 | VTM7 |
CST | 0.74% | 1.25% | 1.47% | 0.99% | 0.91% |
DQ | 1.39% | 1.36% | 1.24% | 1.32% | 1.33% |
CCLM | 4.09% | 4.20% | 4.00% | 3.33% | 3.37% |
MTS | 1.25% | 0.80% | 0.36% | 0.68% | 0.69% |
ALF | 3.61% | 3.73% | 4.79% | 4.65% | 4.64% |
AFF | 2.42% | 2.46% | 2.38% | 2.84% | 2.81% |
SbTMVP | 0.52% | 0.43% | 0.40% | 0.48% | 0.44% |
AMVR | 0.98% | 1.13% | 1.14% | 1.59% | 1.58% |
TPM | 0.43% | 0.43% | 0.41% | 0.39% | 0.44% |
BDOF | 1.02% | 0.63% | 0.66% | 0.68% | 0.67% |
CIIP | 0.43% | 0.51% | 0.31% | 0.24% | 0.23% |
MMVD | 0.81% | 0.52% | 0.59% | 0.52% | 0.51% |
BCW | 0.48% | 0.45% | 0.45% | 0.43% | 0.41% |
MRLP | 0.24% | 0.18% | 0.16% | 0.18% | 0.15% |
IBC | 0.07% | 0.00% | 0.05% | -0.01% | -0.02% |
ISP | 0.24% | 0.12% | 0.20% | 0.31% | |
DMVR | 0.80% | 0.87% | 0.87% | 0.88% | |
SBT | 0.33% | 0.33% | 0.32% | 0.32% | |
LMCS | 0.62% | 0.57% | 1.03% | 1.37% | |
SMVD | 0.26% | 0.24% | 0.27% | 0.26% | |
BDPCM | -0.02% | -0.03% | -0.01% | ||
MIP | 0.27% | 0.32% | 0.36% | ||
LFNST | 0.75% | 0.61% | 0.76% | ||
JCCR | 0.34% | 0.42% | 0.41% | ||
SAO | 0.81% | 0.64% | 0.17% | 0.13% | 0.12% |
PROF | 0.41% | 0.40% |
PSNR-Y vs encoding runtime ratio of VTM with VTM tool tests (VTM anchor)
PSNR-Y vs decoding runtime ratio of VTM with VTM tool tests (VTM anchor)
PSNR-Y vs weighted runtime ratio (a = 6) of VTM with VTM tool tests (VTM anchor)
The AHG recommends the following:
- Consider the reported tool test results during tool adoption decision making
- Review related contributions
- Refine list of tested tools and test methodology for the next meeting cycle
- Consider the reported tool test results as a benchmark for CE tests
- Consider including reporting of compute system information for testers and cross-checkers
Three tools were mentioned as having less than 0.3% BD benefit in RA configuration without a compensating subjective rationale:
- Combined intra/inter prediction (CIIP)
- Multi-reference line prediction (MRLP)
- Symmetric motion vector difference (SMVD)
It was commented that it would not be difficult to somewhat improve the CIIP with encoder optimization if that is desired.
It was commented that these three features are not difficult from a decoder perspective.
It was also commented that design stability favours not making changes, that people have already started implementing the draft standard, and that the tradeoffs are sometimes quite different in a real implementation. There are also interactions between features, such that trying to remove things could have unexpected side effects.
It was noted that the adaptive colour transform (ACT) is primarily intended for RGB content and does not provide a significant benefit for YCbCr sequences.
It was noted that we do not have a CTC for 4:4:4, and it was suggested that such CTC should be established.
It was also suggested that having a way to routinely test RPR would be desirable (although the rationale for this was suggested for a somewhat different purpose – just testing whether the feature functions properly). Having a way of exercising and testing the coding efficiency impact of tiles was also suggested.
As noted previously, it would be highly desirable to improve the test sequence selection for SCC.
Development of the following types of tests was planned:
- The CE2 coordinators were asked to work on preparing a CTC (which should include both camera and SCC content and RGB as well as YCbCr content and 4:4:4, 4:2:2 and monochrome testing).
- For an RPR functionality confirmation testing (FCT) output, we can base this on the prior CE test scheme. J. Luo volunteered to prepare that.
- For lossless and near lossless, we can produce CTC based on CE3 conditions, requesting this to be prepared by the CE3 coordinators.
Such test condition specifications were produced as outputs of this meeting.