JVET-P0003 JVET AHG report: Test model software development (AHG3) [F. Bossen, X. Li, K. Sühring]
This AHG report was discussed Thursday 3 October 0945 (GJS & JRO).
This report summarizes the activities of the AhG3 on Test model software development that has taken place between the 15th and 16th JVET meetings.
VTM software development
Development was continued on the GitLab server, which allows participants to register accounts and use a distributed development workflow based on git.
The server is located at:
https://vcgit.hhi.fraunhofer.de
The registration and development workflow is documented at:
https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/wikis/VVC-Software-Development-Workflow
The VTM software can be found at
https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/
CTC Performance
The following table shows VTM 6.1 performance over HM 16.20:
All Intra | |||||
Over HM-16.20 | |||||
Y | U | V | EncT | DecT | |
Class A1 | -28.12% | -33.44% | -33.64% | 1627% | 174% |
Class A2 | -27.86% | -20.72% | -13.06% | 2625% | 183% |
Class B | -21.10% | -19.86% | -27.27% | 2927% | 190% |
Class C | -21.81% | -19.77% | -23.98% | 4098% | 204% |
Class E | -25.16% | -21.29% | -25.60% | 2393% | 176% |
Overall | -24.23% | -22.48% | -24.96% | 2716% | 187% |
Class D | -17.70% | -14.41% | -15.71% | 4741% | 198% |
Class F | -38.92% | -39.43% | -41.80% | 4338% | 186% |
Random Access | |||||
Over HM-16.20 | |||||
Y | U | V | EncT | DecT | |
Class A1 | -37.32% | -38.56% | -44.50% | 843% | 162% |
Class A2 | -41.64% | -38.65% | -32.64% | 932% | 174% |
Class B | -33.98% | -41.65% | -43.13% | 902% | 160% |
Class C | -28.65% | -32.77% | -34.87% | 1144% | 174% |
Class E | |||||
Overall | -34.76% | -38.06% | -39.10% | 954% | 167% |
Class D | -26.40% | -29.45% | -30.18% | 1231% | 178% |
Class F | -40.66% | -45.33% | -46.77% | 640% | 142% |
Low Delay B | |||||
Over HM-16.20 | |||||
Y | U | V | EncT | DecT | |
Class A1 | |||||
Class A2 | |||||
Class B | -26.95% | -30.69% | -31.68% | 861% | 150% |
Class C | -22.91% | -23.65% | -25.50% | 1005% | 153% |
Class E | -25.56% | -31.33% | -33.57% | 435% | 121% |
Overall | -25.25% | -28.50% | -30.09% | 764% | 143% |
Class D | -21.34% | -19.29% | -21.28% | 1024% | 169% |
Class F | -37.72% | -41.80% | -43.27% | 526% | 116% |
Low Delay P | |||||
Over HM-16.20 | |||||
Y | U | V | EncT | DecT | |
Class A1 | |||||
Class A2 | |||||
Class B | -31.87% | -34.39% | -35.33% | 809% | 149% |
Class C | -25.45% | -25.00% | -26.93% | 964% | 160% |
Class E | -29.82% | -36.00% | -37.95% | 425% | 123% |
Overall | -29.22% | -31.66% | -33.18% | 730% | 145% |
Class D | -23.47% | -21.35% | -22.26% | 987% | 171% |
Class F | -37.62% | -41.91% | -43.00% | 575% | 122% |
The following table shows VTM 6.0 performance compared to VTM 5.2:
All Intra | |||||
Over VTM-5.2 | |||||
Y | U | V | EncT | DecT | |
Class A1 | -1.91% | 9.60% | 4.82% | 73% | 99% |
Class A2 | -3.68% | 7.65% | 7.61% | 75% | 100% |
Class B | -0.81% | 6.57% | 7.39% | 77% | 101% |
Class C | -0.69% | 2.95% | 4.11% | 82% | 102% |
Class E | -0.74% | 3.55% | 5.32% | 84% | 98% |
Overall | -1.43% | 5.95% | 5.92% | 78% | 100% |
Class D | -0.69% | 4.75% | 6.19% | 82% | 106% |
Class F | -1.38% | 0.54% | 0.84% | 91% | 102% |
Random Access | |||||
Over VTM-5.2 | |||||
Y | U | V | EncT | DecT | |
Class A1 | -3.37% | -0.93% | -5.52% | 95% | 102% |
Class A2 | -3.85% | -4.75% | -4.40% | 97% | 103% |
Class B | -1.77% | -8.48% | -7.39% | 92% | 102% |
Class C | -0.99% | -8.26% | -6.28% | 92% | 101% |
Class E | |||||
Overall | -2.30% | -6.16% | -6.12% | 93% | 102% |
Class D | -0.73% | -7.13% | -4.31% | 85% | 99% |
Class F | -1.25% | -7.69% | -6.67% | 88% | 101% |
Low Delay B | |||||
Over VTM-5.2 | |||||
Y | U | V | EncT | DecT | |
Class A1 | |||||
Class A2 | |||||
Class B | -1.27% | -13.27% | -12.74% | 107% | 107% |
Class C | -0.55% | -7.38% | -6.24% | 105% | 113% |
Class E | -0.12% | -11.51% | -9.23% | 99% | 116% |
Overall | -0.74% | -10.87% | -9.69% | 105% | 111% |
Class D | -0.08% | -7.48% | -6.89% | 100% | 107% |
Class F | -2.10% | -9.06% | -8.59% | 98% | 109% |
Low Delay P | |||||
Over VTM-5.2 | |||||
Y | U | V | EncT | DecT | |
Class A1 | |||||
Class A2 | |||||
Class B | -2.01% | -14.22% | -14.41% | 109% | 107% |
Class C | -1.12% | -8.10% | -7.08% | 108% | 112% |
Class E | -1.20% | -12.64% | -10.95% | 107% | 116% |
Overall | -1.51% | -11.78% | -11.10% | 108% | 111% |
Class D | -0.65% | -8.45% | -7.17% | 100% | 106% |
Class F | -2.09% | -9.57% | -9.29% | 102% | 110% |
The following table shows VTM 6.1 performance compared to VTM 6.0:
All Intra | |||||
Over VTM-6.0 | |||||
Y | U | V | EncT | DecT | |
Class A1 | 0.01% | 0.01% | 0.01% | 99% | 93% |
Class A2 | 0.00% | 0.00% | 0.00% | 100% | 93% |
Class B | 0.01% | 0.01% | 0.01% | 103% | 96% |
Class C | 0.03% | 0.03% | 0.03% | 100% | 97% |
Class E | 0.03% | 0.03% | 0.03% | 101% | 96% |
Overall | 0.02% | 0.02% | 0.02% | 101% | 95% |
Class D | 0.05% | 0.03% | 0.03% | 100% | 91% |
Class F | 0.03% | 0.02% | 0.02% | 100% | 96% |
Random Access | |||||
Over VTM-6.0 | |||||
Y | U | V | EncT | DecT | |
Class A1 | 0.00% | -0.03% | 0.01% | 98% | 87% |
Class A2 | 0.00% | -0.04% | 0.04% | 98% | 90% |
Class B | 0.00% | 0.00% | 0.00% | 99% | 88% |
Class C | 0.00% | -0.01% | 0.05% | 99% | 88% |
Class E | |||||
Overall | 0.00% | -0.01% | 0.02% | 99% | 88% |
Class D | 0.00% | 0.03% | 0.08% | 100% | 83% |
Class F | 0.02% | 0.01% | 0.01% | 102% | 89% |
Low Delay B | |||||
Over VTM-6.0 | |||||
Y | U | V | EncT | DecT | |
Class A1 | |||||
Class A2 | |||||
Class B | 0.00% | -0.09% | -0.13% | 99% | 85% |
Class C | 0.00% | 0.06% | -0.06% | 99% | 81% |
Class E | 0.02% | 0.23% | 0.06% | 100% | 83% |
Overall | 0.01% | 0.04% | -0.06% | 99% | 83% |
Class D | -0.02% | 0.29% | -0.09% | 100% | 83% |
Class F | 0.03% | 0.02% | -0.03% | 99% | 81% |
Low Delay P | |||||
Over VTM-6.0 | |||||
Y | U | V | EncT | DecT | |
Class A1 | |||||
Class A2 | |||||
Class B | 0.00% | -0.04% | 0.07% | 98% | 84% |
Class C | 0.00% | 0.14% | 0.00% | 99% | 81% |
Class E | 0.13% | 0.01% | 0.26% | 100% | 82% |
Overall | 0.03% | 0.03% | 0.10% | 99% | 82% |
Class D | 0.06% | -0.57% | -0.30% | 100% | 83% |
Class F | -0.03% | -0.11% | 0.25% | 101% | 82% |
Full.results are attached to this AHG report as Excel files.
Issues encountered during the software implementation process
Several issues were encountered during software development:
- Vacation time caused delays because people responsible for submitting software patches were not available.
- The implementation of “JVET-O0050: avoid small intra prediction with a local dual-tree technique” was much larger than expected (~1000 lines of code changed) and the merge request was provided close to the VTM-6.0 deadline. It is generally a problem when large patches are provided late, because this gives software coordinators little time to review and test them and may thus cause delays in the release process.
- It was discovered that “JVET-O0119: Base palette mode for 4:4:4” causes a significant increase in memory usage, even if not enabled (as in CTC). Coding tools should only have minimal impact on memory and coding time, when disabled. This needs to be fixed.
- It is generally concerning that for many HLS implementations work seemed to start only close to the deadline or after the deadline. HLS implementations can start in parallel to the low-level implementations, because there are few conflicts to be expected.
- Some software patches were submitted completely untested.
- The quality of submitted SIMD code is generally not very high. A lot of SIMD code has been rewritten.
- Configuration files were incorrect in VTM 6.0 release candidate 1 (incorrect naming of DBPCM parameter). This resulted in BDPCM not being enabled in the set (based on rc1) that was used to retrain initial CABAC states.
- There was some confusion about how to configure QP settings in CTC for HDR PQ sequences following the adoption of JVET-O0650 (chroma QP mapping). In particular, the confusion arose from the fact that chroma offsets signaled in the PPS are now applied after QP mapping instead of before. This has a significant effect for PQ sequences, where custom chroma offsets are computed. The luma/chroma balance for PQ content may thus be significantly different in VTM 6 than in VTM 5. This should be addressed at the 16th meeting.
- JVET-O0282 (low QP bug fix) was integrated under the macro of JVET-O0199 (base palette for 444), which led to confusion. Such practice should be avoided in the future.
- An issue was identified regarding text and software integration of JVET-N0278, and the expectations of proponents, what would have been integrated. Draft 5 version 1 integrated the exact text of JVET-N0278, which defines an input filter for the decoder, that removes all layers, except for the target layer (LIdTarget). Software was provided implementing this filter into VTM. Later, in Draft 5 version 4 “fixes for JVET-N0278” were integrated that refer to layers in multiple locations of the specification, including definitions, reference picture list construction, and constraints on different NAL unit types. These changes are likely language from layered HEVC, but not required in the case, where a decoder would see only one specific value of nuh_layer_id. Proponents modified these editorially added constraints later on in the Gothenburg meeting. They expected the original constraints to be implemented in software. Also, for the implementation of scalable coding, proponents expected that the software would already contain some infrastructure for multi-layer coding, which does not exist.
At the beginning of the 16th meeting, the following implementations were still pending:
- JVET-O0357 Maximum TU size for chroma format 4:2:2 and 4:4:4 [L. Li, J. Nam, J. Lim, S. Kim (LGE)]. O0389 was noted to be related [W. Cai, J. Zhu, J. Yao, K. Kazui (Fujitsu)]. The current text says that for chroma, the max TU size is:
- For 4:4:4, max is 64x64
- For 4:2:2, max is 32x64
- For 4:2:0, max is 32x32
- JVET-O0357 Maximum TU size for chroma format 4:2:2 and 4:4:4 [L. Li, J. Nam, J. Lim, S. Kim (LGE)]. O0389 was noted to be related [W. Cai, J. Zhu, J. Yao, K. Kazui (Fujitsu)]. The current text says that for chroma, the max TU size is:
Proponents of O0357 and O0389 were asked to check these aspects for text and software. This was later discussed as a non-CE6 topic. It was confirmed that the current software operates as described above, and it was clarified that O0389 which had proposed 64x64 transform block for chroma was not adopted.
- JVET-O0235: Constraints on nal_unit_type, TemporalId, etc. (Ericsson working on this)
- JVET-O1143. Subpictures
- JVET-O1159: Scalability
For the following implementations, issues remain:
- JVET-O0147/JVET-O0226: A constraint was changed in the contributions to enable the use of more efficient field coding GOPs. An example field coding config file should be provided to guide users.
- JVET-O0145/JVET-O0215: The number of entry points is derived instead of being signalled. For this the number of bricks needs to be known in the slice header. The software needs to be restructured to allow derivation of the slice/tile/brick related variables within the slice header parsing process. An initial version of the code has been merged, but disabled.
- JVET-O0042: The syntax was included with JVET-O0041, but no configuration options exist for frame repetitions. Also, the decoder does not repeat frames.
Software manual
Few merge requests included the required contributions to the software manual. Updates were only provided after being requested by the software coordinators.
Many parameters still have their outdated HEVC documentation and need to be updated. Other parameters for VVC tools are completely missing.
To make the software manual a valuable document, those missing parts need to be added.
CE software
For each CE a group was created in GitLab and CE coordinators were given owner rights to the group. This way they could clone VTM as required, create branches for different tests and assign user access to the group themselves.
The CE development workflow is described at:
https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/wikis/Core-experiment-development-workflow
CE read access is available using shared accounts: One account exists for MPEG members, which uses the usual MPEG account data (as announced on the appropriate email lists). A second account exists for VCEG members. The account information for VCEG members is available in the TIES system:
https://www.itu.int/ifa/t/2017/sg16/exchange/wp3/q06/vceg_account.txt
Bug tracking
The bug tracker for VTM and specification text is located at:
https://jvet.hhi.fraunhofer.de/trac/vvc
The bug tracker uses the same accounts as the HM software bug tracker. Users may need to log in again due to the different sub-domain. For spam fighting reasons account registration is only possible at the HM software bug tracker at
https://hevc.hhi.fraunhofer.de/trac/hevc
Participants are requested to please file all issues related to the VVC reference software into the bug tracker. They should try to provide all the details, which are necessary to reproduce the issue. Patches for solving issues and improving the software are always appreciated.
The AHG recommended to:
- Continue to develop the VTM reference software
- Improve documentation, especially the software manual
- Resolve any normative issues resulting from the large number of integrations in the most recent development cycle
- Encourage people to test VTM software more extensively outside of common test conditions.
- Encourage people to report all (potential) bugs that they are finding.
- Encourage people to submit bitstreams/test cases that trigger bugs in VTM.
- Encourage people to submit non-normative changes that reduce encoder run time without significantly sacrificing compression performance
- Make sure that contributions considered for adoption in the future are subject to adequate text and software review by the JVET at large
- Identify which recent additions contribute to the increase of encoder run time for low delay configurations, in particular for low QP values
In the discussion, it was noted that Broadcom and Allegro had been particularly helpful in the interim period with checking and bug-fixing both software and text. This was greatly appreciated.
It was commented that integration of HLS aspects in software has sometimes been difficult. One issue has been the need for clarity on who is responsible for software for some of these aspects. Making sure we have a clear identification of one person responsible was a suggestion for how to improve this.