JVET-AG0003 JVET AHG report: Test model software development (AHG3) [F. Bossen, X. Li, K. Sühring (co-chairs), E. François, Y. He, K. Sharman, V. Seregin, A. Tourapis (vice chairs)]
Test software model versions
The software model versions prior to the start of the meeting were:
- VTM 23.0 (Dec. 2023)
- HM-18.0 (Apr. 2023)
- HM-16.21+SCM-8.8 (Mar. 2020)
- SHM 12.4 (Jan. 2018)
- HTM 16.3 (Jul. 2018)
- JM 19.1 (Apr. 2023)
- JSVM 9.19.15
- JMVC 8.5
- 3DV ATM 15.0 (no version history)
- HDRTools 0.24 (March 2023)
Software for MFC and MFCD is only available as published by ITU-T and ISO/IEC. It is planned to create repositories with the latest versions available in ITU-T H.264.2 (02/2016). All development history was reported to have been lost (or never provided).
Software development
Development was continued on the GitLab server, which allows participants to register accounts and use a distributed development workflow based on git.
The server is located at:
https://vcgit.hhi.fraunhofer.de
The registration and development workflow are documented at:
https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/wikis/VVC-Software-Development-Workflow
Although the development process is described in the context of the VTM software, it can be applied to all other software projects hosted on the GitLab server as well.
Between the 31st and 32nd meetings, the GitLab server was affected by a huge number of spam issues which were generated using newly registered accounts. Administrator user account confirmation was enabled to protect the server from spam postings. Proponents should plan for some time for server admins to confirm new accounts.
Before the 33rd meeting an issue was found, which prevents the GitLab server from sending out emails to external addresses. This prevents confirmation of new email addresses and thus the registration of new accounts. Email notifications for changes (e.g. updates on merge requests) are also not sent out. Proponents and software coordinators should check manually for updates. The issue was only identified recently and is expected to be resolved soon.
VTM related activities
The VTM software can be found at
https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/
The software development continued on the GitLab server. VTM versions 22.1 and 22.2 were tagged on Oct. 22, and VTM version 23.0 was tagged on Dec. 12. VTM 23.1 is expected during the 33rd JVET meeting.
VTM 22.1 was tagged on Oct. 22, 2023. Changes include:
- FIX #1611: use pic->reconstructed and pic->layerId for computing HashME condition
- FIX #1613: Fix transform skip cost for non-CTC conditions
- Update software manual on max transform size (#1614)
- Clean up IBC-related code
- Fix #1616: NNPFC SEI nnpfc_chroma_sample_loc_type_frame value is checked even when not signalled
- Fix #1612: PO-SEI message
- JVET-AE0156: Message wrapping and importance indication for the SEI processing order SEI message
- Free vector data memory when destroying CodingStructure buffers
- Port CS mem reduction from ecm/ECM!525
- Port using actual CTU size instead of MAX_CU_SIZE at encoder from ecm/ECM!516
- Relax strict equality when comparing floating-point numbers
- Port picture buffer refactoring from ecm/ECM!523
- Fix unreleased RPR temp buffer
- Fix #1615: Encoding exception with VS compiler
- Fix y4m header writing when RPR is used and UpscaledOutput=2
VTM 22.2 was tagged on Oct. 23, 2023. Changes include:
- Remove macros from previous cycle
VTM 23.0 was tagged Dec. 12, 2023. Changes include:
- Add SIMD implementations of transforms
- TileIdx as a new type
- Fix crash for class F: set SPS in CodingStructure
- Fix debugBitstream functionality
- Fix out-of-bounds access in EncCu::updateRdCheckingNum
- Remove parameter UseBLambdaForNonKeyLowDelayPictures
- Remove toLast and unused variables in ClpRngs
- Fix #1622: Initialising NNPFC SEI attributes in the default constructor
- Fix #1621: Compilation issue in non-X86 environments
- JVET-AF0122-SW: Lagrange multiplier optimization for ALF
- Fix debug bitstream for multilayer
- JVET-AF0310: Processing Order Nesting SEI message
- Cleanup: replace TAB characters with spaces in config files
VTM 23.1 was expected to be tagged during the 33rd JVET meeting (i.e. the current meeting). Changes were expected to include bug fixes and code optimizations.
CTC Performance
The following tables shows VTM 20.0 performance over HM 17.0 (not updated; results identical with previous versions).
All-Intra Main 10 | |||||
Over HM-17.0 | |||||
Y | U | V | EncT | DecT | |
Class A1 | -29.50% | -32.81% | -33.83% | 1273% | 166% |
Class A2 | -29.73% | -24.39% | -21.58% | 2142% | 174% |
Class B | -22.32% | -27.21% | -30.98% | 2397% | 175% |
Class C | -22.89% | -19.53% | -23.19% | 3348% | 179% |
Class E | -26.04% | -25.90% | -24.15% | 1869% | 158% |
Overall | -25.50% | -25.75% | -27.02% | 2187% | 171% |
Class D | -18.80% | -13.85% | -13.68% | 4270% | 189% |
Class F | -39.49% | -40.23% | -42.90% | 4220% | 171% |
Random Access Main 10 | |||||
Over HM-17.0 | |||||
Y | U | V | EncT | DecT | |
Class A1 | -40.60% | -40.71% | -47.16% | 586% | 158% |
Class A2 | -44.02% | -41.73% | -40.78% | 679% | 170% |
Class B | -37.41% | -50.12% | -48.51% | 672% | 161% |
Class C | -33.86% | -36.28% | -38.21% | 923% | 164% |
Class E | |||||
Overall | -38.42% | -42.87% | -43.95% | 713% | 163% |
Class D | -31.72% | -32.60% | -31.95% | 970% | 161% |
Class F | -46.07% | -49.76% | -50.65% | 490% | 145% |
Low Delay B Main 10 | |||||
Over HM-17.0 | |||||
Y | U | V | EncT | DecT | |
Class A1 | |||||
Class A2 | |||||
Class B | -30.40% | -37.43% | -35.12% | 629% | 152% |
Class C | -28.18% | -21.23% | -21.06% | 817% | 167% |
Class E | -31.16% | -35.72% | -28.79% | 309% | 140% |
Overall | -29.85% | -31.60% | -28.85% | 575% | 154% |
Class D | -26.75% | -16.50% | -14.88% | 881% | 179% |
Class F | -42.17% | -44.02% | -44.14% | 435% | 137% |
Low Delay P Main 10 | |||||
Over HM-17.0 | |||||
Y | U | V | EncT | DecT | |
Class A1 | |||||
Class A2 | |||||
Class B | -34.52% | -38.34% | -36.12% | 526% | 160% |
Class C | -28.07% | -17.73% | -18.09% | 670% | 176% |
Class E | -33.06% | -37.89% | -31.10% | 280% | 146% |
Overall | -32.01% | -31.36% | -28.85% | 487% | 161% |
Class D | -26.68% | -12.73% | -11.47% | 762% | 175% |
Class F | -40.13% | -41.24% | -41.72% | 437% | 145% |
According to common test conditions in random access configuration HM is using a GOP size of 16 pictures compared to VTM using a GOP of 32 pictures. Random access points are inserted approximately every second aligned with a GOP boundary of GOP 32 in both VTM and HM. VTM uses two more reference pictures in random access than HM (due to more memory being availably in typical level settings).
The following table shows VTM 23.0 performance over VTM 22.0 using SDR CTC:
All-Intra Main 10 | |||||
Over VTM-22.0 | |||||
Y | U | V | EncT | DecT | |
Class A1 | 0.00% | 0.00% | 0.00% | 94% | 91% |
Class A2 | 0.00% | 0.00% | 0.00% | 96% | 94% |
Class B | 0.00% | 0.00% | 0.00% | 97% | 94% |
Class C | 0.00% | 0.00% | 0.00% | 98% | 97% |
Class E | 0.00% | 0.00% | 0.00% | 97% | 93% |
Overall | 0.00% | 0.00% | 0.00% | 96% | 94% |
Class D | 0.00% | 0.00% | 0.00% | 98% | 98% |
Class F | 0.00% | 0.00% | 0.00% | 90% | 97% |
Random Access Main 10 | |||||
Over VTM-22.0 | |||||
Y | U | V | EncT | DecT | |
Class A1 | 0.00% | -0.08% | 0.01% | 93% | 94% |
Class A2 | 0.06% | 0.07% | 0.06% | 93% | 96% |
Class B | 0.02% | -0.03% | -0.03% | 93% | 96% |
Class C | 0.02% | 0.03% | -0.05% | 93% | 96% |
Class E | |||||
Overall | 0.02% | 0.00% | -0.01% | 93% | 96% |
Class D | 0.03% | -0.25% | 0.06% | 96% | 97% |
Class F | 0.14% | 0.17% | 0.12% | 91% | 96% |
Low Delay B Main 10 | |||||
Over VTM-22.0 | |||||
Y | U | V | EncT | DecT | |
Class A1 | |||||
Class A2 | |||||
Class B | 0.06% | 0.14% | 0.44% | 93% | 94% |
Class C | 0.03% | 0.02% | -0.25% | 94% | 96% |
Class E | 0.04% | 0.20% | -0.28% | 97% | 96% |
Overall | 0.04% | 0.12% | 0.03% | 94% | 95% |
Class D | 0.04% | 0.23% | -0.73% | 97% | 98% |
Class F | 0.21% | 0.32% | -0.22% | 92% | 96% |
Low Delay P Main 10 | |||||
Over VTM-22.0 | |||||
Y | U | V | EncT | DecT | |
Class A1 | |||||
Class A2 | |||||
Class B | 0.04% | 0.24% | 0.07% | 94% | 95% |
Class C | -0.02% | 0.22% | -0.11% | 97% | 99% |
Class E | 0.11% | 0.39% | -0.25% | 96% | 95% |
Overall | 0.04% | 0.27% | -0.07% | 95% | 96% |
Class D | 0.04% | -0.35% | -0.34% | 98% | 98% |
Class F | 0.30% | 0.12% | -0.20% | 92% | 96% |
For the high bit depth CTCs, there is no change in coding performance or run time between VTM 23.0 and VTM 22.0 for the low QP range. For the standard QP range (22-37), the simulations were run using VTM 23.0 with the merged fix for DualITree=0 (otherwise the software crashes).
Relative to VTM 22.0, some small differences can be seen:
Random Access | ||||||||||
VTM23.0 + fix over VTM22.0, with CTC described in VTM-JVET-AA2018-STDQP | ||||||||||
wPSNR | PSNR | |||||||||
DE100 | PSNR-L100 | Y | U | V | Y | U | V | EncT | DecT | |
Class H1 | -0.07% | -0.01% | 0.00% | -0.09% | -0.10% | 0.00% | -0.07% | -0.13% | 97% | 95% |
Class H2 | 0.05% | 0.15% | 0.08% | 97% | 95% | |||||
Overall | -0.07% | -0.01% | 0.00% | -0.09% | -0.10% | 0.03% | 0.04% | -0.03% | 97% | 95% |
All-Intra | ||||||||||
VTM23.0 + fix over VTM22.0, with CTC described in VTM-JVET-AA2018-STDQP | ||||||||||
wPSNR | PSNR | |||||||||
DE100 | PSNR-L100 | Y | U | V | Y | U | V | EncT | DecT | |
Class H1 | -0.06% | 0.00% | 0.00% | -0.09% | -0.08% | 0.00% | -0.05% | -0.07% | 100% | 97% |
Class H2 | 0.00% | 0.00% | 0.00% | 98% | 96% | |||||
Overall | -0.06% | 0.00% | 0.00% | -0.09% | -0.08% | 0.00% | -0.03% | -0.03% | 99% | 97% |
Results using HDR CTC show slight differences between VTM 23.0 and VTM 23.0, as depicted in the table below.
Random Access | ||||||||||
Over VTM22.0 | ||||||||||
wPSNR | PSNR | |||||||||
DE100 | PSNR-L100 | Y | U | V | Y | U | V | EncT | DecT | |
Class H1 | -0.03% | 0.00% | 0.00% | -0.13% | 0.19% | 0.01% | -0.09% | -0.10% | 100% | 100% |
Class H2 | 0.00% | -0.03% | 0.10% | 100% | 102% | |||||
Overall | 0.08% | 0.02% | 0.02% | 0.25% | 0.19% | 0.00% | -0.07% | -0.03% | 100% | 101% |
All-Intra | ||||||||||
Over VTM22.0 | ||||||||||
wPSNR | PSNR | |||||||||
DE100 | PSNR-L100 | Y | U | V | Y | U | V | EncT | DecT | |
Class H1 | -0.08% | 0.00% | 0.00% | -0.09% | -0.06% | 0.00% | -0.06% | -0.04% | 102% | 100% |
Class H2 | 0.00% | 0.00% | 0.00% | 100% | 100% | |||||
Overall | -0.08% | 0.00% | 0.00% | -0.09% | -0.06% | 0.00% | -0.04% | -0.02% | 101% | 100% |
The following tables show VTM 22.0 performance over HM 18.0 using HDR CTC:
Random Access | ||||||||||
Over HM18.0 | ||||||||||
wPSNR | PSNR | |||||||||
DE100 | PSNR-L100 | Y | U | V | Y | U | V | EncT | DecT | |
Class H1 | -39.50% | -37.55% | -36.66% | -54.93% | -48.46% | -33.67% | -49.42% | -40.87% | 288% | 101% |
Class H2 | -31.96% | -57.71% | -63.32% | 257% | 92% | |||||
Overall | -39.50% | -37.55% | -36.66% | -54.93% | -48.46% | -33.07% | -52.43% | -49.03% | 276% | 98% |
All-Intra | ||||||||||
Over HM18.0 | ||||||||||
wPSNR | PSNR | |||||||||
DE100 | PSNR-L100 | Y | U | V | Y | U | V | EncT | DecT | |
Class H1 | -41.45% | -27.31% | -26.79% | -57.74% | -52.79% | -24.00% | -52.93% | -45.13% | 1537% | 117% |
Class H2 | -21.76% | -47.21% | -50.57% | 1274% | 109% | |||||
Overall | -41.45% | -27.31% | -26.79% | -57.74% | -52.79% | -23.18% | -50.85% | -47.11% | 1436% | 114% |
Issues in VTM affecting conformance
The following issues in VTM master branch may affect conformance:
- Missing HLS features (see sections below)
Status of implementation of proposals of previous JVET meetings
The following list contains all adoptions of the Q and R meetings that were not marked as merged (or submitted) or specification only change in the software coordinator tracking sheet:
- JVET-Q0112
- JVET-Q0154: Disallow mixing of GDR and IRAP (Disallow mixing of GDR with any non-GDR).
- JVET-Q0164
- JVET-Q0402
- JVET-R0178: Require that when no_aps_constraint_flag is equal to 1, sps_lmcs_enabled_flag and sps_scaling_list_enabled_flag shall be equal to 0
- JVET-R0221
- JVET-R0046: Change the description of the bitstream extraction process per the value of max_tid_il_ref_pics_plus1[ ][ ] (aspect 1.2 per JVET-R0046-v4).
- JVET-R0065: Specify that GDR AUs shall be complete – i.e., all of the layers in the CVS shall have a picture in the AU (as with IRAP AUs).
- JVET-R0191: Update the range value for num_ols_hrd_params_minus1.
- JVET-R0222 aspect 1: Infer vps_max_sublayers_minus1 to be equal to 6 when sps_video_parameter_set_id is equal to 0 (i.e. VPS is not present). The exact editorial expression is at the discretion of the editor.
- JVET-S0196 (JVET-S0144 item 17)
- JVET-S0227 (JVET-S0144 item 22)
- JVET-S0077 (JVET-S0139 item 5)
- JVET-S0174 aspect 2 (JVET-S0139 item 18.b)
- JVET-S0156 aspect 3 (JVET-S0139 item 21)
- JVET-S0139 item 26 (no source listed, text only?)
- JVET-S0188 aspect 1 (JVET-S0139 item 28)
- JVET-S0139 item 40 (item does not exist)
- JVET-S0042 (JVET-S0142 item 1.b)
- JVET-S0174 aspect 1 (JVET S0143 item 19)
- JVET-S0096 aspect 3 (JVET-S0140 item 10)
- JVET-S0096 aspect 4 (JVET-S0140 item 13)
- JVET-S0159 aspect 3 (JVET-S0140 item 16)
- JVET-S0171 (JVET-S0256)
- JVET-S0118 (JVET-S0141 item 7)
- JVET-S0102 (JVET-S0141 item 9.a)
- JVET-S0157 item 2 (JVET-S0141 item 13)
- JVET-S0157 item 4 (JVET-S0141 item 14)
- JVET-S0175 aspect 3 (JVET-S0141 item 16)
- JVET-S0175 aspect 1, 2 (JVET-S0141 item 17)
- JVET-S0175 aspects 4 and 5 (JVET-S0141 item 18)
- JVET-S0175 aspect 6 (JVET-S0141 item 19)
- JVET-S0198/ JVET-S0223 (JVET-S0141 item 24)
- JVET-S0173 aspect 2 (JVET-S0141 item 40.b)
- JVET-S0173 item 1 (JVET-S0141 item 51)
- JVET-S0173 item 3 (JVET-S0141 item 52)
- JVET-S0173 item 5 (JVET-S0141 item 53)
- JVET-S0173 item 6 (JVET-S0141 item 54)
- JVET-S0173 item 4 (JVET-S0141 item 56)
- JVET-S0176 item 4 (JVET-S0141 item 60)
- JVET-S0154 aspect 5 (JVET-S0141 item 68)
- JVET-S0154 aspect 6 (JVET-S0141 item 69)
- JVET-S0154 aspect 8 (JVET-S0141 item 71)
- JVET-S0095 aspect 5 (JVET-S0145 item 5)
- JVET-S0095 aspect 6 (JVET-S0145 item 6)
- JVET-S0100 aspect 1, depends on JVET-R0193 (JVET-S0147 item 2)
- FINB ballot comments
- Make high tier support up to 960.
SEI TuC software
Per the decision during the 32nd JVET meeting, an SEI TuC repository was created based on VTM-22.2. The repository is located at:
https://vcgit.hhi.fraunhofer.de/jvet-tuc/VVCSoftware_VTM
VTM-22.2-TuC-1.0 is expected to be tagged during the 33rd JVET meeting. Changes so far include:
- JVET-AF0088
- JVET-AF0167 MPIISEI message
- JVET-AF0107-TuC: Implementation of Encoder Optimization Info SEI message
- JVET-AF2032 SPTI SEI message
HM related activities
There had not been any further developments to HM during this meeting cycle.
The following MRs are pending (with status indicated):
- Implement phase indication SEI message (JVET-AE0101) [waiting review]
- Port the Y4M support [one issue remains]
- Mark the current picture as short-term ref (for SCM) [need SCC expert reviewer]
The HM SCC (SCM) branch (HM-16.21+SCM-8.8) has not been updated for the recent HM versions. Updating SCM to, for example, HM-18.0+SCM-8.8 should be considered. It may though be helpful to move SCC related functionality into separate source files. Volunteer work towards merging the branches would be appreciated.
As reported in the previous reports, further information on lambda optimization in HM would be appreciated, including comparison of allocation of bits within the GOP structures between HM and VTM.
Otherwise the HEVC bug tracker lists:
- 43 tickets for “HM”, most of which are more than 5 years,
- 1 ticket for “HM RExt”,
- 9 tickets for “HM SCC”, most of which are at least 3 years old,
- 1 ticket for “RExt Text” (8 years old)
- 1 ticket for “SCC Text” (8 years old)
- 6 tickets for text (3-5 years old)
- 2 tickets for encoder description (3-9 years old)
Help to address these tickets would be appreciated.
360Lib related activities
The latest 360Lib software can be found at https://vcgit.hhi.fraunhofer.de/jvet/360lib/-/tags/360Lib-13.6
The following table is for the projection formats comparison using VTM-23.0 according to 360-degree video CTC (JVET-U2012) compared to that using VTM-22.0 (VTM-22.0 as anchor).
End-to-end WS-PSNR | End-to-end S-PSNR-NN | |||||
Y | U | V | Y | U | V | |
Class S1 | 0.00% | -0.01% | 0.05% | 0.00% | 0.00% | 0.05% |
Class S2 | 0.08% | 0.16% | 0.34% | 0.08% | 0.15% | 0.34% |
Overall | 0.03% | 0.06% | 0.17% | 0.03% | 0.06% | 0.16% |
The following table compares generalized cubemap (GCMP) coding and padded equi-rectangular projection (PERP) coding using VTM-23.0 (PERP as anchor).
End-to-end WS-PSNR | End-to-end S-PSNR-NN | |||||
Y | U | V | Y | U | V | |
Class S1 | -11.52% | -5.47% | -6.09% | -11.50% | -5.41% | -6.03% |
Class S2 | -3.65% | 0.98% | 1.48% | -3.62% | 1.08% | 1.56% |
Overall | -8.37% | -2.89% | -3.06% | -8.35% | -2.82% | -3.00% |
The following tables are for PERP and GCMP coding comparison between VTM-23.0 and HM-16.22 (HM as anchor), respectively.
End-to-end WS-PSNR | End-to-end S-PSNR-NN | |||||
Y | U | V | Y | U | V | |
Class S1 | -30.91% | -38.82% | -41.24% | -30.91% | -38.88% | -41.24% |
Class S2 | -36.84% | -37.23% | -39.43% | -36.82% | -37.27% | -39.49% |
Overall | -33.28% | -38.19% | -40.51% | -33.28% | -38.23% | -40.54% |
End-to-end WS-PSNR | End-to-end S-PSNR-NN | |||||
Y | U | V | Y | U | V | |
Class S1 | -34.64% | -40.04% | -42.14% | -34.59% | -40.02% | -42.09% |
Class S2 | -38.34% | -38.18% | -40.06% | -38.32% | -38.19% | -40.11% |
Overall | -36.12% | -39.30% | -41.31% | -36.08% | -39.29% | -41.30% |
SCM related activities
There had not been any further developments to SCC’s SCM during this meeting cycle.
SHM related activities
There had not been any further developments to SHVC’s SHM during this meeting cycle.
HTM related activities
There had not been any releases of HTM of MV-HEVC and 3D-HEVC.
One merge request was pending:
- JVET-AE0295: MV Main 10 profile support (waiting for proponent response)
The next release will include the following changes:
- JVET-Z0209: Early termination during calculating RDcost of depth
HDRTools related activities
There had not been any further developments to HDRTools during this meeting cycle.
JM, JSVM, JMVM related activities
There had not been any further developments to JM, JSVM or JMVM during this meeting cycle.
For the previous release of JM, only basic testing was performed to confirm that the code still works, but performance testing was pending. Performance changes are unlikely but may occur due to bug fixes. A possible encoder/decoder mismatch was reported privately, but needs more investigation.
Bug tracking
The bug tracker for VTM and specification text is located at:
https://jvet.hhi.fraunhofer.de/trac/vvc
The bug tracker uses the same accounts as the HM software bug tracker. Users may need to log in again due to the different sub-domain. For spam fighting reasons account registration is only possible at the HM software bug tracker at
https://hevc.hhi.fraunhofer.de/trac/hevc
Bug tracking for HDRTools is located at:
https://gitlab.com/standards/HDRTools/-/issues
Please file all issues related to the VVC reference software and HDRTools into the appropriate bug tracker. Try to provide all the details, which are necessary to reproduce the issue. Patches for solving issues and improving the software are always appreciated.
CTC alignment and merging
There were currently 8 JVET CTC documents: