JVET-AF0003 JVET AHG report: Test model software development (AHG3) [F. Bossen, X. Li, K. Sühring (co-chairs), E. François, Y. He, K. Sharman, V. Seregin, A. Tourapis (vice chairs)]
The software model versions prior to the start of the meeting were:
- VTM 22.0 (Sept. 2023)
- HM-18.0 (Apr. 2023)
- HM-16.21+SCM-8.8 (Mar. 2020)
- SHM 12.4 (Jan. 2018)
- HTM 16.3 (Jul. 2018)
- JM 19.1 (Apr. 2023)
- JSVM 9.19.15
- JMVC 8.5
- 3DV ATM 15.0 (no version history)
- HDRTools 0.24 (March 2023)
Software for MFC and MFCD is only available as published by ITU-T and ISO/IEC. It is planned to create repositories with the latest versions available in ITU-T H.264.2 (02/2016). All development history is lost.
Software development
Development was continued on the GitLab server, which allows participants to register accounts and use a distributed development workflow based on git.
The server is located at:
https://vcgit.hhi.fraunhofer.de
The registration and development workflow are documented at:
https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/wikis/VVC-Software-Development-Workflow
Although the development process is described in the context of the VTM software, it can be applied to all other software projects hosted on the GitLab server as well.
VTM related activities
The VTM software can be found at
https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/
The software development continued on the GitLab server. VTM versions 20.1 and 20.2 were tagged on May 8, and VTM version 21.0 was tagged on Jul. 9. VTM 21.1 is expected during the 31st JVET meeting.
VTM 21.1 was tagged on July 21, 2023. Changes include:
- JVET-AD0057 item 1: Allow inclusion of NNPFC and NNPFA SEI messages in suffix SEI NAL units
- JVET-AD0054 item 3: NNPF interpolated pictures constraint
- Add encoder parameter WriteVuiHrdFromY4m
- fix RPL with inter-layer references
VTM 21.2 was tagged on July 22, 2023. Changes include:
- Remove macros from previous cycle
VTM 22.0 was tagged Sept. 24, 2023. Changes include:
- remove macro JVET_Z0120_SII_SEI_PROCESSING
- Clean up data types in ALF code
- Only one VPS can be active at any time
- JVET-AE0060: Signal nnpfc_chroma_loc_info_present_flag only when nnpfc_out_order_idc is not equal to 0
- JVET-AE0180: Add missing condition on PIC_RECON_WRAP buffer allocation
- JVET-AE0057: MTT split modes early termination
- JVET-AE0162: fix MS-SSIM calculation for RPR/SR
- Use PLTRunMode instead of bool for palette data
- Fix access to unitialized value in SPS
- Clean up code from JVET-AD0045 (encoder DMVR avoidance)
- Use std::vector to avoid memory deallocation/reallocation
- Port JVET-Z0150 on memory usage log
- Add TGM config files from ECM
- Remove unnecessary memory allocation for CCALF
- JVET-AE0141: Fix a bug in NNPFC SEI message for colourization
- Fix overflow condition in static_vector
- JVET-AE0135 item2: On NNPF picture rate upsampling constraint
- JVET-AE0126: NNPF cleanup and editorial changes for VSEI include item 2, item 3, item 5, and item 7
- JVET-AE0142 item 3 and item 4: Fix two bugs including an added condition and a nnpfc_purpose related constraint.
- JVET-AE0128 item 2: Update the constraint on when an NNPFC SEI message shall be a repetition of the first NNPFC SEI message in the CLVS
- JVET-AE0122: Fix for ppsid when both GOP based RPR and scalable coding is used
- JVET-AE0048 item 1: Add missing value ranges for nnpfc_pic_width_num_minus1,...
- JVET-AE0048 item 5: Cancel the persistence of an NNPFA SEI message with any...
- JVET_AE0048 item 2: Add missing value ranges for nnpfc_inp_format_idc and nnpfc_out_format_idc
- Clean up CommonDefX86.h
- JVET-AE0049 item 2: Constrain the repeated inference to NNPFs that perform...
- JVET-AE0181: Scaling window support
- JVET-AE0189 Item 2: Check when activating NNPFA with base flag equal to 0
- JVET-AE0050 item 1: Add nnpfa_no_prev_clvs_flag to the NNPFA SEI message
- Clean up AffineGradientSearch SIMD code
- JVET-AE0049 item 3: Change the index range in input picture assignment for...
- JVET-AE0049 item 1: Update NNPF SEI identical content constraints
- JVET-AE0050 item 2: Add nnpfa_no_foll_clvs_flag to the NNPFA SEI message
- JVET-AE0134 item 2: add repeated inference of NNPF for creating pictures corresponding to input pictures at the end of a bitstream or a CLVS under some conditions
- Reduce memory for ALF covariance data
VTM 22.1 was expected to be tagged during the 32nd JVET meeting. Changes are expected to include the remaining implementations of the 31st meeting, issue fixes and code optimizations.
CTC Performance
The following tables shows VTM 20.0 performance over HM 17.0 (not updated, results identical with previous versions).
All Intra Main 10 | |||||
Over HM-17.0 | |||||
Y | U | V | EncT | DecT | |
Class A1 | -29.50% | -32.81% | -33.83% | 1273% | 166% |
Class A2 | -29.73% | -24.39% | -21.58% | 2142% | 174% |
Class B | -22.32% | -27.21% | -30.98% | 2397% | 175% |
Class C | -22.89% | -19.53% | -23.19% | 3348% | 179% |
Class E | -26.04% | -25.90% | -24.15% | 1869% | 158% |
Overall | -25.50% | -25.75% | -27.02% | 2187% | 171% |
Class D | -18.80% | -13.85% | -13.68% | 4270% | 189% |
Class F | -39.49% | -40.23% | -42.90% | 4220% | 171% |
Random access Main 10 | |||||
Over HM-17.0 | |||||
Y | U | V | EncT | DecT | |
Class A1 | -40.60% | -40.71% | -47.16% | 586% | 158% |
Class A2 | -44.02% | -41.73% | -40.78% | 679% | 170% |
Class B | -37.41% | -50.12% | -48.51% | 672% | 161% |
Class C | -33.86% | -36.28% | -38.21% | 923% | 164% |
Class E | |||||
Overall | -38.42% | -42.87% | -43.95% | 713% | 163% |
Class D | -31.72% | -32.60% | -31.95% | 970% | 161% |
Class F | -46.07% | -49.76% | -50.65% | 490% | 145% |
Low delay B Main 10 | |||||
Over HM-17.0 | |||||
Y | U | V | EncT | DecT | |
Class A1 | |||||
Class A2 | |||||
Class B | -30.40% | -37.43% | -35.12% | 629% | 152% |
Class C | -28.18% | -21.23% | -21.06% | 817% | 167% |
Class E | -31.16% | -35.72% | -28.79% | 309% | 140% |
Overall | -29.85% | -31.60% | -28.85% | 575% | 154% |
Class D | -26.75% | -16.50% | -14.88% | 881% | 179% |
Class F | -42.17% | -44.02% | -44.14% | 435% | 137% |
Low delay P Main 10 | |||||
Over HM-17.0 | |||||
Y | U | V | EncT | DecT | |
Class A1 | |||||
Class A2 | |||||
Class B | -34.52% | -38.34% | -36.12% | 526% | 160% |
Class C | -28.07% | -17.73% | -18.09% | 670% | 176% |
Class E | -33.06% | -37.89% | -31.10% | 280% | 146% |
Overall | -32.01% | -31.36% | -28.85% | 487% | 161% |
Class D | -26.68% | -12.73% | -11.47% | 762% | 175% |
Class F | -40.13% | -41.24% | -41.72% | 437% | 145% |
According to common test conditions in random access configuration HM is using a GOP size of 16 pictures compared to VTM using a GOP of 32 pictures. Random access points are inserted approximately every second aligned with a GOP boundary of GOP 32 in both VTM and HM. VTM uses two more reference pictures in random access than HM (due to more memory being availably in typical level settings).
There was no change in coding performance or run time between VTM 22.0 and VTM 21.1 using SDR CTC.
For the high bit depth CTCs, there is no change in coding performance or run time between VTM 22.0 and VTM 21.0 for the low QP range; however, a change in coding performance and run-time had been observed for the standard QP range (22-37), as shown in the summary tables below. The coding performance changes were attributed to an optimization of ALF causing small floating point calculation discrepancies (ALF is turned off for the low QP range experiments).
Random Access | ||||||||||
VTM 22.0 vs VTM21.0 for High Bit Depth, Standard QP range (JVET-AA2018) | ||||||||||
| wPSNR |
| PSNR |
|
| |||||
DE100 | PSNR-L100 | Y | U | V | Y | U | V | EncT | DecT | |
Class H1 | 0.17% | 0.02% | 0.00% | 0.12% | 0.23% | 0.00% | 0.08% | 0.21% | 99% | 105% |
Class H2 |
|
|
|
|
| 0.00% | 0.00% | 0.00% | 99% | 104% |
Overall | 0.17% | 0.02% | 0.00% | 0.12% | 0.23% | 0.00% | 0.04% | 0.11% | 99% | 105% |
All Intra | ||||||||||
VTM 22.0 vs VTM21.0 for High Bit Depth, Standard QP range (JVET-AA2018) | ||||||||||
| wPSNR | PSNR |
| |||||||
DE100 | PSNR-L100 | Y | U | V | Y | U | V | EncT | DecT | |
Class H1 | 0.06% | 0.00% | 0.00% | 0.09% | 0.08% | 0.00% | 0.05% | 0.07% | 98% | 102% |
Class H2 |
|
|
|
|
| 0.00% | 0.00% | 0.00% | 100% | 102% |
Overall | 0.06% | 0.00% | 0.00% | 0.09% | 0.08% | 0.00% | 0.03% | 0.03% | 99% | 102% |
The coding performance differences had been almost eliminated in a pending merge request (!2682), as shown in the table below:
Random Access | ||||||||||
(VTM 22.0 + !2682) vs VTM21.0 for High Bit Depth, Standard QP range (JVET-AA2018) | ||||||||||
wPSNR | PSNR | |||||||||
DE100 | PSNR-L100 | Y | U | V | Y | U | V | EncT | DecT | |
Class H1 | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | -0.01% | 0.01% | 102% | 107% |
Class H2 | 0.00% | 0.00% | 0.00% | 102% | 107% | |||||
Overall | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | -0.01% | 0.00% | 102% | 107% |
All Intra | ||||||||||
(VTM 22.0 + !2682) vs VTM21.0 for High Bit Depth, Standard QP range (JVET-AA2018) | ||||||||||
wPSNR | PSNR | |||||||||
DE100 | PSNR-L100 | Y | U | V | Y | U | V | EncT | DecT | |
Class H1 | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 101% | 104% |
Class H2 | 0.00% | 0.00% | 0.00% | 102% | 103% | |||||
Overall | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 101% | 103% |
Results using HDR CTC show slight differences between VTM 22.0 and VTM 21.0, as depicted in table below.
Random Access | ||||||||||
Over VTM21.0 | ||||||||||
wPSNR | PSNR | |||||||||
DE100 | PSNR-L100 | Y | U | V | Y | U | V | EncT | DecT | |
Class H1 | 0.08% | 0.02% | 0.02% | 0.25% | 0.19% | 0.01% | 0.22% | 0.20% | 101% | 101% |
Class H2 | 0.00% | 0.00% | 0.00% | 101% | 101% | |||||
Overall | 0.08% | 0.02% | 0.02% | 0.25% | 0.19% | 0.00% | 0.14% | 0.13% | 101% | 101% |
All Intra | ||||||||||
Over VTM21.0 | ||||||||||
wPSNR | PSNR | |||||||||
DE100 | PSNR-L100 | Y | U | V | Y | U | V | EncT | DecT | |
Class H1 | 0.08% | 0.00% | 0.00% | 0.09% | 0.06% | 0.00% | 0.06% | 0.04% | 99% | 100% |
Class H2 | 0.00% | 0.00% | 0.00% | 98% | 99% | |||||
Overall | 0.08% | 0.00% | 0.00% | 0.09% | 0.06% | 0.00% | 0.04% | 0.02% | 98% | 100% |
The following tables show VTM 22.0 performance over HM 18.0 using HDR CTC:
Random Access | ||||||||||
Over HM18.0 | ||||||||||
wPSNR | PSNR | |||||||||
DE100 | PSNR-L100 | Y | U | V | Y | U | V | EncT | DecT | |
Class H1 | -39.51% | -37.55% | -36.66% | -54.89% | -48.36% | -33.68% | -49.39% | -40.75% | 289% | 102% |
Class H2 | -31.97% | -57.69% | -63.36% | 257% | 90% | |||||
Overall | -39.51% | -37.55% | -36.66% | -54.89% | -48.36% | -33.06% | -52.41% | -48.97% | 277% | 97% |
All Intra | ||||||||||
Over HM18.0 | ||||||||||
wPSNR | PSNR | |||||||||
DE100 | PSNR-L100 | Y | U | V | Y | U | V | EncT | DecT | |
Class H1 | -41.41% | -27.31% | -26.79% | -57.69% | -52.79% | -24.00% | -52.89% | -45.12% | 1506% | 117% |
Class H2 | -21.76% | -47.21% | -50.57% | 1280% | 109% | |||||
Overall | -41.41% | -27.31% | -26.79% | -57.69% | -52.79% | -23.18% | -50.83% | -47.10% | 1420% | 114% |
Issues in VTM affecting conformance
The following issues in VTM master branch may affect conformance:
- Missing HLS features (see sections below)
Status of implementation of proposals of previous JVET meetings
The following list contains all adoptions of the Q and R meetings that were not marked as merged (or submitted) or specification only change in the software coordinator tracking sheet:
- JVET-Q0112
- JVET-Q0154: Disallow mixing of GDR and IRAP (Disallow mixing of GDR with any non-GDR).
- JVET-Q0164
- JVET-Q0402
- JVET-R0178: Require that when no_aps_constraint_flag is equal to 1, sps_lmcs_enabled_flag and sps_scaling_list_enabled_flag shall be equal to 0
- JVET-R0221
- JVET-R0046: Change the description of the bitstream extraction process per the value of max_tid_il_ref_pics_plus1[ ][ ] (aspect 1.2 per JVET-R0046-v4).
- JVET-R0065: Specify that GDR AUs shall be complete – i.e., all of the layers in the CVS shall have a picture in the AU (as with IRAP AUs).
- JVET-R0191: Update the range value for num_ols_hrd_params_minus1.
- JVET-R0222 aspect 1: Infer vps_max_sublayers_minus1 to be equal to 6 when sps_video_parameter_set_id is equal to 0 (i.e. VPS is not present). The exact editorial expression is at the discretion of the editor.
- JVET-S0196 (JVET-S0144 item 17)
- JVET-S0227 (JVET-S0144 item 22)
- JVET-S0077 (JVET-S0139 item 5)
- JVET-S0174 aspect 2 (JVET-S0139 item 18.b)
- JVET-S0156 aspect 3 (JVET-S0139 item 21)
- JVET-S0139 item 26 (no source listed, text only?)
- JVET-S0188 aspect 1 (JVET-S0139 item 28)
- JVET-S0139 item 40 (item does not exist)
- JVET-S0042 (JVET-S0142 item 1.b)
- JVET-S0174 aspect 1 (JVET S0143 item 19)
- JVET-S0096 aspect 3 (JVET-S0140 item 10)
- JVET-S0096 aspect 4 (JVET-S0140 item 13)
- JVET-S0159 aspect 3 (JVET-S0140 item 16)
- JVET-S0171 (JVET-S0256)
- JVET-S0118 (JVET-S0141 item 7)
- JVET-S0102 (JVET-S0141 item 9.a)
- JVET-S0157 item 2 (JVET-S0141 item 13)
- JVET-S0157 item 4 (JVET-S0141 item 14)
- JVET-S0175 aspect 3 (JVET-S0141 item 16)
- JVET-S0175 aspect 1, 2 (JVET-S0141 item 17)
- JVET-S0175 aspects 4 and 5 (JVET-S0141 item 18)
- JVET-S0175 aspect 6 (JVET-S0141 item 19)
- JVET-S0198/ JVET-S0223 (JVET-S0141 item 24)
- JVET-S0173 aspect 2 (JVET-S0141 item 40.b)
- JVET-S0173 item 1 (JVET-S0141 item 51)
- JVET-S0173 item 3 (JVET-S0141 item 52)
- JVET-S0173 item 5 (JVET-S0141 item 53)
- JVET-S0173 item 6 (JVET-S0141 item 54)
- JVET-S0173 item 4 (JVET-S0141 item 56)
- JVET-S0176 item 4 (JVET-S0141 item 60)
- JVET-S0154 aspect 5 (JVET-S0141 item 68)
- JVET-S0154 aspect 6 (JVET-S0141 item 69)
- JVET-S0154 aspect 8 (JVET-S0141 item 71)
- JVET-S0095 aspect 5 (JVET-S0145 item 5)
- JVET-S0095 aspect 6 (JVET-S0145 item 6)
- JVET-S0100 aspect 1, depends on JVET-R0193 (JVET-S0147 item 2)
- FINB ballot comments
- Make high tier support up to 960.
HM related activities
There had not been any further developments to HM during this meeting cycle.
The following MRs are pending [with status indicated]:
- Implement phase indication SEI message (JVET-AE0101) [waiting review]
- Port the Y4M support [one issue remains]
- Mark the current picture as short-term ref (for SCM) [need SCC expert reviewer]
The HM SCC (SCM) branch (HM-16.21+SCM-8.8) has not been updated for the recent HM versions. Updating SCM to, for example, HM-18.0+SCM-8.8 should be considered. It may though be helpful to move SCC related functionality into separate source files. Volunteer work towards merging the branches would be appreciated.
As reported in the previous reports, further information on lambda optimization in HM would be appreciated, including comparison of allocation of bits within the GOP structures between HM and VTM.
Otherwise the HEVC bug tracker lists:
- 43 tickets for “HM”, most of which are more than 5 years,
- 1 ticket for “HM RExt”,
- 9 tickets for “HM SCC”, most of which are at least 3 years old,
- 1 ticket for “RExt Text” (8 years old)
- 1 ticket for “SCC Text” (8 years old)
- 6 tickets for text (3-5 years old)
- 2 tickets for encoder description (3-9 years old)
Help to address these tickets would be appreciated.
360Lib related activities
The latest 360Lib software can be found at https://vcgit.hhi.fraunhofer.de/jvet/360lib/-/tags/360Lib-13.6
The following table is for the projection formats comparison using VTM-22.0 according to 360-degree video CTC (JVET-U2012) compared to that using VTM-21.0 (VTM-21.0 as anchor).
End-to-end WS-PSNR | End-to-end S-PSNR-NN | |||||
Y | U | V | Y | U | V | |
Class S1 | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% |
Class S2 | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% |
Overall | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% | 0.00% |
The following table compares generalized cubemap (GCMP) coding and padded equi-rectangular projection (PERP) coding using VTM-22.0 (PERP as anchor).
End-to-end WS-PSNR | End-to-end S-PSNR-NN | |||||
Y | U | V | Y | U | V | |
Class S1 | -11.50% | -5.44% | -6.08% | -11.49% | -5.37% | -6.02% |
Class S2 | -3.69% | 0.81% | 1.28% | -3.67% | 0.90% | 1.37% |
Overall | -8.38% | -2.94% | -3.13% | -8.36% | -2.86% | -3.07% |
The following tables are for PERP and GCMP coding comparison between VTM-22.0 and HM-16.22 (HM as anchor), respectively.
End-to-end WS-PSNR | End-to-end S-PSNR-NN | |||||
Y | U | V | Y | U | V | |
Class S1 | -30.91% | -38.82% | -41.28% | -30.91% | -38.88% | -41.28% |
Class S2 | -36.89% | -37.33% | -39.65% | -36.88% | -37.36% | -39.71% |
Overall | -33.30% | -38.23% | -40.63% | -33.29% | -38.27% | -40.65% |
End-to-end WS-PSNR | End-to-end S-PSNR-NN | |||||
Y | U | V | Y | U | V | |
Class S1 | -35.60% | -40.82% | -42.80% | -35.56% | -40.75% | -42.75% |
Class S2 | -39.84% | -39.52% | -41.44% | -39.85% | -39.52% | -41.48% |
Overall | -37.29% | -40.30% | -42.25% | -37.27% | -40.26% | -42.24% |
SCM related activities
There had not been any further developments to SCC’s SCM during this meeting cycle.
SHM related activities
There had not been any further developments to SHVC’s SHM during this meeting cycle.
HTM related activities
There had not been any releases of HTM of MV-HEVC and 3D-HEVC.
One merge request is pending:
- JVET-AE0295: MV Main 10 profile support (waiting for proponent response)
The next release will include the following changes:
- JVET-Z0209: Early termination during calculating RDcost of depth
HDRTools related activities
There had not been any further developments to HDRTools during this meeting cycle.
JM, JSVM, JMVM related activities
There had not been any further developments to JM, JSVM or JMVM during this meeting cycle.
For the previous release of the JM only basic testing was performed to confirm that the code still works, but performance testing is pending. Performance changes are unlikely but may occur due to bug fixes. An possible encoder/decoder mismatch was reported privately, but needs more investigation.
Bug tracking
The bug tracker for VTM and specification text is located at:
https://jvet.hhi.fraunhofer.de/trac/vvc
The bug tracker uses the same accounts as the HM software bug tracker. Users may need to log in again due to the different sub-domain. For spam fighting reasons account registration is only possible at the HM software bug tracker at
https://hevc.hhi.fraunhofer.de/trac/hevc
Bug tracking for HDRTools is located at:
https://gitlab.com/standards/HDRTools/-/issues
Participants were requested to file all issues related to the VVC reference software and HDRTools into the appropriate bug tracker and to try to provide all the details which are necessary to reproduce the issue. Patches for solving issues and improving the software are always appreciated.
Software repositories
Git repositories that were previously assigned to the JCT-VC group on the GitLab server were re-assigned to the JVET group. The old URLs are still working and will forward the user to the new location, with the display of a warning suggesting to update bookmarks to the new location. The SVN repository for 360Lib was converted to git and development was moved to the GitLab server. Historical branches can still be accessed in the SVN repository.
CTC alignment and merging
There are currently 8 JVET CTC documents: