JVET-X0003 JVET AHG report: Test model software development (AHG3) [F. Bossen, X. Li, K. Sühring, Y. He, K. Sharman, V. Seregin, A. Tourapis]

Abstract

This report summarizes the activities of the AhG3 on Test model software development that has taken place between the 23^rd and 24^th JVET meetings.

The mandates given to the AHG are:

• Coordinate development of test model (VTM, HM, SCM, SHM, HTM, MFC, MFCD, JM, JSVM, JMVM, 3DV-ATM, 360Lib, and HDRTools) software and associated configuration files.
• Produce documentation of software usage for distribution with the software.
• Enable software support for recently standardized additional SEI messages.
• Discuss and make recommendations on the software development process.
• Propose improvements to the guideline document for developments of the test model software.
• Perform comparative tests of test model behaviour using common test conditions.
• Suggest configuration files for additional testing of tools.
• Investigate how to minimize the number of separate codebases maintained for group reference software.
• Coordinate with AHG on Draft text and test model algorithm description editing (AHG2) to identify any mismatches between software and text, and make further updates and cleanups to the software as appropriate.
• Prepare drafts of merged CTC documents for HM and VTM, as applicable.

The software model versions prior to the start of the meeting were:

• VTM 14.0 (Aug. 2021)
• HM-16.24 (Oct. 2021)
• HM-16.21+SCM-8.8 (Mar. 2020)
• SHM 12.4 (Jan. 2018)
• HTM 16.3 (Jul. 2018)
• JM 19.0
• JSVM 9.19.15
• JMVC 8.5
• 3DV ATM 15.0 (no version history)
• HDRTools 0.23 (October 2021)

Software for MFC and MFCD is only available published by ITU-T and ISO/IEC. It is planned to create repositories with the latest versions available in ITU-T H.264.2 (02/2016). All development history is lost.

Software development

Development was continued on the GitLab server, which allows participants to register accounts and use a distributed development workflow based on git.

The server is located at:

https://vcgit.hhi.fraunhofer.de

The registration and development workflow are documented at:

https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/wikis/VVC-Software-Development-Workflow

Although the development process is described in the context of the VTM software, it can be applied to all other software projects hosted on the GitLab server as well.

VTM related activities

The VTM software can be found at

https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/

The software development continued on the GitLab server. VTM versions 13.2 and 13.3 were tagged on Jul. 19 and Jul. 20, and VTM version 14.0 was tagged on Aug. 26. VTM 14.1 is expected during the 24^th JVET meeting.

VTM 13.1 was tagged on Jul. 19, 2021. Changes include:

• JVET-V0061 display orientation SEI message
• JVET-U0084: EDRAP SEI message
• JVET-U0082 SDI SEI and three other SEIs: MAI, ACI, and DRI.
• JVET-V0108: Colour transform information SEI
• JVET-V0111: Decoding Unit Information
• JVET-Q0443: Implement MinCR check for subpictures using Subpicture SEI message
• Fix for Ticket#1489 - Cleanup of variable names for m_iSourceWidth and m_iSourceHeight
• Fix TMVPMode=2: don't search affine candidates when there are none
• Fix TestSAODisableAtPictureLevel=1: reset picture when SAO disabled for picture
• Fix EncDbOpt=1: make sure reference luma block has same size as chroma block
• Fix #1492: Values in cfg_sliFractions and cfg_sliNonSubpicLayersFractions should have a range of 0 to 255

VTM 13.2 was tagged on Jul. 20, 2021. Changes include:

• Fix #1490: Some pictures are not output when bitstream has mixed NALUs with CRA subpicture
• Remove macros from previous cycle

VTM 14.0 was tagged Aug. 26, 2021. Changes include:

• JVET-W0134 Uniform metrics log
• JVET-W0070 & W0121: Condition the signalling of sps_ts_residual_coding_rice_present_in_sh_flag on...
• JVET-W0043: Alignment of smooth block QP control with adaptive QP in VTM
• JVET-W0129: change default encoder cfg: enable using true original samples for ALF
• JVET-W0178: Bitsream constraints on RExt tools for low bit-depth
• JVET-W0046: reverse last significant coefficient position
• JVET-W2005: Operation range extension profiles based on JVET-W0136 and meeting decisions
• Fix high bit depth support after !2081 (JVET-V0108)
• Fix segmentation fault when calling xCreateUnavailablePicture() on RPL1
• fix #1496: add missing condition on cu height
• Fix memory leak after !2081
• Fix #1497: Encoder crash for RGB content when LFNST and TransformSkip disabled
• Fix #1498: Fix decoder mismatch for RGB content when LFNST is disabled
• Fix #1499: Fix a bitsream constraint on RExt tools for low bit-depth (JVET-W0178)
• Fix #1503: Added check for transform skip being disabled if TSRCRicePresent is enabled
• Fix #1502: Low bit-depth encoding fails due to the check in riceStatReset
• Fix #1495: scalable encoding fails when GDR_ENABLED=1
• Fix parameters sets used for unavailable picture creation
• Fix #1495 and #1501: GDR-related issues
• Fix for #1505: access to m_pProfile, when no profile is set (Profile=none)
• Change hbd.cfg etc file to set appropriate Profile for HBD CTC
• Revert chroma scaling changes introduced in !2095

VTM 14.1 is expected to be tagged during the 24^th JVET meeting. Changes include:

• JVET-W0133: Constrained RASL encoding for bitstream switching
• JVET-S0154 aspect9 and JVET-S0158 aspect4: bitstream extraction of extracting non-nested SEI from nested SEI
• JVET-S0117 Virtual boundary rewriting for subpicture extraction
• JVET-W0078: Multiview View Position SEI message
• Fix #1508: Use correct chroma format in call to verifyPlane
• Fix #1509: lossless coding of RGB content
• Fixes for multi-layer (spatial scalability) including fix for #1510
• (open merge requests)

CTC Performance

The following tables show VTM 14.0 performance over HM 16.24:

According to common test conditions in random access configuration HM is using a GOP size of 16 pictures compared to VTM using a GOP of 32 pictures. Random access points are inserted approximately every second aligned with a GOP boundary of GOP 32 in both VTM and HM. VTM uses two more reference pictures in random access than HM (due to more memory being availably in typical level settings).

The following tables show VTM 14.0 performance compared to VTM 13.2:

Full results are attached to this AHG report as Excel files.

Issues in VTM 14.x affecting conformance

The following issues in VTM master branch (Oct. 6, 2021) affect conformance:

• Handling of NoOutputOfPriorPicFlag is disabled due to crash issues (issue #1415)
• Missing HLS features (see sections below)

There are no known issues in VTM that affect processing of current VVC v1 conformance bitstreams, as issue #1490 was resolved (handling of CRA subpicture with mixed NAL units).

Status of implementation of proposals of previous JVET meetings

The following list contains all adoptions of the Q and R meetings that were not marked as merged (or submitted) or specification only change in the software coordinator tracking sheet:

• JVET-Q0112
• JVET-Q0154: Disallow mixing of GDR and IRAP (Disallow mixing of GDR with any non-GDR).
• JVET-Q0164
• JVET-Q0402
• JVET-R0178: Require that when no_aps_constraint_flag is equal to 1, sps_lmcs_enabled_flag and sps_scaling_list_enabled_flag shall be equal to 0
• JVET-R0221
• JVET-R0046: Change the description of the bitstream extraction process per the value of max_tid_il_ref_pics_plus1[ ][ ] (aspect 1.2 per JVET-R0046-v4).
• JVET-R0065: Specify that GDR AUs shall be complete – i.e., all of the layers in the CVS shall have a picture in the AU (as with IRAP AUs).
• JVET-R0191: Update the range value for num_ols_hrd_params_minus1.
• JVET-R0222 aspect 1: Infer vps_max_sublayers_minus1 to be equal to 6 when sps_video_parameter_set_id is equal to 0 (i.e. VPS is not present). The exact editorial expression is at the discretion of the editor.
• JVET-S0196 (JVET-S0144 item 17)
• JVET-S0227 (JVET-S0144 item 22)
• JVET-S0077 (JVET-S0139 item 5)
• JVET-S0174 aspect 2 (JVET-S0139 item 18.b)
• JVET-S0156 aspect 3 (JVET-S0139 item 21)
• JVET-S0139 item 26 (no source listed, text only?)
• JVET-S0188 aspect 1 (JVET-S0139 item 28)
• JVET-S0139 item 40 (item does not exist)
• JVET-S0042 (JVET-S0142 item 1.b)
• JVET-S0174 aspect 1 (JVET S0143 item 19)
• JVET-S0096 aspect 3 (JVET-S0140 item 10)
• JVET-S0096 aspect 4 (JVET-S0140 item 13)
• JVET-S0159 aspect 3 (JVET-S0140 item 16)
• JVET-S0171 (JVET-S0256)
• JVET-S0118 (JVET-S0141 item 7)
• JVET-S0102 (JVET-S0141 item 9.a)
• JVET-S0117 (JVET-S0141 item 11)
• JVET-S0157 item 2 (JVET-S0141 item 13)
• JVET-S0157 item 4 (JVET-S0141 item 14)
• JVET-S0175 aspect 3 (JVET-S0141 item 16)
• JVET-S0175 aspect 1, 2 (JVET-S0141 item 17)
• JVET-S0175 aspects 4 and 5 (JVET-S0141 item 18)
• JVET-S0175 aspect 6 (JVET-S0141 item 19)
• JVET-S0198/ JVET-S0223 (JVET-S0141 item 24)
• JVET-S0173 aspect 2 (JVET-S0141 item 40.b)
• JVET-S0173 item 1 (JVET-S0141 item 51)
• JVET-S0173 item 3 (JVET-S0141 item 52)
• JVET-S0173 item 5 (JVET-S0141 item 53)
• JVET-S0173 item 6 (JVET-S0141 item 54)
• JVET-S0173 item 4 (JVET-S0141 item 56)
• JVET-S0176 item 4 (JVET-S0141 item 60)
• JVET-S0154 aspect 5 (JVET-S0141 item 68)
• JVET-S0154 aspect 6 (JVET-S0141 item 69)
• JVET-S0154 aspect 8 (JVET-S0141 item 71)
• JVET-S0095 aspect 5 (JVET-S0145 item 5)
• JVET-S0095 aspect 6 (JVET-S0145 item 6)
• JVET-S0100 aspect 1, depends on JVET-R0193 (JVET-S0147 item 2)
• FINB ballot comments
• Make high tier support up to 960.

HM related activities

HM 16.24 is was tagged on Oct. 5, 2021. Changes include:

• JVET-V0056: Changes to MCTF
• JVET-V0078: QP control for very smooth blocks
• JCTVC-AM0023: Illustration of the film grain characteristics SEI message in HEVC
• JCTVC-AM0024: Illustration of the shutter interval info SEI message in HEVC Draft
• Update Conformance Window code (backport from VTM)
• Update of copyright year

The following actions have yet to be included:

• JVET-T0050: Add ability to detect static objects to encoder

Merge requests are available, but have pending discussions.

The following tables show HM 16.24 performance compared to HM 16.23:

As reported in the previous report, further information on lambda optimisation in HM would be appreciated, including comparison of allocation of bits within the GOP structures between HM and VTM.

The HEVC bug tracker lists:

• 38 tickets for “HM”, most of which are more than 5 years,
• 1 ticket for “HM RExt”, which was created during this reporting period,
• 7 tickets for “HM SCC”, all of which are at least 3 years old,

Help to address these tickets would be appreciated.

One merge request is available related to HM SCC for ticket #1511 on SCC reference picture marking. We would appreciate help to confirm that the proposed change matches the SCC text.

360Lib related activities

There is no update for 360Lib software.

The following table is for the projection formats comparison using VTM-14.0 according to 360-degree video CTC (JVET-U2012) compared to that using VTM-13.0 (VTM-13.0 as anchor).

The following table compares generalized cubemap (GCMP) coding and padded equi-rectangular projection (PERP) coding using VTM-14.0 (PERP as anchor).

The following tables are for PERP and GCMP coding comparison between VTM-14.0 and HM-16.22 (HM as anchor), respectively.

SCM related activities

There had not been any further developments to SCC’s SCM during this meeting cycle.

SHM related activities

There had not been any further developments to SHVC’s SHM during this meeting cycle.

HTM related activities

There had not been any updates to the HTM of MV-HEVC and 3D-HEVC.

HDRTools related activities

HDRTools version 0.23 was tagged on October 5th, 2021. Changes include:

• Add fix to support integer N->M bitdepth conversion.
• Also add support for log conversion of MSSSIM (JVET module))
• Add start frame number CLI option
• clean up old project files
• add a few guidelines on how to use libpng with Xcode

New development is now being added under the branch named 0.24-dev.

JM, JSVM, JMVM related activities

There had not been any updates to the JM, JSVM and JMVM software.

Bug tracking

The bug tracker for VTM and specification text is located at:

https://jvet.hhi.fraunhofer.de/trac/vvc

The bug tracker uses the same accounts as the HM software bug tracker. Users may need to log in again due to the different sub-domain. For spam fighting reasons account registration is only possible at the HM software bug tracker at

https://hevc.hhi.fraunhofer.de/trac/hevc

Bug tracking for HDRTools is located at:

https://gitlab.com/standards/HDRTools/-/issues

Please file all issues related to the VVC reference software and HDRTools into the appropriate bug tracker. Try to provide all the details, which are necessary to reproduce the issue. Patches for solving issues and improving the software are always appreciated.

Software repositories

Git repositories that were previously assigned to the JCT-VC group on the GitLab server were re-assigned to the JVET group. The old URLs are still working and will forward the user to the new location, with the display of a warning suggesting to update bookmarks to the new location.

The subversion repositories for SHM and HTM were converted to git and are now stored on the GitLab server to unify access and development process.

CTC alignment

The following differences were found in CTC alignment between HEVC and VVC:

• For HM two test configurations are described: one for 8-bit coding bit depth for Main profile and a second one for 10-bit coding bit depth for Main 10. VTM only specifies a 10-bit test case. These should be aligned, so that the same templates can be used.

Merging of CTC documents related to HEVC and VVC in each area of interest should be considered. JVET-W0152 was submitted to the 23^rd JVET providing an overview of the existing CTC documents and suggesting merge options.

There was no activity during this meeting cycle on creating merged CTC documents.

Recommendations

The AHG recommended to:

• Continue to develop reference software
• Improve documentation, especially the software manual
• Encourage people to test VTM and other reference software more extensively outside of common test conditions.
• Encourage people to report all (potential) bugs that they are finding.
• Encourage people to submit bitstreams/test cases that trigger bugs in VTM and other reference software.
• Encourage people to submit non-normative changes that either reduce encoder run time without significantly sacrificing compression performance or improve compression performance without significantly increasing encoder run time
• Design and add configuration files to the VTM software for testing of HLS features
• Review VTM-related contributions and determine whether features should be added (or removed) from the software
• Continue to investigate the merging of branches.
• Continue to investigate merging of CTC documents.
• Keep common test conditions aligned for the different standards.
• Consider documents (including late documents) related to AHG3 activities