JVET-U0003 JVET AHG report: Test model software development (AHG3) [F. Bossen, X. Li, K. Sühring, K. Sharman, V. Seregin, A. Tourapis]

The mandates given to the AHG were:

• Coordinate development of test model (VTM, HM, SCM, SHM, HTM, MFC, MFCD, JM, JSVM, JMVM, 3DV-ATM, and HDRTools) software and associated configuration files.
• Produce documentation of software usage for distribution with the software.
• Enable software support for recently standardized additional SEI messages.
• Discuss and make recommendations on the software development process.
• Propose improvements to the guideline document for developments of the test model software.
• Perform tests of test model behaviour using common test conditions.
• Suggest configuration files for additional testing of tools.
• Investigate how to minimize the number of separate codebases maintained for group reference software.
• Coordinate with AHG on Draft text and test model algorithm description editing (AHG2) to identify any mismatches between software and text, and make further updates and cleanups to the software as appropriate.
• Coordinate with AHG6 for integration with 360lib software.

The software model versions prior to the start of the meeting were:

• VTM 11.0 (Nov. 2020)
• HM-16.22 (Jul. 2020)
• HM-16.21+SCM-8.8 (Mar. 2020)
• SHM 12.4 (Jan. 2018) [svn]
• HTM 16.3 (Jul. 2018) [svn]
• JM 19.0
• JSVM 9.19.15
• JMVC 8.5
• 3DV ATM 15.0 (no version history)
• HDRTools 0.19.1 (Sep. 2019)

Software for MFC and MFCD is only available in the version published by ITU-T and ISO/IEC. It is planned to create repositories with the latest versions available in ITU-T H.264.2 (02/2016). No development history detail was found.

Software development

Development was continued on the GitLab server, which allows participants to register accounts and use a distributed development workflow based on git.

The server is located at:

https://vcgit.hhi.fraunhofer.de

The registration and development workflow are documented at:

https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/wikis/VVC-Software-Development-Workflow

Although the development process is described in the context of the VTM software, it can be applied to all other software projects hosted on the GitLab server as well.

Only SHM and HTM are still located in subversion repositories. It is suggested to convert and move these repositories to GitLab as well.

VTM related activities

The VTM software can be found at

https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/

The software development continued on the GitLab server. VTM versions 10.1 and 10.2 were tagged on Oct. 20 and Oct. 21, and VTM version 11.0 was tagged on Nov. 30. VTM version 11.1 is expected during the 21st JVET meeting.

VTM 10.1 was tagged on Oct. 20, 2020. Changes include:

• JVET-S0102 Aspect 3: Place PT SEI messages in individual SEI NAL units when general_same_pic_timing_in_all_ols_flag is equal to 1
• JVET-S0102 Aspect 4: Mandate same value for buffering period syntax elements
• Fix #1103: PPS extension data flag cleanup
• JVET-S0158: On the general sub-bitstream extraction process
• Bugfix and code cleanup for Ticket #1095.
• Refactor/Bugfix: VPS handling
• Fix for Ticket #1319 - encoder is crashed when setting DecodingRefreshType equal to 2
• Fix segmentation fault when decoding a VPS after 07afae5e
• Fix #1320: Add process for decoding NAL_UNIT_FD when filler data NAL units are in the bitstream
• JVET-S0097: set default values in VPS
• Fix cfg files for field coding
• Cleanups and fixes for software manual / remove unused encoder option
• Fix intra-layer references in enhancement layer IRAP
• Allow build with OpenMP on macOS/Xcode
• JVET-R0227 aspect 3: Rename gci_no_qp_delta_constraint_flag to gci_no_cu_qp_delta_constraint_flag
• Fix decoding if bitstream starts with a CRA picture with RASL
• Fix for ticket #1373: Repeated conversion of the Intra Chroma minQT setting from chroma unit to luma unit for multiple SPSs
• JVET-R0249: Prefix syntax elements in the VPS, SPS, PPS, PH, and SH with vps_,...
• JVET-R0251 aspect 1: Renaming sps_XXX_pic_present_flag to...
• JVET-S0172 item 2: Constraints on GCMP
• Fix syntax order of sublayer_hrd_parameters() to align with specification
• Bugfix for Ticket #1291: Mismatch between spec and software in BPSEI and PTSEI
• JVET-S0047 and JVET-S0211: Add "_minus1" to VB pos SEs and signal num VBs with ue(v)
• Fix RPL construction: allow short-term to be marked as long-term (regression after #1868)
• Fix #1382: modify list construction and verification
• Fix decoding when using long term in the SPS RPL
• Fix #1383: ALF APS management in multilayer streams
• Add .layer to filenames when outputting multiple layers
• Add layer ID to output picture log file
• Fix for IBC buffer initialization when CTU size changes
• JVET-S0212 aspect 2: Check that bitdepth_minus8 is in the range of 0..2, inclusive
• JVET-S0154 and JVET-R0068 Apsect 5: On the subpicture sub-bitstream extraction process
• JVET-S0208 aspect7: Prefix all syntax elements in dpb_parameters() with 'dpb_'
• Fix #1148: POC management
• Fix #1385: reset adaptive maximum BT stats on IRAPs
• JVET-S0162: Subpicture merge app
• JVET-S0208: aspects 1 (extra header bits) and 6 (OLS count)
• JVET-S0208 aspect8: Rename SEI syntax elements with descriptive prefixes
• Fix #1395: don't adapt CABAC init flag for IRAPs
• Fix #1396 : Wrong output order of multiple layers
• Fix #1353: Deblocking, ISP, and 4:2:2/4:4:4 chroma formats
• JVET-S0202: At least one picture with PictureOutputFlag equal to 1 in the bitstream
• Fix check for SEI presence in single layer case after #1473
• Fix output picture present in bitstream checking (#1794)
• JVET-S0121 aspect 2: coding of ph_deblocking_filter_override_flag
• JVET-S0115 aspect 2: rename syntax element in VPS
• JVET-T0061: remove unnecessary call and fix memory release
• various bug fixes and cleanups

VTM 10.2 was tagged on Oct.21, 2020. Relative to VTM 10.1 only macros of the previous cycle are removed.

VTM 11.0 was tagged Nov. 30, 2020. Changes include:

• JVET-S0219 aspect1, JVET-R0193 and JVET-S0141 item47
• Enable temporal prefilter
• Rename RA config file with GOP size 32 to encoder_randomaccess_vtm.cfg
• Fix #1408: Replace int with TCoeff to fix zeroing out of coefficients.
• Fix #1409: Don't scale 15/16 bit sequences to lower bit depths if...
• JVET-T0091: LMCS encoder overflow fix at high bit-depth for SDR
• JVET-S0090: All slices shall have the same value of nuh_layer_id when there is only one layer
• JVET-T0062: Extension of rate control to support GOP size of 32 (Ticket #1402).
• Fix: set the default for MaxTidILRefPicsPlus1 to 7
• JVET-S0163: On target OLS and sublayers for decoding (Operating Point Information OPI NAL Unit)
• JVET-S0078-Rebased: Handling NoOutputOfPriorPicFlag
• Port MS-SSIM from HM-16.19
• Change default of gci_present_flag
• JVET-R0266 proposal 5: no_gdr_constraint_flag semantics
• JVET-S0084 and JVET-S0110: Specify RPL constraint on RADL picture
• JVET-T2018: configuration files for high bit depth CTC
• Disable temporal filtering for high bit depth CTC
• JVET-T0065: Add level 6.3
• JVET-S0096 aspect 1: an RPL constraint
• Fix for ticket #1418: Mismatch between spec and VTM on signalling of sn_layer_id
• Temporarily disable JVET_S0078_NOOUTPUTPRIORPICFLAG
• Check for value range of dpb_max_num_reorder_pics
• Encoder improvement: use IDR_N_LP NAL unit type if GOP does not specify reordering of pictures
• Fix #1299: handling of independent subpics flag
• Fix #1404: condition for coding weighted prediction num_l1_weights.
• Fix handling of RPR with subpictures
• Fix #1405: decoded picture hash SEI message syntax (JVET-R0481)
• Fix #1426: Move initialization of bestBcwCost
• JVET-S0175 aspect 5: use u(8) instead of u(4) for ffi_display_elemental_periods_minus1 and pt_display_elemental_periods_minus1
• Fix #1407: Add missing HRD syntax conditions (JVET-S0175 aspect 6 option 2)
• JVET-R0046 aspect2
• Fix potential race condition when frame field information SEI arrives before picture timing SEI
• Update output YUV shift if the bitdepth changes between two sequences.
• Fix #1431: Avoid clipping after inverse transform when RExt__HIGH_BIT_DEPTH_SUPPORT is enabled
• JVET-T0064: Addition of ALF filter strength control to VTM
• Ticket #1433: Mark GDR with RecoveryPocCnt==0 as immediately decodable picture
• Add support for pps_single_slice_per_subpic_flag in subpicMergeApp
• Fix for ticket #1429: Temporal Filter does not work with field coding
• various other cleanups and bug fixes

It was expected that VTM 11.1 would be tagged during or short after the 21st JVET meeting. Changes include:

• JVET-R0264: IRAP constraint
• Fix #1422: dpb parameters inference when subLayerInfoFlag = 0
• Fix access to data that might be deleted
• JVET-T0053: Adding support for Annotated Regions SEI message
• Fix memory allocation when decoding a stream changing bitdepth between CVS
• Fix #1439: GOP32 configuration for larger intra periods
• Fix #1442: Fix for PTL signalling in VPS

CTC Performance

The following tables show VTM 11.0 performance over HM 16.22:

In the random access configuration, the HM is using a GOP size of 16 pictures compared to the VTM using a GOP size of 32 pictures. Since random access points are inserted approximately every second, aligned with a GOP boundary, the intra period differs in some frame rates between VTM and HM. The VTM uses two more reference pictures in random access than the HM (due to more memory being availably in typical level settings).

The following tables show VTM 11.0 performance compared to VTM 10.0:

It is noted that VTM10 used the same intra period as HM16.22, therefore the intra period in RA is not the same as for VTM11.

Full results were attached to this AHG report as Excel files.

The following issues in VTM 11.0 affect conformance:

• Decoding of multi-layer bitstreams broken (issues #1438, #1444)
• Handling of NoOutputOfPriorPicFlag is disabled due to crash issues (issue #1415)
• Crashes related to CRA and RASL pictures (issues #1413 and #1414)
• Missing HLS features

Some HLS features are still missing (see sections below) and there are several tickets related to multilayer coding that are still open.

It was mentioned that there may also be a problem with GDR implementation which needs further investigation (see the AHG5 report).

It should be noted that the syntax of the decoded picture hash SEI message was corrected between VTM versions 10.1 and 11.0 (issue #1405 / JVET-R0481). Thus, VTM 11.0 is incompatible with “draft5” conformance bitstreams provided at

https://www.itu.int/wftp3/av-arch/jvet-site/bitstream_exchange/VVC/draft_conformance/draft5/.

Status of implementation of proposals of previous JVET meetings

The following list contains all adoptions of the Q, R and S meetings that were not marked as merged (or submitted) or specification only change in the software coordinator tracking sheet:

• • • • JVET-Q0112
      • JVET-Q0154
      • JVET-Q0164
      • JVET-Q0402
      • JVET-Q0443
      • JVET-R0073
      • JVET-R0070
      • JVET-R0178
      • JVET-R0221
      • JVET-R0046
      • JVET-R0065
      • JVET-R0191
      • JVET-R0222 aspect 1
      • JVET-R0255 item3
      • JVET-S0196 (JVET-S0144 item 17)
      • JVET-S0227 (JVET-S0144 item 22)
      • JVET-S0077 (JVET-S0139 item 5)
      • JVET-S0139 item 18.b
      • JVET-S0139 item 21
      • JVET-S0139 item 26
      • JVET-S0139 item 28
      • JVET-S0139 item 40
      • JVET-S0210
      • JVET-S0050 aspect 4 / S0054 aspect 4 (JVET-S0138 item 31)
      • JVET-S0042 (JVET-S0142 item 1.b)
      • JVET-S0174 aspect 1 (JVET S0143 item 19)
      • JVET-S0096 aspect 3 (JVET-S0140 item 10)
      • JVET-S0096 aspect 4 (JVET-S0140 item 13)
      • JVET-S0159 aspect 3 (JVET-S0140 item 16)
      • JVET-S0171 (JVET-S0256)
      • JVET-S0141 item 7
      • JVET-S0141 item 9.a
      • JVET-S0141 item 11
      • JVET-S0141 item 13
      • JVET-S0141 item 14
      • JVET-S0141 item 15
      • JVET-S0141 item 16
      • JVET-S0141 item 17
      • JVET-S0141 item 18 (JVET-S0175 aspects 4 and 5)
      • JVET-S0141 item 19
      • JVET-S0141 item 24
      • JVET-S0141 item 40.b (JVET-S0173 aspect 2)
      • JVET-S0141 item 51 (JVET-S0173 item 1)
      • JVET-S0141 item 52 (JVET-S0173 item 3)
      • JVET-S0141 item 53 (JVET-S0173 item 5)
      • JVET-S0141 item 54 (JVET-S0173 item 6)
      • JVET-S0141 item 56 (JVET-S0173 item 4)
      • JVET-S0141 item 60 (JVET-S0176 item 4)
      • JVET-S0141 item 68 (JVET-S0154 aspect 5)
      • JVET-S0141 item 69 (JVET-S0154 aspect 6)
      • JVET-S0141 item 71 (JVET-S0154 aspect 8)
      • JVET-S0145 item 5 (JVET-S0095 aspect 5)
      • JVET-S0145 item 6 (JVET-S0095 aspect 6)
      • JVET-S0147 item 2 (JVET-S0100 aspect 1, depends on JVET-R0193)
      • FINB ballot comments to make the high tier support up to 960.

Status of proposals of the 20th JVET meeting (Online)

The following list contains all adoptions of the T meeting that were not marked as merged or specification only change in the software coordinator tracking sheet:

• • • • JVET-T0055 aspect 4
      • JVET-T0055 item 2

It should be noted that pending merge requests are available for both items.

There had not been any further developments to the HM software during this meeting cycle.

The following actions had yet to be included:

• • • • JCTVC-AM0023 (Illustration of the film grain characteristics SEI message in HEVC)
      • JCTVC-AJ0028 (Encoder-only Supplemental Motion Vector Estimation for Point cloud Coding content)
      • JVET-T0050: Add ability to detect static objects to encoder

Merge requests were available, but merging was pending final review.

As reported in the previous report, further information on lambda optimization in HM would be appreciated, including comparison of allocation of bits within the GOP structures between HM and VTM.

The HEVC bug tracker lists:

• 38 tickets for “HM”, most of which are more than 5 years,
• 1 ticket for “HM RExt”, which was created during this reporting period,
• 7 tickets for “HM SCC”, all of which are at least 3 years old,

Help to address these tickets would be appreciated.

There had not been any further developments to SCC’s SCM during this meeting cycle.

There had not been any further developments to SHVC’s SHM during this meeting cycle.

There had not been any updates to the HTM of MV-HEVC and 3D-HEVC.

There had not been any updates of the HDRTools.

There had not been any updates to the JM, JSVM and JMVM software.

The bug tracker for VTM and specification text is located at:

https://jvet.hhi.fraunhofer.de/trac/vvc

The bug tracker uses the same accounts as the HM software bug tracker. Users may need to log in again due to the different sub-domain. For spam fighting reasons account registration is only possible at the HM software bug tracker at

https://hevc.hhi.fraunhofer.de/trac/hevc

Please file all issues related to the VVC reference software into the bug tracker. Try to provide all the details, which are necessary to reproduce the issue. Patches for solving issues and improving the software are always appreciated.

On the GitLab server the different reference software was assigned to either the JVET group or the JCT-VC group resembling the organization structure. With the merge of JCT-VC into JVET, the repositories should also be moved into the JVET server location. This changes the URLs of software locations and there is no forwarding available from the old to the new location. The change will be made during or shortly after the 21st JVET meeting and announced to the JVET reflector.

The remaining subversion repositories should be converted to git and stored on the GitLab server to unify access and development process.

The following differences were found in CTC alignment between HEVC and VVC:

• In random access configuration VTM uses a different intra period due to the change to GOP 32

It was agreed that the HM intra period should be aligned with the one that VVC uses in GOP 32 (this particularly applies to low frame rate sequences). Also check for verification test setup.

• For HM two test configurations are described: one for 8-bit coding bit depth for Main profile and a second one for 10-bit coding bit depth for Main 10. VTM only specifies a 10-bit test case. These should be aligned, so that the same templates can be used.

This was to be further considered when the new proposal on an 8-bit profile for VVC is discussed.

The AHG recommended to:

• Continue to develop reference software
• Improve documentation, especially the software manual
• Resolve any normative issues resulting from the large number of integrations in the most recent development cycle
• Encourage people to test VTM and other reference software more extensively outside of common test conditions.
• Encourage people to report all (potential) bugs that they are finding.
• Encourage people to submit bitstreams/test cases that trigger bugs in VTM and other reference software.
• Encourage people to submit non-normative changes that reduce encoder run time without significantly sacrificing compression performance
• Design and add configuration files to the VTM software for testing of HLS features
• Review VTM-related contributions and determine whether features should be added (or removed) from the software
• Continue to investigate the merging of branches.
• Keep common test conditions aligned for the different standards.

Many of the not-yet-implemented issues relate to decoder-side checks for illegal bitstreams.

It is noted that scalable bitstreams can be generated (including spatial scalablity and multiview), but it is not known how efficient those are.