JVET-T0005 JVET AHG report: Conformance testing (AHG5) [J. Boyce, W. Wan, E. Alshina, F. Bossen, I. Moccagatta, K. Kawamura, K. Sühring, X. Xu]

This document summarizes the activity of AHG5: “Conformance testing” between the 19th Meeting (teleconference, 22 June – 1 July 2020) and the 20th Meeting (teleconference, 7–16 Oct. 2020).

Timeline

The progress on the Conformance testing specification is consistent with the preliminary timeline agreed at the 16th JVET meeting, as follows:

• 17th meeting Jan. 2020: Preliminary guidelines for bitstream preparation (e.g., naming conventions),
• improved list of conformance bitstreams
• 18th meeting Apr. 2020: Final guidelines for bitstream preparation and improved list of conformance
• bitstreams with identified responsible experts, initial bitstreams provided
• 19th meeting July 2020: Confirmed list of bitstreams to be included in v1, collection of bitstream
• candidates for CD ballot at next meeting
• 20th meeting Oct. 2020: CD of conformance specification
• 21st meeting Jan. 2021: Final bitstreams provided, DIS ballot in ISO/IEC
• 22nd meeting April 2021: No action pending DIS ballot
• 23rd meeting July 2021: Final conformance specification

Status on bitstream submission

The status at the time of preparation of this report is as follows:

• 102 bitstream categories have been identified
• Volunteers have been identified for all categories
• 206 bitstreams in 77 bitstream categories have been provided for VTM 10.0
• 25 bitstream categories have no VTM 10.0 bitstreams
• 12 bitstream categories have no provided bitstreams (for any VTM version)

Activities and Discussion

The AHG activities are on schedule with the preliminary timeline shown in section 2. It is expected to output CD of the Conformance specification from this meeting.

Output document JVET-S2008 “Conformance testing for versatile video coding (Draft 4)” was published on 5 October 2020.

Because CD is planned to be issued from this October meeting, all bitstream volunteers had been requested to update their bitstreams using VTM 10.0. At least one bitstream has been provided in 77 of the 102 identified categories for VTM 10.0. Bitstream submitters were also requested to provide text descriptions of the bitstreams for inclusion in the conformance specification on a shared document, but relatively few have been updated.

A modification was made to the VTM master branch to include layer ID for each picture in the .opl files such as to distinguish between pictures within an access unit. A new version of the VTM is expected to include the opl change, as well as other changes to multilayer coding, missing HLS features, and a bug fix to 4:2:2/4:4:4 deblocking. Bitstreams that have already been provided in those categories may need to be updated, or the submitters are waiting for the availability of the new VTM to provide initial bitstreams.

The change to the .opl format affects bitstream packages that have already been submitted. It is expected that bitstream packages can be updated to the new format using a script and individual packages do not need to be resubmitted.

It was suggested to add bitstreams with 9-bit depth.

The regular JVET e-mail reflector was used for discussions (jvet@lists.rwth-aachen.de).

The AHG5 chairs and JVET chairs can be reached at jvet-conformance@lists.rwth-aachen.de. Participants should not subscribe to this list but may send emails to it.

There was one related contribution, discussing gaps in the existing conformance bitstream suite.

• JVET-T0100, AHG5: On gaps in conformance bitstreams [F. Bossen (Sharp)]

The procedure to exchange the bitstream (ftp cite, bitstream files, etc.) is specified in Sec 2 “Procedure” of JVET-R2008. The ftp and http sites for downloading bitstreams are

ftp://ftp3.itu.int/jvet-site/bitstream_exchange/VVC

https://www.itu.int/wftp3/av-arch/jvet-site/bitstream_exchange/VVC/

The ftp site for uploading bitstream file is as follows.

ftp://ftp3.itu.int/jvet-site/dropbox/

(user id: avguest, passwd: Avguest201007)

If using FileZilla, using the Site Manager with the following configuration was suggested:

The AHG recommended the following:

• Consider if should separate the draft Conformance specification into two output documents: the Conformance specification and instructions for generating conformance bitstreams, or should move the instructions to an Annex
• Encourage volunteers of missing conformance bitstreams to provide them quickly
• Encourage conformance bitstream providers to provide text descriptions of provided bitstreams for the Conformance specification online at http://mpeg.expert/live/nextcloud/index.php/f/37368
• Encourage conformance bitstream providers to provide updated bitstreams with corrected naming for those bitstreams indicated in the JVET-S2008 document as needing name changes to be consistent with the naming convention
• Discuss and refine the list of conformance bitstreams and the conformance specification, especially consider adding 9-bit depth and solicit volunteers
• Review submitted bitstreams and consider if the flexibility of the tested tool is sufficiently exercised, including consideration of input contribution JVET-T0100
• Proceed with CD issuance as an output of this meeting

There was discussion of whether the conformance testing spec really needs to be normative, and whether the dataset really needs to be part of the approved standard, or could be something external to it that is referenced informatively. (The tests are far from complete in any case.)

The question was discussed of whether to issue an ISO/IEC TR rather than a standard (removing all “shall” and perhaps all “should”).

This was initially agreed – with a long editing period – and to issue a request to change the plan from producing an IS to producing a TR. However, the matter was then further discussed later in the meeting.

The same approach was discussed for the software.

The fact that we have called prior reference software and conformance test data normative causes a problem with basically normative dependencies on non-standard external packages. On the other hand, some users may like the normative nature as providing assurance of correctness.

It was noted that there was a software maintance problem previously for 3D-AVC, such that the living codebase was removed from the web.

There would be less naming convenience on the ITU side as H.266.1 and H.266.2 if the ITU handles it as a Technical Paper or Technical Report or Supplement rather than a Recommendation.

This aspect was later further discussed (see also the notes for JVET-T0100), and it was concluded that we should issue these specifications as regular standards.