JVET-X0207 JVET BoG Report: VVC v1/v2 Conformance Testing [I. Moccagatta, D. Rusanovskyy]
This is a report of activities from the BoG on VVC v1/v2 Conformance Testing. The BoG call was held during the 24rd JVET meeting, on October 12, at 5:00 – 7:15 UTC. The meeting time and Zoom link were announced over the JVET reflector and made available through JVET calendar.
As output of the meeting, the BoG recommended the following:
- Approve the text of the document JVET-X0161-v2 (VVCv1 conformance testing).
- Approve the text of the document JVET-X0185 (VVCv2 conformance testing).
- Approve the collaborative procedure for VVCv2 conformance bitstreams cross-check. The cross-check was to be done offline, prior to the upload to the JVET FTP. Progress of the bitstream generation and cross-check is to be tracked by an Excel document maintained by the conformance testing coordinators.
- Submit conformance bitstreams generated according to the JVET-W2026 specification to the VVCv2 CDAM ballot.
- Extend the VVCv2 conformance (in next versions) by adding a new category on WPP (JVET-X0128).
- Use independently designed VVCv2 decoder for the cross-check, is available. Experts having access to such decoders are encouraged to volunteer for the conformance testing.
- Discuss in JVET if the list of AhG5 chairs need to be extended by adding more experts contributing to the VVCv2 development and cross-check.
The BoG met with following agenda:
- Review of the input documents on conformance testing:
- JVET-X0161 AHG5: Editors update on VVC conformance testing [J. Boyce, E. Alshina, F. Bossen, K. Kawamura, I. Moccagatta, W. Wan].
- JVET-X0185 AHG5: Editors update on conformance testing for VVC operation range extensions [D. Rusanovskyy, H.-J. Jhu, I. Moccagatta, M. Sarwer, Y. Yu, T. Zhou].
- Discuss organization of the VVCv2 conformance testing:
- Agree on cross-check procedure for VVCv2 and bitstream exchange.
- Complete the list of volunteers for bitstream generation and cross-check.
- Extension of the VVCv2 conformance test set.
Input documents review
Document JVET-X0161 was presented by I. Moccagatta.
No objections to the content were received on the text of the document. The following items were discussed:
- Editorial changes:
- Removal of informational section 6.5.4. No voices were voiced on preserving this section in the text.
- Adding of IBC_E_Tencent bitstream, section 6.6.2.14.5.
- Table numbering update.
- Comments from Summary of voting on ISO/IEC DIS 23090-15 conformance (m57766):
- Comment disposition suggestions from Gary, Iole, Jill and Frank were received and reflected in the DoCR document.
- Further review of the DoCR document was encouraged.
- No reports on corrupted bitstreams or missing coverages were received during the BoG.
The BoG recommended to approve the document JVET-X0161-v2 as basis of the FDIS text. This recommendation was agreed by the JVET plenary in session 21.
The DoCR document was to be reviewed on Friday.
Document JVET-X0185 was presented by Dmytro.
No objections to the content had been received on the text of the document. The following items had been discussed:
- Editorial section of the document, incl. first two tables.
- Collaborative cross-check procedure for VVCv2 conformance testing has been presented and agreed in principle.
- Excel document to maintain of the progress on VVCv2 conformance bitstreams generation was presented (X2026_StatusTable). Document to record following stages of preparation:
- List of categories and bitstreams.
- Volunteers for bitstream generation and cross-check.
- Availability of the bitstream and results of the cross-check, incl. notes and software version.
- It was suggested to include names of PoC in the conformance testing coordination documents.
- Status of the bitstream generation:
- 97 streams had been generated (with 85 bitstreams being submitted to the dropbox).
- No cross-check was conducted yet due to FTP access issues.
- Few categories had been identified as missing of volunteers:
- Sony and KKDI volunteered to generate the bitstreams for those categories.
- 14 bitstreams had been identified as missing cross-checkers:
- Qualcomm was to take 6 additional bitstreams.
- OPPO was to take 4 additional bitstreams.
- 4+ bitstreams were still missing cross-checkers.
- The BoG recommended volunteers to step forward to cover the missing 4.
- Bitstreams description in JVET-X0161:
- The BoG recommended that editing of the conformance to add bitstream description be done collaboratively. Details to be discussed off line. Bitstream description should follow template provided in conformance v2.
- VVCv2 conformance test set extension is needed due to adoption of the JVET-X0128.
The BoG recommended to approve the document JVET-X0185 as the basis for next draft (and CDAM).
The BoG recommended to extend the VVCv2 conformance set by adding a new category on WPP (JVET-X0128).
These recommendations were agreed by the JVET plenary in session 21.
The CDAM request document was under preparation to be available by Friday.
Organization of the VVCv2 conformance testing
VVCv2 collaborative cross-check procedure
The procedure was agreed at the last BoG and implemented in output document JVET-W2026.
This process specifies that volunteers for the bitstream generation will also contribute to the cross-check of the submitted bitstreams from other volunteers. Bitstreams assignment, progress on the generation and cross-check as well as comments (reports) to be recorded in the Excel document maintained by the VVCv2 conformance test coordinators. Goal is to reduce amount of cross-checking efforts of the AHG5 chairs.
The BoG recommended the cross-checking to be done collaboratively.
Two options to organize cross-check and were discussed:
- Option 1: Streams will be uploaded to the JVET drop-box and made available for cross-check by moving from dropbox to under_test.
- Option 2: More efficient to have the cross-check done off-line before uploading to the dropbox.
The BoG recommended Option 2.
After cross-checked streams are uploaded to dropbox, the coordinators will take over.
Note: Iole was requested to delete streams currently in the dropbox as not cross-checked. New bitstreams were to be uploaded after offline cross-check.
The BoG recommended experts who have access to independently decodable decoder to step up and help.
Extension of the conformance test set and SW version.
The following items were discussed:
- JVET-X0128 low-level adoption impacts WPP use case, will not affect existing bitstreams if they generated with WFS disabled.
- HLS adoptions have been assessed as not affect parsing of the existing bitstreams.
- Current bitstreams have been generated by VTM-14.0, as specified in JVET-W2026 do not cover adoptions from this meeting.
- BoG recommended submitting existing streams to the CDAM, and the new adoptions to be reflected in the bitstream generated and cross-checked using the next version of the SW.
Tomonori Hashimoto will contribute to the work on VVC v2 conformance testing and AhG5.
It was suggested to extend the existing list of AhG5 chairs with a larger number of VVCv2 contributing experts.
Software development (0)
Section kept for future use.
Implementation studies (2)
Contributions in this area were discussed in session 13 at 0910–0920 UTC on Monday 11 Oct. 2021 (chaired by JRO).