JVET-AQ0202 AHG17: Suggested initial text of "Template for Proposal Description Documents of Responses to the Joint CfP on Video Compression with Capability beyond VVC" [J.-R. Ohm, M. Wien]
This document was developed based on JVET-I1003 (Template for Proposal Description Documents for Responses to the Joint CfP on Video Compression with Capability beyond HEVC). For convenience, a version with changemarks of that previous document is included in the zip container. It is suggested to include JVET-AO2040 (complexity reporting template) or an update thereof in the proposal description package template, which allows removal of two previous annexes. A first version of the proposal description template should be issued as an output document from the 43rd meeting, with the option of further updating into a final version by the time of the 44th meeting. It is further suggested not to attach the .csv files as part of the proposal description packages, as those are mandatory to be provided by proponents when submitting results, and can be made available to JVET members (in case that proponents agree or if the proposal is selected to be considered for further investigation in JVET) by the time of the 45th meeting. This would also require some further updates of wording in the final CfP JVET-AQ2027.
Initial discussion July 7 1700-1840 (chaired by JRO and MW)
It was expressed that it should be left to proponents if they want to include results from the main and supplemental submission packages. On the other hand, due to the cross-checking of csv files, it would be impossible to keep proponents anonymous.
It was also commented that including results to some extent is useful to get the full image about a proposal, but not to the detail that the csv files give.
Another important aspect that needs to be clarified (in the CfP): To which extent results will be made publicly available (agreement of proponents, at which point will proposals be identified, only subjective results?), and what will be done with the data collected in the csv files.
The following was agreed:
It is asserted to be important that proposals can be judged by both subjective and objective results, e.g. if subjective is not available for all encoder run times, or compare performance for both main and supplemental submission. The best solution for this is that the input report contains both without associating proposal numbers with description document numbers.
The individual objective results of a given proposal in the form they would be reported could be provided to proponents early enough to allow them a decision if they want to attach it to their description. At this moment, the proposal will no longer be anonymous.
Follow-up discussion on Wednesday 8 July, JVET-AQ0202v2 was presented that reflects the previous discussion by modifying section 4 in alignment with the newest version of the CfP; some wording improvements were made. In the discussion, it was further agreed that it is not practical to include submission of additional functionality as part of the main submission package as test conditions cannot be defined beforehand. Instead reporting of results on that functionality was in section 4.2.
Regarding sections 5 and 6 (decoder/encoder complexity analysis) it was suggested to better integrate the content of JVET-AO2040 word document, sections/tables on codec-level complexity analysis and module-level complexity analysis, investigate whether aspects need to be added for encoder complexity assessment from section 6. Furthermore, an Excel reporting template should be developed similar as the Excel from JVET-AO2040 for VVC module-level, to be filled by proponents containing
- 1 sheet on codec-level analysis, without run time that is available from .csv
- 1 sheet on module-level for decoder, with spare columns for additional modules
- 1 sheet on module-level for encoder, listing some default modules such as motion estimation, quantization, filter coefficients derivation, RDO for mode decision, etc.; this could be filled separately for the different test cases.
X. Li was asked to integrate a correspondingly extended version of JVET-AO2040 into JVET-AQ0202 in coordination with M. Wien for follow-up discussion.
Further edits on sections 2 and 3 were done in the subsequent discussion to add more detail on what should be described in the different sections. As it was not practical to do this in the large group, a side activity (P. de Lagrange, X. Li, J. Gan, R. Chernyak, E. Alshina, F. Bossen) was tasked to further continue on top of JVET-AQ0202v3, and also coordinating with the complexity reporting template prepared by X. Li.
The result was presented on Thursday 9 July at 1420-1530. So far, mostly section 2 was edited.
It was agreed that a better separation of sections 2.4 and 2.5 would be such that only elements that make use of reconstructed samples or need to generate prediction samples to determine parameters are subsumed under the latter.
Re-structuring, adding of missing sections, or removal of unnecessary sections should be done as appropriate.
A first version of integrating the complexity reporting template was presented, including a table with parameters for encoder complexity. As it is expected that for lower encoder complexity also the decoder module complexity and codec-level complexity may become lower (e.g. by disabling tools), a separate complexity reporting template of all three sheets should be submitted for each test case. It was suggested to include some mechanism to easily identify the differences (e.g. differential sheet, color marking?).
It was suggested that the example complexity reporting template might also contain some aspects of ECM or NNVC modules, if modules of such type are not available in VTM.
The side activity was mandated to continue after the closing of the main session on Thursday, and met until 1800. The result was made available in JVET-AP0202v4 and was presented at 1600 in the main session before lunch break on Friday.
Further edits were made based on the discussion, with result uploaded as JVET-AP0202v5. Filling the subsections in 4 (encoder description) and section 7 (software implementation) still need to be filled with text description. Furthermore, some improvements on the complexity analysis (sections 5 and 6 plus spreadsheet) were deemed necessary (e.g., including internal bitdepth/register length, integer/float, etc.). It was requested to continue the side activity over the weekend to make the draft more complete, and conduct a follow-up review in JVET main track on Monday morning (to be chaired by F. Bossen).
Follow-up review on July 13 0900-1025 (chaired by FJB)
Further edits were made during the review of JVET-AP0202v6 (main document and attached spreadsheet). Most edits were editorial in nature. Clarifications were added, for example in section 4.3 where reporting of encoder optimization performance could also be made by disabling optimizations in the proposal. A new version JVET-AP0202v7 is to be uploaded.
JVET-AQ0202v7 was finally reviewed in the July 13th 16:00h session (chaired by M. Wien). The edits made over the weekend and in the morning were presented. On the decoder run time measurement, the activation of writing YUV at the decoder in order to assert full operation of post processing tools. It was confirmed that the VTM would operated accordingly in case the output is directed to /dev/null.