Abstract not available in document
JVET-O1133 BoG report on PROF/BDOF harmonization contributions (CE4 & CE9 related) [S. Esenlik, H. Yang]
The BoG met in two sessions in order to discuss CE9 (decoder motion vector derivation) related contributions. The sessions were held on 7 July 2019 from 11:30 to 13:45 and from 15:00 to 17:00.
Presented in track B Tuesday 1130-1330
Recommended for adoption:
JVET-O1133 BoG report on PROF/BDOF harmonization contributions (CE4 & CE9 related) [S. Esenlik, H. Yang]
See section 6.9.
List of actions taken affecting the draft text of VVC, the VTM, and 360Lib
The following is a summary, in the form of a brief list, of the actions taken at the meeting that affect the text of the VVC draft text, VTM or 360Lib description. Both technical and editorial issues are included. This list is provided only as a summary – details of specific actions are noted elsewhere in this report and the list provided here may not be complete and correct. The listing of a document number only indicates that the document is related, not that it was adopted in whole or in part. The description given in the “Tool” column is a best effort for the sake of understanding but may not precisely reflect the functionality of the tool. It is also noted that in cases where several contributions proposed the same method, usually only one of the is listed as adoption below; refer to the meeting notes about the adoption to see which other contributions are related.
Category | Sub-Category | Rationale | Tool | Document |
In-loop Filter |
|
|
| |
Deblocking | Subjective quality | Sub-sample MV threshold | ||
Deblocking | Cleanup | Deblocking tC table defined for 10-bit video | ||
Deblocking | Bug fix | Chroma line selection for gradient computation | ||
Deblocking | Subjective quality | Deblocking at 4x4 block boundaries | ||
ALF | Coding efficiency (chroma) | Alternative chroma filters + CTU chroma filter selection | ||
ALF | Simplification | Modification of clipping value signalling | ||
ALF | Simplification | Modification of ALF coefficient signalling | ||
ALF | Cleanup | Save the signalling of the APS index in cases where the number is <= 2 | ||
ALF | Bug fix | Apply VB when the bottom CTU boundary is a slice/tile/brick or “360 virtual” boundary | ||
ALF | Simplification | Modified ALF filtering for Slice, Brick and Virtual boundaries | ||
ALF | Simplification | Simplification of ALF Coefficient Signalling | ||
ALF | Cleanup | Syntax constraints of ALF APS | ||
LMCS | Cleanup/Bug fix | LMCS related clean-ups | ||
LMCS | Simplification | Unification of chroma residual scaling | ||
LMCS | Simplification | Simplified inverse luma mapping | ||
Intra |
|
|
|
|
IBC | Simplification | Single HMVP table inside shared merge list region for IBC | ||
IBC | Simplification | Disabling IBC for chroma in case of dual tree | ||
IBC | Bug fix | IBC mvp flag conditioned on MaxNumMergeCand>1 | ||
IBC | Cleanup | Number of IBC merge candidates independent for P/B slices | ||
IBC | Simplification | Bitstream conformance with a virtual IBC buffer concept | ||
CCLM | Complexity reduction | CCLM restrictions for dualtree to reduce latency | ||
Palette | Coding efficiency (444/SCC) | Base palette mode for 4:4:4 | ||
Block sizes | Complexity reduction | Small chroma block size restrictions for shared tree | ||
Block sizes | Bug fix | MinCbSize >= 8 | ||
ISP | Complexity reduction | 4xN prediction regions for 4xN and 8xN (N > 4) ISP-coded blocks | ||
BDPCM | Bug fix/cleanup | Unified TS and BDPCM signalling | ||
BDPCM | Simplification | Intra prediction mode alignment for BDPCM | ||
PCM | Simplification | Remove PCM | ||
Prediction | Simplification | Small blocks (4x4, 4x8, 8x4) are not smoothed for all angles | ||
Prediction | Simplification | Always apply ‘DCT-IF’ interpolation to ISP | ||
Prediction | Simplification | Intra prediction simplifications | ||
Prediction | Simplification | MRL reference samples for DC | ||
Prediction | Simplification | Signal all 67 intra modes for ISP (MPM, remaining modes); apply PDPC | ||
Prediction | Bug fix | Wide-angle in chroma intra angle mapping table for 4:2:2 | ||
Mode coding | Bug fix | Chroma DM mode is planar in case of IBC | ||
Mode coding | Cleanup | Intra chroma mode coding cleanup | ||
MIP | Simplification | MIP 8-bit coefficient and simplifications | ||
Transform |
|
|
| |
LFNST | Simplification | Simplification of 48x16 LFNST matrices | ||
LFNST | Simplification | LFNST index signalling depends on last position | ||
LFNST | Simplification | Disable LFNST for non-DCT2 MTS candidate | ||
LFNST | Cleanup | Disable LFNST and MIP for implicit MTS | ||
LFNST | Cleanup | LFNST transform set selection for a CCLM | ||
LFNST | Complexity reduction | Limit LFNST up to max TU size | ||
MTS | Cleanup | Context modelling for MTS index | ||
MTS | Cleanup | Context modelling for signalling MTS index | ||
MTS | Bug fix | Decouple intra transform selection from inter MTS related SPS flag | ||
MTS | Bug fix | MTS for IBC | ||
Block sizes | Cleanup | Configurable maximum transform size | ||
Block sizes | Cleanup | Maximum TU size for chroma format 4:2:2 and 4:4:4 | ||
Quantization |
|
|
| |
QP signalling | Bug fix | CU > 64x64, always send cu_qp_delta in first TU | ||
QP signalling | Functionality | Signalling of chroma QP tables | ||
QP signalling | Functionality | CU level chroma QP control | ||
Scaling matrices | Cleanup | Use inter matrices for IBC | ||
Scaling | Bug fix | QP clipping in scaling process for transform skip | ||
CABAC |
|
|
|
|
Coeff. coding | Troughput/worst case | TB-level constraints on context-coded bins | ||
Coeff. coding | Closing an open detail | Joint chroma residual coding with multiple modes | ||
Coeff. coding | Coding efficiency (SCC) | Sign context, level mapping, and bitplane coding for TS residual coding | ||
Coeff. coding | Complexity reduction | 3-pass residual coding for transform skip | ||
Coeff. coding | Cleanup | Exclude coded_subblock_flag in TSRC max ctx coded bin count | ||
Coeff. coding | Cleanup | Ctx derivation of coded block flag | ||
Coeff. coding | Cleanup | Context model reduction for sig_coeff_flag | ||
Coeff. coding | Cleanup | Disallow joint chroma coding for non-I CUs | ||
Coeff. coding | Cleanup | High-level flag for joint chroma coding | ||
Engine | Simplification | CABAC init | ||
Inter |
|
|
|
|
AMVR | Coding efficiency | HPEL AMVR extension with alt. IF | ||
PROF | Coding efficiency | Affine prediction refinement with optical flow | ||
PROF/ | Cleanup | Prediction sample padding unification for BDOF and PROF | ||
PROF/ | SImplification | Unified gradient calculations | ||
BDOF | Simplification | BDOF 4x4 early termination threshold | ||
BDOF | Simplification | Reduction of number of multiplications | ||
DMVR/ | Simplification | disabling DMVR and BDOF for CIIP | ||
DMVR/ | Cleanup | Unify allowed DMVR and BDOF block sizes | ||
DMVR/ | Simplification | Disabled DMVR, BDOF and BCW for CIIP | ||
DMVR | Simplification | modified SAD for the center coordinate | ||
DMVR | Simplification | Simplification for DMVR padding process | ||
Triangle merge | Simplification | MV storage in TPM | ||
Merge syntax | Cleanup | Change merge mode flags syntax tree | ||
SbTMVP | Bug fix | max. num. of subblock merge cand. Signalling | ||
SbTMVP | Cleanup | Remove switching between L0 and L1 for temporal MV | ||
AMVP | Simplification | remove spatial AMVP cand scaling | ||
Affine / AMVR | Cleanup | SPS affine AMVR control flag | ||
Affine / BCW | Simplification | copy BCW index from first spatial affine merge candidate | ||
SMVD | Cleanup | SMVD value range | ||
SMVD | Cleanup | condition sym_mvd_flag on mvd_l1_zero_flag | ||
SMVD | Simplification | SMVD only for STRPs | ||
SMVD | Bug fix | Add missing initialization | ||
MVD | Cleanup | Extent MVD range to [-2^17, 2^17-1] | Plenary | |
BDOF | Simplification | Simplification of BDOF | ||
BDOF | Cleanup | Unified syntax for JVET-O0184/O0250/O0504 on DMVR and BDOF flag | ||
Partitioning |
|
|
| |
Chroma formats | Cleanup | Do not signal SH chroma tool flags in case of 4:0:0 | ||
Coding tree | Bug fix | Store CuPredMode[x][y], cqtDepth[x][y], CbWidth[x][y], CbHeight[x][y], CbPosX[x][y] and CbPosY[x][y] for separate trees | ||
CTU size | Simplification | Increase Min CTU size to 32x32 | ||
HLS |
|
|
|
|
VUI | Bug fix | update BT.2100 references and ICTCP HLG coding equations | ||
RPR | Functionality | Integrated Specification Text for Reference Picture Resampling: including elements of the following documents: JVET-O0133, JVET-O0134, JVET-O0182, JVET-O0204, JVET-O0242, JVET-O0303, JVET-O0319, and JVET-O0395. | ||
High level syntax decision from BoG report JVET-O1076 review | notes on JVET-O1076 | |||
Software |
|
|
|
|
Pre-filter | Coding efficiency | Encoder-only GOP-based temporal filter | ||
Triangle merge | Speedup | Modified mode decision process for triangle mode | ||
Triangle merge | Speedup | SIMD for triangle merge mode | ||
Subblock merge | Bug fix | Parse merge_subblock_flag conditioned on MaxNumSubblockMergeCand | ||
Subblock merge | Bug fix | Different ctx models (but same derivation) for inter affine flag and subblock merge flag | ||
Palette mode | Bug fix | 4:4:4 low QP bug fix | ||
Motion search | Coding efficiency | Motion estimation improvements | ||
MVD | Bug fix | MVD clipping | ||
BCW | Bug fix | BCW index binarization | ||
LMCS | Bug fix | CRS luma lambda computation | ||
LMCS | Coding efficiency , simplification | LMCS encoder improvement | ||
MTS | Bug fix | SPS control of ISP and SBT transform selection | ||
PROF | Bug fix | Align bit-shift operation with BDOF | ||
Coeff. coding | Bug fix | Align SW to text regarding position of chroma CBFs for SBT TUs | ||
Dep. quant. | Speedup | Fast encoder with adjusted threshold | ||
QP signalling | Functionality | Chroma QP table signalling | ||
IBC | IBC virtual buffer in decoder SW | |||
Project planning
Core experiment planning
Core experiment plans were established, and initial drafts of the plans were reviewed at the meeting.
Drafting of specification text, encoder algorithm descriptions, and software
The following agreement has been established: the editorial team has the discretion to not integrate recorded adoptions for which the available text is grossly inadequate (and cannot be fixed with a reasonable degree of effort), if such a situation hypothetically arises. In such an event, the text would record the intent expressed by the committee without including a full integration of the available inadequate text.
Plans for improved efficiency and contribution consideration
The group considered it important to have the full design of proposals documented to enable proper study.
Adoptions need to be based on properly drafted working draft text (on normative elements) and HM encoder algorithm descriptions – relative to the existing drafts. Proposal contributions should also provide a software implementation (or at least such software should be made available for study and testing by other participants at the meeting, and software must be made available to cross-checkers in EEs).
Suggestions for future meetings included the following generally supported principles:
- No review of normative contributions without draft specification text
- VTM algorithm description text is strongly encouraged for non-normative contributions
- Early upload deadline to enable substantial study prior to the meeting
- Using a clock timer to ensure efficient proposal presentations (5 min) and discussions
The document upload deadline for the next meeting was planned to be Tuesday 24 September 2019.
As general guidance, it was suggested to avoid usage of company names in document titles, software modules etc., and not to describe a technology by using a company name.
General issues for experiments
It was emphasized during the opening plenary on January 9 that those rules which had been set up or refined during the 12th meeting should be observed. In particular, for some CEs, results were available late, and some changes in the experimental setup (particularly in CE4) were not discussed on the JVET reflector.
Group coordinated experiments have been planned as follows:
- “Core experiments” (CEs) are the coordinated experiments on coding tools which are deemed to be interesting but require more investigation and could potentially become part of the draft standard by the next meeting.
- A CE is a test of a specific fully described technology in a specific agreed way. It is not a forum for thinking of new ideas (like an AHG). The CE coordinators are responsible for making sure tha the CE description is complete and correct and has adequate detail. Reflector discussions about CE description clarity and other aspects of CE plans are encouraged.
- A description of each experiment is to be approved at the meeting at which the experiment plan is established. This should include the issues that were raised by other experts when the tool was presented, e.g., interference with other tools, contribution of different elements that are part of a package, etc. The experiment description document should provide the names of individual people, not just company names.
- Software for tools investigated in a CE will be provided in one or more separate branches of the software repository. Each CE will have a “fork” of the software, and within the CE there may be multiple branches established by the CE coordinator. The software coordinator will help coordinate the creation of these forks and branches and their naming. All JVET members have read access to the CE software branches (using shared read-only credentials as described below).
- During the experiment, revisions of the experiment plans can be made, but not substantial changes to the proposed technology.
- The CE description must match the CE testing that is done. The CE description needs to be revised if there has been some change of plans.
- The CE summary report must describe any changes that were made in the process of finalizing the CE.
- By the next meeting it is expected that at least one independent cross-checker will report a detailed analysis of each proposed feature that has been tested and confirm that the implementation is correct. Commentary on the potential benefits and disadvantages of the proposed technology in cross-checking reports is highly encouraged. Having multiple cross-checking reports is also highly encouraged (especially if the cross-checking involves more than confirmation of correct test results). The reports of cross-checking activities may (and generally should) be integrated into the CE report rather than submitted as separate documents.
It is possible to define sub-experiments within particular CEs, for example designated as CEX.a, CEX.b, etc., where X is the basic CE number.
As a general rule, it was agreed that each CE should be run under the same testing conditions using one software codebase, which should be based on the group test model software codebase. An experiment is not to be established as a CE unless there is access given to the participants in (any part of) the CE to the software used to perform the experiments.
The general agreed common conditions for SDR single-layer coding efficiency experiments are described in the prior output document JVET-N1010.
Experiment descriptions should be written in a way such that it is understood as a JVET output document (written from an objective “third party perspective”, not a proponent perspective – e.g. not referring to methods as “improved”, “optimized”, etc.). The experiment descriptions should generally not express opinions or suggest conclusions – rather, they should just describe what technology will be tested, how it will be tested, who will participate, etc. Responsibilities for contributions to CE work should identify individuals in addition to company names.
CE descriptions contain a basic description of the technology under test, but should not contain excessively verbose descriptions of a technology (at least not unless the technology is not adequately documented elsewhere). Instead, the CE descriptions should refer to the relevant proposal contributions for any necessary further detail. However, the complete detail of what technology will be tested must be available – either in the CE description itself or in documents that are referenced in the CE description that are also available in the JVET document archive.
Any technology must have at least one cross-check partner to establish a CE – a single proponent is not enough. It is highly desirable have more than just one proponent and one cross-checker.
For each CE, a group is created in GitLab by the software coordinators, and CE coordinators are given owner rights to the group. This way they can clone VTM as required, create branches for different tests and assign user access to the group themselves.
The CE software development workflow is described at:
https://vcgit.hhi.fraunhofer.de/jvet/VVCSoftware_VTM/wikis/Core-experiment-development-workflow
CE read access is available using shared accounts: One account exists for MPEG members, which uses the usual MPEG account data (as announced on the appropriate email lists). A second account exists for VCEG members. The account information for VCEG members is available in the TIES system:
https://www.itu.int/ifa/t/2017/sg16/exchange/wp3/q06/vceg_account.txt
Some agreements relating to CE activities were established as follows:
- Only qualified JVET members can participate in a CE.
- Participation in a CE is possible without a commitment of submitting an input document to the next meeting. Participation is requested by contacting the CE coordinator.
- All software, results, and documents produced in the CE should be announced and made available to JVET in a timely manner.
- A JVET CE reflector will be established and announced on the main JVET reflector. Discussion of logistics arrangements, exchange of data, minor refinement of the test plans, and preparation of documents shall be conducted on the JVET CE reflector, with subject lines prefixed by “[CEx: ]”, where “x” is the number of the CE. All substantial communications about a CE other than such details shall take place on main JVET reflector. In the case that large amounts of data are to be distributed, it is recommended to send a link to the data rather than the data itself, or upload the data as an input contribution to the next meeting.
General timeline for CEs
Reviewed Thursday 11 July 2019 at 1900 (chaired by Y. Ye).
T1= 3 weeks after the JVET meeting: To revise the CE description and refine questions to be answered. Questions should be discussed and agreed on JVET reflector. Any changes of planned tests after this time need to be announced and discussed on the JVET reflector. Initially assigned description numbers shall not be changed later. If a test is skipped, it is to marked as “withdrawn”.
T2 = Test model software release + 2 weeks or X XX, whichever is earlier: Integration of all tools into a separate CE branch of the VTM is completed and announced to JVET reflector.
- Initial study by cross-checkers can begin.
- Proponents may continue to modify the software in this branch until T3
- 3rd parties are encouraged to study and make contributions to the next meeting with proposed changes
T3: 3 weeks before the next JVET meeting or T2 + 1 week, whichever is later: Any changes to the CE test branches of the software must be frozen, so the cross-checkers can know exactly what they are cross-checking. A software version tag should be created at this time. The name of the cross-checkers and list of specific tests for each tool under study in the CE plan description shall be documented in an updated CE description by this time.
T4: Regular document deadline – 1 week: CE contribution documents including specification text and complete test results shall be uploaded to the JVET document repository (particularly for proposals targeting to be promoted to the draft standard at the next meeting).
The CE summary reports shall be available by the regular deadline. This shall include documentation about crosscheck of software, matching of CE description and confirmation of the appropriateness of the text change, as well as sufficient crosscheck results to create evidence about correctness (crosscheckers must send this information to the CE coordinator at least 3 days ahead of the document deadline). Furthermore, any deviations from the timelines above shall be documented. The numbers used in the summary report shall not be changed relative to the description document.
CE reports may contain additional information about tests of straightforwared combinations of the identified technologies. Such supplemental testing needs to be clearly identified in the report if it was not part of the CE plan.
New branches may be created which combine two or more tools included in the CE document or the VTM (as applicable).
It is not necessary to formally name cross-checkers in the initial version of the CE description document. To adopt a proposed feature at the next meeting, we would like see comprehensive cross-checking done, with analysis that the description matches the software, and recommendation of value of the tool given tradeoffs.
The establishment of a CE does not indicate that a proposed technology is mature for adoption or that the testing conducted in the CE is fully adequate for assessing the merits of the technology, and a favourable outcome of CE does not indicate a need for adoption of the technology.
Availability of spec text is important to have a detailed understanding of the technology and also to judge what its impact on the complexity of the spec will be. There must also be sufficient time to study it in detail. CE contributions without sufficiently mature draft spec text in the CE input document should not be considered for adoption.
Plans for the CEs to be conducted were established Thursday 11 July (chaired by Y. Ye); CE plan documents were reviewed Thursday 11 July (chaired by Y. Ye).
Lists of participants in CE documents should be pruned to include only the active participants. Read access to CE software will be available to all members.
Software development and anchor generation
The planned timeline for software releases was established as follows:
- VTM6.0 was planned to be released by 2019-08-12, including all adoptions necessary for CTC and CE basis references. VTM6.1 with non-CTC adoptions were to be released later. Further versions of VTM may be released for additional bug fixing, as appropriate.
- Preparation of the VTM software will include immediate removal of macros that were added in the previous meeting cycle. The software coordinator has the discretion to retain some such macros.
- No change of of 360lib was noted in response to this meeting.
- See JVET-O0756 regarding a recent enhancement of HDRTools.
Establishment of ad hoc groups
The ad hoc groups established to progress work on particular subject areas until the next meeting are described in the table below. The discussion list for all of these ad hoc groups was agreed to be the main JVET reflector (jvet@lists.rwth-aachen.de).
Title and Email Reflector | Chairs | Mtg |
Project Management (AHG1)
| J.-R. Ohm, G. J. Sullivan (co-chairs) | N |
Draft text and test model algorithm description editing (AHG2)
| B. Bross, J. Chen (co-chairs), J. Boyce, S. Kim, S. Liu, Y.-K. Wang, Y. Ye (vice-chairs) | N |
Test model software development (AHG3)
| F. Bossen, X. Li, K. Sühring (co-chairs) | N |
Test material and visual assessment (AHG4)
| T. Suzuki, M. Wien (co-chairs), V. Baroncini, R. Chernyak, A. Norkin, J. Ye (vice-chairs) | N |
Memory bandwidth consumption of coding tools (AHG5)
| R. Hashimoto (chair), T. Ikai, X. Li, D. Luo, H. Yang, M. Zhou (vice-chairs) | N |
360° video coding tools, software and test conditions (AHG6)
| J. Boyce and Y. He (co-chairs), K. Choi, J.-L. Lin, Y. Ye (vice-chairs) | N |
Coding of HDR/WCG material (AHG7)
| A. Segall (chair), E. François, W. Husak, S. Iwamura, D. Rusanovskyy (vice-chairs) | N |
Layered coding and resolution adaptivity (AHG8)
| S. Wenger and A. Segall (co-chairs), M. M. Hannuksela, Hendry, S. McCarthy, Y.-C. Sun, P. Topiwala, M. Zhou (vice-chairs) | N |
Neural networks in video coding (AHG9)
| S. Liu and Y. M. Li (co-chairs), B. Choi, K. Kawamura, Y. Li, L. Wang, P. Wu, H. Yang (vice-chairs) | N |
Encoding algorithm optimization (AHG10)
| A. Duenas, A. Tourapis (co-chairs), S. Ikonin, A. Norkin, R. Sjöberg, J. Le Tanou, J.-M. Thiesse (vice-chairs) | N |
Screen content coding (AHG11)
| S. Liu (chair), J. Boyce, A. Filippov, Y.-C. Sun, J. Xu, H. Yang (vice-chairs) | N |
High-level parallelism and coded picture regions (AHG12)
| S. Deshpande (chair), M. M. Hannuksela, R. Sjöberg, R. Skupin, W. Wan, Y.-K. Wang S. Wenger (vice-chairs) | N |
Tool reporting procedure (AHG13)
| W.-J. Chien, J. Boyce (co-chairs), W. Chen, Y.-W. Chen, R. Chernyak, K. Choi, R. Hashimoto, Y.-W. Huang, H. Jang, R.-L. Liao, S. Liu (vice-chairs) | N |
Operation modes for low latency support (AHG14)
| J.-M. Thiesse (chair), S. Deshpande, A. Duenas, Hendry, K. Kazui, R. Sjöberg, A. Tourapis (vice-chairs) | N |
Quantization control (AHG15)
| R. Chernyak (chair), E. François, C. Helmrich, S. McCarthy, A. Segall (vice-chairs) | N |
Implementation studies (AHG16)
| M. Zhou (chair), J. An, E. Chai, K. Choi, S. Sethuraman, T. Hsieh, X. Xiu (vice-chairs) | N |
High-level syntax (AHG17)
| R. Sjöberg, J. Boyce (co-chairs), S. Deshpande, M. M. Hannuksela, R. Skupin, A. Tourapis, Y.-K. Wang, W. Wan, S. Wenger (vice-chairs) | N |
Lossless and near-lossless coding tools (AHG18)
| T. Nguyen and T.-C. Ma (co-chairs), M. Ikeda, S. Iwamura, H. Jang, X. Zhao (vice-chairs) | N |
Output documents
The following documents were agreed to be produced or endorsed as outputs of the meeting. Names recorded below indicate the editors responsible for the document production. Where applicable, dates of planned finalization and corresponding parent-body document numbers are also noted.
It was reminded that in cases where the JVET document is also made available as MPEG output document, a separate version under the MPEG document header should be generated. This version should be sent to GJS and JRO for upload.