JVET-C0104 Report of BoG on test material [T. Suzuki]

The BoG on test material selection was generated with following mandates:

• Review the class A1/A2 selection made by the last meeting, and propose possible changes
• Establish work plan towards the next meeting for investigating the 1080p sequences (with the goal to establish new class or replace class B by the next meeting
• Summarize the material offered for VR, identify whether it covers all common methods of projection/rendering/stitching, and discuss possible methods of quality assessment

For the VR material, it should also be discussed with parent bodies how to coordinate the different activities in this area. The work of JVET should not be dominated by VR.

The new material for screen content should be brought to the attention of JCT-VC, and could be used in the SCC verification test. Currently, the development of higher compression technology specifically for screen content is not in the focus of JVET.

This document reports the results of the BoG discussion.

Summary of activities

The BoG met at Room C2 (ITU-T building) between 2:00 PM – 6:00 PM, May.29 2016. The BoG reviewed the following contributions. The discussion for each contribution is summarized in the Annex of the BoG report.

• JVET-C0021 GoPro test sequences for Virtual Reality video coding [A.Abbas (GoPro)]
• JVET-C0028 Suggested 1080P Test Sequences Downsampled from 4K Sequences H. Zhang, X. Ma, H. Yang (Huawei)]
• JVET-C0029 Surveillance sequences for video coding development [H. Zhang, X. Ma, H. Yang (Huawei), W.Qiu (Hisilicon)]
• JVET-C0033 On comparison criteria for Virtual Reality video coding schemes [E. Thomas (TNO)]
• JVET-C0041 Proposed test sequences for 1080p class [A. Norkin (Netflix)]
• JVET-C0044 Response to B1002 Call for test materials: Five test sequences for screen content video coding [J. Guo, L. Zhao, T. Lin (Tongji Uni.),
• H. Yu (Futurewei)]
• JVET-C0048 Lens distorted test sequence by an action camera for future video coding [K. Kawamura, S. Naito (KDDI Corp.)]
• JVET-C0050 Test sequence formats for virtual reality video coding [K. Choi, V. Zakharchenko, M. Choi, E. Alshina (Samsung)]
• JVET-C0064 Nokia test sequences for virtual reality video coding [J. Ridge, M. M. Hannuksela (Nokia)]
• JVET-C0067 Ultra High Resolution (UHR) 360 Video [C. J. Murray (Panoaction)]

VR test sequences:

• JVET-C0021 (GoPro): 9 VR test sequences are proposed. For each sequence, both equirectangular and cube-4x3 formats are provided. All contents proposed at this meeting are compressed. Those are captured as a compressed bitstream (AVC). But the proponent can provide an uncompressed version. Until uncompressed files are available, the bitstreams at the above sites can be used to understand the nature of VR test sequences.
• JVET-C0064 (Nokia): This contribution offers the stereoscopic equirectangular panorama sequences. The camera-captured sequences are several tens of seconds long, out of which Nokia is willing to provide 10-second excerpts (with start points as agreed by JVET) for standardization. There was a question on the possibility to provide whole portion of video, and the proponent needs to confirm the permission to provide complete sequences. The proponent can provide original fish eye content (8 cameras). The resolution of each camera video is 2Kx2K (8 bit)
  • Proponent will select 10 sec for each sequences and will upload to the ftp site. Availability of complete sequences for original fish eye camera need to be confirmed.
• JVET-C0067 (Panoaction): No presentation. 8kx4k and 14x7k equirectangular sequences are proposed.
• JVET-C0033 (TNO): This is not a proposal of test material, but proposes to consider spatial random access for the evaluation of virtual reality test sequences. There are several representation of omnidirectional video, e.g. cube, tile, etc). Full HD per view point.
• JVET-C0050 (Samsung): This is not a proposal of test material, but proposes representations of virtual reality test sequences (equirectangular, cube and icosahedron format). Those were converted from the equirectangular test sequences proposed in JVET-C0021. From test sequence perspective, proposed format can be converted from equirectangular sequences, Proponent can provide conversion tools, for further testing.

BoG recommendations:

• To keep all proposed test sequences as candidates of test sets
• To continue to study by AHG. AHG will investigate the following issues
  • Study format of VR sequences to decide test conditions
    • Study of format also includes the study of random access (both temporal and spatial), low delay, etc.
  • Evaluation method
  • Test conditions
• VR test sequences
  • Equirectangular sequences maybe sufficient at this moment. Tool can convert into other formats
  • Some companies may have camera original (video before stitching) and those are also useful for further study
  • FTP site has sufficient to space to upload all sequences.
    • All portions of sequences can be uploaded, and then discuss appropriate portion by the next meeting.
    • Nokia needs to confirm whether it is allowed to upload full length
    • Nokia will pre-select an appropriate portion (uncompressed), and then upload
    • GoPro test sequences: only compressed sequences are available now. At the next meeting, uncompressed sequences will be available.
• To discuss how we should proceed on VR
  • Scope of JVET on VR issues (was clarified in joint meeting of parent bodies)

From follow-up discussion in JVET:

• It was verbally expressed that the sequences from JVET-C0064 will be provided with acceptable licensing conditions (at least similar to previous cases). This will apply to 10s excerpts from the sequences.
• Among the three contributions on VR material, C0064 provides stereo, the other two (GoPro, Panoaction) are monoscopic.
• Evaluation method for VR sequences should be further studied in AHG. For example, PSNR could be measured after backprojection from equirectangular to 2D.

4K test sequences

BoG recommendations:

• To study 4K sequences proposed in C0029 and some test sequences from the test sets studied in the last meeting.
• The following sequences are pre-selected during BoG and are to be studied further by the next meeting as a replacement of the current class A sequences. (5 sequences)
  • ParkRunning1
  • BuildingHall
  • CrossRoad1
  • Runners (from previous test sets)
  • Crosswalk (from previous test sets)

From the follow-up discussion in JVET

• No clear opinion exists whether some of the test material in class A1/A2 is inappropriate
• It is planned to generate test cases for the A1/A2 classes and the 5 sequences listed above, HM/JEM at approximately same bit rates (Possible additional sequences: Market 2, Time lapse)
  • An initial idea for rates in class A 60 Hz could be 2.5, 4, 7, 12, 18 for RA
  • If impossible to reach for certain sequences, extend the range appropriately
  • For 30 Hz scale down by 1.5, etc.
• Perform subjective viewing at the next meeting, for assessing the subjective quality in terms of appropriateness for a formal subjective viewing and selecting appropriate rate points
• This material could also be used in the context of AHG5 for investigating objective metrics, likely after the next meeting.

1080p test sequences

BoG recommendations:

• To study 1080p sequences proposed in C0028, C0041 and C0048.
  • Encoding time is less than class A and all proposed sequences can be tested.
• To study further on the design of test classes.
  • New test class can be considered. The followings were discussed
• Not necessary to restrict current class B number of sequences
• New class B with 10 bit
• New class for specific applications, e.g. surveillance
• New class for special features of content (wind & nature, toddler fountain, complex motion, water, complex texture (grass), etc)
• New class for smaller picture size (smaller than 1080p)
• New class could be optional (e.g. for subjective test)
• The number of test sequences is
  • 12 sequences from C0028 (up to 600 frames. Huawei will provide which part should be used)
  • 9 sequences from C0041
  • 1 sequences from C0048
  • Totally 22 sequences for 1080p

HDR test sequences

BoG recommendations:

• To study further if HDR test sequences can be added.
  • Technicolor and Netflix can provide HDR sequences for JVET.
• To study evaluation methods before including HDR sequences.
• AHG should study evaluation methods and then discuss at the next meeting if HDR sequences can be added.

A workplan document was issued as an output. Its table with volunteers is to be filled in by contacting T. Suzuki.

List of actions taken affecting the JEM3

The following is a summary, in the form of a brief list, of the actions taken at the meeting that affect the text of the JEM3 description. Both technical and editorial issues are included. This list is provided only as a summary – details of specific actions are noted elsewhere in this report and the list provided here may not be complete and correct. The listing of a document number only indicates that the document is related, not that it was adopted in whole or in part.

• • Encoder only or software changes
    • JVET-C0037
      • IRAP-level parallel encoding with precise bitstream concatenation
    • JVET-C0046
      • Bug fixes for enabling of RExt tools in JEM
  • Syntax/semantics/decoding process changes
    • JVET-C0024
      • QTBT replaces quadtree in main branch of JEM3
    • JVET-C0025
      • Simplification/unification of MC filters for affine prediction
    • JVET-C0027
      • Simplification/improvement of BIO
    • JVET-C0035
      • ATMVP simplification
    • JVET-C0038
      • Modifications of ALF: Diagonal classification, geometric transformations of filters, prediction of coefficients from fixed set, alignment of luma and chroma filter shapes, removal of context coded bins for filter coefficient signalling
    • JVET-C0042/JVET-C0053
      • unified binarization of NSST index
    • JVET-C0046
      • Enabling TS with 64x64 transform blocks
    • JVET-C0055
      • Simplified derivation of MPM in intra prediction
    • NSST & TS
      • Disable NSST and do not code NSST index if all components in a block use TS; otherwise, if NSST is on, it shall not be used for a block of a component that uses TS.

Project planning

JEM description drafting and software

The following agreement has been established: the editorial team has the discretion to not integrate recorded adoptions for which the available text is grossly inadequate (and cannot be fixed with a reasonable degree of effort), if such a situation hypothetically arises. In such an event, the text would record the intent expressed by the committee without including a full integration of the available inadequate text.

Plans for improved efficiency and contribution consideration

The group considered it important to have the full design of proposals documented to enable proper study.

Adoptions need to be based on properly drafted working draft text (on normative elements) and JEM encoder algorithm descriptions – relative to the existing drafts. Proposal contributions should also provide a software implementation (or at least such software should be made available for study and testing by other participants at the meeting, and software must be made available to cross-checkers in CEs).

Suggestions for future meetings included the following generally-supported principles:

• No review of normative contributions without draft specification text
• JEM text is strongly encouraged for non-normative contributions
• Early upload deadline to enable substantial study prior to the meeting
• Using a clock timer to ensure efficient proposal presentations (5 min) and discussions

The document upload deadline for the next meeting was planned to be Wednesday 5 October 2016.

As general guidance, it was suggested to avoid usage of company names in document titles, software modules etc., and not to describe a technology by using a company name.

General issues for Experiments

Group coordinated experiments have been planned. These may generally fall into one category:

• "Exploration experiments" (EEs) are the coordinated experiments on coding tools which are deemed to be interesting but require more investigation and could potentially become part of the main branch of the JEM by the next meeting.
• A description of each experiment is to be approved at the meeting at which the experiment plan is established. This should include the issues that were raised by other experts when the tool was presented, e.g., interference with other tools, contribution of different elements that are part of a package, etc. (E. Alshina will edit the document based on input from the proponents, review is performed in the plenary)
• Software for tools investigated in EE is provided in a separate branch of the software repository
• During the experiment, further improvements can be made
• By the next meeting it is expected that independent parties (at least one at a minimum) will report a detailed analysis about the tool, confirm that the implementation is correct, and give reasons to include (or not include) the tool in JEM
• As part of the experiment description, it should be captured whether performance relative to the JEM as well as the HM (with all other tools of the JEM disabled) should be reported by the next meeting.

It is possible to define sub-experiments within particular EEs, for example designated as EEX.a, EEX.b, etc., where X is the basic EE number.

As a general rule, it was agreed that each EE should be run under the same testing conditions using one software codebase, which should be based on the JEM software codebase. An experiment is not to be established as an EE unless there is access given to the participants in (any part of) the EE to the software used to perform the experiments.

The general agreed common conditions for single-layer coding efficiency experiments are described in the output document JVET-B1010.

Experiment descriptions should be written in a way such that it is understood as a JVET output document (written from an objective "third party perspective", not a company proponent perspective – e.g. referring to methods as "improved", "optimized" etc.). The experiment descriptions should generally not express opinions or suggest conclusions – rather, they should just describe what technology will be tested, how it will be tested, who will participate, etc. Responsibilities for contributions to EE work should identify individuals in addition to company names.

EE descriptions should not contain excessively verbose descriptions of a technology (at least not unless the technology is not adequately documented elsewhere). Instead, the EE descriptions should refer to the relevant proposal contributions for any necessary further detail. However, the complete detail of what technology will be tested must be available – either in the CE description itself or in referenced documents that are also available in the JVET document archive.

Any technology must have at least one cross-check partner to establish an EE – a single proponent is not enough. It is highly desirable have more than just one proponent and one cross-checker.

Some agreements relating to EE activities were established as follows:

• Only qualified JVET members can participate in an EE.
• Participation in an EE is possible without a commitment of submitting an input document to the next meeting.
• All software, results, documents produced in the EE should be announced and made available to all EE participants in a timely manner.

Further discussed Tuesday AM, chaired by JRO and J. Boyce.

A separate branch under the experimental section will be created for each new tool include in the EE. The proponent of that tool is the gatekeeper for that separate software branch. (This differs from the main branch of the JEM, which is maintained by the software coordinators.)

New branches may be created which combine two or more tools included in the EE document or the JEM. Requests for new branches should be made to the software coordinators.

Don’t need to formally name cross-checkers in the EE document. To promote the tool to the JEM at the next meeting, we would like see comprehensive cross-checking done, with analysis that the description matches the software, and recommendation of value of the tool given tradeoffs.

Timeline:

T1 = JEM3.0 SW release + 4 weeks: Integration of all tools into separate EE branch of JEM is completed and announced to JVET reflector.

Initial study by cross-checkers can begin.

Proponents may continue to modify the software in this branch until T2

3^rd parties encouraged to study and make contributions to the next meeting with proposed changes

T2: JVET-D meeting start – 3 weeks: Any changes to the exploration branch software must be frozen, so the cross-checkers can know exactly what they are cross-checking. An SVN tag should be created at this time and announced on the JVET reflector.

This procedure was again confirmed during the closing plenary of the third JVET meeting. It was further confirmed that the Common Test Conditions of JVET-B1010 are still valid, however the CTC encoder setting will be reflected in the config file that is attached to the JEM3.0 package.

Software development

Software coordinators will work out the detailed schedule with the proponents of adopted changes.

Any adopted proposals where software is not delivered by the scheduled date will be rejected.

The planned timeline for software releases was established as follows:

• JEM3.0 including all adoptions from section 9.4 will be released within 3 weeks (2016-06-24)
• JEM3.1 including adaptive QP will be released 2 weeks later.
• The results about coding performance of JEM3.0 will be reported by 2016-07-07

Output documents and AHGs

The following documents were agreed to be produced or endorsed as outputs of the meeting. Names recorded below indicate the editors responsible for the document production.