JVET-AG0331 BoG on gaming content compression [S. Puri, J. Sauer, R. Chernyak, A. Duenas, L. Wang]

This report summarizes the discussion during the BoG on gaming content compression established during the JVET-AG meeting.

The first BoG session was conducted on Jan. 24, 11:20-12:20 (UTC).

AHG15 on gaming content compression was established at the 32^nd JVET meeting. Mandates of this ad hoc group were:

• Identify gaming content application scenarios and their requirements for codec operation.
• Identify and characterize required types of content; solicit contributions, collect, and make a variety of gaming content available, in coordination with AHG4 and AG 5.
• Propose test conditions appropriate for gaming applications.
• Evaluate JVET test models (such as ECM, VTM, NNVC, etc.) under the proposed test conditions.
• Investigate possibilities to enhance compression capability for gaming content.

Testing conditions for gaming content

This section describes simulations’ setup that was used within AhG15 prior to the meeting.

Anchor: VTM-11ecm11 (https://vcgit.hhi.fraunhofer.de/ecm/ECM/-/tree/VTM-11.0ecm11.0)

Test: ECM-11.0 (https://vcgit.hhi.fraunhofer.de/ecm/ECM/-/tree/ECM-11.0)

Configurations: AI/LDB/LDP

Testing conditions: JVET CTC with QP set [27, 32, 37, 42]

Content: 5s; TBD

Extra parameters: with and without ClassF cfg file

Discussion on using classF config:

• One expert suggested not to use CSGO sequence as it is relatively simple
• ClassF config has only high gains for very simple sequences
• One expert commented that classF runtime increase could be due to individual tools. Suggested to investigate each tool separately
• It was commented that classF seems to work least well for sequences needing high rates
• Surprising that CSGO works well with classF, as the selected segment of the sequence had a lot gameplay and not menus
• Proposed to take best and worst results out of consideration
• It was commented that classF was tested to see if sequences behave more like natural or screen content, not to investigate classF/SCC tools. It was suggested not to use the classF config.
• In LDB results, some encoder runtimes seem decreasing with classF config. This is likely an artefact in result processing due to some missing simulation results
• It was commented that LDB is the most important config for gaming content compression and its desirable to be able to encode fast, i.e. avoid runtime increases
• It was commented that for sequences that one proponent simulated runtime increases were consistent around 20% when enabling classF config

Decision: Do not use classF config for further investigation of gaming content compression. Any tools that may bring gains including classF elements can be suggested to be included in test conditions in the future by bringing test results for said tools.

Discussion on reporting results template:

What to include in spreadsheet?

So far used spreadsheet based on JVET-U0127. It targets collection of info on many candidate sequences.

Build on this?

Could add:

• YUV PSNR 10-1-1
• Relative encoder / decoder runtime
• Others?

Switch to sheet based on JVET CTC sheet instead?

Two experts suggested to use CTC test sheets as basis.

Decision: Use the JVET CTC sheet as the basis.

Discussion on tested QPs:

Question was raised if quality range for current QPs (27-42) is adequate for this content.

One of the outputs of BoG should be mandates for next AhG group. Suggested to have new mandate: Solicit contribution from industry on typical bitrate/quality/resolution used for gaming content compression.

It was commented that Nvidia GeForce NOW uses transmission settings of

These are production numbers (tuned for lowest latency). It is expected that reference software default settings would require less rate, but is not suitable for real-time operation.

It was commented that results for some sequences have quite low PSNR numbers (26, 27), but there were no artefacts visible in a simple visual examination.

Input documents discussed

JVET-AG0302: AHG15: Description of gaming content sequences proposed by Huawei

This contribution proposes 5 new sequences to be used for the study of gaming content compression by AHG15. A short description for each sequence is provided in this document. A copyright notice is part of this contribution file and also attached ty o the file source.

The sequences have been uploaded to the jvet ftp server. They are available at:
ftp://jvet@ftp.ient.rwth-aachen.de/ahg/candidates/JVET-AG0302

Particular format for auxiliary information for depth, optical flow and camera parameters was proposed.

Discussion:

It was commented that DesertTown1 seems the most interesting/challenging of these sequences

Auxiliary information for gaming sequences

Various additional information may be available from the gaming engine. This can include:

Discussion:

Which auxiliary data is desired?

Of particular interest are:

• Depth
• Motion Vector
• Camera Information

InterDigital offered to convert provided depth map and optical flow to same format as Huawei. It is suggested to also convert Huawei format to InterDigital’s format and provide both at this point. InterDigital can also provide camera information.

Comment: How will 4x4 camera matrices be stored? It is suggested to use a json based format.

Other forms of auxiliary data that could be studied in the future are:

• Albedo
• Specular
• Normal
• Roughness

Suggested sequence segments for collected sequences

Of each sequence a segment of 5 seconds was selected for encoding for January meeting. This data should help identifying the most interesting sequences for a new class of gaming content sequences.

The table below lists the selected segments for each sequence. Note that for some sequences more than one potentially interesting segment were identified. The collected simulation results can be found in JVET-AG0015.

Gaming content compression test set and testing conditions

Comments:

Two experts commented: Reasonable to have two sets. Better to be able to distinguish between sequences that have or do not have auxiliary information.

Suggested to add RA, since its faster to simulate. Can be good if LDB not available in time. Also there are some applications, i.e. streaming gaming content.

Suggested to have AhG telco and choose sequences based on RA results. No need to do crosschecks at this point. If there are concrete proposals (e.g. tools) crosscheck will be needed.

Comment: last time used QP 27 to 42. Two proponents also tested QP22, it is not that slow. Suggested to test QPs 22-42. Only one or two rate points very high.

Comment: QPs 22-42 is also used by AhG11, can re-use excel template of them. Template also reports results for 4 lowest rate points (27-42). Can still use it if QP22 is not finished in time.

Comment: simulations should be split between several parties. As there are many candidates (27?). Not reasonable that everyone simulates all.

Question: Do we need LDP?

Question: Do we run 4K sequence for RA. No results for LDB/LDP. Comments: better to run for RA, but not for low delay. Twitch recently also tests 4K streaming of content (RA)

Comments:

• for lowest latency important. In actual implementation LDP is more used than LDB.
• Good to use only one to save some resources
• Agreed to use both of LDP/LDB
• So far there are no tools exploiting auxiliary information. If tools are developed, simulations results need to be shown with and without the corresponding tool.

Decision:

• Use QPs 22-42 for AI/RA/LDB/LDP (we also add RA).
• Simulation work should be split throughout the group.
• No testing of 4K sequences for low-delay.
• Use Excel template by AhG11 as it already supports 5 QPs. It shows results for 5 QPs and 4 QPs.

Proposed testsets contains all candidate sequences, but two classes: with (class G1) and without (class G2) auxiliary information:

Licenses

Need to clarify.

Comments:

• Should ask lawyers to be sure. We are not legal experts.
• Some sequences (e.g. 3GPP ones) have license information. This could be added to the report.
• It was suggested to add a table similar to the one below to the BoG report. License information should be checked by some lawyers.
• Can make two tables: sequences + source. Source + license
• While some sequence have licenses we are not sure we understand the licenses
  • Difficult to do for us (non-legals).
  • Make table stating existence of license information and clarity of license
  • Licenses should be reviewed by some legal expert

Note: The Twitch sequences from the table above come with the following copyright statement: “These video game recordings are placed in the public domain. The recordings may contain copyrighted work which can only be used in compliance with fair use.”

Recommendations

The BoG recommended to:

• Revise testing condition for gaming content compression
  • Add RA
  • Add QP22
  • Use AhG11 excel template (supports 5 QPs)
  • No longer test classF config. Initial tests indicate that gaming content behaves more like natural than like screen content.
  • Complete crosschecks not necessary at this point, but some partial cross-checks would be desirable to increase robustness of the results . But will be needed if new tools are proposed.
• Define new gaming content test set with two classes
  • Class G1: sequences with auxiliary data
  • Class G2: sequences without auxiliary data
• Work on auxiliary data
  • converting data to several formats such that all formats are available for each sequence that has auxiliary data
  • add camera matrices to auxiliary data
• Plan joint telco with AG5 until April meeting
• Work on clarifying license situation of all sequences
• Add new mandate: Solicit contributions from industry on typical bitrate/quality/resolution used for gaming content compression.

It is planned to further narrow down the number of test sequences. Most sequences are HD, two are 4K. Those would fall into class G2. Clarify if this class could become a mix of resolutions?

In the end, the classes should not have more than 4 cases each. Needs to be clarified later if the classes would be mandatory in CTC (considering the computational burden).

Licensing needs further checking, clarify if sequences can be copied to JVET ftp. Sequences which would be finally selected for the Gx classes shall be made available in the ftp, to guarantee that they can be accessed in longer term.

It was commented that sequences that would not be selected for classes Gx could later be used as training materials.

Generative face video (AHG16) (4+4)

Contributions in this area were discussed at 0720–0940 on Tuesday 23 Jan. 2024, and at 0605 on Thursday 25 Jan. 2024 (chaired by JRO).

JVET-AG0331 BoG on gaming content compression [S. Puri, J. Sauer, R. Chernyak, A. Duenas, L. Wang]

See section 4.15.