JVET-M0023 CE3: Summary report on intra prediction and mode coding [G. Van der Auwera, J. Heo, A. Filippov]

This contribution was discussed Wednesday 9 January 1750–2030 (chaired by GJS)

This is the summary report of the third Core Experiment (CE3). The goal of CE3 is to study intra prediction tools, including mode coding, for potential inclusion into the VVC standard.

The following is the list of defined sub-tests in CE3:

• CE3.1: Intra prediction modes (5 tests)
• CE3.2: Cross-component prediction (11 tests)
• CE3.3: Intra mode coding (10 tests)

This document summarizes the objective results (BD-rates, runtimes), cross-check reports, and related input contributions.

The source codes of the tests and full test results are uploaded by proponents into the following CE3 GitLab repository:

https://vcgit.hhi.fraunhofer.de/JVET-L-CE3/VVCSoftware_VTM.git

The following changes were made to the CE3 test description document after the T2 deadline (December 6, 2018) had passed. Besides changes to contact persons and assignment of cross-checkers, the following changes were requested on the JVET reflector and discussed for clarification, if needed:

• CE3.2:
  • Added test CE3.2.2.1 (MMLM; combination of tests CE3.2.1 and CE3.2.2)
  • Added test CE3.2.6.2 (CCLM; testing 3 columns of neighbouring samples on left side of block, 1 row above block)
• CE3.3:
  • Reduction and clarification of test CE3.3.1 from 7 subtests to 4 (decoder-side intra mode derivation)
  • Additional result for test 3.5 (multiple DM for chroma)

CE3.1 on ‘Intra prediction modes’

Regarding 1.1.x, it was commented that this does not really increase decoder complexity. Another participant commented that this has a difficult pipeline dependency. However, other difficult pipline dependencies are already in the VTM, and this is not increasing the difficulty of the worst case.

CE3.1: Related contributions

CE3.1: Additional test results

It was noted that 1.1.x shows more gain on Class F (SCC content) than on other content, and Class F is not included in the average. A non-proponent participant focused on implementation issues indicated that they had analysed it and found it acceptable for implementation.

This was further discussed in the plenary on Sunday 13 January, and it was agreed to revisit this topic since it has an aspect of reversal of decoding order which may cause it to be difficult to implement. This aspect was studied further and was agreed to be removed. See the notes of the two plenary discussions of this in section 9.1.

Decision: Adopt CE3-1.1.1 proposal (without the reverse coding order aspect); text was provided in a revision of JVET-M0102.

Regarding 1.2.1, this has (in the decoder perspective) a selection of a matrix of stored fixed values among a set of such matrices, followed by a matrix multiply applied to boundary sample values to generate the prediction signal in the frequency domain and then an inverse transform is applied to generate a spatial domain prediction, followed by an ordinary residual difference.

It was noted that this has a bit more gain on RA than is usual for intra coding efficiency proposals (0.85/1.36=0.625 versus the usual ~0.5).

This has some decoder runtime increase. For 1.2.1 there is an increase in computational operations.

It was commented that the need to include an inverse transform in the 1.2.1 variant is an additional functional block unlike anything typically done for intra.

The amount of stored coefficient data is another issue, especially for the 1.2.1 variant (~300 kbytes). The 1.2.2 variant omits the inverse transform and has a (simple 2-tap one-dimensional average) downsampling that reduces the size of the matrices (to about ~18 kbytes – a total of around 14,000 numbers of 10 bits each), with a corresponding (bilinear) upsampling in the decoder. The proponent pointed to CPR as an instance where added storage of a greater amount is needed (although, for screen content, that has quite high gain).

The encoding complexity is another significant concern; both proposed variants increase that by ~50%. Additional encoder-only variants are reported in JVET-M0043 with different trade-offs.

The proponent said the training set did not include the CTC test set.

Between 1/3 and 1/2 of the intra blocks were reportedly using this mode (which is a lot).

Further study was encouraged (not necessarily in a CE). Side activity was encouraged during the meeting to potentially come up with a plan for further study.

For 1.3, the proposal is to add an additional mode; the measured gain was quite small and the encoder runtime increased by about 6%, so no action was taken on this.

CE3.2 on ‘Cross-component prediction’

The VTM has 3 CCLM modes (CCLM, CCLM-above, and CCLM-left). The selection between these modes is signalled, but the model parameters are not. The tests in the CE are to improve coding efficiency by adding more models.

• 2.1 uses 3 columns to the left and 2 lines above (except at the CTU boundary)
• 2.2 uses 5 columns to the left and 4 lines above (except at the CTU boundary)
• Between 2.1, 2.2, and 2.2.1, it was suggested to focus on the combination test 2.2.1, for using fewer lines and cleaner signalling

(*) Summary of test 2.4:

Some comments:

• Chroma gain tends to roughly translate to luma gain at about a 1:10 ratio
• Chroma gain in the CTC is often dominated by one particular test sequence: CampFire
• The MDLM scheme adopted at the previous meeting provided about 2.7% chroma gain
• The proposed MMLM schemes double the number of models derived and used.

Notes of the previous meeting: “MMLM (and its add-ons MNLM, MFLM) need to determine two models. Whereas the number of samples that is used to compute the models is the same in total, it cannot be foreseen how many samples fall into which class. Therefore, it is more difficult for pipeline processing than CCLM. The classification step, though it is a simple averaging criterion, also may impose some additional pipelining issues. Gain of MMLM standalone is 0.3% luma, approx. 2.5% for chroma [for AI; RA: 0.1% luma, approx. 2.0% for chroma – for 5.2.3.4 scheme]. It is recommended to further study whether the complexity concerns are less valid in combination with the LM computation of JVET-L0191, and whether the gain would still be preserved. It should also be investigated if MMLM and CCLM can use same building blocks.”

In response to the above notes:

• The gain is somewhat less now: AI: -0.27%, -1.71%, -2.26%, RA: -0.13%, -1.30%, -1.73%
• The complexity concern (e.g., latency added by determining the threshold and applying the classification step; although the cross-component prediction may not be in a critical-latency path) remains valid.
• It was remarked that even with the current scheme, the complexity is a problem, esp. for small blocks (2x2 chroma).

Due to complexity concerns, no action was taken on 2.1/2.2/2.2.1. This was also requested to be further discussed in the CE-related BoG to clarify the concerns and recommend what to further study.

2.3 has an adaptive number of models, sometimes more than two per component, and showed less gain, so no action was taken on that.

Regarding 2.4.x, the current scheme is optimized for type 0 chroma (the usual chroma per Rec. 709); 2.4 compensates for co-sited chroma (type 2 per BT.2100). The 2.4 test was done only on the PQ sequences. A decoder might need to support two chroma types, with encoder high-level selection of which to apply.

Testing was not done to determine the potential penalty of running the type 2 optimized scheme on type 0 content. Test results for this were requested.

A participant remarked that he had done some testing on self-generated content that used type 1 (centered both horizontally and vertically) and found not so much (~1-2%) benefit for customizing for that chroma siting. Here the experiments show a much more substantial gain for optimization to type 2.

It was commented that proper chroma alignment may be more visually important than what shows up in objective metrics.

The kernel of the 2.4.c 5-tap filter (except at CTU boundaries, where only one line above is used) is:

0 1 0

1 4 1

0 1 0

Decision: Adopt 2.4.c with a high-level flag to switch between two chroma format type optimizations.

Test results for applying the type 2 scheme to type 0 content. were discussed in a plenary. See notes in section 9.1 about that.

For 2.5.x, there was a focus on the first two variants. These are adding some additional complexity. The gain seemed insuffient to justify changing the current scheme.

2.6.1 reduces the number of neighbour lines used to compute the model parameters to 1 (where the current VTM scheme uses 3 columns to the left and 2 rows above when within the current CTU). 2.6.2 uses 3 columns to the left but only one row above. These would make the processing consistent whether within a CTU or at the boundary.

But it was commented that adoption of the 2.6.1 or 2.6.2 scheme would not be reducing the line buffering needed in a decoder, since it does not apply for the other chroma type. It was also commented that the question of 1 or 2 lines had been discussed at the previous meeting and the 2 line scheme was considered not a problem. So no action was taken on the 2.6.x proposals.

CE3.3 on ‘Intra mode coding’

CE3.3: Additional test results

For 3.1.x, the proposal is to add another mode in which the intra prediction mode is inferred by decoder processing rather than signalled. This adds encoder (and some decoder) complexity, but the test results do not show much benefit from this, so no action was taken on this.

For 3.2, the proposal is to restrict the number of selectable intra prediction modes to 32 of the 67. Some gain is observed, but only a small amount, and it was commented that the restriction of what modes an encoder would be allowed to select would restrict encoder freedom on how to make its mode decisions (not allowing an encoder to choose a mode until after it is able to determine which 32 of the 67 are allowed for selection), so no action was taken on this.

3.3.1, 3.3.2, 3.4.1, 3.4.2, and 3.5 are regarding chroma mode coding, which is dependent on luma mode. 3.3.1 and 3.4.1 would increase decoder complexity. One key aspect is how many luma points are considered for deriving the chroma mode candidates (1 or 2). Another key aspect is how many chroma mode candidates there are (3 or 5).

Schemes 3.3.2 and 3.4.2 reduce the number of candidates from 5 to 3. As tested, this reduced encoding time since the encoder checked fewer modes, although the decoder complexity is higher than for the current scheme (because it uses a 2-point check for direct mode selection). It was commented that the DC, planar, horizontal and vertical modes are especially important for some encoder implementations. If these are not always selectable, it would force a dependency between luma and chroma for encoding decisions. These two schemes did not provide much coding gain, although in the way they were tested, they reduced encoding time. The lack of significant coding gain, together with that dependency, did not appear to justify action on those.

Results for an additional scheme called 3.5.1 (proposed in JVET-M0203) were included in the CE report. This was a late addition that was not in the CE plan, so it was considered a non-CE proposal.

3.3.1 and 3.4.1 check two luma locations, whereas the VTM checks only 1 (the central position of the luma block). The VTM sends a flag on whether to use that mode; if not, it sends a CCLM mode flag; if not, it sends 2 bypass-coded bins to select between four modes. If the luma mode was not DC, planar, horizontal or vertical, then those are the four modes; otherwise the luma mode is replaced with the vertical diagonal mode to determine the four modes.

3.3.1 and 3.4.1 check two luma locations and perform some comparison flowchart operations to determine what is the primary selectable mode and what are the other four modes. The DC and planar modes are always among the 5 selectable modes. It was noted that this forces a dependency between the luma and chroma mode decisions unless the encoder only used DC and planar modes for chroma.

The possibility of supporting both the current scheme and the alternative was discussed. The gain seemed insufficient to want to need two different ways to be supported in the decoder.

3.5 checks one luma location (same as VTM); if the luma mode is DC or planar and the block shape is vertical, then instead of the vertical diagonal mode being considered special, the horizontal diagonal mode is considered special; if the luma mode is angular, the other selectable modes are determined by a flowchart (but the horizontal and vertical modes are not always available). This has basically the same forced cross-component dependency as 3.3.1 and 3.4.1.

Since the gain is relatively small and the 3.3.1, 3.3.2, 3.4.1, 3.4.2, and 3.5 proposals introduce an undesirable cross-component dependency for encoders, no action was taken on on these.