JVET-M0030 CE10: Summary report on combined and multi-hypothesis prediction [C.-W. Hsu, M. Winken]

Five sub CEs are created to test different methods of combined predictions, including:

• CE10.1: multi-hypothesis prediction,
• CE10.2: overlapped block motion compensation,
• CE10.3: multiple shape prediction partitions,
• CE10.4: diffusion filtering of inter- and intra-prediction signals,
• CE10.5: local illumination compensation.

There are 13, 4, 1, 2 and 2 tests for each sub CE, respectively.

CE proposals are listed as follows,

CE10.1: Multi-hypothesis prediction

The test results for this aspect are summarized as follows,

According to the CE description, for investigating the impact of multi-hypothesis inter prediction on cache-related aspects, the cache model compiled into the reference decoder software by setting #define JVET_J0090_MEMORY_BANDWITH_MEASURE to 1 is used, in conjunction with the cache config file provided in JVET-K0451. For each bit stream, the decoder outputs a total hit ratio when using the cache model. The following table compares the average hit ratios (in percentage) of the tests with those of VTM-3.0:

Data quoted from JVET-M0176:

Data quoted from JVET-M0425:

1.1.a uses two hypotheses in uni prediction and merge. Basically, the same prediction could be invoked by using bi prediction. However, only one AMVP list is generated, and it may save some signalling. Encoder time increases by 7%, no gain in LB.

Are gains of 1.1.a/b additive? They are said to have been almost additive before this CE cycle, but in the current CE the combination was not tested.

Worst case memory BW of 1.1.a is uncritical, of 1.1.b was analysed as roughly 80% of VTM.

Combination of 1.1.a/b would be somewhat similar to 1.2.a, which uses multi hypothesis for both merge and AMVP, and additionally allows kind of different weighting of the hypotheses. 1.2.a is the “full set” of functionality of this proposal, whereas b..d are somewhat simplifications, mainly for the benefit of saving local storage (b), or memory bandwidth (c,d). d still uses up to 4 hypotheses (2 each in bi with >=8x16 block size restriction), but only for luma, which is the reason for worse performance in chroma. It is reported that worst case memory BW is 0.97%

It is also mentioned that potentially gains of 1.1.a might add up to 1.2.x.

Generally, the gain of all 1.1.x and 1.2.x proposals is significantly less than it was over VTM2 (cut by half or even more). In particular, the 1.1.b and 1.2.x proposals add need for more building blocks. 1.1.a is relatively simple and ce re-use existing memory access structures and MC logic, but for getting the gain of 0.17% for RA (no gain for LB), the encoder runtime also increases by 7%.

By doing more encoder checks and increasing runtime by 7%, probably it should be possible to get similar gain without a syntax change.

1.1.3…5 target simplification of VTM3 (CIIP)

For simplification aspects, the tests and corresponding results are summarized as follows, where the current design in VTM-3.0 and the differences between each test and VTM-3.0 are listed

* Not in original CE description.

Test results for this aspect are summarized as follows,

It is commented that 10.1.3.b/d was not originally planned in the CE.

In the context of reviewing 10.1.3.x proposals, the following aspects were identified in CIIP:

• does it need a specific MPM derivation? If there was only one mode (as in 10.1.3a/d) it is not needed at all, and it is mentioned that there are proposals just suggesting fixed length coding
• CIIP does not have any serious latency issue. Therefore, simplifications that remove some processing steps that are otherwise used in intra prediction is not necessary.
• Sample-wise unequal weighting is not an implementation issue, whereas equal weighting would be preferrable, unless it costs compression performance or causes qualty problems.

Beyond the approachs tested in 10.1.3 more study is necessary on these aspects.

10.1.4 requests restricting the maximum block size of CIIP to 32x32 (now it is 64x64). The motivation is saving need for additional memory. As however the maximum size of a inter-only or intra-only prediction block is large anyway, the real need for such a restriction is not obvious.

10.1.5 is combining some 10.1.3.x and 10.1.4.

CE10.2: OBMC

All proposals impose a CU size constraint >= 64 samples. It is requested to provide a more detailed analysis of the memory bandwidth. Likely, for processing on-the-fly, the worst case would be that the current block is 8x8, and all of the top and left neighbours are 4x4. Another option would be to store samples from the current block that were already fetched to perform interpolation in the neighbour blocks. In that case, it should be reported how large the additional local buffer would be, for all four methods.

Additional analysis is shown Saturday afternoon for 10.2.1 and 10.2.2 (to be provided in updated version of JVET-M0178/9). This analysis indicates that for those tw approaches the worst-case memory bandwidth of VVC is not increased for on-the-fly fetch (and even a little less for 10.2.2), or if the pre-generation method is used, the memory BW is even lower, but 1.96/1.28 kByte is necessary as local buffers.

In terms of processing, it is reported that worst case number of sample interpolations is not increased relative to the current bi prediction case. Method 10.2.1 uses uni prediction in a 8x8 block, and additionally needs to interpolate four 4x4 areas with other vectors for OBMC. Method 10.2.2 does not use any interpolations for OBMC. For 10.2.1, weighted superposition requires at most 4 shifts and 2 adds per sample. Furthermore, since the operations are locally varying, some additional logic is necessary. For 10.2.2., the latter numbers duplicate. Furthermore, 10.2.2 is more challenging in terms of local memory acces, as 6 different sources need to be blended.

10.2.1 seems manageable from complexity perspective (but definitely adds some complexity).

The test results are summarized as follows,

As a general note, the gain in compression performance is lower than it was with VTM2 (approx. half). However, from the results, OBMC even gives more gain for LB than for RA, and LB is in overall performance of VVC still worse than RA, this is assessed to be valuable enough. Some support, and no opposition is raised in Track B against adopting it.

It was initially agreed in Track B to adopt JVET-M0178. Specification text was available. This decision was later reverted in the JVET Sunday plenary (see the notes in section 9.1).

This would have a high-level flag for disabling it.

CE10.3: Multiple shape prediction partitions

In CE10.3, the goal is to test AMVP to be combined with non-rectangular prediction partitions within one CU. The tests and corresponding results are summarized as follows:

From the results (low gain versus high increase in encoder run time), not worthwhile to consider

CE10.4: Diffusion filters for intra and inter prediction

In CE10.4, the goal is to test prediction to be combined using filtering, where three types of filters, two directional filters and one uniform filter are used. The filters are FIR filters, applied on the prediction signal, not iterative. The 1D filters are 9-tap, symmetric, only 1 multiplication, otherwise shifts. The 2D filter is a 5-tap diamond shape, only shift/add operations. At the boundaries, pixel replication is used.

The tests and corresponding results are summarized as follows,

Generally, it was agreed that this method gives interesting gain, and is straightforward to implement at the decoder. Two concerns are raised:

• As it needs to be run after the prediction signal is generated, it produces additional delay in the intra prediction loop.
• For inter blocks, it can be used for 128x128 CU, which would break concepts of 64x64VPDU.

It is requested to provide additional results without using the method in intra prediction (and the intra part of CIIP), and restrict the largest CU size to 64x64.

It is also reported that a late contribution (JVET-M0848) provides new results for the same method of 10.4.2 with some (small) encoder speedup and slightly increased performance (-0.44% for luma).

Furthermore, it would be desirable to use only the prediction samples (no reconstructed samples from current picture). A version which did that was shown in the previous CE.

This was revisited (Track B Thursday 17 January1630) after new results were made available in JVET-M0042v3. The restrictions were implemented as requested, including not using reconstructed samples from neighbour blocks. Results are as follows:

The spec text was available and straightforward.

One expert points out that the multiplication by 6 can be implemented by shift and add.

LB results are not complete yet, but by tendency similar as in the original scheme, lower gain than for RA.

Further study in CE, along with LIC and post rec filters. This CE should also identify how potentially gains add up. The restriction of not using recosntructed samples from neighbouring inter blocks may not be necessary.

CE10.5: Local illumination compensation

In CE10.5, the goal is to test prediction to be combined using linear model derived from reconstructed and reference neighbouring samples. The tests and corresponding results are summarized as follows:

The method of 10.5.2 is mainly for the benefit of encoders, and does not solve the latency issue that LIC imposes on decoder pipeline. The problem is that a current block needs to wait for reconstruction of the neighbours, and then it requires a certain number of cycles until the parameters of the linear model are computed. Of particular concern is the fact that many decoder implementations target processing inter and intra coded blocks independently in order to make best benefit of parallel processing. Typically, inter coded blocks of a CTU are decoded first. For this, has a non-negligible complexity impact (in particular for parallel processing) if LIC of an inter coded block uses the reconstruction of intra coded neighbours. If such an approach (disabling LIC from intra coded neighbours, including CPR) would be combined with the approach of 10.5.3 (only using reconstructed inter coded samples from current CTU), the situation would be better, however would still mean that the inter reconstruction within a CTU would need to be sequential (similar as the situation in intra is). Regardless of that, LIC has some additional processing complexity (which doubles in case of bi prediction) which needs to be justified by performance. The processing should also be aligned with the 64x64 VPDU concept. (See further notes on these aspects under JVET-M0873.)

Further study was recommended on these aspects.

A BoG (coordinated by C.-W. Hsu and M. Winken) was established to review CE10 related proposals, and suggest candidates for further study in a CE. See the further notes for the discussion of the BoG report JVET-M0873.

The subsequent notes only contain abstracts copied from the documents. Actions taken are noted above under JVET-M0029.