JVET-L0032 CE12: Summary report on mapping functions [E. François, D. Rusanovskyy, P. Yin]
This contribution provides a summary report of Core Experiment 12 on mapping functions. CE12 aims at evaluating approaches for mapping of HDR and SDR content. The considered technologies are out-of-loop dynamic range adaptation, in-loop reshaping, in-loop cross-component chroma refinement. Test results against VTM2.0.1 anchors are provided for each performed test. Crosschecking reports are integrated in this contribution.
- HDR-related
- CE12-1: out-of-loop dynamic range adaptation (JVET-K0298/JVET-L0205)
- CE12-2: in-loop reshaping for HDR (JVET-K0308/JVET-L0245)
- CE12-3: in-loop chroma refinement for HDR (JVET-K0298/JVET-L0206)
- SDR-related
- CE12-4: in-loop reshaping for SDR (JVET-K0309/JVET-L0246)
- CE12-5: in-loop chroma refinement for SDR (JVET-K0468/JVET-L0206)
- CE12-related in-loop reshaping
- JVET-L0247: CE12-related: Universal low complexity reshaper for SDR and HDR video
- Results tested in CE (CE12-2.1a2 / CE12-4)
- CE12-related out-of-loop reshaping
- JVET-L0490: CE12-related: HDR Coding with Backward Compatibility Options
- JVET-L0247: CE12-related: Universal low complexity reshaper for SDR and HDR video
All Intra | ||||||||||||
Test# | DE100 | PSNRL100 | wPsnrY | wPsnrU | wPsnrV | psnrY | psnrU | psnrV | EncT | DecT | ||
out-of-loop mapping | DRA+K0298+QPHARM+REFLC | 12-1.1 | -6.4% | -4.0% | -2.0% | -8.4% | -20.1% | 2.0% | -1.1% | -10.7% | 99% | 102% |
DRA+K0308+QPHARM+REFLC | 12-1.2 | -7.5% | -2.9% | -2.3% | -9.5% | -21.0% | -0.3% | -2.7% | -12.1% | 101% | 102% | |
in-loop mapping | ILM+K0308+ILFOPT3 | 12-2.1a1 | 1.5% | -2.4% | -2.0% | 2.7% | 1.3% | 0.0% | 8.4% | 7.9% | 104% | 100% |
ILM+K0308+ILFOPT0 | 12-2.1a2 | 1.4% | -2.0% | -1.7% | 2.8% | 1.2% | 0.1% | 8.4% | 7.8% | 100% | 101% | |
ILM+K0298+ILFOPT3 | 12-2.1b | 2.8% | -3.6% | -1.7% | 4.0% | 1.8% | 2.3% | 10.1% | 8.6% | 103% | 102% | |
ILM+K0308+ILFOPT3+QPHARM | 12-2.2 | 1.2% | -2.4% | -2.0% | 1.9% | 0.9% | 0.1% | 7.1% | 7.4% | 103% | 102% | |
ILM+K0308+ILFOPT3+ILREFC | 12-2.3a | -0.4% | -2.4% | -2.0% | -2.9% | -9.0% | 0.1% | 2.6% | -1.7% | 102% | 102% | |
ILM+K0308+ILFOPT3+ILREFLC | 12-2.3b | -0.6% | -2.5% | -2.3% | -2.9% | -9.0% | -0.4% | 2.6% | -1.7% | 104% | 98% | |
in-loop refint | ILREFC | 12-3. | -2.4% | 0.0% | 0.1% | -6.2% | -11.3% | 0.1% | -5.9% | -10.2% | 97% | 100% |
Random | Access | |||||||||||
Test# | DE100 | PSNRL100 | wPsnrY | wPsnrU | wPsnrV | psnrY | psnrU | psnrV | EncT | DecT | ||
out-of-loop mapping | DRA+K0298+QPHARM+REFLC | 12-1.1 | -8.4% | -4.8% | -2.6% | -6.0% | -24.4% | 1.7% | 1.6% | -14.7% | 99% | 107% |
DRA+K0308+QPHARM+REFLC | 12-1.2 | -9.0% | -3.0% | -2.5% | -6.5% | -24.8% | -0.5% | 0.6% | -15.4% | 100% | 108% | |
in-loop mapping | ILM+K0308+ILFOPT3 | 12-2.1a1 | 2.5% | -2.3% | -2.1% | 5.0% | 6.3% | -0.2% | 10.6% | 12.6% | 103% | 104% |
ILM+K0308+ILFOPT0 | 12-2.1a2 | 2.3% | -2.0% | -1.8% | 4.8% | 5.5% | -0.1% | 10.3% | 11.8% | 101% | 103% | |
ILM+K0298+ILFOPT3 | 12-2.1b | 3.2% | -3.6% | -2.2% | 5.3% | 6.3% | 1.2% | 11.1% | 12.7% | 101% | 106% | |
ILM+K0308+ILFOPT3+QPHARM | 12-2.2 | 2.4% | -2.3% | -2.1% | 4.2% | 5.8% | -0.2% | 9.6% | 12.0% | 102% | 106% | |
ILM+K0308+ILFOPT3+REFC | 12-2.3a | 0.8% | -2.3% | -2.1% | 1.3% | -0.9% | -0.1% | 7.0% | 6.4% | 101% | 106% | |
ILM+K0308+ILFOPT3+REFLC | 12-2.3b | 0.8% | -2.3% | -2.2% | 1.3% | -1.0% | -0.2% | 7.0% | 6.4% | 100% | 100% | |
in-loop refint | ILREFC | 12-3. | -1.8% | 0.0% | 0.0% | -4.1% | -7.4% | 0.0% | -3.7% | -6.3% | 103% | 102% |
No viewing was performed during the meeting, but proponents of in/out loop approaches performed mutual crosschecks and confirmed that
- Both in-loop and out-of-loop outperform the anchors objectively
- No subjective visual difference between in-loop and out-of-loop
- CE12-1.2 and CE12-2.3.b perform similarly for wPSNRY (HDR) (AI diff 0.0%, RA diff 0.3%)
- CE12-1.2 outperforms CE12-2.3.b for wPsnrU/V, DE100, and PSNR L100.
The out-of-loop reshaping shows same (or objectively higher) benefit than in-loop, and both are claimed to outperform the anchors. It would be premature at the current status of standardization to define the correct place of signalling (as HL syntax is just starting to be developed). However, it might be useful to be used as anchor in the future after confirmation of subjective benefit. Possibility of subjective viewing prior to next meeting to be identified.
CE12.4: In-loop reshaping for SDR
All Intra | Test# | psnrY | psnrU | psnrV | EncT | DecT |
in-loop mapping | 12-4. | -1.0% | 2.6% | 2.1% | 107% | 105% |
in-loop refint | 12-5. | 0.0% | −1.1% | −0.9% | 101% | 100% |
Random Access | Test# | psnrY | psnrU | psnrV | EncT | DecT |
in-loop mapping | 12-4. | −1.3% | 2.1% | 1.6% | 106% | 105% |
in-loop refint | 12-5. | 0.0% | −1.3% | −1.0% | 101% | 101% |
Inverse reshaping is generally done before the loop filter, and motion comp is applied in the original domain. This means that in the decoding process, reshaping is applied after motion comp, inverse reshaping after adding the residual. The same is applied in intra and inter prediction for inter slices. At the decoder, the inverse reshaping must be performed for each block after reconstruction. After computing the prediction in the original domain, it must be reshaped such that the residual can be added.
At the encoder, the original is once reshaped as whole picture, and then the decoder in the encoder loop has to perform the process above (reshaping of prediction, inverse reshaping of reconstruction).
Both reshaping and inverse reshaping are LUT operations, only applied for luma. Additionally, the chroma residual is scaled depending on the luma.
In case of RA, intra slice is not reshaped for UHD, as it was found that the reshaped signal increases the rate for QP 22 significantly. In case of AI, reshaping is done to the entire picture, prediction and coding are performed in reshaped domain, and inverse reshaping before loop filters
In the encoder for RD decision, weighted PSNR is used.
Some of the PSNR curves are crossing. Generally, the gain is becoming lower at higher rates.
It is generally that the reshaped version has higher rate than the anchor, so the quality of B pictures is likely better than the anchor, while the I picture should be the same.
The gain is highly sequence dependent, highest gain for Marketplace (5.2%).
Several aspects require further study:
- Implementation, regarding the impact on pipelining of the block-wise prediction loop, dependency between luma and chroma, etc., interdependency with CCLM
- investigate performance in low QP range, to see if quality saturates
- Since the quality difference of I vs B pictures is changed, and rate allocation is spatially varying impact on visual quality (compared to anchors at lower bit rate points). Informal viewing to be announced.
Viewing was done Monday. Experts who participated did not observe visual differences, such that it can be judged that the method does not produce visual artefacts.
Further investigate in a CE the behaviour at different (and also lower) QP. Currently, the same reshaping function was used for QP points. Investigate the possibility to make it rate adaptive, or disable towards higher rates.
It is also inconsistent that for AI the reshaping was done at the picture level (before in-loop filtering), and for RA not at all in I slices in UHD sequences. This should be unified. Cases should also be studied (in RA) where the rate for the inter pictures stays similar as in CTC.
CE12-5 applies an additional in-loop filter (not a block-wise operation as in CE12.4) to the chroma component after the deblocking filter. This is a cross-component operation, sample-wise scaling of the chroma values depending on the luma value. The mapping function used for scaling is a piecewise linear function, which is designed in a way (and signalled per frame) such that the chroma values are coming closer to the original. For SDR, this results in chroma gains of around 1%, obviously no luma gain. Overall, the bit rate reduction (or PSNR improvement) seems to be very low. It would also require 2-pass coding to determine the LUT. No benefit for in-loop operation.
CE12 will continue on the investigation of using mapping functions for SDR content. E. François and P. Yin to coordinate the CE description. The HDR viewing prior to next meeting (mentioned somewhere else in context of HDR AHG) will further investigate the subjective benefit of in-loop and out-loop reshaping for HDR content, to identify possible necessary actions.