JVET-W0132 AHG11: 1.5x/2.0x Upsample method for NN-Based Super-Resolution Post-Filters [Y. Yasugi, T. Chujoh, T. Ikai (Sharp)]
This contribution presents several experimental results of NN-based super-resolution post-filter. In VVC, RPR (Reference Picture Re-sampling) has been introduced, and in several 4K sequences with low bit-rate, it was reported that there are some coding gains by changing the resolution of the whole sequence. In this experiment, the performance of NN-based filters with different network structures are compared. As shown in experimental results, the use of interpolation for upscaling in feature space can reduce the number of parameters more than 30% without significant loss of performance.
The super-resolution 1.5x post-filter used in this experiment is based on a simplified ESRGAN
- Configuration A: 2x, PixelShuffle
- Configuration B: 2x, Bicubic interpolation (2.0x)
- Configuration C: 1.5x, PixelShuffle + Bicubic interpolation (0.75x)
- Configuration D: 1.5x, Bicubic interpolation (1.5x)
The number of parameters for configurations A-D were 483597, 335884, 483597, and 335884, respectively.
Experimental results of configuration A (2.0x, PixelShuffle)
Random access Main 10 | |||||
BD-rate Over VTM-11.0&JVET-V0056 RPR20 QP=22,...,42 | |||||
Y-PSNR | U-PSNR | V-PSNR | EncT | DecT | |
Class A1 | -1.81% | -2.83% | -4.56% | 100% | 100% |
Class A2 | -6.31% | -4.49% | -3.25% | 100% | 100% |
Experimental results of configuration B (2.0x, Bicubic interpolation)
Random access Main 10 | |||||
BD-rate Over VTM-11.0&JVET-V0056 RPR20 QP=22,...,42 | |||||
Y-PSNR | U-PSNR | V-PSNR | EncT | DecT | |
Class A1 | -1.64% | -2.13% | -4.67% | 100% | 100% |
Class A2 | -6.03% | -4.68% | -2.08% | 100% | 100% |
Experimental results of configuration C (1.5x, PixelShuffle+Bicubic interpolation)
Random access Main 10 | |||||
BD-rate Over VTM-11.0&JVET-V0056 QP=27,...,47 | |||||
Y-PSNR | U-PSNR | V-PSNR | EncT | DecT | |
Class A1 | -1.43% | 1.76% | -0.12% | 100% | 100% |
Class A2 | -3.33% | 0.95% | 2.74% | 100% | 100% |
Experimental results of configuration D (1.5x, Bicubic interpolation)
Random access Main 10 | |||||
BD-rate Over VTM-11.0&JVET-V0056 QP=27,...,47 | |||||
Y-PSNR | U-PSNR | V-PSNR | EncT | DecT | |
Class A1 | -1.71% | 0.67% | -1.62% | 100% | 100% |
Class A2 | -3.39% | 0.99% | 3.03% | 100% | 100% |
Question: Why is there slight loss for 2x, slight gain for 1.5x? May need further analysis
The behaviour may also be different as different QP ranges are used for th two cases.
Question: Do the PSNR/rate graphs cross? Needs to be checked
Results comparing against full resolution anchor (as in EE CTC), and also other classes would be interesting to compare against other proposals.
A BoG (coordinated by A. Segall) was asked to perform further analysis about the complexity/compression tradeoffs of the various loop filter and super resolution proposals, align the reports with AHG11 reporting conditions, and refine as necessary. The unbalance between luma and chroma gains should also be further analysed in detail. Also suggest candidates for upcoming EE.
Other (4)
Contributions in this area were discussed in session 14 at 2150–2300 UTC on Monday 12 July 2021 (chaired by JRO).