JVET-AC0090 AhG11: Neural Network-based Reference CU Quality Enhancement for Motion Compensation Prediction [Y. Chu, Z. Wang, W. Zhang, S. Li (Hisense)]
This contribution proposes to enhance the Coding Unit (CU) before motion compensation with neural network. The proposed method is mandatorily performed on CU with minimum size of 16×16. Experimental results demonstrate that, compared with VTM-11.0_nnvc-2.0, the proposed method achieves 0.86% BD-rate reductions for Y component on Class D under the LDP configuration.
Part (a) of the figure below is the process of integrating neural network into conventional motion compensation while part (b) is the process of integrating the neural network into affine motion compensation. In the conventional motion compensation, the VVC standard method finds the reference CU from the current CU according to the MV. Since the fractional interpolation filter will reduce the image size, it is necessary to expand the boundary of the reference CU by 4 pixels and then perform DCTIF to ensure that the CU size is consistent before and after filtering. The residual CU is obtained by subtracting the filtered prediction CU from the original CU, then it is transformed and quantized. Our mothed is designed to input the padded image patch into neural network, and the output image patch is then interpolated and filtered. In the affine motion compensation, the VVC standard method is to split the current CU into several 4 × 4 sub-CUs, and calculate the MV of each sub-CU according to the 4-parameter or 6-parameter affine models. The reference sub-CUs will then be found through MV. By performing padding, fractional interpolation on the reference sub-CUs, the final predicted sub-CUs can be obtained. Finally, the predicted sub-CUs will be combined to gets the predicted CU. In our method, we first get the max region containing all the reference sub-CUs, and then pad the region and input it to the neural network. Then, based on the corresponding position of the output image patch, the padding sub-CUs can be filtered by DCTIF to get the predicted sub-CUs.
Three networks were trained, one for the CU block with size 64×64, one for the CU block with size 64×32, 32×64, 32×32, and the last one for the CU block with size 16×64, 64×16, 32×16, 16×32, 16×16.
(a) Conventional motion compensation
(b) Affine motion compensation
Motion compensation in VVC
The structure of the proposed neural network.
Training information.
Training information | ||
Mandatory | GPU Type | NVIDIA Geforce RTX 3090 |
CPU Type | Intel(R) Core(TM) i9-10900X CPU @ 3.70GHz | |
Framework: | PyTorch 1.8.0 | |
Number of GPUs per Task: | 1 | |
Epoch: | 240 | |
Batch size: | 64 | |
Total Parameter Number: | 738.43k/model (3 models in total) | |
Parameter Precision (Bits): | 32 (F) | |
Training data information: | BVI-DVC | |
Training configurations for generating compressed training data (if different to VTM CTC): | QP =22 | |
Optional | Patch size | 64x64, 32x32, 16x16 |
Learning rate: | 1e-4 | |
Optimizer: | ADAM | |
Loss function: | MSE | |
Inference information | ||
Mandatory | GPU Type | N/A |
CPU Type | Intel(R) Core(TM) i9-10900X CPU @ 3.70GHz | |
Framework: | PyTorch 1.8.0 | |
Number of GPUs per Task | 0 | |
Total Parameter Number | 738.43k/model (3 models in total) | |
Parameter Precision (Bits) | 32 (F) | |
Memory Parameter (KB) | 2611K/model (3 models in total) | |
Optional | Total Conv. Layers | 22 |
Total FC Layers | 0 | |
Batch size: | 1 block at a time | |
Patch size | Depending on the block (64x64, 32x32, 16x16) | |
The goal is enhancement of prediction.
Further study was recommended, with the following comments:
- Complexity reduction should be possible
- Would it still have gain when a NN-based loop filter is used?
- Performance on larger resolutions?
- Where is the gain coming from? Correction of affine subblocks?