JVET-AD0109 AHG11: Neural network loop filter [R. Yang, M. Santamaria, N. Zou, F. Cricri, R. G. Youvalari, J. Lainema, H. Zhang, M. M. Hannuksela (Nokia)]
This document describes a multi-scale attention-based neural network loop filter. It is reported that the proposed NN-based loop filter achieves x.xx% (Y), x.xx% (Cb), x.xx% (Cr) BD-Rate gains compared to NNVC 3.0 in the RA configuration (results were not yet available at the time this summary was prepared).
The proposed NN loop-filter is a multi-scale convolutional neural network (next figure below) that includes Attention Splitting Blocks (ASB), described in the second figure below. Three scales are utilized.
NN architecture. ASB stands for Attention Splitting Block. The mid number of channels is 16
Architecture of an ASB
The input to the network is made of reconstructed samples, predicted samples, boundary strength, splitting signal computed as the average CU value, sequence QP, slice QP, slice type and temporal layer id.
Network Information for NN-based Video Coding Tool Testing in Training Stage
Network Information in Training Stage | ||
Mandatory | GPU Type | Nvidia A100-SXM-80GB |
Framework: | PyTorch v1.12.1 | |
Number of GPUs per Task | 1 | |
|
| |
Epoch: | 60 | |
Batch size: | 128 | |
Loss function: | L1 | |
Training time (for 1 model): | 2d 4h | |
Training data information: | BVI-DVC & DIV2K | |
Training configurations for generating compressed training data (if different to VTM CTC): | 17, 19, 22, 27, 32, 37, 42 | |
Optional |
|
|
Number of iterations | ||
Patch size | 128x128 luma, 64x64 chroma (random positions) | |
Learning rate: | 1,00E-03 | |
Learning rate update strategy | Every 10 epochs, the LR is drop by 0,1 | |
Optimizer: | ADAM | |
Preprocessing: | Top-left patch is not considered | |
Mini-batch selection process: |
| |
Training data update strategy: | None. Single stage training | |
Other information: |
| |
|
| |
Network Information for NN-based Video Coding Tool Testing in Inference Stage
Network Information in Inference Stage | ||
Mandatory | HW environment: | |
GPU Type | N/A | |
Framework: | LibTorch v1.12.1 CPU | |
Number of GPUs per Task | N/A | |
|
| |
Number of Parameters (Each Model) | 164481 for luma model 166354 for chroma model | |
Total Number of Parameters (All Models) | 330835 | |
Parameter Precision (Bits) | Float 32 | |
Memory Parameter (MB) | 1,26203537 | |
Multiply Accumulate (kMAC/pixel) | Luma 169 Chroma 45 | |
Calculation Method | Block basis | |
Optional |
|
|
Total Conv. Layers | 60 | |
Total FC Layers | ||
Total Memory (MB) |
| |
Batch size: | 1 | |
Patch size | CTU | |
Changes to network configuration or weights required to generate rate points | ||
Peak Memory Usage (Total) |
| |
Peak Memory Usage (per Model) |
| |
Border handling | ||
Other information: |
| |
|
| |
BD-rate over NNVC 3.0 in RA config (partial results)
BD-rate Over NNVC-3.0 |
| ||||||||
Y-PSNR | U-PSNR | V-PSNR | Y-MSIM | U-MSIM | V-MSIM | EncT | DecT CPU | PSNR Overlap | |
Class A1 | |||||||||
Class A2 | |||||||||
Class B | -3.42% | -14.57% | -12.38% | -3.00% | -14.31% | -12.58% | 711% | 96834% | 99% |
Class C | -4.21% | -14.04% | -12.90% | -2.84% | -11.39% | -10.54% | 152% | 89186% | 99% |
Class E |
|
|
|
|
|
| |||
Overall | |||||||||
Class D | -6.16% | -14.78% | -14.81% | -3.09% | -12.29% | -10.25% | 155% | 85752% | 99% |
Class F | -2.07% | -9.75% | -7.65% | -1.46% | -10.10% | -8.68% | 755% | 51046% | 99% |
Class H | |||||||||
It was pointed out that quantization of the softmax operation would be necessary.
Complexity/performance-wise, this would be somewhat between the “low” and “high” architecture configurations that are intended to be defined.
Interest was expressed to study the multi-scale aspect further, also in the context of other architectures, but this would be premature for the EE cycle initiated at the current meeting.