Back to Search Document details
29th Meeting: by teleconference, DE, January 2023 2023-01-13 23:57
EE1-related: Reduced complexity through channel redistribution in NN head
Authors: P. Wennersten Ericsson J. Ström Ericsson Du Liu Ericsson
Abstract
This contribution proposes to modify the number of channels in the head of the luma NN loopfilter to reduce the complexity. Comparing to the models in JVET-AB-EE1-1.5.3, it is reported that the proposed the luma model reduces the worst-case complexity by 9% (from 673 kMAC/pix to 615 kMAC/pix in block-basis), while the loss is 0.01% in RA. It is further reported that, also compared to the models in JVET-AB-EE1-1.5.3, the number of parameters goes down 2% (from 3.10 M to 3.03 M) and the training time is decreased by 12% (from 12 days to 10.5 days).
Meeting Notes
JVET-AC0126 EE1-related: Reduced complexity through channel redistribution in NN head [P. Wennersten, J. Ström, D. Liu (Ericsson)]

This contribution proposes to modify the number of channels in the head of the luma NN loopfilter to reduce the complexity. Comparing to the models in JVET-AB-EE1-1.5.3, it is reported that the proposed the luma model reduces the worst-case complexity by 9% (from 673 kMAC/pix to 615 kMAC/pix in block-basis), while the loss is 0.01% in RA. It is further reported that, also compared to the models in JVET-AB-EE1-1.5.3, the number of parameters goes down 2% (from 3.10 M to 3.03 M) and the training time is decreased by 12% (from 12 days to 10.5 days).

Result over NNVC-3.0, int16 Y: RA -9.60%, AI -7.31%, LDB -8.17%

Result over NNVC-3.0 filter set#1, int16 Y: RA -0.13%, AI 0.23%, LDB 0.37%

Result over JVET-AC0089 (JVET-AB-EE1-1.5.3), int16 Y: RA 0.01%, AI 0.08%, LDB 0.12%

Head of luma network. The inputs are combined to form the input y to the next part of the network.

The next figure below shows the k-th residual block. In this contribution, there are eight residual blocks.

The k-th residual block (k=0..7). The output y of the head is fed into a first residual block with input z0=y, which also takes the inputs rec, pred, part, bs, qp, and IPB. The output z1 is then fed into another such residual block.

The next figure below shows the last part of the network. These two are the same as in JVET-AB-EE1-1.5.3.

The output of the last residual block is fed into this last part of the network.

Input with 192 channels for reconstruction, 24 channels for prediction, 12 channels bs/qp/IPB

Network Information in Training Stage

Mandatory

GPU Type

GPU: NVIDIA A100-PCIE-40GB

Framework:

PyTorch v1.9

Number of GPUs per Task

1

 

 

Epoch:

340

Batch size:

32

Loss function:

Weighted L1 and L2

Training time:

10.5 days

Training data information:

DIV2K, BVI-DVC

Training configurations for generating compressed training data (if different to VTM CTC):

VTM-11.0, JVET-W-EE1-1.6, qp {17, 22, 27, 32, 37, 42}

Optional

 

 

Number of iterations

Patch size

256x256

Learning rate:

1e-4 and 1e-5

Optimizer:

ADAM

Preprocessing:

Mini-batch selection process:

 

Other information:

 

 

Network Information in Inference Stage

Mandatory

HW environment:

GPU Type

N/A

Framework:

SADL

Number of GPUs per Task

0

 

 

Total Parameter Number

See next table below

Parameter Precision (Bits)

Int16

Memory Parameter (MB)

See next table below

Multiplay Accumulate (MAC)

See next table below

Optional

 

 

Total Conv. Layers

Total FC Layers

Total Memory (MB)

 

Batch size:

1

Patch size

128x128, 256x256

Changes to network configuration or weights required to generate rate points

Peak Memory Usage (Total)

 

Peak Memory Usage (per Model)

 

Border handling

Other information:

 

 

 

Summary of complexity and performance (int16) for the luma model

Test/Luma model

Nbr. of models

kMAC/pix (Y)

kMAC/pix (YUV) worst case (frame-basis)

kMAC/pix (YUV) worst case (block-basis)

Num. of param. in total (106)

Total mem. in 16bit integer (106 bytes)

RA BDR-Y/U/V (vs. NNVC-3.0)

RA BDR-Y/U/V (vs. NNVC-3.0 filter set #1)

NNVC-3.0 filter set #1

4

intra 415

inter 418

525

664

6.21

12.42

−9.49%/

−20.71%/

−20.40%

0

JVET-AB
EE1-1.5.3:
IPB luma + chroma noIPB

2

425

532

673

3.10

6.21

−9.60%/

−20.93%/

−21.49%

−0.13%/

−0.24%/

−1.41%

Proposed luma model

2

379

486

615

3.03

6.06

−9.60%/

−20.93%/

−21.41%

−0.13%/

−0.25%/

−1.32%

It was commented that the approach to give different priority to the different inputs of the network (by varying number of channels) is an interesting approach to reduce complexity and increase performance, and that a similar approach could be used for filter set#0.

Support was expressed to investigate this in an EE. In this context, also the impact of the selection of the number of channels could be relevant. For example, it was pointed out that in SIMD implementation channel numbers which are powers of 2 are most effective to reduce run time.

Cited:
JVET-AC0312 X
ARCHIVE :
JVET-AC0126-v2.docx JVET-AC0126.pptx JVET-AC0126_over_AB-EE1-1.5.3-v2.xlsm JVET-AC0126_over_NNVC3.0-v2.xlsm JVET-AC0126_over_NNVC3.0_filterset1-v2.xlsm
Decisions
Support was expressed to investigate this in an EE. In this context, also the impact of the selection of the number of channels could be relevant. For example, it was pointed out that in SIMD implementation channel numbers which are powers of 2 are most effective to reduce run time.
Citation