JVET-AP0023 EE1: Summary report of exploration experiment on neural network-based video coding [E. Alshina, R. Chang, F. Galpin, Yue Li, Yun Li, M. Santamaria, T. Shao, J. Ström, Z. Xie (EE coordinators)]

Code base for EE1 tests was NNVC16.0, the anchor for the majority of EE1 tests is default configuration of NNVC-16.0 (NN-Intra and LOP7 filter enabled). NNVC common test conditions, results and complexity reporting template were used.

For majority of proposals AhG11 training set, which consists of DIV2K, BVI-DVC and TVD was used. NN-based Inter tools were trained using Vimeo 90K triplet. Comparison is done between tests which use the same sub-set of training data.

It was agreed for tests competing with technologies in NNVC to configure the proposed solution targeting the complexity close to existing NNVC tool, but not exceeding it:

1. kMAC/pxl of EE1 test ≤ kMAC/pxl NNVC (must),
2. the number of channels (both input and output) in neural network modules which are modified must be keep multiple of 16,
3. Number of Parameters EE1 test ≤ Number of Parameters NNVC (if possible).

According to the EE description only tests with results provided to quantized (int 16) model are considered for adoption to NNVC, float point model results are provided just for information.

Inference cross-check is required for all EE1 tests. Candidates for adoption to NNVC are required to undergo training cross-check, training scripts (including model quantizing algorithm) must be made available, implementation must be compatible with SADL.

Exact parameters settings were announced by proponents in the 2^nd AhG11/14 teleconference on March 20.

List of tests in this round of exploration experiment includes:

• EE1-1: LOP in-loop filter
  • EE1-1.1 – Dynamic convolution for LOP7 neural in-loop filtering JVET-AP0066 (KHU, KBS).
  • EE1-1.2 – AHG11: Enhancing LOP7 with Re-Exploited Boundary Strength Guidance withdrawn (to be confirmed)
• EE1-2: VLOP in-loop filter
  • EE1-2.1 – VLOP with new backbone block based on Spatial-Channel Mixing JVET-AP0201 (KHU, ETRI).
• EE1-3 NN-Inter
  • EE1-3.1 – Very Small Deep Reference Frame Generation Network for Inter Prediction Enhancement JVET-AP0051 (Wuhan Uni)
• EE1-4: Framework for externally coded pictures
  • EE1-4.1 – Multi-layer framework
  • EE1-4.2 – Multi-layer framework with frame level control
  • EE1-4.3 – Single-layer framework with different E2E AI codec
  • EE1-4.4 – Single-layer framework with frame level control

JVET-AP0080 (Huawei, InterDigital), JVET-AP0182, JVET-AP0183, JVET-AP0184, JVET-AP0185 (all Nokia), JVET-AP0232 (Huawei)

• EE1-5: Operational bit-exact reproducibility
  • EE1-5.1 – [AHG11] Stable float convolution for neural network inference JVET-AP0218 (Nokia)

EE1-1: LOP in-loop filter

Anchor is NNVC-16, LOP7 filter and NN-Intra enabled.

Comparison point NNVC LOP7: kMAC/pxl = 16.5, NumParameters =0.24 M

• • EE1-1.1 – Dynamic convolution for LOP7 neural in-loop filtering JVET-AP0066 (KHU, KBS).
    • EE1-1.1.1 – SADL int16 model generation for the proposed dynamic convolution in LOP
    • EE1-1.1.2 – kMAC/pxl reduction of dynamic convolution (make it lower than LOP7)

kMAC/pxl = 16.59, NumParameters = 0.84 M

• • • EE1-1.1.3 – Parameter reduction of dynamic convolution in LOP7

kMAC/pxl = 16.52, NumParameters = 0.83 M

• • • EE1-1.1.4 – Optimal combination of kMAC/pxl and parameter reduction for dynamic convolution with coding-efficiency trade-off

In this test fixed-kernel static convolution layers in the LOP7 in-loop filter are replaced with dynamic convolution. Unlike static convolution, which uses a single fixed kernel for all inputs, dynamic convolution generates a per-sample effective kernel as a weighted combination of multiple expert kernels, where the weights are derived from the input feature via a lightweight router as shown in the figure below.

Difference between static and dynamic convolutions.

Changes compared to LOP7 are illustrated in the next figure: Blue-shaded CONV blocks are replaced with dynamic convolution (DynConv); the yellow-shaded CONV block indicates the chroma input convolution whose output channel count is d₂ (the parameter varied across sub-tests). Subtests use different number of expert kernels and different chroma input channels (d₂).

The decoder run time increase in this test (~130% of decoding with LOP7) is associated with the dynamic kernels derivation.

Training time for the model (only stage 3 as agreed in EE1) is 10 days (same training for LOP7 requires 7 days).

Modifications compared to LOP7

EE1-1.1 tests summary

Proponents conclude that test EE1-1.1.2 provide the best complexity-performance trade-off (EE1-1.1.1 and EE1-1.3 are not fulfilling criteria of max number of kMAC/pixel).

Feedback from the cross-checkers: cross-check of inference is done, training is partially finished. There is perfect match in inference, tests after re-training are expected to be ready during meeting (second week), so far maximum deviation between original and re-trained model is within 0.1% BD-rate.

It is expected that the decoding run time can be further reduced by optimizing the implementation

It is noted that the number of parameters is significantly increased, which may have some memory usage aspect. This is not a criterion in the exploration, but might be relevant in standard development. EE1-1.4 has lower number of parameters, but also lower gain.

1.1.3 and 1.1.4 were also cross-checked (only inference).

It was asked and confirmed that learning rate was not changed relative to the one used for LOP7 training.

EE1-1.1.2 was initially identifies as candidate for adoption, provided that full cross-check would be finished.

A presentation AP0066-v3 was given on Thursday 30 April 1140 (presentation deck not yet included).

Training crosscheck not finalized yet, and quite some divergence (0.3% worse in RA, 0.1% in AI) is observed after 52 epochs.

It was concluded that the EE should be further continued during the next meeting cycle.

EE1-2: VLOP in-loop filter

EE1-2.1 – VLOP with new backbone block based on Spatial-Channel Mixing JVET-AP0201 (KHU, ETRI).

The key idea of Spatial-Channel mixing is Fusion of depth-wise 3×3 convolution and skip connection during inference. Inspired by inspired by the RepViT, but batch normalization is not used, GeLU was replaced with PReLU.

Complexity:

• Parameter counts: 72.98k (VLOP4: 72.72k)
• MACs/pixel: 5.06k (VLOP4: 5.09k)

Two variants were tested

• a non-re-parameterized model (training = inference), and
• a re-parameterized model (training ≠ inference), where the skip connections are removed only during inference.

EE1-2.1 tests summary

Changes relatively to VLOP4 (depicted in the next figure below) include:

• Replacement VLOP4 blocks by the SCM-based backbone block
• Channel Reduction in 1×1 convolution via SCM-based backbone block (complexity reduction)
• Tail convolution optimization for SCM-based backbone block (complexity reduction)
• Components removed when applying reparameterization (complexity reduction)

Modifications compared to VLOP4

Feedback from the cross-checkers: cross-check of inference and training is on-going. There is perfect match in inference, tests after re-training can be ready during meeting.

Re-parametrized model: training (left) and inference (right).

It was commented that the benefit of the channel spatial mixing is not obvious from the results – the benefit in complexity/runtime reduction rather comes from other changes in the backbone blocks.

It was commented that higher loss occurs for campfire, by tendency all UHD sequences have losses, while some lower resolution sequences have more gains.

Not obvious that the complexity advantage is significant enough to justify the losses in compression, in particular for the higher resolution classes.

EE1-3 NN-Inter

EE1-3.1 – Very Small Deep Reference Frame Generation Network for Inter Prediction Enhancement JVET-AP0051 (Wuhan Univ.)

Several variants for very-small DRFG network have been studied. Difference between EE1-3.1.1 and EE1-3.1.2 is the number of channels is made multiple of 16, so decoder speed-up is observed.

In test EE1-3.1.3 the DRFG tool (same design as EE1-3.1.2) alone (w/o LOP7 and NN-Intra) was tested vs NNVC configured as VTM.

Feedback from the cross-checkers: no information about cross-check at the moment report was created

Very small DRFG block diagram

EE1-3.1 tests summary (RA cfg)

Standalone gain (without other NNVC tools) is higher according to EE1-3.1.3

It was commented that the extra reference picture requires more memory. This may not be too relevant for the exploration of the technology.

EE1-3.1.2 would be the most preferred solution, according to proponents.

It was commented that the runtime numbers found by cross-checkers would be lower.

From the numbers in the table above, decoder runtime of NNVC default would increase 3.7x. This might also impact assessment of other tool modifications.

3.1.2 was initially identified as candidate for adoption

Further review was conducted after completion of training cross-check. It was reported that the result of retraining by cross-checker is below 0.1% difference when number of channels is not multiple of 16 (3.1.1), with integer accuracy in inference. For the case 3.1.2 where number of channels is multiple of 16, 0.01% difference was found in case of inference with FP, but 0.16% after integerization. Also, significantly larger decoder run time was found.

A better proposal is planned to be investigated in EE, no urgency for adoption in EE. For inclusion in the NNVC default, it would be desirable to further reduce the decoding run time.

EE1-4: Framework for externally coded pictures

Four different configurations for externally E2E AI coded picture have been tested:

• EE1-4.1 - 'Multi-layer framework (I-frame is replaced by E2E AI picture + residual + in-loop filters + multi-layer HLS, unconditionally)
• EE1-4.2 - Multi-layer framework with frame level control (I-frame is replaced by E2E AI picture + residual + in-loop filters + multi-layer HLS, based on encoder choice)
• EE1-4.3 - Single-layer framework with different E2E AI codec (I-frame is replaced by E2E AI picture unconditionally)
• EE1-4.4 - Single-layer framework with frame level control (I-frame is replaced by E2E AI picture, based on encoder choice)

Experiments were performed with different E2E AI image codec. Test results for DCVC-RT as E2E AI image coding are summarized in the next table. In the random access configuration test, the best BD-rate performance (1.7% Luma gain) was demonstrated by multi-layer framework with frame level RDO decision for inclusion or not E2E AI coded picture into the reference picture list. Frame level RDO increases CPU encoder run time by 6…8% (in average) but reduces CPU decoder run time roughly by factor ×2 since E2E AI decoding is not used only in ~60% cases. Frame level RDO improves BD-rate performance by 0.1% in presence of residual (multi-layer framework), and by 0.4% in absence of residual (single layer framework).

For single layer framework encoding run time is 4–6% faster than for multilayer frame work (no encoder search at enhancement layer). Also, there is no high-level-syntax overhead in single layer framework, so in all-intra test configuration single layer framework shows better performance compared to the multi-layer. But in random access cfg (more relevant to video coding) residuals and in-loop filters of enhancement layer improve BD-rate performance by 1.4–1.7%.

Random access test results for EE1-4 test with DCVC-RT as E2E AI image codec.

Random access test results for EE1-4 test with NLIC as E2E AI image codec.

A similar tendency is observed in NLIC (E2E AI image codec by Nokia). Partial test results for NLIC as E2E AI image coding are summarized in a table below. Full results are not available by the time of this report preparation. Partial data show slightly lower BD-rate gain and roughly twice higher CPU decoding run time compared to DCVC-RT.

None of the E2E AI image codec in this test has full SADL implementation, but for some E2E AI image codec SADL implementation and quantizing to int 16 is ongoing.

Participants of this EE1 category cross-checked inference of each other. More details can be found in JVET-AP0080 (Huawei, InterDigital), JVET-AP0182, JVET-AP0183, JVET-AP0184, JVET-AP0185 (all Nokia), JVET-AP0232 (Huawei).

Statistic for use of E2E AI image codec in multi-layer framework, depending on QP and resolution is reported in EE1-related contribution JVET-AP0144.

Conditional encoding of AI intra in the next table shows benefit (this is not currently implemented in NNVC16 SW)

Results above are only for RA, corresponding complete tables for DCVC-RT are shown below:

It is asserted from the row “no” that single layer would not give benefit at least in combination with DCVC-RT. Corresponding complete single-layer results for NLIC are not available yet.

From the current results, it can be concluded that implementing frame-level switching in the multi-layer framework gives benefit in improving compression, and reducing decoding run time. This was tested and cross-checked with both AI intra codecs.

Decision(SW): Adopt the RDO based multi-layer frame-level switching used in JVET-AP0080 and JVET-AP0183.

Further investigation in EE appears necessary on identifying potential benefits of single-layer approach.

EE1-5: Operational bit-exact reproducibility

• EE1-5.1 – [AHG11] Stable float convolution for neural network inference JVET-AP0218 (Nokia)

This test uses the mechanism known as "stable float" for neural network-based video codec inference implemented floating-point data. This technique was claimed to ensure bit-exact reproducibility by converting operands involved in convolution operations to integers, with the integer values represented as floating-point numbers. Implementation requires multi-pass (at least double pass) over latent tensor: 1) compute in and max of input data 2) re-scale and round 3) perform convolution 4) apply inverse scaling 5) add bias.

Compared to float point SADL 16 bit integer implementation for LOP7 (default in NNVC) there is minor BD-rate performance deviation in convolutions are replaced by stable convolutions. Roughly ×2 increase of CPU decoder run-time, but almost twice faster decoding using GPU was demonstrated.

Summary of EE1-5 test results (stable convolutions in NNLF) compared to int 16 SADL (default NNVC).

Bit-exact reproducibility is ensured for ‘traditional video’ coding tools by design (typically assuming no more than 16 bits precision for multipliers and no more than 32 bits for register). The comparison of stable convolution and other well-known video coding tools (like in-loop filters and transforms) is shown in the next table.

Stable convolution in comparison with traditional video coding tools.

Feedback from the cross-checkers:

• Match between CPU and GPU encoding / decoding for NNVC CTC was reported by one cross-checker (JVET-AP0278).
• Bit-exactness is only guaranteed with a full control of the environment and implementation of the inferring backend (JVET-AP0274).
• For LOP7 model parameters and intermediate buffer (between convolution) are both 16 bits. For convolutions with 3×3 kernel and just 16 channel the worst-case register accumulation is 39 bits.
• Based on specification float point format used in this implementation bit-exact operations are ensured if the overall dynamic range of accumulator is within 23 bits. Proponent confirmed that in the worst-case stable convolution would be equivalent to quantizing the model parameters and intermediate buffer to 7 bits (6 bits magnitude, one sign).

It is asserted that more investigation is necessary to resolve the problem. Integer implementation is stable, but not typically supported by GPUs so far, and FP is implemented differently on different GPUs. To be further investigated in AhG11/14 (clarify in mandates), based on the software that is available in the EE branch.