JVET-AN0023 EE1: Summary report of exploration experiment on neural network-based video coding [E. Alshina, R. Chang, F. Galpin, Yue Li, Yun Li, M. Santamaria, T. Shao, J. Ström, Z. Xie (EE coordinators)]

Code base for the EE1 tests was NNVC14.0, anchor is default configuration of NNVC-14.0 (NN-Intra and LOP6 filter enabled). NNVC common test conditions, results and complexity reporting template were be used.

For proposals in all categories, proponents used AhG11 training set, which consists of DIV2K, BVI-DVC, TVD. NN-based Inter tools were trained using Vimeo 90K triplet. Comparison is done between tests which use the same sub-set of training data.

For tests competing with technologies in NNVC it was agreed to configure the proposed solution targeting the complexity close to existing NNVC tool, but not exceeding it:

1. kMAC/pxl of EE1 test ≤ kMAC/pxl NNVC (must),
2. the number of channels (both input and output) in neural network modules which are modified must be keep multiple of 16,
3. Number of Parameters EE1 test ≤ Number of Parameters NNVC (if possible).

If it is not possible to respect both constrains 1) and 2) at the same time then two sub tests need to be performed: first test respecting the constrain 1) only – in order to prove the value of the proposed change, and the second test respecting constrain 2) with minor violation of 1) - in order to provide software and hardware friendly design.

According to the EE description only tests with results provided to quantized (int 16) model are considered for adoption to NNVC, float point model results are provided just for information.

Exact parameters settings were announced by proponents by 2^nd AhG11/14 teleconference on August 28.

Inference cross-check is required for all EE1 tests. Candidates for adoption to NNVC are required to undergo training cross-check, implementation must be compatible with SADL.

This round of EE1 tests includes:

EE1-1: LOP in-loop filter

EE1-2: VLOP in-loop filter

EE1-3: NN-inter prediction

EE1-4: NN-based super-resolution

Anchor for ALL tests is NNVC-14.0 with NN-Intra and LOP6 enabled (default configuration).

EE1-1: LOP in-loop filter

• EE1-1.1 – LOP with Overlapped Feature Integration  JVET-AN0127 (UESTC, Transsion) (crosscheck JVET-AN0281)

Key changes in head block of LOP filter are additional operations over Luma reconstruction and prediction input:

1. Boundary Cropping: Crop 1 pixel from the left/right columns and top/bottom rows of the original block, retaining the central (W-2)×(H-2) region.
2. Mirror Symmetric Padding: Restore the original block size by padding mirrored values from the cropped boundaries.
3. 2×2 DCT Transformation: Apply 2x2 DCT-II to the padded block.

The changes introduced by proposed solution (on a right) relatively to LOP6 (on a left) are highlighted in the diagram below:

Ablation study:

• EE1-1.1.1: retraining LOP6 w/o change of architecture, use changed learning rate.
• EE1-1.1.2: only change architecture w/o overlapped feature.
• EE1-1.1.3.1: DCT II added to Sub-test 1.1.2 (full proposed package).
• EE1-1.1.3.2: DCT II added to Sub-test 1.1.2 (full proposed package), use changed learning rate.

Complexity measured in kMAC/pxl remains close to the LOP6 (16.57 kMAC/pxl), number of parameters are not reported, training time (only stage 3) is 7 days. PSNR drop for SlideShow is 0.4% (all-intra cfg test).

Encoder run time is unreliable, decoder run time claimed to be reliable, indicates reduction of decoding time (10%) due to SIMD acceleration for the group convolution (not yet available in SADL, but proposed in JVET-AN0128) and aligning channels number with multiple of 16. Cross-checker reported status of cross-check via e-mail:

• for inference:
  • successful inference xcheck of EE1-1.1.1, EE1-1.1.2, EE1-1.1.3.1 and EE1-1.1.3.2 under AI configuration, all results matched; and
  • all tests under RA configuration are still under-going and results are not available.
• for training:
  • as requested by proponents, only training of EE1-1.1.3.2 was cross-checked
    • successful training xcheck of EE1-1.1.3.2 under AI configuration, {Y: 0.01%, U: -0.16%, V: 0.12%} compared with JVET-AN0127.
    • RA configuration are still under-going, RA results are expected to be obtained within 5 days.
• EE1-1.2 –Backbone Block Enhancement of LOP In-Loop Filter with Over-Parameterized Training and Variable Channels  JVET-AN0083 ((training cross-check JVET-AN0335 )

Re-parametrization is only training modification. Three variants of over-parametrized modules were used during training, but in the inference, model returned to the original network structure. Training time increase 10h – 190 h.

Normative changes are in the chroma branch, to provide BBB(x) with richer input information, the proposed method increases the number of input channels of BBB(1) and gradually reduce the number of input channels of BBB(0) accordingly. The number of input channels of the five BBB(x)s in the chroma branch gradually decreases from 128 to 64.

SIMD acceleration for the depth-wise separable convolution at the boundary was provided. This acceleration is also applicable to the anchor. Run time reported relatively to accelerated NNVC-14.0.

Computational complexity reduced 16.60 -> 16.56 kMAC/pxl.

Training was successfully cross-checked (two runs of training ended up in very close performance).

EE1 LOP in-loop filter modifications

It was commented that gain of both 1.1 and 1.2 is relatively low (within the 0.1% margin of training uncertainty). Basically, both could be combined, but then re-training in combination would be necessary.

For 1.1, training cross-check is not finalized yet. It shows an interesting decoding runtime reduction, which however is due to SIMD optimization in SADL which might likewise be applied to the anchor.

For 1.2, both inference and training cross-check are finalized. A concern was raised how much of the gain comes from over-parameterization using the new training strategy, and how much gain is due to the architectural change.

1.2 should be further investigated in an EE on the aspect of benefit of the architectural change. It appears that the overparameterization could stabilize the training of LOP6 also without architectural change (which was not tested so far).

It was commented that the reported gain of both 1.1 and 1.2 appears too small to justify architectural changes. Proponents might consider if a combination could be beneficial to be investigated in an EE.

1.1.1 is retraining of LOP6 without architectural change.

Further consideration of variant 1.1.3.2 was planned after completion of training cross-check and potential confirmation of the slightly higher gain (to be compared against 1.1.1 which does not apply an architectural change).

Further discussion Friday 10 Oct. 12:00

In an update of JVET-AN0023, more results are reported that after completion of training cross-check, the results on AI were within the usual margin, but in RA a larger deviation was observed.

Cross-check JVET-AN0281 not yet available.

Further study was planned in EE1.

EE1-2: VLOP in-loop filter

• EE1-2.2 – Decomposed Content-Adaptive VLOP JVET-AN0153 (registered but not uploaded cross-check JVET-AN0282)

The comparison for this test is caVLOP4 (content-adaptive VLOP4).

Claimed performance benefits over caVLOP4 are as follows.

EE1 cVLOP in-loop filter modifications

Performance of ‘VLOP’ family in-loop filters in NNVC vs VTM (NNVC all tools off)

Comments from cross-checker: inference cross-check matched, training is still ongoing.

Gain for the content-adaptive VLOP4 appears interesting. However, the relatively large increase in decoding time is of concern, which appears larger than with the current CA-VLOP1 implementation. This may be an implementation issue, or may be caused by the decomposition approach. One reason may be that the model needs to be re-compiled and re-loaded each time it is changed.

Further consideration on Friday 10 Oct. after completion of cross-check. The proponents were asked to investigate why the decoding runtime is more significantly increased than for the existing CA-VLOP1 implementation. In the crosscheck, the gain is confirmed. After further investigation, it was found that the decoder runtime is accurate and does not significantly increase (in the initial report, an error was made in the reference to compare run time)..

Decision: Adopt JVET-AN0153 test EE1-2.2.

EE1-3: NN-inter prediction

Anchor for tests in this category is NNVC-14.0 with NN-Intra and LOP6 enabled (default configuration).

Deep Reference Frame (DRF) was adopted to NNVC at JVET 39^th meeting. It is disabled by default.

Results for DRF in NNVC are provided as reference. As was agreed during NNVC teleconference adopted version of DRF is not enabled in LDB cfg (model was not trained for those purposes).

• EE1-3.1 – Deep Reference Frame Generation for Inter Prediction Enhancement with Structural Re-parameterization JVET-AN0193 (training and inference cross-check JVET-AN0304)

Significant reduction of model size and number of multiplications by using light version of DRF and structural re-parameterization. Training uses sophisticated loss: CharbonnierLoss + TernaryLoss. Training set is Vimeo-90K triplet. Training takes approx. 80 hours.

Cross-checker performed re-training, after re-training deviation from proponent’s results in average is {0.15%, -0.53%,0.64%} BD-rate for {Y, U, V} components, but bigger difference (0.3%) is observed for classes A2 , 0.14% for A1 and B, for other classes within usual margin.

• EE1-3.2 – RA/LDB Reference Frame Synthesis for VVC Inter Coding  JVET-AN0084 (cross-check JVET-AN0306 ongoing)

Training set is Vimeo-90K triplet, Loss is L1. Training uses video data compressed with QPs 7,12,17,22,27,32,37,42). Training takes approx. 120 hours. Decoding is still in progress (partial data available).

Cross-check status has been reported by cross-checker via e-mail:

1. Inference crosscheck failed for LDB (decoder crash)
2. Inference crosscheck for RA was not bit exact (they have floating point operations in their inference code), there are also a crash in Class A
3. Training crosscheck for RA model was done – Chroma results varied by 1.21%. Luma was okay.
4. Inference crosscheck of float models did not show the same results. Maybe the models they provided were wrong.

• EE1-3.5 – Retrained DRF in NNVC14.0 JVET-AN0195 (training and inference cross-check JVET-AN0305)

During porting of adopted DRF version to NNVC, colour conversion process was modified compared to original version, so inference was different from training. Retraining was needed to align training and inference. Training takes approx. 50 hours.

Cross-checker performed re-training, after re-training deviation from proponent’s results is negligible (below 0.1%).

EE1 DRF family of tools in NNVC

EE1-3.1 would be interesting as additional operation point with lower complexity of DRF.

It was reported by proponents of EE1-3.1 that they are currently are trying to identify the problems leading to partial deviation in training crosscheck. The cross-check was not finalized by the end of the meeting, and will be finalized in further EE study.

EE1-3.2 is immature (not to be continued in an EE)

Decision: Adopt JVET-AN0195 EE1-3.5 (re-training the current DRF model).

EE1-4: NN-based super-resolution

Per agreement during the second NNVC teleconference test conditions for NN-based super-resolution tests are aligned with RPR in NNVC:

NNSR testing conditions are:

• Multiple scaling ratios are used (x1.5 and x2)
• Reduced resolution is allowed for QP>=32 (base QP)
• Enabled only for 4K classes
• QP offset for chroma is the same for A1 and A2 (no per sequence adaptation).

EE1-4.1   NNSR with new backbone block based on Spatial-Channel Mixing (SCM) JVET-AN0201 (registered, but not yet available cross-check JVET-AN0277)

The key element is a new backbone block for NNSR that performs spatial mixing followed by channel mixing. No run time provided. Some (0.1…0.4%) BD-rate gain in Chroma compared to NNSR in NNVC-14.0.

Two variants are provided with C_Y1=32 (kMAC/pxl 4.05, Nparam 17k) and C_Y1=16 (kMAC/pxl 3 Nparam 12k).

• EE1-4.2 Cross-component enhanced NNSR JVET-AN0238 (cross-check JVET-AN0256)

Proposed changes enhance feature interaction between luma and chroma components. Some (0.1…0.4%) BD-rate gain in Chroma compared to NNSR in NNVC-14.0. Increase of kMAC/pxl compared to NNSR in NNVC-14.0

Anchor for tests in this category is NNVC-14.0 with NN-Intra and LOP6 enabled (default configuration).

Results (generated by AhG14) for RPR and NN-SR in NNVC are provided as reference.

Cross-check of inference – match, after re-training by cross-checked Chroma BD-rate drop 0.1..0.2%.

Run times are reported to be similar to current NNVC-NNSR.

EE1-4.1.2 could be interesting, having significant complexity reduction compared to current NNSR without impact on gain. After completion of cross-check JVET-AN0277, it was found that the significant gains in class A1 that had been reported by proponents could not be confirmed, the cross-check reports slight losses. For other classes, the method did not change much in terms of compression results. Further investigation in an EE was planned.

EE1-4.2 is not attractive, no benefit in compression, slight complexity increase compared to current NNSR.