JVET-AQ0023 EE1: Summary report of exploration experiment on neural network-based video coding [E. Alshina, R. Chang, F. Galpin, Yue Li, Yun Li, M. Santamaria, T. Shao, J. Ström, Z. Xie (EE coordinators)]

Summary of conducted tests

• EE1-1: LOP in-loop filter

Comparison point NNVC-LOP in-loop filter:

• Parameter counts: 242 k
• MACs/pixel: 16.5 k

• • EE1-1.1 – Dynamic convolution for LOP7 neural in-loop filtering JVET-AQ0071 (KHU, KBS). Cross-checker: Huawei (training JVET-AQ0178), Samsung inference JVET-AQ0193).
• EE1‑1.1.1: Integer model of dynamic convolution.
• EE1‑1.1.2: 1.1.1 + Y–UV training‑path refinement for balanced coding gain –priority #3 (withdrawn)
• EE1-1.1.3: 1.1.1/1.1.2 + Reduced training-time strategy.

All tests will have the same complexity

• • Parameter counts: 835 k
  • MACs/pixel: 16.52 k

In this test fixed-kernel static convolution layers in the LOP7 in-loop filter are replaced with dynamic convolution. Unlike static convolution, which uses a single fixed kernel for all inputs, dynamic convolution generates a per-sample effective kernel as a weighted combination of multiple expert kernels, where the weights are derived from the input feature via a lightweight router (fully connected layer) as shown in Figure 1-Figure 2.

Changes compared to LOP7 are illustrated on Figure 2: Blue-shaded CONV blocks are replaced with dynamic convolution (DynConv); the yellow-shaded CONV block indicates the chroma input convolution whose output channel count is d₂ (the parameter varied across sub-tests). Subtests use different number of expert kernels and different chroma input channels (d₂).

The only difference between Tests EE1‑1.1.1 and EE1‑1.1.3 is faster training time. Proponent reports that Stage 3 training for EE1‑1.1.1 requires 9.4 days, but for EE1‑1.1.3 the same training requires only 5.4 days. For anchor (LOP7) the same training requires 4.6 days. The training configuration used in EE1-1.1.3 relies on optimizations designed for NVIDIA Ampere (sm_80) or newer architectures, including Automatic Mixed Precision (AMP), torch.compile, and cudnn.benchmark. Accordingly, the reported training time and memory usage are representative only on Ampere-class or newer GPUs (e.g., A100, RTX A6000 Ada, and H100). On earlier architectures, such as the V100, these optimizations may not be fully supported, resulting in different memory usage and training time.

Figure 1 Difference between static and dynamic convolutions.

Figure 2 Process of router-based expert kernel generation.

Figure 3 Modifications compared to LOP7

^

Table 1 EE1-1.1 tests summary

Feedback from the cross-checkers:

For both EE1-1.1.1 and EE1-1.1.3 perfect match was observed during inference cross-check.

After re-training of EE1-1.1.1 the BD-rate difference (Y component) of 0.1% (RA cfg) and 0.0% (all intra cfg test) was observed.

Re-training of EE1-1.1.3 is still ongoing (about a half of epochs is done).

Observations /conclusions:

1. dynamic convolution introduced into NN-based LOP in-loop filter, improve compression performance 0.6..0.7% BD-rate (same kMAC/pxl, ×4 larger model)
2. The increase of training time compared to the LOP7 has been reduced to just 20% (5.4 days vs 4.5 days)

From discussion in JVET Wednesday 8 July:

- Gain of 1.1.1 and 1.1.3 is attractive, kMAC not increased relative to LOP7, but both come with substantial increase of memory usage, and increase in training time (less for 1.1.3, but training crosscheck not yet finished).

- Also increase in decoding time (on CPU), and could also be more difficult to implement on GPU.

- For the NNVC exploration, the dynamic convolution is interesting, even though it may be less practical for implementation than LOP7 – Candidate for adoption.

In the discussion in the JVET session on July 13^th 1600h it was confirmed by the software coordinator that the SADL implementation has a sufficient quality and does not break existing tools.

It was further reported that EE-1.1.1 has been confirmed previously. The simulations confirming the performance of EE-1.1.3 were started after successful training. It was considered unlikely to receive full results of the simulations by the end of the meeting. It was suggested to conditionally adopt EE-1.1.3 based on confirmation in the AHG phase. Otherwise, EE-1.1.1 would be adopted. EE-1.1.1 was suggested to be integrated into the NNVC software and potentially be replaced later upon successful performance confirmation.

Decision: Adopt JVET-AQ0071 EE1-1.1.1. as LOP8 (also replacing LOP7 in NNVC default). If the training crosscheck of EE1-1.1.3 is successful by the time of the AHG telco planned on July 30, the model from 1.1.3 shall be used instead for LOP8.

• EE1-2 NN-Inter

Comparison points LDRF1 / HDRF1:

• Parameter counts: 3600 k / 6064 k
• MACs/pixel: 62 k / 477 k
  • EE1-2.1 – Very Small Deep Reference Frame Generation Network for Inter Prediction Enhancement JVET-AQ0047 (Wuhan Uni) Cross-checker: InterDigital (inference JVET-AQ0137), Huawei (training and inference - JVET-AQ0217)

EE1-2.1/2 VLDRF / VLDRF(16)

• Parameter counts: 243 / 257 k
• MACs/pixel: 8.8 / 9.7 k
  • EE1-2.2 – Improved H-DRF with Weighted Fusion and Optimized YUV Processing JVET-AQ0048 (Wuhan Uni) Cross-checker: InterDigital (inference JVET-AQ0137), Huawei (model quantizer and inference – JVET-AQ0206, JVET-AQ0216)
• Parameter counts: 6064 k
• MACs/pixel: 477 k

Changes are

• • Frame fusion is changed from simple averaging to HardSigmoid-based (analog of clipping) weighted fusion.
  • YUV-to-RGB conversion is removed from optical-flow computation by keeping H-DRF and L-DRF inference in the YUV domain.
  • The down/up sampling method for YUV420/YUV444 of H-DRF is changed from bilinear to Lanczos-3, aligning with the L-DRF inference procedure.
  • The quantization method for the GridSample layer is optimized.

Figure 4 Processing flow of the GridSample layer

Figure 4 shows the processing flow of the `GridSample` layer. `GridSample` uses the input tensor and optical flow to generate a flow-warped output tensor. During grid generation (green block), terms such as 2/(W-1) and 2/(H-1) are involved, which can introduce additional error during float32 to int16 conversion. Therefore, quantization precision of grid generation (green block) is increased when W/H become larger to improve coding performance.

The quantization operation is performed as:

In the previous implementation, the quantization parameter for the grid generation part was fixed to Q=11 for all spatial resolutions.

In the optimized version, the quantization parameter is empirically selected according to the spatial resolution. Q=14 is essential for 256×256 to reduce the quantization error, while lower Q values are sufficient for other resolutions with limited performance impact:

This adaptive quantization improves the precision of the grid generation process and reduces the float32-to-int16 conversion error.

The GridSample grid generation contains two parts. The base grid generation is handled during the offline quantization process. During inference, the optical flow also needs to be normalized to the [-1,1] coordinate system by applying the scale factors 2/(W−1) and 2/(H−1) before being added to the base grid.

• • • EE1-2.2.1 changes are applied to the HDRF (the most more complex DRF model in NNVC)--> HDRF2
    • EE1-2.2.2 same changes applied to the LDRF (middle complexity DRF model in NNVC)-->LDRF2
  • EE1-2.3 – Deep Reference Frame Generation for Inter Prediction Enhancement with motion compensation JVET-AQ0049 (InterDigital) Cross-checker: Wuhan Uni (only inference, training is not modified JVET-AQ0134)

Figure 5 illustration of the motion compensation of the input patches 0 and 1 (EE1-2.3)

The patches used as input of a DRF models are first displaced (pel motion) using the collocated motion vector.

For YUV420/YUV444 instead of bi-linear of Lanzoc-3 (EE1-2.2) VVC-RPR filters are used.

• • EE1-2.4 – Combination of test 2.3 (Deep Reference Frame Generation with motion compensation) and tests 2.1 and 2.2 JVET-AQ0050 (Wuhan Uni, InterDigital) Cross-checker: Huawei (training and inference – JVET-AQ0213)

Table 2 EE1-1.2 tests summary

Feedback from the cross-checkers:

EE1-2.1.1 was successfully cross-checked (training and inference).

EE1-2.2.1 (HDRF) training is still not yet cross-checked successfully

EE1-2.2.2 (LDRF) after re-training the difference is 0.13% (BD-rate Y).

Inference cross-check is succesful.

Figure 6 DRF test results visualized: BD-rate vs complexity

Observations /conclusions:

1. both encoding and decoding run time for LDRF relatively to NNVC has been reduced compared to the last EE1 round (likely due to the SADL optimization).
2. Improvements from EE1-2.2 (Weighted Fusion and Optimized YUV Processing) improve performance of both LDRF and HDRF by 0.3%.
3. Patched displacement from EE1-2.3 improve compression performance of all versions of DRF by 0.3..0.4%. This change is generic and complementary to improvements from EE1-2.2 as was shown in EE1-2.4.

From discussion in JVET July 8:

- 2.1.1 and 2.1.2 (VLDRF) successfully crosschecked, provide gain of 0.4%/0.5%, seem only partially overlapping with other NNVC tools (2.1.3*).

- With similar changes to optical flow estimation applied to HDRF (2.2.1) and LDRF (2.2.2) additional gain is achieved, but decoder runtime shows substantial increase in particular in case of HDRF

- 2.3.x is adding a global motion estimation to initialize the local patches in the existing models of LDRF and HDRF, and uses correct chroma location of sequences in interpolation/downsampling

- 2.4.x is applying 2.3.x to new models of VLDRF, LDRF, and HDRF.

- 2.4.2 shows 0.8% gain for VLDRF(16), 2.4.3 2% gain for LDRF, both fully cross-checked

- Opinion was expressed that, though the tradeoff of best VLDRF vs. decoding time increase of 3x is not attractive as such, it would not put a burden in running the NNVC default (encoding time is not increased)

Decision: Adopt JVET-AQ0050, containing 2.4.2 VLDRF(16) to be added to the NNVC default, 2.4.3. to replace current LDRF1 (to become LDRF2), and 2.4.4. to replace current HDRF1 (to become HDRF2)

Follow-up discussion on Tuesday 14 July, to reflect some intent in the original discussion on July 8, where the following was forgotten to be marked as revisit. EE1-2.2.1 (HDRF modifications) cross-check completed, was added to the adoptions under JVET-AQ0050.

• EE1-3: Single layer framework for externally coded pictures

Withdrawn

• EE1-4: NN-Intra bit-exact reproducibility

Comparison point NNVC-NN-Intra:

• Model weights - 16 bits, ‘off-line’ quantization
• Activations - 16 bits (bit-depth of temporal buffer between convolutions)
  • EE1-4.1 – Adaptive Quantization and Hardware Optimization for NNIP JVET-AQ0182 (Qualcomm) Cross-checker: Huawei (inference, training is confirmed in earlier tests – JVET-AQ0218), InterDigital (inference – JVET-AQ0205).

• EE1-4.1.1, 10-bit dynamic activation quantization and 8-bit static weight quantization: 0.00% luma loss in RA configuration.
• EE1-4.1.2, 8-bit dynamic activation quantization and 8-bit static weight quantization: 0.03% luma loss in RA configuration.
• EE1-4.1.3: withdrawn?
• EE1-4.1.4a, 10-bit static activation quantization and 8-bit static weight quantization: 0.18% luma loss in RA configuration.
• EE1-4.1.4b, 12-bit static activation quantization and 8-bit static weight quantization: 0.04% luma loss in RA configuration.
• EE1-4.1.4c, 14-bit static activation quantization and 8-bit static weight quantization: 0.00% luma loss in RA configuration.

Dynamic quantization approach, each 8-element activation block is quantized at runtime based on its local dynamic range. For every 8‑element block, the maximum absolute value is determined, and a corresponding right shift is computed so that all values fit within the target bit‑depth.

Static quantization in this test: for each 8‑element block, required shift values are collected off-line over a representative dataset (BVI-AOM), and a single shift value is selected based on a predefined criterion. During inference, each block is quantized using its precomputed shift and clipped to the target bit‑depth. During the matrix multiplication, the applied block‑wise shifts are compensated in the accumulation process to preserve the correct numerical scaling.

Minor mismatch between results from proponent and cross-checkers have been observed (4^th digit after decimal point). Proponent and cross-checkers exchanged streams, logs, found perfect match in per-frame results. So the issue relates to parsing scripts.

Figure 7 EE1-4.1 Trade-offs results (BD-rate numbers is from random access configuration test).

Beside of bit-depth reduction for weights and activations two changes which speed up NNIP have been introduced:

• Fusion of Activation Functions into Matrix Multiplication (leaky ReLU is fused with fully connected layer) – modification of NNIP which potentially changes results (less values will be clipped)
• SIMD‑Friendly Sparse Matrix Multiplication (sparse weight matrices are packed in block sizes aligned to SIMD execution 8-elements units) – implementation aspect, doesn’t change results

Model parameters quantized statically in all tests. Activations are quantized dynamically or statically. Shift values for each 8-elements block becomes a part of the model. Total number of parameters increases by 12.5%. Since bit-depth of model parameters reduced to 8 bits (in all tests) the total size of memory for model parameters is reduced by half (dynamic quantization) and by 43% in case of static quantization. In the Table 3 Enc and Dec run time are from proponent.

One cross-checker also performed limited test out-side of NNVC CTC, using synthetic image.

Table 3 EE1-1.3 tests summary (anchor NNVC17.: LOP filter on, NN-Intra on)

Observations /conclusions:

1. static quantizing of NNIP model parameters to 8 bits doesn’t lead to the performance degradation
2. additionally dynamic quantizing of activations in NNIP model to 8 bits lead to the negligible performance degradation (0.0% random access and 0.1% in all-intra tests)
3. in performed tests dynamic quantizing of activations in NNIP increases decoding run time 5% compared to the static quantizing (stat. sa10w8 and dyn. a10w8 comparison)
4. the performance of NNIP is much more sensitive to the bit-depth of activations: static quantizing of activations to 14 bits – no drop, to 10 bits 0.6% BD-rate drop in ‘all intra’ test.
5. bit-depth of model parameters has been reduced to 8 bits, so the total size of memory for model parameters is reduced by half in case of dynamic quantization and by 43% in case of static quantization.

From discussion in JVET July 8:

- The main intent of EE1-4.x is to avoid overflow that could accidentally happen in NN intra, and could cause mismatch between different devices.

- Cross-check JVET-AQ0218 shows for the example of a non-CTC image (screen content with large flat area) that dynamic quantization (4.1.1/4.1.2) provides >1% better performance than static. It is asserted that this may not be too relevant as the image is simple.

- Concern was raised that the dynamic quantization solutions could be problematic in SADL.

- Among the static solutions, 4.1.4c has practically no loss under CTC, and would be a safe choice, solving the issue without problems in implementation.

- It was commented that in a standard specification, it would need to be clearly defined how to handle overflow such that no encoder/decoder mismatch occurs. In this context, implementation friendliness as well as performance would be relevant. In the context of the NNVC exploration, 4.1.4c is a suitable solution to solve the issue.

The adoption of 4.1.4c was discussed in the session of July 13^th 1600h. It was reported that the implementation inside SADL would imply difficulties and that the logic pertaining to model quantization should be integrated into the NNVC software. It was reported that some small changes to SADL would still be expected. It was agreed tentatively adopt 4.1.4c subject to review of the implementation in a early meeting of the AHG and confirming the appropriateness of the integration.

Decision: Conditional adoption of JVET-AQ0182 EE1-4.1.4c. To be confirmed by AHG telco on July 30, provided that the corresponding is confirmed to be appropriate by the software coordinator.