Back to Search Document details
36th Meeting: Kemer, TR, November 2024 2024-11-01 09:11
EE1 Related: Transformer-Based Reference Frame Synthesis for VVC Inter Coding
Authors: Qipu Qin Xidian Univ. Cheolkon Jung Xidian Univ.
Abstract
This contribution presents Transformer-based reference frame synthesis for VVC inter coding, named TRFS. TRFS operates between the decoded picture buffer (DPB) and reference picture lists (RPLs) in VTM-11.0_nnvc-10.0 to generate new reference frames through spatiotemporal compensation between previously reconstructed frames. TRFS is composed of a hierarchical feature extractor (HFE), a scale-adaptive flow estimator (SAFE), and a U-Net frame enhancer (UFE). The experimental results show that:
Meeting Notes
JVET-AJ0148 EE1 Related: Transformer-Based Reference Frame Synthesis for VVC Inter Coding [Q. Qin, C. Jung (Xidian Univ.)]

This contribution presents Transformer-based reference frame synthesis for VVC inter coding, named TRFS. TRFS operates between the decoded picture buffer (DPB) and reference picture lists (RPLs) in VTM-11.0_nnvc-10.0 to generate new reference frames through spatiotemporal compensation between previously reconstructed frames. TRFS is composed of a hierarchical feature extractor (HFE), a scale-adaptive flow estimator (SAFE), and a U-Net frame enhancer (UFE). The experimental results show that:

  • For network complexity, TRFS has only 2.58M parameters and a multiply-accumulate (MAC) operation of 330kMAC/pixel.
  • Based on VTM-11.0_nnvc-10.0 (NN-Tools ON) under RA configuration, TRFS achieves bitrate reductions of {4.43%, 6.43%, 7.04%} for {Y, U, V} components, respectively.
  • Compared with VTM-11.0_nnvc-10.0 (NN-Tools OFF) under RA configuration, TRFS achieves bitrate reductions of {6.61%, 13.86%, 13.30%} for {Y, U, V} components, respectively.
  • A figure below illustrates the entire framework of the proposed TRFS-Net integrated into VVC. Two decoded frames {RA: and } from DPB are input to TRFS-Net to generate a reference frame that is inserted into RPLs for subsequent processing of inter prediction. As shown in the figure, TRFS-Net is composed of a hierarchical feature extractor (HFE), a scale-adaptive flow estimator (SAFE), and a U-Net frame enhancer (UFE).

Entire framework of TRFS-Net for VVC reference frame generation.

Illustration of the hierarchical feature extractor (HFE) in TRFS-Net
based on Transformer blocks and residual down-sampling blocks.

It was commented that an integer implementation might be difficult for the strong nonlinearities in the transformer stages.

For training, vimeo was used.

It was agreed to investigate this in an EE. Also study the impact of trying integerization by replacing strong nonlinearities, and impact of training materials (using BVI and TVD). It was further requested to study the impact of keeping the number of reference frames constant rather than extending by the new generated frame, and its position (first in L0 might be useful).

ARCHIVE :
JVET-AJ0148-v3.docx JVET-AJ0148_slides.pdf TRFS_vs_NNVC10.0_NNTools_OFF.xlsm TRFS_vs_NNVC10.0_NNTools_ON.xlsm
Decisions
It was agreed to investigate this in an EE. Also study the impact of trying integerization by replacing strong nonlinearities, and impact of training materials (using BVI and TVD). It was further requested to study the impact of keeping the number of reference frames constant rather than extending by the new generated frame, and its position (first in L0 might be useful).
Citation