Back to Search Document details
32nd Meeting: Hannover, DE, October 2023 2023-10-16 17:13
AHG8: An exemplar software implementation for temporal resampling for VCM
Authors: Shurun Wang Alibaba Jie Chen Alibaba Y. Ye Alibaba Shurun Wang CityU
Abstract
Sections on using temporal resampling to improve coding efficiency for video coding for machines have been included in JVET-AD2030 “Optimization of encoders and receiving systems for machine analysis of coded video content (draft 2)” at the conceptual level, but without providing any detail. This contribution proposes a temporal resampling software implementation for inclusion in JVET-AD2030. The proposed implementation can achieve 22.12%, 30.03% and 75.68% average Pareto BD-rate improvements for random access (RA), low delay (LD) and all intra (AI) configurations for SFU-HW dataset under the common test conditions for optimization of encoders and receiving systems for machine analysis of coded video content as defined in JVET-AE2031. For TVD dataset, 45.39%, 35.50% and 73.72% average Pareto can be achieved for RA, LD and AI configurations, respectively.
Meeting Notes
JVET-AF0060 AHG8: An exemplar software implementation for temporal resampling for VCM [S. Wang, J. Chen, Y. Ye (Alibaba), S. Wang (CityU)]

Sections on using temporal resampling to improve coding efficiency for video coding for machines have been included in JVET-AD2030 “Optimization of encoders and receiving systems for machine analysis of coded video content (draft 2)” at the conceptual level, but without providing any detail. This contribution proposes a temporal resampling software implementation for inclusion in JVET-AD2030. The proposed implementation can achieve 22.12%, 30.03% and 75.68% average Pareto BD-rate improvements for random access (RA), low delay (LD) and all intra (AI) configurations for SFU-HW dataset under the common test conditions for optimization of encoders and receiving systems for machine analysis of coded video content as defined in JVET-AE2031, when VTM version is 20.0 and temporal resampling ratio is 4. For TVD dataset, 45.39%, 35.50% and 73.72% average Pareto can be achieved for RA, LD and AI configurations, respectively.

When VTM version is 12.0, it is reported that the following performance could be achieved:

For object tracking

  • temporal resampling ratio 2: -30.32% (RA), -23.41% (LD)
  • temporal resampling ratio 4: -44.20% (RA), -34.38% (LD), -77.35% (AI)

For object detection

  • temporal resampling ratio 2: -19.80% (RA), -20.52% (LD), -52.79% (AI)
  • temporal resampling ratio 4: -21.34% (RA), -29.67% (LD), -75.05% (AI)

It was asked what the benefit of the frame interpolation actually is? How would it compare against a VTM where just the same frame skipping is done at the encoder, and the temporally subsampled video is fed into the machine vision task? How would it compare to a simpler conventional upsampling?

What is the complexity of the additional processing at the decoder? Roughly 20-30x runtime increase.

It was commented that in video surveillance frame dropping might not be appropriate.

It was pointed out that when temporal subsampling is done, the access period should still be retained as 1 second in RA case. In the experiments, the original period of 64 frames (which would be relevant for 60 fps) was used, but with a subsampling by factor 4 it should have been 16. This might significantly increase the rate.

Cited:
JVET-AF0308 X
ARCHIVE :
._JVET-AF0060-VTM12_2x.xlsm ._JVET-AF0060-VTM12_4x.xlsm ._JVET-AF0060-VTM20_4x.xlsm ._JVET-AF0060_v2.docx ._JVET-AF0060_v2.pptx JVET-AF0060-VTM12_2x.xlsm JVET-AF0060-VTM12_4x.xlsm JVET-AF0060-VTM20_4x.xlsm JVET-AF0060_v2.docx JVET-AF0060_v2.pptx
Decisions
It was pointed out that when temporal subsampling is done, the access period should still be retained as 1 second in RA case. In the experiments, the original period of 64 frames (which would be relevant for 60 fps) was used, but with a subsampling by factor 4 it should have been 16. This might significantly increase the rate.
Citation