JVET-AG0216 AHG8: Multi-layer VVC for hybrid machine-human consumption [J. Laitinen, T. Partanen, A. Mercat, J. Vanne (Tampere University), A. Aminlou, M. M. Hannuksela, F. Cricri, H. Zhang (Nokia)] [late]

This contribution investigates VVC multi-layer coding to facilitate hybrid machine-human consumption, where machine-optimized input is coded as the base-layer and the original content is encoded as an enhancement-layer for human consumption. Machine-optimized input is generated using pre-processing by extracting Region-of-Interest (ROI) areas from the background. Two multi-layered schemes are tested by blurring or greying-out the background. For each multi-layered schemes four different spatial scaling ratios of 1x, 0.75x, 0.5x, and 0.25x are tested and compared to coding of original content using VVC (VVC only), and simulcasting of machine-optimized and original contents using VVC (simulcasting). The comparison for machine and human consumption are performed using mAP and PSNR, respectively.

In addition to BD-rate comparisons between the tested schemes, the contribution presents break-even points, defined as the fraction of time that the full multi-layer bitstream is required for human consumption, where the multi-layered schemes are beneficial over the VVC-only case. Based on the presented results, it is concluded that both the VVC multi-layer coding and simulcast are valid options for hybrid machine-human consumption. The contribution proposes to add a section on hybrid machine-human consumption in the technical report on optimization of encoders and receiving systems for machine analysis of coded video content.

In v2, the following items have been added:

• A detailed analysis of 0.25x downsampling of machine consumption using extended QP
• The proposed text to be included in the technical report
• Excel files of results
• Presentation slides

A potential application is security monitoring, where the machine (getting base layer) is analysing the video all the time, and the human only sporadically. It would not be beneficial for storage, where typically the single layer VVC at a quality for human consumption would be stored. For streaming, a benefit is claimed when the human does not watch all the time. Potentially, instead using scalable coding, a switch to a higher-quality single layer stream could be made.

Another expert commented that in terms of coding efficiency there could also be cases where using the human stream as the base layer is better.

More evidence about the benefit would be necessary for inclusion in the technical report. There are many side assumptions to demonstrate that bit rate is saved.

Further study of this was requested.