JVET-J0034 AHG9: CNN-based driving of block partitioning for intra slices encoding [F. Galpin, F. Racapé, P. Bordes, F. Le Léannec, E. François (Technicolor)]
This contribution describes in more details the convolutional neural network (CNN) based algorithm used in the MTT codec presented in the CfP response JVET-J0022, for driving the block partitioning in intra slice encoding.
A CNN-based encoding approach is explored to partly substitute heuristics-based encoder speed-ups by a systematic and automatic process. The approach allows controlling the trade-off between complexity and coding gains, in intra slices, with a single parameter. This contribution reports, in AI configuration, a BD-rate gain of 6% for the method in the MTT codec presented in JVET-J0022, compared to JEM7 at the same encoding runtime, and for the same BD-rate performance as JEM7, the average encoding runtime is reportedly reduced by a factor of 4.3.
The luma CTU size is 256x256, with the first split inferred; the second split based on RDO, with splits from 64x64 and lower being CNN based.
The CNN was derived from the ResNet architecture, with one CNN for luma and one shared for Cb and Cr. The input to the CNN is 65x65 patches and associated QP values. The output is a vector with partition boundary probabilities.
The network is trained with partition boundary choices obtained by conventional RDO.
With CNN based decisions, QT/ABT reportedly outperforms the JEM starting from 50% runtime.
Questions and comments from the discussion included:
- Currently this was only used for intra coding; an investigation for inter was reportedly ongoing.
- It was asked why ResNet was selected. No other architectures had been investigated.
- It was asked what influences the runtime. Ultimately, the number of split candidates that are checked by conventional RDO is the control mechanism , which are selected as the ones that the network marks as the most probable.