JVET-AM0073 [AHG11] Comparison of execution time of NN on different devices [A. Karabutov, E. Alshina, F. Brand (Huawei)]

This contribution provides information about the execution time of neural networks across different devices using the JPEG-AI framework. Two devices were tested: a Huawei Mate50 Pro and a high-end PC with an RTX 3090 GPU. Experiments focused on decoding a 4K stream and varying image sizes, with the prediction and synthesis parts of the codec optimized into a single model for efficiency. The results showed that the ratio between running time of a NN model on CPU and GPU depends on the model complexity in terms of kMAC/px and number of processed pixels. It was proposed empirical formula to calculate the ratio of inferencing time on CPU/GPU for used hardware in the experiments. It is applicable to convolutional type of networks. For attention-based networks it was observed non-linear dependency between CPU and GPU running time, therefore they were excluded from consideration. The formula shows that considering only CPU runtime and complexity in kMAC/px NN model is not sufficient for prediction actual time of its inference on GPU.

For JPEG-AI simple and base profiles, the ratio between kMAC/p and run times is almost linearly depending on image size (but linear factor is different for different block sizes), for high profile not even the linear dependency on image size is given (likely due to higher number of attention mechanisms such as transformer). However, the ratio between CPU and GPU is not as constant.

As a consequence, comparing different architectures only by kMAC/pixel does not allow concluding about runtime.

Further study necessary to get better criteria to judge complexity of NN architectures, and also investigate impact for more devices with different property in processor type, memory, etc.