JVET-AP0211 AHG14: New memory management policy in SADL [C. Hallam, F. Galpin, J. Pohl, S. Cizel (InterDigital)]

This contribution presents an alternative memory management for Neural Network inference in SADL. The new memory management allows to decrease significantly the memory usage in some cases. It is proposed as an alternative memory policy for models with large intermediate tensors or long chain of layers. The new memory policy is deactivated by default, and the behaviour is then similar to the legacy memory manager. While the legacy memory management is usually faster for small memory footprint models, the proposed new policy can reduce significantly the memory footprint for large models and even speed-up the inference in that case.

The method is asserted to have benefit for large models in particular.

No noticeable impact on runtime.

Decision(SW): Adopt JVET-AP0211 as an alternative method for memory allocation (up to the choice of users to select this method or the currently existing method)It was pointed out that it might be relevant to report in the context of experiments which of the memory allocation methods was used, or ask proponents for that.