JVET-AJ0372 BoG Report: Group of ECM tools with large memory access [X. Li (BoG chair)]

This was presented Thu 7 Nov. 1515

This document contains the report of the BoG on group of tools with large memory access. The BoG met on 11/6 Wednesday 10:15am –11:35am.

Mandates

With regard to AHG7 activities, it was agreed to establish a new group of tools with large memory access, e.g by needing large lookup tables. It is however also necessary to take into account the granularity of usage, e.g. it is different if a large amount of coefficient values needs to be reloaded for every 4x4 block or for an entire CTU.

This BoG was to further work on criteria, and set up a list of tools that would be added to such a group.

Recommendations

• Tool list
  • PDP
  • Enhanced MTS
  • LFNST extension
  • NSPT
  • Fixed filter in ALF
• Cfg Options (For tools which extended the tools in VVC, the group off test should only turn off the extended part while leaving the VVC tool unchanged)
  • PDP control is available
  • The proponents of Enhanced MTS and LFNST extension will try to
  • The proponents NSPT and Fixed filter in ALF will implement the control
• Summary table

Note that the lookup table sizes are counted based on the ECM-14 implementation, which has not been fully optimized for memory usage.

The tools with lookup table size larger than 64Kbytes. Note that Kbytes =1000bytes in the following discussions.

BoG discussions

Agenda

The tools with lookup table size larger than 64Kbytes. Note that Kbytes =1000 bytes in the following discussions.

For tools which extended the tools in VVC, the group off test should only turn off the extended part while leaving the VVC tool unchanged.

Discussions

JVET-AJ0369 was used as a starting point for the discussions.

If reliable numbers could not be achieved in this BoG, AHG7 was to be tasked to have an input document (with crosscheck) on the analysis to the next JVET meeting.

PDP

• Total size of the lookup tables (KB): 7672.232
  • PDP: 3868.544
  • PDP-MIP: 3803.688
• Worst-case memory reading
  • 16x16 block, 264 bytes/sample
• Worst-case multiplication per sample
  • 132 per sample
• Cfg Option
  • PDP

Enhanced MTS (newly added lookup tables)

• Total size of the lookup tables (KB)
  • 1573.955 (total, sum of row g_aiTr2 to g_aucTrIdxToTr in section 2.5 MTS of JVET-AJ0369) - 32.7 (VVC MTS) = 1541.255 (Note that some parts of the table are not actually used)
• Worst-case memory reading
  • 2 bytes/sample
• Worst-case multiplication per sample
  • To be calculated (to be included in AHG7 report)
• Cfg Option
  • Qualcomm will try it

LFNST extension

• Total size of the lookup tables (KB)
  • 564.577
• Worst-case memory reading
  • 4x4/8x16 block, 16x1=16 bytes/sample
• Worst-case multiplication per sample
  • 4x4/8x16 block, 16 multiplication/sample (to be included in AHG7 report)
• Cfg Option
  • LGE will try it

NSPT

• Total size of the lookup tables (KB)
  • 3601.92
• Worst-case memory reading
  • 8x16/16x8: 40 bytes/ sample
• Worst-case multiplication per sample
  • 8x16/16x8: 40 multiplications
• Cfg Option
  • LGE will do it

Fixed filter in ALF

• Total size of the lookup tables (KB):
  • 1949.696 (Note that some tables only need 2 bits/element, the total size could be reduced by 40%)
• Worst-case memory reading
  • 2x2 level, 3 filters (2 9x9 filters and 1 13x13 filter): 2*41+64=146 elements, 146 *273 bytes.
• Worst-case multiplication per sample
  • 146 multiplication / sample
• Cfg Option
  • Qualcomm will do it

From discussion in JVET:

Recommendations of the BoG were confirmed.

It was agreed that NN based intra prediction also shall be included in this category (see notes under adoption of JVET-AJ0249). This adoption was made after the BoG meeting, and proponents are requested to provide the corresponding numbers for an update of the BoG report.

It was commented that some of the ECM tools have larger need for local memory access than NN based loop filters, where the model is only loaded once per slice.

It was commented that the analysis should not only be based on the ECM software implementation, but also consider optimizations e.g. in hardware.

The methodology of analysing the impact on memory access will be very useful in analysing tools in future standardization.

Further investigation is necessary whether the tools of this group can be disabled altogether in software.