JVET-AG0006 JVET AHG report: ECM software development (AHG6) [V. Seregin (chair), J. Chen, R. Chernyak, F. Le Léannec, K. Zhang (vice-chairs)]
Software development
ECM software repository is located at https://vcgit.hhi.fraunhofer.de/ecm/ECM.
ECM software is based on VTM-10.0 with enabled MCTF including the update from JVET-V0056, and GOP32, which is very close to VTM-11.0.
VTM-11.0ecm anchor https://vcgit.hhi.fraunhofer.de/ecm/ECM/-/tree/VTM11_ANC is used for ECM performance evaluation.
The following changes were integrated into ECM-10.1:
Fix: high QP for AA0093 (MR 519)
Fix: reconstruction buffer copied when BIF is not performed for a sub-TU (MR 522)
Fix for uninitialized values usage reported by valgrind (MR 524)
Reduce MCTF peak memory allocation by computing applyMotion in place
Fix: memory reduction: use actual CTU size instead of MAX_CU_SIZE in encoder (MR 520)
ALF optimization - compute only Upper triangular part of covariance matrix (MR 521)
PU memory reduction by moving intraMPM member from PU IntraPredictionData to IntraPrediction (MR 518)
Fix: CU memory reduction. move some members from CU to intra search (MR 517)
Refactor picture buffer to allocate buffers for current frame and not for the entire GOP (MR 523)
CS memory reduction by allocating temporary CodingStructure data only for the current picture (MR 525)
Shrink some deblocking buffers and reduce length of memset() initializations (MR 526)
Memory reduction for sign prediction (MR 527)
Bugfix for multi-model BVG-CCCM (JVET_AE0100_BVGCCCM) (MR 530)
Use transposed buffers (MR 529)
Refactor Cabac ctx (MR 528)
Fix: fill picture level recon buffer in TU reconstruction code for BIF (MR 531)
Fix for derivation of cost in IntraTMP CandList (MR 532)
Fix the problem with disabling AE0159 macro (MR 533)
Fix unreleased RPR temp buffer (MR 534)
The following adopted aspects were integrated into ECM-11.0:
JVET-AF0133: Retraining I-slice context models (Test 6.2) (MR 537)
JVET-AF0201: Incorporate decoder memory measurement (MR 547)
JVET-AF0111: Updates the threshold for MTT modes early termination (MR 538)
JVET-AF0237: SAO/ALF encoder memory reduction (lossless) (MR 539)
JVET-AF0163: TM based subblock motion refinement (Test 3.4a) (MR 546)
JVET-AF0128: LIC flag derivation for merge candidates with template costs (Test 3.2) (MR 544)
JVET-AF0190: Enabling template-based reordering tools and LIC for scaled pictures in the RPR (Test 4.1b) (MR 542)
JVET-AF0197: Luma Residual Tap in CCALF (MR 548)
JVET-AF0079: IntraTMP block vector storing (Test 2.6c) (MR 535)
JVET-AF0177: Change data type of ALF/CCALF covariance from double to float (MR 541)
JVET-AF0073: Inter cross-component prediction merge (Test 3.1d) (MR 536)
JVET-AF0059: AHG12: Fix to interpolation filter for intra prediction (MR 554)
JVET-AF0066: Enable DBV mode in single tree configuration (MR 552)
JVET-AF0159: Affine subblock BDOF refinement (MR 550)
JVET-AF0112: Dynamic Scaling of Bilateral Filter (BIF) (Test 5.1a) (MR 556)
JVET-AF0101: Lossless code optimization by moving initialization to init() function (MR 555)
JVET-AF0057: Encoder only method for robust MV derivation in DMVR (MR 557)
Optimizations:
Memory reduction by allocating picture margin according to whether RPR is used (MR 561)
Bug fixes:
Fixes to DebugStream tool (MR 562, 566, 569, 570)
Fix gcc 13.2 compiling issues (MR 560)
Fix SAO/ALF memory reduction when RPR is enabled (MR 563)
Fix tracing (MR 565)
Fix: out of bounds access in simdFilter (MR 572)
Fix: out of bounds access in BI-GPM (MR 571)
Fix: valgrind error in Affine Merge candidates management in DecCu::xDeriveCUMV (MR 574)
Fix: inconsistent block stat by JVET-AD0222 when virtual boundary is enabled (MR 575)
The following adopted aspects were integrated into VTM-11ecm anchor:
JVET-AF0165: align VTM-11.0ecm CTCs to ECM CTCs for low delay (MR 545)
JVET-AF0111: MTT split modes early termination threshold is updated and enabled in RA and LDB CTC (MR 540)
ECM-10.1, ECM-11.0, and VTM-11ecm11.0 were tagged on November 21, 2023.
CTC Performance
In this section, ECM test results following ECM CTC configuration descried in JVET-AF2017 are summarized.
ECM-10.1 performance over ECM-10.0 anchor is summarized in the tables below.
All-Intra Main 10 | ||||||
Over ECM-10.0 | ||||||
Y | U | V | EncT | DecT | VmPeak | |
Class A1 | 0.00% | 0.01% | -0.03% | 98.0% | 98.4% | 88.2% |
Class A2 | -0.01% | -0.15% | -0.16% | 97.7% | 97.8% | 88.0% |
Class B | -0.01% | -0.15% | -0.16% | 93.4% | 94.3% | 57.5% |
Class C | 0.00% | 0.03% | 0.03% | 91.5% | 93.0% | 36.5% |
Class E | 0.01% | 0.06% | -0.15% | 93.7% | 92.6% | 44.6% |
Overall | 0.00% | -0.05% | -0.09% | 94.5% | 94.9% | 57.5% |
Class D | 0.00% | -0.09% | -0.06% | 92.7% | 92.4% | 29.9% |
Class F | -0.09% | -0.10% | -0.13% | 99.0% | 99.2% | 44.8% |
Class TGM | -0.21% | -0.24% | -0.27% | 100.8% | 98.9% | 56.9% |
Random Access Main 10 | ||||||
Over ECM-10.0 | ||||||
Y | U | V | EncT | DecT | VmPeak | |
Class A1 | 0.02% | -0.21% | -0.01% | 97.7% | 98.8% | 78.5% |
Class A2 | 0.01% | -0.17% | 0.00% | 96.2% | 97.8% | 78.1% |
Class B | -0.05% | -0.01% | -0.20% | 94.5% | 98.4% | 60.1% |
Class C | 0.01% | 0.06% | -0.07% | 92.6% | 96.3% | 40.4% |
Class E | ||||||
Overall | -0.01% | -0.07% | -0.09% | 94.9% | 97.8% | 60.1% |
Class D | -0.01% | -0.23% | -0.16% | 93.0% | 96.0% | 32.2% |
Class F | 0.03% | 0.10% | 0.17% | 101.8% | 101.0% | 51.5% |
Class TGM | -0.11% | -0.16% | -0.06% | 104.0% | 103.1% | 67.5% |
Low Delay B Main 10 | ||||||
Over ECM-10.0 | ||||||
Y | U | V | EncT | DecT | VmPeak | |
Class A1 | ||||||
Class A2 | ||||||
Class B | -0.01% | 0.18% | 0.26% | 95.5% | 94.2% | 58.9% |
Class C | -0.04% | -0.60% | -0.32% | 91.4% | 93.5% | 37.8% |
Class E | -0.03% | 1.00% | -1.48% | 97.8% | 96.4% | 45.9% |
Overall | -0.02% | 0.13% | -0.37% | 94.7% | 94.5% | 47.8% |
Class D | 0.09% | -0.25% | -0.55% | 93.1% | 93.8% | 30.6% |
Class F | 0.26% | 0.20% | 0.31% | 98.3% | 98.2% | 49.1% |
Class TGM | -0.03% | 0.02% | -0.05% | 99.7% | 96.7% | 63.2% |
Low Delay P Main 10 | ||||||
Over ECM-10.0 | ||||||
Y | U | V | EncT | DecT | VmPeak | |
Class A1 | ||||||
Class A2 | ||||||
Class B | 0.02% | -0.23% | 0.09% | 96.4% | 97.7% | 58.7% |
Class C | 0.02% | 0.05% | -0.03% | 94.5% | 96.6% | 37.5% |
Class E | -0.09% | 0.75% | -1.10% | 96.8% | 94.3% | 45.5% |
Overall | -0.01% | 0.11% | -0.25% | 95.9% | 96.4% | 47.4% |
Class D | 0.05% | -0.51% | 0.17% | 94.7% | 95.8% | 30.4% |
Class F | 0.18% | -0.20% | 0.39% | 98.4% | 99.1% | 48.2% |
Class TGM | 0.05% | 0.15% | 0.05% | 99.4% | 96.4% | 62.8% |
Next tables show ECM-11.0 performance over ECM-10.1 anchor.
All-Intra Main 10 | |||||||
Over ECM-10.1 | |||||||
Y | U | V | EncT | DecT | EncVmPeak | DecVmPeak | |
Class A1 | -0.30% | -0.01% | -0.11% | 94.6% | 94.6% | 78.2% | |
Class A2 | -0.31% | -0.13% | -0.28% | 96.6% | 94.7% | 78.7% | |
Class B | -0.32% | -0.28% | -0.15% | 100.6% | 96.4% | 60.7% | |
Class C | -0.38% | -0.33% | -0.27% | 96.8% | 88.6% | 79.6% | |
Class E | -0.50% | -0.60% | -0.17% | 99.5% | 97.7% | 70.2% | |
Overall | -0.36% | -0.27% | -0.20% | 97.9% | 94.2% | 71.9% | |
Class D | -0.34% | -0.17% | -0.34% | 99.3% | 90.6% | 90.9% | |
Class F | -0.43% | -0.65% | -0.50% | 100.8% | 92.7% | 70.1% | |
Class TGM | -0.21% | -0.33% | -0.18% | 102.2% | 97.9% | 60.3% | |
Random Access Main 10 | |||||||
Over ECM-10.1 | |||||||
Y | U | V | EncT | DecT | EncVmPeak | DecVmPeak | |
Class A1 | -0.32% | -0.97% | -1.19% | 102.4% | 102.7% | 86.3% | |
Class A2 | -0.65% | -1.54% | -2.24% | 104.4% | 109.5% | 87.0% | |
Class B | -0.30% | -1.63% | -1.21% | 106.4% | 107.0% | 79.7% | |
Class C | -0.38% | -1.09% | -1.05% | 107.0% | 109.1% | 85.8% | |
Class E | |||||||
Overall | -0.39% | -1.34% | -1.37% | 105.4% | 107.2% | 84.1% | |
Class D | -0.29% | -0.82% | -0.64% | 106.1% | 106.8% | 90.7% | |
Class F | -0.60% | -1.18% | -1.15% | 103.2% | 108.0% | 83.0% | |
Class TGM | -0.14% | -0.45% | -0.64% | 94.5% | 95.8% | 80.3% | |
Low Delay B Main 10 | |||||||
Over ECM-10.1 | |||||||
Y | U | V | EncT | DecT | EncVmPeak | DecVmPeak | |
Class A1 | |||||||
Class A2 | |||||||
Class B | -0.08% | -5.40% | -5.43% | 95.9% | 101.6% | 77.5% | |
Class C | -0.23% | -2.93% | -2.92% | 99.5% | 101.1% | 86.4% | |
Class E | -0.12% | -4.41% | -2.61% | 92.6% | 97.2% | 80.9% | |
Overall | -0.14% | -4.33% | -3.89% | 96.2% | 100.3% | 81.2% | |
Class D | -0.16% | -4.60% | -3.34% | 101.7% | 103.3% | 94.1% | |
Class F | -0.53% | -3.94% | -4.09% | 98.6% | 101.3% | 83.4% | |
Class TGM | -0.14% | -2.26% | -2.46% | 96.7% | 106.9% | 78.9% | |
Low Delay P Main 10 | |||||||
Over ECM-10.1 | |||||||
Y | U | V | EncT | DecT | EncVmPeak | DecVmPeak | |
Class A1 | |||||||
Class A2 | |||||||
Class B | -0.03% | -5.83% | -4.68% | 96.4% | 100.1% | 76.8% | |
Class C | -0.17% | -3.40% | -3.45% | 99.0% | 108.6% | 87.4% | |
Class E | -0.29% | -4.28% | -2.81% | 94.0% | 107.8% | 81.1% | |
Overall | -0.14% | -4.64% | -3.80% | 96.6% | 104.8% | 81.3% | |
Class D | -0.10% | -5.00% | -4.71% | 101.6% | 108.1% | 94.2% | |
Class F | -0.95% | -3.78% | -4.60% | 101.3% | 110.8% | 83.6% | |
Class TGM | 0.06% | -2.38% | -2.78% | 97.7% | 98.7% | 79.3% | |
The below tables show ECM-11.0 performance comparing to VTM-11.0ecm11.0 anchor.
All-Intra Main 10 | |||||||
Over VTM-11.0ecm11.0 | |||||||
Y | U | V | EncT | DecT | EncVmPeak | DecVmPeak | |
Class A1 | -10.81% | -21.80% | -29.33% | 859.1% | 400.6% | ||
Class A2 | -16.95% | -28.77% | -31.78% | 860.2% | 405.4% | ||
Class B | -11.31% | -26.84% | -25.03% | 784.4% | 411.0% | ||
Class C | -11.41% | -16.72% | -17.24% | 810.7% | 381.9% | ||
Class E | -15.01% | -24.77% | -23.07% | 750.1% | 431.9% | ||
Overall | -12.81% | -23.73% | -24.81% | 808.6% | 405.0% | ||
Class D | -9.48% | -14.56% | -14.65% | 795.1% | 383.1% | ||
Class F | -26.60% | -35.72% | -35.58% | 513.6% | 392.7% | ||
Class TGM | -39.53% | -47.43% | -46.69% | 457.3% | 461.2% | ||
Random Access Main 10 | |||||||
Over VTM-11.0ecm11.0 | |||||||
Y | U | V | EncT | DecT | EncVmPeak | DecVmPeak | |
Class A1 | -22.47% | -27.84% | -36.88% | 875.4% | 769.1% | ||
Class A2 | -26.01% | -36.30% | -40.11% | 824.7% | 958.3% | ||
Class B | -20.80% | -35.53% | -33.16% | 714.1% | 774.7% | ||
Class C | -22.22% | -27.13% | -27.05% | 748.2% | 829.1% | ||
Class E | |||||||
Overall | -22.56% | -31.91% | -33.67% | 775.1% | 821.9% | ||
Class D | -23.10% | -28.71% | -28.75% | 768.1% | 898.2% | ||
Class F | -28.70% | -37.47% | -37.77% | 614.7% | 505.2% | ||
Class TGM | -36.82% | -43.33% | -43.43% | 566.6% | 413.2% | ||
Low Delay B Main 10 | |||||||
Over VTM-11.0ecm11.0 | |||||||
Y | U | V | EncT | DecT | EncVmPeak | DecVmPeak | |
Class A1 | |||||||
Class A2 | |||||||
Class B | -17.76% | -41.31% | -38.53% | 661.6% | 649.7% | ||
Class C | -19.49% | -31.72% | -31.66% | 618.2% | 678.1% | ||
Class E | -16.28% | -30.12% | -30.39% | 587.1% | 406.5% | ||
Overall | -17.97% | -35.32% | -34.21% | 627.8% | 586.1% | ||
Class D | -21.22% | -34.09% | -33.02% | 659.2% | 772.5% | ||
Class F | -24.87% | -39.63% | -38.99% | 575.6% | 491.8% | ||
Class TGM | -35.47% | -44.37% | -44.64% | 538.8% | 424.4% | ||
Low Delay P Main 10 | |||||||
Over VTM-11.0ecm11.0 | |||||||
Y | U | V | EncT | DecT | EncVmPeak | DecVmPeak | |
Class A1 | |||||||
Class A2 | |||||||
Class B | -16.37% | -40.73% | -37.81% | 592.0% | 625.0% | ||
Class C | -18.40% | -30.91% | -30.87% | 545.9% | 688.5% | ||
Class E | -15.76% | -29.13% | -29.84% | 571.9% | 461.5% | ||
Overall | -16.89% | -34.56% | -33.51% | 571.2% | 598.4% | ||
Class D | -21.11% | -33.25% | -33.49% | 588.5% | 750.2% | ||
Class F | -24.13% | -39.44% | -39.00% | 609.4% | 513.2% | ||
Class TGM | -32.65% | -42.90% | -43.19% | 600.4% | 387.9% | ||
ECM memory consumption
ECM encoder memory consumption (VmPeak, GiB) is provided in ECM encoder log files and is summarized in the table below as maximum class-wise consumption rounded up to GiB.
AI | RA | LB | |
Class A (A1& A2) | 7 | 14 | |
Class B | 3 | 5 | 4 |
Class C | 2 | 2 | 2 |
Class D | 2 | 2 | 2 |
Class E | 2 | 3 | |
Class F | 3 | 5 | 4 |
Class TGM | 3 | 7 | 6 |
It is encouraged to care about memory allocation when developing and integrating tools into ECM. In particular, it is strongly recommended to re-use already existed memory wherever possible, rather than systematically allocating new memory, and allocate only the required amount of memory if new memory is needed.
Recommendations
The AHG recommended to:
- Continue to develop ECM software.
- Improve the software documentation.
- Encourage people to report all (potential) bugs that they are finding using GitLab Issues functionality https://vcgit.hhi.fraunhofer.de/ecm/ECM/-/issues.
- Encourage people to submit merge requests fixing identified bugs.
- Encourage people to continue working on ECM memory consumption reduction.
- Encourage people to continue working on speeding up ECM encoder to reduce the simulation time.
It was asked what the memory consumption is compared to the VTM. This was not reported, but it is estimated to be larger by a factor of approximately 2x for the current ECM software implementation.