JVET-AP0049 [AHG11] On NNVC training sets [E. Alshina, N. Bhaskar, V. Khamidullin (Huawei), F. Galpin, T. Dumas (InterDigital), J. N. Shingala (Ittiam)]
This document clarifies the situation regarding the training sets available for JVET and specifies which sequences were used for each NNVC tools training.
Training sets, their location and permission to be used for standardization
Data set | Can use? | Location |
Yes | https://data.bris.ac.uk/data/dataset/3hj4t64fkbrgn2ghwp9en4vhtn | |
Yes | ||
Not clear | ||
Yes | http://download.opencontent.netflix.com.s3.amazonaws.com/index.html?prefix=bvi_aom_dataset/ | |
Not clear | ||
Not clear | ||
Yes | ||
Research purpose only | ||
Yes | https://vqa.lfb.rwth-aachen.de/index.php/apps/files/files/550079?dir=/jvet/ctc/gaming/G1. | |
Yes |
DIV2K appears critical, as it is only made available for research purposes.
Suggestions made in the document:
For now training materials are hosted by third parties and there is a risk materials will be lost or modified. For the next generation video coding standard development or even before CfP it would be good to copy all materials JVET has permission to be used in standardization process to https://vqa.lfb.rwth-aachen.de.
In order to simplify re-training only stage 3 of NN-based loop filter it is suggested to make stage 3 materials available on to https://vqa.lfb.rwth-aachen.de. Currently HOP and LOP use different set of training materials for stage 3 (LOP and VLOP use the same). It is suggested to unify and use the only one (from previous study in AhG11 difference would be small). Streams, reconstructions, patches – all materials needed for stage 3 to which simplify data loading will take about 4 TB.
For AhG11 and EE1 proposals it must be very clearly mentioned which data set / sub-set was used for training and if training of NN-filter was stage 3 only or filter was re-trained from the scratch.
It is suggested to up-date the NNVC training sets, includes two missed BVI-DVC sequences, 66 unused before TVD sequences, 9 sequences proposed in JVET-AH0120 and 239 sequences from BVI-AOM data set. DIV2K is available for research purposes only. JVET might consider excluding it from NNVC training set. Training of DRF could benefit if vimeo90K-septuplet will be used additionally to vimeo90K-triplet.
It was commented that copying the material and making it available on the JVET server would not be allowed for most of the material. Storing processed material (e.g. for retraining stages of loop filters) should be possible.
No immediate action was possible at the moment, but the aspects of availability of training material usable for standard development will have high impact on investigations and decisions after CfP.
The information contained in the table should be included in the CTC document where the training material and procedures are described.