JVET-AC0181 AHG4: experiments in preparation of film grain visual tests [P. de Lagrange (InterDigital)]
This document reports about some experiments in preparation of visual tests planned in JVET-AB2022 for FGC SEI visual tests on top of VVC, more specifically the first category which goal is to assess the visual improvement brought by film grain synthesis over a wide range of coding quality. Three sequences that originally contain grain have been encoded with the VTM at every QP from 12 to 51, and checked visually to find when grain is just preserved, when grain is fully removed by encoding alone, and the QP range useful for visual assessment.
The goal was to find:
- Where grain just starts to be impaired by encoding (or is “just preserved”), with rating close to 10
- Where grain is removed, but the underlying content hopefully very well preserved (rating close to 9), the encoder acting as a denoiser
- The QP range useful for visual quality assessment (rating typically from 2 to 9)
From the joint meeting on Monday:
It was commented that content from the SVT sequences may not be very representative, and that for other kind of material just using “VTM as denoiser” may not be appropriate.
Having clean material and adding different levels of noise might be a viable approach (provided that an appropriate artistic synthesis was available).
It was further suggested to divide the test into two classes of “light grain” and “heavy grain”. It was commented that the latter could either be artificially generated or coming from old film or generated by purpose with heavy-grain film materials (or even be natural grain strengthened by artificial grain).
A follow-up joint meeting was held on Wednesday 18 January at 1530-1620 UTC. The following notes in this section are from that discussion.
- Clarify what the goal of a first round of subjective test would be: Content as currently available in our test sets, removal for purpose of simpler coding, synthesis after decoding, and comparison against original (this would be the main focus of the current draft), could be done with naïve subjects.
- Acquire wider variety of material, and extend the test in the bullet point above. If this includes more heavy film grain, this might require further development on noise removal and parameter derivation in the software (still with naïve subjects).
For the second bullet, it might be considered to use cropped material if tha can easier be acquired to circumvent copyright problems. Update the draft test plan from the contribution JVET-AC0181, and also include initial plans on the second bullet.
It had been commented before that superimposing synthetically generated film grain (done in a studio) over a clean sequence (which could be some of our exising test sequences) might give some advantage, as it would allow using the clean sequence as ground truth, e.g.
- Testing how good a film grain removal algorithm is, as this allows mimicking “perfect removal”
- Testing if film grain synthesis superimposed upon the coded sequence can hide coding artifacts
- Testing how close film grain synthesis generated via SEI works to reproduce similar grain as the one orginally superimposed.
How to implement this is for further discussion.
It was mentioned that in this context, the problem exists that reproducing “real” film grain is sophisticated. On the other hand, most film grain present today may be synthetic.