JVET-Q0199 AHG8: Support of ROI (Region-Of-Interest) RPR [T. Lu, F. Pu, P. Yin, S. McCarthy, W. Husak, T. Chen (Dolby)]
This contribution was initially discussed Saturday 11 Janurary 1815–1845 in Track B (chaired by JRO).
Further discussion was held Tuesday 14 January at 1230 in Track A (chaired by GJS).
This contribution proposes syntax and semantics to support additional functionality and coding efficiency for reference picture resampling (RPR) in VVC. In particular, this contribution proposes a generalization of reference picture resampling (RPR) in which a region of interest (ROI) within a picture – rather than the entire picture – may be resampled to create a reference picture. The proposed syntax and semantics also support the case in which a reference picture is used to predict a region of interest within a current picture rather than the entire current picture. The coding tool proposed in this contribution may be thought of as either RPR using ROI (ROI RPR), or as reference region resampling (RRR) in which RPR is a special case. It is proposed to signal ROI RPR window offset parameters to substitute for the default RPR scaling window offset parameters when ROI RPR is enabled. The additional decoding complexity increase is reportedly minimal.
For test clips with a bitstream comprising regular frames and frames with ROI, test results indicate an average coding performance improvement of approximately 11%, compared to VTM7.0 without ROI support, for LDB and LDP test conditions. When using the ROI feature, decoder complexity relative to VTM7.0 is 102% for LDB and 100% for LDP. Encoder complexity is 105% for LDB and 104% for LDP. There is no change in performance if there is no ROI in the bitstream.
An example application is a video sequence where a part of the entire scene is switched to an ROI with larger resolution for a certain duration, and then switches back to the entire scene. It is claimed that compression benefit can be achieved at the switching points.
At the switching point, the bit rate savings is quite high, about 60%.
It is reported that approx. 11% and 10% bit rate reduction can be achieved when the scaling factor is 2x / 0.5x for the switches between entire scene and ROI (ROI then coded with full res) and back. The switch is assumed to happen every 0.5 s, which might be unrealistic. For example, if switching would happen only every 2 seconds, the bit rate saving would only be 1/4.
It would probably be necessary to impose some constraints.
Further discussion in HLS.
The idea is that with the current picture, for each reference picture, the encoder would send a rectangle in the current picture and a rectangle in the reference picture. In contrast, in the current VVC draft, one window is sent for the current picture and the window in the reference picture is the window that was sent when that picture was sent.
It is proposed to signal a list of window positions in the SPS. It was commented that some picture/PPS-level override or just moving the syntax to that level might be more desirable. Something similar to what was done for RPL was suggested.
Syntax for the PH and SH is proposed, and semantics are proposed so that the rectangle pair shall be the same.
The HLS was modified in the -v4 proposal.
An example use would be zooming in or out relative to an area of the picture.
It was commented that this could enable additional functionity. For example, fo viewport-dependent streaming. It was noted that SHVC had ROI scalability capabilities that are not in currently VVC.
Low-latency applications were mentioned as potentially benefitting from this.
Another participant mentioned that zooming in particular could be a potentially benefitting usage.
There had been a previous contribution JVET-P0336. At the previous meeting, the report said “Concern was expressed about whether this functionality would have a complexity problem. There is interest in this in concept, further study with provision of text and software was encouraged.”
It was suggested to consider a simplified variant in which only one reference picture would be associated with this functionality, in order to simplify the signalling, and to use signalling only in the PH.
It was commented that software had been provided.
It was commented that the proposed text is well written.
Concern was expressed over whether introducing this at this stage could have side-effects and risk the ability to stabilize the design and complete the standard.
Further study toward a v2 version of the standard was encouraged, but there was not a consensus to put this into the first version.