Video can be understood as a 2D view of a scene carved into a sequence of slices or frames. These frames are implicitly sampled at a known, regular time interval, and are intended to be played back at a certain number of frames/sec by a display device. However, if they are played back (rendered) at a faster rate, then the action appears faster, but that is a playback artifact, not a property of the video sequence itself.
Motion-Compensated FRC is the process whereby a sequence of input frames is used to generate a new sequence of output frames, but with a different time interval between the output frames compared to the input frames. As a consequence, if a point in time of a desired converted frame does not lie exactly at an input frame time, then a new frame (that never existed in the input sequence) needs to be generated to correspond to that time instant. Objects may have moved from the input frame immediately prior to that time to the input frame immediately following it. Therefore the new frame has to be reconstructed by interpolating where those objects would have appeared had the camera sampled at that same time instant. This process is called Motion Compensated Frame Interpolation (MCFI). This type of interpolation requires true frame-to-frame motion estimation on each pixel in the image, so it is very compute-intensive. The isovideo
Legato/Viarte converter uses MCFI right down to each individual pixel, giving very high quality on complex motion scenes. To be able to do this in realtime, it uses GPU acceleration to perform the computational "heavy lifting".
MCFRC can mitigate an impairment called
motion judder on playback,
which is often caused by a mismatch between the rendered rate sent to the display,
and the display refresh rate. Most playback systems know the display refresh rate,
so when asked to present a sequence at some other rate, they will automatically
duplicate or drop entire input frames (as appropriate) in order to get the desired average displayed frame rate.
A repeated pattern of duplicating or dropping frames is called a playback cadence.
Playback cadence patterns can have repeat frequencies that lie within a sensitive region of our
vision system (about 2Hz to 20Hz), leading to bad visual artifacts.
True frame-rate converters allow the input frame rate to be converted to the display rate,
so playback cadences are no longer required. Frame-rate conversion may introduce
more playback impairments than benefits, so choose your FRC system wisely!
For example, a common problem is displaying 24 FPS sources (e.g. film) on 60 Hz refresh rate progressive display devices. Display systems will replicate frames in an alternating 2:3 cadence (also called a 3:2 pulldown, for historical reasons). Essentially an input frame is shown three times on the display, and then the next input frame is shown twice, and then the pattern is repeated as a cadence. The average number of times each input frame is displayed, as expected, is (3+2)/2 = 2.5 = 60/24. Judder impairments are more visible for large-area motion (e.g. a camera pan) in your field of view.
Share:
