Thread Rating:
  • 1 Vote(s) - 5 Average
  • 1
  • 2
  • 3
  • 4
  • 5
Super resolution
#1
Multi-image or multi-frame super resolution (SR) is a technique used to restore detail, while upscaling low resolution images or video frames. Although SR can be applied to native low resolution images or video frames, here we will focus on the case where a high resolution source has been downscaled to a low resolution, as is the case for commercial DVDs.
We will start with an example of a single image. When you downscale a high resolution image to a lower resolution you lose information. Each pixel in the low resolution image becomes a weighted average of a number of frames in the high resolution image. When we upscale the low resolution image to the resolution of the high resolution image the lost information does not magically reappear. The result is a blurry image that lacks detail, as can be seen in the following example. Here the resolution has first been reduced by a factor of two, and than upscaled to the original size.

http://screenshotcomparison.com/comparison/125890

The information about the stripes on the girl's pants and scarf has obviously been lost as a consequence of the downscale. However, if we have multiple images with subpixel shifts much of this information can be retrieved. This effect can be simulated by shifting the high resolution image in multiple directions by one pixel. Each of these shifted images is then downscaled and upscaled in the same way as before. If we then align, average, and deblur these shifted images, we obtain the following result:

http://screenshotcomparison.com/comparison/125895

As you can see much of the original detail has been retrieved.
We have seen that it is possible to obtain a high resolution image by combining multiple low resolution images. The same principle can be applied to video. Video can be viewed as a set of shifted images. By aligning very similar objects in multiple frames, averaging the pixel information, and deblurring, the resolution of a video can be increased. In practise this is of course quite complicated, but the basic concept is the same as the above example for a single image.
An example using the bluray for Star Wars can be seen in the following screenshot comparisons. As before the video has first been downscaled by a factor of two, and then upscaled using SR.

Simple upscale versus SR:

http://screenshotcomparison.com/comparison/125527

The methods used for image and video super resolution have been extensively described in scientific literature. Just type in (video) super resolution in Google Scholar, if you're interested. Commercial software is also available at a reasonable price. As a Virtualdub and Avisynth user I have found that for video upscaling the best performing option at this point is the Infognition software (standalone, Virtualdub plugin, Avisynth plugin). Why don't you try it out?
Reply }
Thanks given by: nickdiba , bttfbrasilfan
#2
DrDre, thanks for this informative article; simple, well written, and interesting; great job, you'd be a good teacher!
Reply }
Thanks given by:
#3
Indeed, thanx for posting this. Awesome stuff! I checked out the comparison images that they have posted there (listing numerous ways of "upscaling" and their "method") and the results are very impressive. In fact, you're right about the VERY reasonable price and I will be getting this on payday.

Really, thank you very much for posting this and referencing the software as it will be a huge help for the "Top Gun" project I am starting work on (along with all the other things I'm working on) where I need to upscale PAL and NTSC DVD's (before layering some of them at 1080p) and this will help produce a highly superior result to what I was getting so far.

Hope this info helps others that work with upscaled footage... the results speak for themselvesWink
Reply }
Thanks given by:
#4
Thanks for the nice replies! Here's another one of my ramblings. ;-)

To clarify the difference between sharpening, deblurring, and super resolution, I will use another example programmed in MATLAB.

When an image (or video frame) is downscaled two things happen: the image is blurred and compressed. Blurring causes the information content to be distributed differently among the pixels, resulting in loss of detail, but in principle no information is lost. Compression results in loss of information.  We can subsequently upscale the downscaled image back to its original size through interpolation. When we compare the upscale to the original, we obviously get a blurred image with less detail:

http://screenshotcomparison.com/comparison/126569

Sharpening is the same as applying an unsharp mask filter. An unsharp mask filter enhances the high frequency content in the image. In other words sharpening results in edge enhancement. No lost details are recovered, the existing details are simply made more visible:

http://screenshotcomparison.com/comparison/126570

A side effect of sharpening is ringing. Although sharpening enhances detail, it does not necessarily result in a better representation of the original high res image. It simply represents a subjective reimagining of the low resolution image.

Deblurring is the process of undoing the effects of blurring. Provided a reasonable estimate of the blur function is known, many details can be recovered:

http://screenshotcomparison.com/comparison/126571

The following comparison shows the difference between sharpening and deblurring. The deblurred image obviously has more detail, and is a better representation of the high res image. The deblurred image also has less artifacts, such as reduced ringing:

http://screenshotcomparison.com/comparison/126572

However, the loss of detail due to compression cannot be undone by deblurring. If we have multiple images with subpixel shifts the images can be super resolved, and part of the detail lost due to compression can be recovered. The objects in the images are registered, aligned, averaged, and deblurred:

http://screenshotcomparison.com/comparison/126585

The super resolved image has more (accurate) detail, less artifacts, such as reduced ringing, and is an even better representation of the high res image, when compared to the deblurred single image:

http://screenshotcomparison.com/comparison/126586

Summarizing, sharpening enhances existing details, deblurring reconstructs details lost as a result of blurring, super resolution reconstructs details lost as a result of blurring and compression. Any of these methods results in artifacts, the foremost being ringing. However, sharpening generally has the most artifacts, while super resolution has the least.

The sucess of super resolution in practise, hinges on accurate data fusion, and the accuracy of the estimated point spread function (blur function).

For more information on the limits of super resolution, I refer you to this scientific paper:

http://homepage.tudelft.nl/e3q6n/publica...TPLVKS.pdf
Reply }
Thanks given by: nickdiba
#5
That's great, thanks DrDre!
Reply }
Thanks given by:
#6
Fantastic post and an interesting piece of software.
Lets say I wanted to upscale a DVD I have that was mastered in HD
Would this have to be applied on a shot by shot basis?
Could you put a whole film untouched through the filter / Programme with the same results?
Reply }
Thanks given by:
#7
(2015-05-13, 02:49 PM)CSchmidlapp Wrote: Fantastic post and an interesting piece of software.
Lets say I wanted to upscale a DVD I have that was mastered in HD
Would this have to be applied on a shot by shot basis?  
Could you put a whole film untouched through the filter / Programme with the same results?

Thank you for your reply! In principle yes. The software allows you to import an entire movie (such as an mp4, mkv or mpeg). It will then upscale the movie on a frame by frame bases. The program is reasonably fast. A full movie of 2 hours will take about 3-5 hours to process, depending on the chosen color space (YV12 is faster, but RGB is more accurate in my experience). Sometimes you may want to use other filters to reduce noise, artifacts, etc. In that case Virtualdub and/or Avisynth is a better choice, although there are a large number of filters available in the stand alone software as well.
Reply }
Thanks given by:
#8
Has anybody tried this method on a fair enough HD (let's say 1080p) source?
Presumably you'd have more detail in each frame for the software to work with, thus generating a more detailed image.

For example when the master used on the blu-ray is dated and has been color graded in guidance by a man who uses his trustworthy four-sided dice and a sun
dial for reference in a moat he's dug with a broken vase. On second thought, you can ignore this last part.
Reply }
Thanks given by:
#9
Let's ignore all that last part... Tongue

A serious test should be made using a "mastered in 4K" BDs... in theory, Super Resolution could bring back a lot of details, because, according to Sony, they are made using a 2160p (which is 3840x2160 and NOT 4K, which is 4096x2160) master, hence a perfect resolution to make a downscale...

I wonder if someone would like to spend some time to make few demos (1 min. each will do) and also some 4K (oops, UHDTV, sorry!) display owner would like to make some tests, with original clips upscaled inside the display, and the SR upscaled clips...
Reply }
Thanks given by:
#10
What does it do with film damage? Like spots, etc. There's no data in the adjacent frames for it. I presume it doesn't touch 'em as there's no data matching them.
Reply }
Thanks given by:


Forum Jump:


Users browsing this thread: 1 Guest(s)