
Super-resolution from a single image - nkoren
http://www.wisdom.weizmann.ac.il/~vision/SingleImageSR.html
======
JTxt
I was skeptical of the bottom line of the eye chart image. characters are 3x3
pixels. The o should translate to a dot. there is no hole.

Their explanation for it:

"The lines of letters have been recovered quite well due to the existence of
cross-scale patch recurrence in those image areas. However, the small digits
on the left margin of the image could not be recovered, since their patches
recurrence occurs only within the same (input) scale. Thus their resulting
uniﬁed SR constraints reduce to the “classical” SR constraints (imposed on
multiple patches within the input image). The resulting resolution of the
digits is better than the bicubic interpolation, but suffers from the inherent
limits of classical SR [3, 14]."

So it's guessing based on the larger characters. Neat.

Here's their last line:

DKN?MULJSPAVMRA?C?OVZO

Here's the actual:

DKNTWULJSPXVMRAHCFOYZG

(from what I can read.)

(
[http://1.bp.blogspot.com/-VgvutrSWaFk/T4VI-2tDH2I/AAAAAAAAAX...](http://1.bp.blogspot.com/-VgvutrSWaFk/T4VI-2tDH2I/AAAAAAAAAXY/tx60XCDeAxI/s1600/reading+glasses+eye+chart.jpg)
)

This could probably help ocr.

Other than this, I think Genuine Fractals and BenVista PhotoZoom give similar
results.

~~~
Groxx
So where are the '?' coming from? I think I have to call shenanigans /
database, there aren't any in the image, or even any shapes like it. But then,
I'm not sure what they're implying with this paragraph (this is the whole
paragraph, my additions not italicized):

> _Fig. 6_ [the eye chart] _compares our uniﬁed SR result against ground
> truth. In Fig. 7_ [baby with hat] _we compare our method to results from
> [11,13] and [9]. Note that our results are comparable, even though we do not
> use any external database of low-res/highres pairs of patches [11, 13], nor
> a parametric learned edge model [9]. Other examples with more comparisons to
> other methods can be found in the paper’s website._

~~~
JTxt
I added the ? when I could not read it. Normal SR can only do so much with a
3x3 pixel blur.

But their bottom line is mostly correct because their process also uses larger
versions of the same pattern to replace the smaller ones.

So, there was not a much larger T, so it made a little fuzzball instead. So I
wrote "?".

~~~
Groxx
There are '?' characters in the paper's PDF as well, page 7.

------
alister
Dear Mr. Glasner, Bagon, and Irani (authors of the super-resolution paper):

Please apply this technique to resolve a historically important question about
the assassination of President John F. Kennedy by identifying the license
plate number of the car immediately behind the bus:

[http://i754.photobucket.com/albums/xx183/twsxpress/Forum2/Ye...](http://i754.photobucket.com/albums/xx183/twsxpress/Forum2/Yellow_Transit_Van.png)

[http://i754.photobucket.com/albums/xx183/twsxpress/Forum2/mu...](http://i754.photobucket.com/albums/xx183/twsxpress/Forum2/murry1240.jpg)

Better resolution photos should be available since the ones I've seen printed
in books are better than these. And there are plenty of photos of Texas
license plates from 1963 with which to seed your algorithm.

The reason this is interesting is because Roger Craig, a Deputy Sheriff in
Dallas, after he witnessed the assassination said that he saw Lee Harvey
Oswald run from the Texas School Book Depository and get into Nash Rambler
station wagon driven by a another man.

These photos show a Nash Rambler station wagon that was indeed passing the
Depository just after the assassination. If motor vehicle records still exist
for 1963, then its owner can be identified.

The key point is that the existence of a person picking up Oswald after the
assassination would strongly indicate the possibility that he was not acting
alone.

I'll readily grant that this isn't likely to settle the issue, but I think it
still would be amazing if the super-resolution approach was able to generate
_new information_ about a topic that no one expects to ever see new
information about.

~~~
nkoren
I'm not an expert on this algorithm, but I'm quite sure that the algorithm
does not recover information which exists in the image; rather, it invents
information which categorically does _not_ exist. Since both the natural world
and many built objects are quasi-fractal in nature (repeating windows in a
building, leaves on a plant, etc.), the information which it invents often
happens to be correct. For non-fractal items like license plates, however,
there's no chance that it would produce a useful result.

I'm pretty sure that if you seeded the algorithm with images of many other
Texas license plates, and then asked it to sharpen this image, what you'd get
would simply be the super-resolution sum of all the other Texas license
plates. It would show you something that _looked_ intelligible, but actually
had absolutely nothing to do with the original image. That would do more to
confuse the issue than to shed light on it.

The other point is that I'm not sure how well this would actually work on
_blurry_ images, or whether (as a super-resolution technique) it's very
specialised for working on _pixelated_ images.

That said, if it can work on blurry images too, then it would be a fun and
interesting exercise to see what other historical images could yield insight
(or confusion) with this kind of enhancement.

~~~
regularfry
There was a hack a little while ago where someone demonstrated reading digits
from a pixelated image, with no more information than the font used. That's
_precisely_ what would happen here.

~~~
ef4
The hack you're referring to works because the input space is very small and
well-defined, and the blur filter that's being applied is perfectly
predictable. There's no truly random noise involved.

If instead you printed the document out and blurred it with a wet sponge, it
would be much harder to reverse, because the real world is much noisier.

Similarly, an analog photo at the limit of its resolution is full of real
noise that's hard to remove.

------
jawns
I'd be curious to know at what magnification this breaks down, if any. For
instance, most of the images have been enlarged by about 3x or 4x. What
happens when you blow them up to, say, 20x? It seems like this algorithm does
some sort of vectorization -- because look at how crisp the lines are -- which
would suggest to me that it scale well. But maybe I'm wrong.

~~~
eridius
it could just be emphasizing edges. The very first picture of the baby had a
rather overemphasized edge around his face, and yet it doesn't look like a
vector.

------
jhuckestein
Some of these images are in uncanny-valley territory for me. The bicubic
images just look resized to me, but super-resolution images look real but
"wrong". Given how much more "real" they still look than many computer
graphics (especially realtime), I guess we won't leave uncanny valley behind
us anytime soon.

~~~
madmax108
thank you for saying the same thing I was thinking. It's pretty neat when you
consider the title of the paper, but the actual results just seem wrong. Not
really sure why!

------
kemiller
So this means that CSI's "enhance" is actually more possible than reputed.

~~~
ynniv
This doesn't actually add information, it just creates a magnified image that
looks nicer than bicubic.

~~~
wmf
It's a shame that hamburglar is hellbanned because his comment is insightful.
In some sense, isn't this algorithm "inventing" detail? If you're lucky the
detail corresponds closely to what should be there, but sometimes it doesn't
and thus it can't be relied upon.

~~~
ehsanu1
For those without showdead one, hamburglar's comment:

 _Yeah, and in fact the fact that it looks crisper can be misleading. You
can't rely on a detail that a method like this infers from the low resolution
image. Look at the green tiles on the kitchen floor in the 6th example. The
"Our SR Result" image looks fantastic, but it's obvious that it's false,
because the tiles end up misshapen and unevenly spaced. You really wouldn't
want to convict someone of murder based on this type of enhancement._

~~~
kemiller
I can't reply to him directly, so I'll do it here. You wouldn't want to
convict someone of murder, no. But you could certainly use it to generate
leads.

~~~
sukuriant
The scary thing comes when people stare using it to convict someone of murder.
CSI has trained us to do things like that :/

------
apu
I'm a little late to the party, but super-resolution is a topic of interest in
computer vision, and this approach is one of many that have been recently
proposed. At a high level, I should say that the field is still searching for
the right approach, and there is by no means a consensus that this method (or
more generally, this philosophy) is the right one.

As a general rule of thumb, the performance shown in example figures of a
computer vision paper should be taken as best cases (and in some papers, as
outliers), and even the "failures" shown are often not the worst or most
typical. Similarly, quantitative results are generally as optimistic as
possible, arrived at through "graduate student descent" of the parameter
space.

So I think we are still quite far from a super-resolution method that is truly
practical OR effective.

Some specific points about this paper:

\- This research group has a long history of exploring methods based on
exploiting self-similarity in images for various tasks (super-resolution,
denoising, segmentation, etc.), and although they have shown remarkable
progress on this front, it is generally agreed that using more than just a
single image would improve results drastically for just about any task.

\- The use of more-than-exhaustive self-similarity search is EXTREMELY
expensive, however, and that's why runtime is often not mentioned in these
papers. It's not uncommon for processing times to be on the order of DAYS, for
a single tiny VGA image. (I don't remember whether they quote a number in this
paper or not, but it's certainly not faster than a few hours per image, unless
you start using some hacks.)

\- As others have commented, this method is not actually extracting new
information from the image, but rather "hallucinating" information based on
similar parts of the image. There is certainly reasonable justification for
its use as a prior, but it's not clear whether it's optimal, or even close to
it.

\- My gut feeling is that super resolution methods will find much more
application in compression-related areas rather than CSI-esque forensics. For
example, JPEG compression is roughly 30%. But downsample an image by 2x in
each direction and your compression ratio is already 25%. So if you can
downsample images a few times and then use super-resolution to "hallucinate" a
high-resolution version, that's probably good enough for the vast majority of
common images (e.g., facebook photos), where users aren't very sensitive to
quantitative "correctness". And of course with mobile adoption happening much
faster than mobile bandwidth is increasing, this becomes an ideal application
domain.

~~~
tripzilch
> "graduate student descent" of the parameter space

 _that_ is a wonderful phrase :) going to keep it in mind :)

------
beggi
<http://en.wikipedia.org/wiki/Super-resolution>

Implementation in Python: <http://mentat.za.net/supreme/>

~~~
iwalton3
How do I use the Python Implementation?

~~~
regularfry
Looks like
[http://mentat.za.net/supreme/doc/api/supreme.resolve.html#so...](http://mentat.za.net/supreme/doc/api/supreme.resolve.html#solve)
might help.

------
CodeCube
Maybe Apple can take this algorithm and use it as the scaling algorithm for
low res images on retina displays.

~~~
HardyLeung
This algorithm is likely very slow. In fact, the authors didn't mention
runtime at all.

~~~
ynniv
Yes, there are other high performance candidates that work well with line art.
<http://en.wikipedia.org/wiki/Hq3x>

~~~
rsiqueira
Depixelizing Pixel Art may work even better for some types of images:
<http://research.microsoft.com/en-us/um/people/kopf/pixelart/> "The algorithm
extracts a smooth, resolution-independent vector representation from the
image, which enables magnifying the results by an arbitrary amount without
image degradation."

------
ANH
The first application that comes to my mind is improving the quality of photos
in real estate listings. Half the time it looks like they were taken by a
camera phone circa 1995 by someone who was skipping from room to room.

------
rhplus
I just noticed that this is from a 2009 paper. Has this method made its way
into any products by now?

~~~
jandrewrogers
Techniques like this have been used in some high-end sensor applications since
(at least) the 1990s. It was an option for increasing discrimination
resolution via DSP that could not practically be accomplished via higher
resolution sensors due to things like space constraints given the technology
of the time. It is probably less useful today due to ongoing miniaturization
of imaging sensors.

~~~
linhat
> It is probably less useful today due to ongoing miniaturization of imaging
> sensors.

Not really. Dig into concepts called _sparsity_ and _compressed sensing_ ,
which are quite similar algorithms (not really the algorithms, but the
underlying ideas).

I can't find the authors' original website, only this old wired article from
2010: <http://www.wired.com/magazine/2010/02/ff_algorithm/2/>

So especially with the ongoing miniaturization of imaging sensors, which will
allow you to take even bigger and bigger pictures, there are issues these
algorithms could solve, unless SD cards get really cheap really fast I guess.

~~~
jandrewrogers
I meant less useful for the purposes which such algorithms were originally
designed for two decades ago. They original use cases were discrimination, not
making prettier pictures. I am quite familiar with compressive sampling.

------
petekp
This should be integrated into web browsers. Seems like it would particularly
benefit people with high-density displays.

------
jjcm
Are there any decent photoshop plugins that will achieve a similar result to
this? Would love to have this in my workflow.

~~~
kawera
PhotoAcute does super-resolution very well: <http://www.photoacute.com/>

------
mistercow
The subjective effect reminds me strongly of 2xSaI and hqnx upscaling filters,
although of course those only work well for pixel art.

------
eridius
Is there any information on how computationally expensive this is?

~~~
ansgri
As I understand, the complexity is not much worse than O(A * K), where A is
upscaled image area and K is approximate KNN lookup time in a dataset of all
tiles for all scales.

The question is, how fast is KNN on such high-dimensional data.

------
oulipo
It is the principle of NL-means denoising: since an image is redundant, you
use informations about shapes that are similar and are present at different
scales in the image to guess the shape of the zoomed version of the smallest
ones by using the (more thorough) content of the bigger ones

------
chmike
Applying this on satellite images with much more data available could yield
interesting results. Anyone interested to test this method with google
satellite images ?

------
rburhum
Is there a BSD/LGPL licensed c/c++/ implementation? I'd love to try it out in
some public domain satellite imagery to see the results

------
sp332
The server is already slow, so if it stops working I think this coral cache
version should still load.
[http://www.wisdom.weizmann.ac.il.nyud.net/~vision/SingleImag...](http://www.wisdom.weizmann.ac.il.nyud.net/~vision/SingleImageSR.html)

------
jostmey
I am impressed. Still, I suspect all the images shown on the webpage closely
follow the frequency spectrum of natural images. Different methods will do
better under different frequency spectrums.

------
t3kcit
In the context of computer vision, this paper is pretty "old". Since then
people have more focused on denoising, though.

------
duncan
<http://www.youtube.com/watch?v=Vxq9yj2pVWk>

------
gwern
Interesting how these super-resolution approaches are slowly inching onto
AI/machine learning territory.

------
ck2
It looks like contrasting lines are almost vectorized then expanded?

------
wahnfrieden
Are there any implementations of this available?

------
Sapemeg
i wonder whether this kind of method can be as fast as the older ones?

------
guard-of-terra
The problem with these: Every half a year we hear about new revolutionary
image transformation. Today it's super-resolution, last year it was low-res PC
games graphics interpolation, year before that - filter that could resize
images by one direction (e.g. width) while preserving proportions and throwing
away less interesting parts. The image will become 2x narrower and you'll
hardly notice.

Sadly, this is usually first and last time we see the technology in question.
They do not seem to produce any impact that could increase our quality of
life. They just sit on some dusty shelves somewhere.

~~~
mintplant
Do you have sources for each those, especially the uni-directional resizing?
They sound quite interesting.

~~~
avidal
The technique is called Seam Carving:
<https://en.wikipedia.org/wiki/Seam_carving>

It effectively eliminates the least interesting line when performing a resize,
allowing you, for example, to resize a skyline by eliminating the
uninteresting smaller buildings and gaps.

~~~
eridius
There's even an iOS app called Liquid Scale that will do this right on your
iPhone.

<http://www.savoysoftware.com/liquidscale/>

There may be others but I haven't run across them.

------
stevewilhelm
Just a comment to the authors, my I suggest that the example magnified photos
default to displaying your SR algorithm instead of NN interpolation.

~~~
daten
I personally liked how the examples seamed to start with worst-cast and
progress toward their new possibly "best case" method.

What do you think is gained by starting with the new result?

~~~
stevewilhelm
Since the paper is about the new method, one assumes the examples show the new
method's results. The first impression, is that the new method isn't very
effective.

------
cypherpnks
I claim bullshit. The theoretically optimal algorithm is, and has been known
to be for 40+ years, sinc interpolation. Gimp does this (Lanczos
approximation). Comparing to bicubic leaves aliasing relics and looks bad. I'd
be very interested in seeing the algorithm compared to something that's not a
pure known strawman.

(In CS terms, this is akin to comparing your algorithm to something using a
bubble sort, and ignoring the invention of n log n sorting algorithms)

~~~
pohl
I don't know anything about this area of specialty, but I thought it important
to point out that Huffman compression was also proven to be theoretically
optimal, and then along came arithmetic compression. I only mean to say that
you should be very precise about what the proof of optimality actually showed.

And, with that, I'm bracing myself to get schooled.

 _Edit: This reminds me of the Iterated Fractal Systems proprietary lossy
image compression algorithm they tried to commercialize in the 90s. It was
able to decompress to a larger scale image that introduced synthetic detail
that was often convincing to the eye. Notice how this article talks about
different scales._

~~~
cypherpnks
That wasn't my point. You can do better than theoretically optimal -- indeed,
their results, in many cases, do look better than sinc. Theoretically optimal
makes assumptions that the source image is, in some ways, random, in a way
that the real world doesn't conform to. If you know there are hard edges, you
can do better.

The point is that they should be comparing to sinc. They're comparing to a
known-stupid algorithm to make themselves look better than they are.

