
With Google's RAISR, images can be up to 75% smaller without losing detail - newman314
http://www.pcmag.com/news/351027/google-raisr-intelligently-makes-low-res-images-high-quality
======
tinco
The title is a bit misleading. Detail was lost, that is the law. Raisr makes
up detail based on some learned likelihood of that detail being there.

You could add any image to Raisr and it will make up more detail, you could
let it loop over the same picture over and over and it will probably add
infinite detail, or up to some fixed point.

What's cool is that for many pictures, it's good enough at making up detail
that the result is credible.

~~~
tyingq
Seems similar to image inpainting...educated guesses that can be stunningly
accurate, or stunningly bad :)

~~~
kartan
Yes. And as such it has obvious artistic value, as images look better. But I
can see that people can misuse them CSI style. You can add more detail to an
image, but it is mainly an artistic interpretation that is not the real
original image.

------
jerf
This article and the linked blog post from Google [1] suggest that this has
already been deployed. How does it work in the field? For instance, I can
imagine it being a JS library that can be downloaded and cached for client-
side use, or it could be deployed as an Android module that the Android
browser can use. But I don't know where to go looking for this because I don't
use Google+, have no photos on it, and wouldn't be sure whether or not I was
seeing it.

[1]: [https://www.blog.google/products/google-plus/saving-you-
band...](https://www.blog.google/products/google-plus/saving-you-bandwidth-
through-machine-learning/)

~~~
gwern
>RAISR produces results that are comparable to or better than the currently
available super-resolution methods, and does so roughly 10 to 100 times
faster, allowing it to be run on a typical mobile device in real-time....For
example, in addition to improving digital “pinch to zoom” on your phone, one
could capture, save, or transmit images at lower resolution and super-resolve
on demand without any visible degradation in quality, all while utilizing less
of mobile data and storage plans.

> In our proposed approach, the run-time is more than one to two orders of
> magnitude faster than the best competing methods currently available, while
> producing results comparable or better than state-of-the-art.

From the blog post & article, I think the idea is that you ship a small CNN (a
few MB model size, which will accordingly run in realtime on mobile GPUs) to
the Android device, and then when browsing, it downloads a small thumbnail
(cheap on bandwidth, quick to download), and upscale it as necessary locally
on the device.

~~~
jacobolus
In general they’d get better bandwidth savings for the same quality and decode
effort by shipping the higher-resolution image with more aggressive image
compression (using [http://bellard.org/bpg/](http://bellard.org/bpg/) or
something, not JPEG, and ideally running the decode in hardware). Seems like a
dubious method overall.

The main impression I get is “woo! neural nets can do everything! let’s use
them everywhere!”

I don’t believe their claim that this is “comparable or better than the state
of the art” for image upscaling without much better evidence than they provide
in their paper.

------
tedunangst
If you click enough links, you may eventually find your way to the paper:
[https://arxiv.org/abs/1606.01299](https://arxiv.org/abs/1606.01299)

~~~
itcrowd
Thanks. I'd like to make some comments on the paper \- it is very readable,
even for non-ML folk

\- first author was an intern at Google. Cool for him to have a project like
this handed to him.

\- paper is quite long and I don't think it would ever be published in a
reputable journal. The content/length ratio is too low, for one

\- another reason I think it would not get accepted is the lack of novelty.
What is being done here is a basic interpolation scheme (which explains the
"10-100X speedup over SotA) but with weights determined by a lookup table. The
table itself is learned beforehand by analyzing a large number of other
images.

~~~
almostarockstar
I havn't time to read through it, but based on your comments, I could see it
being accepted to a conference, which is where most CS publications come from
(in europe anyway).

------
nailer
This exactly what Magic Pony (acquired by Twitter last year) does.

[https://www.inverse.com/article/17200-twitter-buys-london-
vi...](https://www.inverse.com/article/17200-twitter-buys-london-video-
company-magic-pony-to-improve-its-live-video)

There's an example of them finding a low res image of Barack Obama, and
filling in the missing detail via ML to make a compelling high resolution
image that still looks like Barack Obama:
[https://fsmedia.imgix.net/25/36/be/9d/fd04/4c51/9f79/ef675b0...](https://fsmedia.imgix.net/25/36/be/9d/fd04/4c51/9f79/ef675b0143e3/magic-
ponys-machine-learning-of-an-image-of-barack-obama.png)

(Upvotes for anyone who a) replies with a better link and b) is Zehan [if b)
then HI ZEHAN!])

------
tornadoboy55
The biggest about this is not being able to 'compress' images to 25%, but that
you can now take (some) images that we're previously only low-res, and upscale
them!

~~~
CamelCaseName
One day we'll be pointing at pictures yelling "Enhance!", and not in jest!

~~~
fulafel
Good idea for a dystopian CSI. "This AI invented image detail shows you are
guilty"

~~~
patates
Well then it means statistically there's a good chance that what produced the
end low res image is incriminating. And of course I'm not serious.

Although on a serious note, does anyone know if and how statistical
probability is accepted as evidence in a court of law?

~~~
tornadoboy55
Yeah, it is, that's called circumstantial evidence. You do need direct
evidence to validate the circumstantial evidence though..

~~~
patates
It even has a name, didn't expect that. Thank you.

------
acqq
From Google blog:

[https://www.blog.google/products/google-plus/saving-you-
band...](https://www.blog.google/products/google-plus/saving-you-bandwidth-
through-machine-learning/)

"To help everyone be able to see the beautiful photos that photographers share
to Google+ in their full glory"

"we’ve only begun to roll this out for high-resolution images when they appear
in the streams of a subset of Android devices."

Anybody has an idea which subset?

~~~
tw04
I'm fairly confident you can assume they're referring to pixel devices.

~~~
Vexs
And it's probably something like 20% of them.

------
SQL2219
I wonder how it compares to this: [https://tinyjpg.com/](https://tinyjpg.com/)

~~~
explorigin
The cool thing is that it's not a compression algorithm so you could use both.
TinyJPG to compress the smaller-sized image, RAISR to upsample the smaller
image. If you've ever played with GIMP (or Photoshop I suppose), RAISR looks
like unsharp mask with an neural network controlling the inputs.

~~~
derefr
Now create a psychophysical compressor that optimizes for the weighted
combination of smallest compressed size and most-faithful RAISR restoration.

I wonder if there's a guaranteed-optimal way to do that, given whatever RAISR
is using as an algorithm; some logical inverse to the particular way it adds
information. Maybe truncated wavelet compression?

------
chubs
I wonder if this could be used to upscale, say, a 1080p movie frame-by-frame
to 4k? Could be fantastic if so. Imagine if it could be built into an FPGA and
done in real-time by high-end TVs. Very impressive!

~~~
andai
The trick is that using a simple frame-by-frame approach means the details
filled in only depend on the frame, not on the details which were filled in to
the previous frame, so you get crazy jitter.

Deepart.io now has a video mode [0] that takes this into account.

[0] [https://deepart.io/video/](https://deepart.io/video/)

------
wodenokoto
The article makes it sound like it is some sort of machine learning algorithm,
which makes me wonder: How is the upscaling done client side?

~~~
sushisource
It's very much within the realm of possibility to have a pre-trained neural
net on the client, which then simply asks it questions.

For example, swiftkey does this already with its keyboard prediction. Prisma
does it with image styling.

~~~
gkbrk
I actually think Prisma sends the image to their servers for processing, their
app size is way too small to contain that many trained networks. Also it
doesn't work without an internet connection if I'm correct.

~~~
HisGraceTheDuck
That used to be the case but recent updates do the processing on the device
now. I don't know how they managed it but they did.

------
ZenoArrow
I'd be interested to see if similar techniques could be used for scaling up
pixel art. Tools to upscale pixel art do exist already, but the results are a
bit hit and miss. Could be a good stress test for the machine learning
algorithms too.

~~~
andai
That sounds interesting. Do you have any links?

I must not understand what you mean by scaling up, I though the best way to
scale up pixel art was to disable interpolation, so each pixel just becomes a
perfectly sharp square?

~~~
ZenoArrow
>"That sounds interesting. Do you have any links?"

Plenty to take a look at here:

[https://en.wikipedia.org/wiki/Pixel_art_scaling_algorithms](https://en.wikipedia.org/wiki/Pixel_art_scaling_algorithms)

~~~
jacobolus
Existing pixel-art-specific upscaling methods are going to be better than
this, for pixel art.

But if they wanted they could probably try to train their neural network for
pixel art specifically, similar to
[https://github.com/nagadomi/waifu2x](https://github.com/nagadomi/waifu2x)

------
ingenter
Here's a list of various image interpolation techniques:

\-
[http://www.ifp.illinois.edu/~jyang29/papers/chap1.pdf](http://www.ifp.illinois.edu/~jyang29/papers/chap1.pdf)

* Image super-resolution: Historical overview and future challenges

\-
[http://www.robots.ox.ac.uk/~vgg/publications/papers/pickup08...](http://www.robots.ox.ac.uk/~vgg/publications/papers/pickup08.pdf)

* Machine Learning in Multi-frame Image Super-resolution

\-
[http://www.cs.huji.ac.il/~peleg/papers/icpr90-SuperResolutio...](http://www.cs.huji.ac.il/~peleg/papers/icpr90-SuperResolutionSequences.pdf)

\-
[http://www.wisdom.weizmann.ac.il/~vision/SingleImageSR.html](http://www.wisdom.weizmann.ac.il/~vision/SingleImageSR.html)

\-
[http://chiranjivi.tripod.com/EDITut.html](http://chiranjivi.tripod.com/EDITut.html)

\-
[http://www.tecnick.com/pagefiles/appunti/iNEDI_tesi_Nicola_A...](http://www.tecnick.com/pagefiles/appunti/iNEDI_tesi_Nicola_Asuni.pdf)

\-
[http://www.eurasip.org/Proceedings/Eusipco/Eusipco2009/conte...](http://www.eurasip.org/Proceedings/Eusipco/Eusipco2009/contents/papers/1569192778.pdf)

\- [http://bengal.missouri.edu/~kes25c/](http://bengal.missouri.edu/~kes25c/)

* [http://bengal.missouri.edu/~kes25c/nnedi3.zip](http://bengal.missouri.edu/~kes25c/nnedi3.zip)

* [http://forum.doom9.org/showthread.php?t=147695](http://forum.doom9.org/showthread.php?t=147695)

\-
[http://arxiv.org/pdf/1501.00092v2.pdf](http://arxiv.org/pdf/1501.00092v2.pdf)

* [http://waifu2x.udp.jp/](http://waifu2x.udp.jp/)

* [https://github.com/nagadomi/waifu2x](https://github.com/nagadomi/waifu2x)

* [http://waifu2x-avisynth.sunnyone.org/](http://waifu2x-avisynth.sunnyone.org/)

* [https://github.com/sunnyone/Waifu2xAvisynth](https://github.com/sunnyone/Waifu2xAvisynth)

\- [http://i-programmer.info/news/192-photography-a-
imaging/1010...](http://i-programmer.info/news/192-photography-a-
imaging/10100-no-comment-super-resolution.html)

* [https://github.com/david-gpu/srez](https://github.com/david-gpu/srez)

\-
[http://arxiv.org/pdf/1609.04802v1.pdf](http://arxiv.org/pdf/1609.04802v1.pdf)

\- [https://github.com/alexjc/neural-
enhance](https://github.com/alexjc/neural-enhance)

~~~
dr_zoidberg
Great list, leaving a comment here to "bookmark" it.

------
nurettin
I think this is what futurama's "better resolution than real life" TV sets are
going to look like.

------
deadcast
Does anyone know if there's something like this available now to experiment
with? I've been looking for a new way to ultra compress images and never heard
of these techniques before. Looks like Google hasn't released any tools yet to
the public, if they ever will.

~~~
randyrand
For storage purposes: If you want, you could just downsample your images right
now, and just wait around for this or similar to be released. I dont recommend
it.

------
jgalt212
How many of these compression plays are driven by a real market need and how
many are driven by Pied Piper envy?

------
mschwaig
Inventing details out of thin air to make a low resolution image look more
detailed feels like cheating to me.

------
EclipseAnatomy
Works kind of like PerfectResize for PS then, except this downscales or I
might be completely wrong..

~~~
andai
I believe the down-scaling is done the old fashioned way. The up-scaling uses
a neural network to guess what the original image was. See waifu2x [0][1].

The new thing here is that they're using it to save bandwidth, and running it
on the user side.

[0]
[https://raw.githubusercontent.com/nagadomi/waifu2x/master/im...](https://raw.githubusercontent.com/nagadomi/waifu2x/master/images/slide.png)

[1] [http://waifu2x.udp.jp/](http://waifu2x.udp.jp/)

