
Depixelizing Pixel Art - epsylon
http://research.microsoft.com/en-us/um/people/kopf/pixelart/supplementary/multi_comparison.html
======
zeidrich
This is a pretty old paper, but I hadn't seen the live demo. That's pretty
cool.

A problem with this algorithm is that it's pretty slow. It would be hard to
use it for real time scaling.

I think that it's pretty cool though for any precalculated conversion to
vector format. It would also be pretty cool if there were software that
allowed you to create a pixel art image with a live preview in this format and
the ability to export into a vector format.

One of the benefits of pixel art IMO is that it's got a pretty distinct style
and its simplicity and restrictions means it's easy to be aware of composition
where if you are given an unrestricted canvas that freedom can make it hard to
know where to start. If you are given a 8x8 pixel grid to indicate "happy"
there's only so many ways to do it, a symbolic heart, a little smiley face,
etc. Given a whole sheet of paper you'll have more difficulty.

With the high res displays, there's more to consider than just the upscaling
of old programs, the fact is people (especially indie game devs) still want to
work with pixel art, and the number of titles with the term "8-bit" in their
name that have come out recently. Part of that's because it's in vogue now,
but a big part of it is simply that a poor artist can make a passable 24x24
sprite.

But it's still a shame that on high res displays they end up looking like a
blocky mess. If you had a program that let you author your sprites under the
same conditions, but let you preview an upscaled version and left you with a
vector image (or high res texture with mip maps) you can make use of the
fidelity while maintaining the simplicity. I think that would be cool.

~~~
glhaynes
_A problem with this algorithm is that it 's pretty slow. It would be hard to
use it for real time scaling._

Even if it's too slow for realtime on today's hardware, I'd _love_ to see some
prerendered sample videos.

~~~
josh-wrale
Here's a video: [https://research.microsoft.com/en-
us/um/people/kopf/pixelart...](https://research.microsoft.com/en-
us/um/people/kopf/pixelart/supplementary/video_multi_comparison_4x_h264.mp4)

~~~
chongli
Am I crazy for preferring the one on the left (nearest-neighbour)? The other
two really mess up certain details such as the point numbers, the stem of the
fruits and pokey's eyes.

They also tend to cause a very subtle distortion of the outlines of foreground
sprites as they move over the background.

~~~
Spittie
No, you aren't. I prefer nearest-neighbor too, I think it preserves the
"spirit" of the game. Generally speaking, I prefer it for any kind of "pixel
art".

That said, those algorithms are impressive, and I can see why someone could
prefer them over a simple nn.

~~~
derefr
> I think it preserves the "spirit" of the game.

The art-direction of games from this era wasn't done with the intention that
these games would be played on pixel-perfect computer monitors, though. The
art was designed and tested for CRT TVs, which would generally give you
something like
[http://i385.photobucket.com/albums/oo299/muddi900/LTTP3xNTSC...](http://i385.photobucket.com/albums/oo299/muddi900/LTTP3xNTSC.png),
or even
[http://4.bp.blogspot.com/_Kzdww8T9fUA/TRplyGIj6PI/AAAAAAAAAt...](http://4.bp.blogspot.com/_Kzdww8T9fUA/TRplyGIj6PI/AAAAAAAAAtA/jylLsrv3b0I/s1600/CRT%252BNTSC-
RGB.png)

~~~
josh-wrale
This reminds me of staring at those tiny screenshots on the back of NES boxes
with sweaty palms as a six year old.. lol.. Pixel art was not yet embraced as
an art form, I guess. (well, not by the adults)

edit: proof:
[https://www.google.com/search?q=nes+box+back&tbm=isch](https://www.google.com/search?q=nes+box+back&tbm=isch)

------
jrockway
I wish they would use this in Windows 8. As far as I can tell, their scaling
algorithm for 4k displays is: scale up 2x using nearest neighbor, then
randomly drop pixels to get back to 125% or 150%. The result is horrifyingly
bad. Not just kind of bad, but unusably bad. (Adding insult to injury, they
ClearType the text _before_ scaling.)

~~~
jjcm
While something similar would probably be better, this is heavily optimized
for pixel artwork. Lots of curves, shading, and most importantly - limited
colors. UI is generally much straighter, less shaded, with far more colors.

Here's an example of UI that's scaled with potrace (a similar algorithm):
[http://i.imgur.com/jDq4M7e.png](http://i.imgur.com/jDq4M7e.png) (left side is
the converted side, right side is nearest-neighbor. Top is 3x, bottom is 2x)

That's with the highest amount of colors/depth that potrace allows. On an i7
processor, it takes a LONG time to compute all of that. Each icon is no longer
just a couple hundred pixels, it's a couple hundred vector paths. Even
rendering it takes a number of seconds to refresh the screen in inkscape.

~~~
dublinben
1\. That looks awful.

2\. We just need vector-based icons already.

~~~
slazaro
I think the ideal would be something similar to fonts: there's the vector
definition for big sizes, and optional pixel-perfect definition for small
sizes, because rendering vectors on very small sizes (16x16, 32x32) might look
worse than hand-crafted pixel work.

~~~
oofabz
These days, font rendering does not often rely on bitmaps, even at small
sizes. Instead, they typically use hinting and subpixel rendering. I would
love to see these effective techniques applied to vector graphics too.

------
frik
hq4x looks better (IMHO), sadly the website is offline (default blank
wordpress site): [http://hiend3d.com/hq4x.html](http://hiend3d.com/hq4x.html)

backup on archive.org:
[https://web.archive.org/web/20120630044334/http://www.hiend3...](https://web.archive.org/web/20120630044334/http://www.hiend3d.com/demos.html)

[http://en.wikipedia.org/wiki/Hqx](http://en.wikipedia.org/wiki/Hqx)

Source:
[https://code.google.com/p/hqx/source/browse/trunk/src/hq4x.c](https://code.google.com/p/hqx/source/browse/trunk/src/hq4x.c)

~~~
lloeki
Only if you look at the first ones. All of Boo, Dolphin, all of SoM, Invaders,
Golden Axe, Windows assets and Lost Vikings look great and have vastly reduced
artifacts compared to hqx. Also, according to the paper abstract[0], it
outputs vector, not scaled up bitmap.

[0]: [http://research.microsoft.com/en-
us/um/people/kopf/pixelart/](http://research.microsoft.com/en-
us/um/people/kopf/pixelart/)

~~~
scott_karana
I agreed with you, but then I realized they're representing hqx deceptively.

There was a video posted here, with them rendered _at the same resolution_
side by side.

[https://news.ycombinator.com/item?id=7307975](https://news.ycombinator.com/item?id=7307975)

It's really a matter of taste in this video.

~~~
corysama
> they're representing hqx deceptively.

A big difference in the techniques is that hq4x is a fix-rate upscaling,
whereas Microsoft's can handle arbitrary target resolutions. At the same rez,
they are pretty equivalent. But, MS has the option to go higher. HQ8x has not
yet been developed. I doubt that HQ(arbitrary)x is possible without completely
changing the algorithm.

~~~
scott_karana
I agree, that's huge advantage. That's why I said it was a _deceptive_
representation, but not an invalid one.

------
anon4
The issue with depixelizing pixel art is that you need an AI that can identify
semantically what the features in the sprite are. When you look at Mario you
know you're looking at a stout little man with a hat, you can identify his
eyes, ears, nose, hair, hat, moustache, shirt, gloves, overalls and shoes.
Given that knowledge, if I were to depixelize Mario's sprite, I'll know the
rough shape of his shoes and where the shading should go and what shape it
should have, i.e. I can reason what the 3D shape is and how the 2D shape was
derived from that. I'll also be able to identify which lines were put there to
make sure separate elements can be legible and not make those thicker than
needed. Or that his eyes blend with his hat simply because there aren't enough
pixels to make them separate.

For simple shapes, like the background, their algorithm works really well, but
for complex objects it fails, because it distorts details that were put in
with very careful thought and completely depend on the resolution. Such small
sprites rely a lot on being looked at by someone who can identify semantically
what they're looking at, and any really successful depixelization solution
will need to be able to understand what basic shapes the sprite is made of
based on what it's supposed to represent.

~~~
Houshalter
I've been thinking about doing something like this. Compress images down to
half resolution, then train a neural network to blow it back up and guess the
pixels as accurately as possible.

~~~
Ziron
There has been a neural network scaler called NNEDI3 available as a plugin for
avisynth for a few years. It was trained in exactly the way you are talking
about and does an amazing job at both deinterlacing and upscaling video.

It does a great job of scaling pixel art too. I'm kinda surprised it hasn't
been implemented in any emulator. With the newer opencl implantation it is
finally able to run in real-time.

[http://screenshotcomparison.com/comparison.php?id=46792](http://screenshotcomparison.com/comparison.php?id=46792)

~~~
Houshalter
Whoa, that's pretty good.

------
kaeso
A C++ LGPL implementation of this has been as been developed as part of last
year Inkscape GSoC and is available as a standalone library:
[https://launchpad.net/libdepixelize](https://launchpad.net/libdepixelize)

It will be released in the next Inkscape major version, expected soon :)

~~~
SimHacker
Perfect! Thanks for the link. I want to run this over the classic SimCity tile
set (and the additional ones in the add-on graphics sets like the moon base,
etc), to generate high resolution pixel and vector tiles for the JavaScript
browser based version of Micropolis (GPL SimCity).

------
rsiqueira
The authors of this paper also have other amazing image
processing/optimization algorithms. Eg: Deblurring, Optimizing Color
Consistency, Content-aware Automatic Photo Enhancement, Object Discovery and
Segmentation in Internet Images and more:

Dani Lischinski's page:
[http://www.cs.huji.ac.il/~danix/](http://www.cs.huji.ac.il/~danix/)

Johannes Kopf page: [http://johanneskopf.de/](http://johanneskopf.de/)

~~~
vanderZwan
Wow, the content-aware downscaling seems very useful!

[http://research.microsoft.com/en-
us/um/people/kopf/downscali...](http://research.microsoft.com/en-
us/um/people/kopf/downscaling/)

> _This paper introduces a novel content-adaptive image downscaling method.
> The key idea is to optimize the shape and locations of the downsampling
> kernels to better align with local image features. [...] Besides natural
> images, our algorithm is also effective for creating pixel art images from
> vector graphics inputs, due to its ability to keep linear features sharp and
> connected._

IIRC, this is one of the major reasons that vector graphics based icons have
not taken off.

------
JasonFruit
Sometimes the Vector Magic results are truly magical; striking poses become
even more striking when the figure is reduced to a collection of simpler
shapes.

~~~
vsviridov
I like a lot of VectorMagic results... Very abstract art!

~~~
zimpenfish
Yeah, I really like the VM ones - particularly how the space invader turns
into a StarTrek badge and the Bombermans(men?).

------
th0ma5
I thought this looked familiar! It is from 2011
[https://research.microsoft.com/en-
us/um/people/kopf/pixelart...](https://research.microsoft.com/en-
us/um/people/kopf/pixelart/)

------
10098
But the blocky versions actually look better than vectorized ones. Most of the
vectorized sprites look hideous with the exception of Boo and dolphin.

~~~
userbinator
Count me as another one who likes the pixel aesthetic. The title made me ask
myself "why would you want to do this?"

(I also loathe most forms of antialiasing, including _Clear_ Type, so that
might have something to do with it... I like sharp, clearly contrasted edges,
including pixel edges.)

------
smackfu
From when this first came out:

[https://news.ycombinator.com/item?id=2601347](https://news.ycombinator.com/item?id=2601347)

[https://news.ycombinator.com/item?id=2578706](https://news.ycombinator.com/item?id=2578706)

------
rkuykendall-com
Does anyone know where I can find an implementation of this? I can't even
figure out what to call it. They just refer to it as "our method."

~~~
jonathanj
Here's a pure Python implementation:
[https://github.com/jerith/depixel](https://github.com/jerith/depixel)

~~~
rkuykendall-com
Thanks!

See? Even he doesn't know what to call it!

------
sp332
I wonder why they all fall down on Peach and Peach 2? None of them look nearly
as good (to me) as bicubic.

~~~
Lerc
I think it comes down to your point of focus. With peach and peach 2, The eyes
change shape quite a bit. Humans notice that much more than other aspects.

In general, I think the "ours" method could benefit by moving the boundaries
between colours in and out until they get an average brightness match with the
similar area in the nearest neighbour version. A lot of the "ours" images have
a higher proportion of black.

------
mberning
Boo is the best example of 'ours'. Looks amazing.

~~~
Trufa
I think bower looks badass! [http://research.microsoft.com/en-
us/um/people/kopf/pixelart/...](http://research.microsoft.com/en-
us/um/people/kopf/pixelart/supplementary/results_ours/smw_bowser_ours_8x.png)

------
josh-wrale
I chose to embrace the Pixels, myself:
[http://vulariter.wordpress.com/tag/jewel-
punk/](http://vulariter.wordpress.com/tag/jewel-punk/)

------
omegote
I'm amazed to find that no one has mentioned that ScummVM has implemented some
of these algorithms for years and they work pretty nice with the classical
games.

~~~
baumgarn
and snes emulators ...10 years ago?

------
gverri
Looks like a normalized version of the EPX algorithm.

------
rangibaby
HQ4X and Ours are the clear winners if you _must_ resample like this; the
downsides are that HQ4X occasionally has jaggies and Ours makes makes a lot of
fine details look "blobby".

The Vector Magic results reminded me a lot of this:
[http://www.cowboybooks.com.au/html/acidtrip1.html](http://www.cowboybooks.com.au/html/acidtrip1.html)

~~~
jatin085
Regarding HQ4X, they used the algorithm to create 4X image but used Nearest
neighbor to create 8X and 16X images. I think doing HQ4X two times would have
resulted into a better 16X images without jags.

------
yzh
I did a GPU version in 2012 for my course project. For small size sprite, it
runs roughly the same or even slower than the original paper. Anyone
interested in finishing this and maybe try this on realtime emulator?
[http://yzhwang.github.io/depixelization/](http://yzhwang.github.io/depixelization/)

------
mjmahone17
Kind of ironic that MSR is using an unregistered version of PhotoZoom.

------
mistercow
I think the most interesting aspect of this is the internal shading. I haven't
seen any other algorithm that manages that while maintaining sharp edges
overall.

I imagine it would end up with pretty weird results though when a sprite was
on top of a background that with similar enough colors to trigger the shading
detection at its edges.

------
tumes
I must be a luddite, I prefer my pixilation like my soup: chunky :)

------
mrbuttons454
to me, hq4x looks better every time.

~~~
Ricapar
I think it depends on what your end goal is. Are you trying to scale and hide
the pixel-ness of the original? The "ours" does pretty darn well, especially
if you look at the Boo example as mentioned above.

However, if you're looking at in the case of an SNES emulator or something..
then hq4x seems like the best choice. It keeps true to the original pixel art,
while appearing significantly better.

~~~
fuzzywalrus
Linked earlier:

Video of Ours vs HQ4X. Ours seems to have a slight advantage to my eyes,
admittedly it is a personal preference. Text has a distinct advantage on HQ4X
but Mario and Yoshi look better even if less "accurate".
[https://research.microsoft.com/en-
us/um/people/kopf/pixelart...](https://research.microsoft.com/en-
us/um/people/kopf/pixelart/supplementary/video_multi_comparison_4x_h264.mp4)

------
galactus
I wrote an implementation of this in common lisp when they published it:

[http://dev.crazyrobot.net/2011/07/12/pixelart-to-
vector/](http://dev.crazyrobot.net/2011/07/12/pixelart-to-vector/)

------
adamnemecek
Technology almost caught up with CSI.

~~~
bradyd
There is a technique that might actually be able to do this called compressed
sensing[1] that I first heard about it in a Wired article from several years
ago[2]. The technique allows reconstructing images, music, etc. by sampling a
portion of the original and applying an algorithm that reconstructs the
missing information and is apparently used in MRI image processing to get by
with shorter scans. Researchers from Rice University constructed a camera with
only a single pixel image sensor that, using this technique, produced legible
images with as little as a 2% sampling of the final image resolution, although
more complex images required 10-20% or more[3]. I don't see why this technique
couldn't be used to take already good quality images and "enhance" areas of
them like in CSI.

[1]
[https://en.wikipedia.org/wiki/Compressed_sensing](https://en.wikipedia.org/wiki/Compressed_sensing)
[2]
[http://www.wired.com/magazine/2010/02/ff_algorithm/](http://www.wired.com/magazine/2010/02/ff_algorithm/)
[3] [http://dsp.rice.edu/cscamera](http://dsp.rice.edu/cscamera)

------
rsiqueira
Amazing image zoom algorithm, t creates a hires version of a low res image. I
would love to see this image zoom feature implemented the in ImageMagick.

------
Lambdanaut
The Space Invaders "ours" were very impressive. The text was always really
good too. The algorithm works well on more regular shapes.

------
neona
It would be interesting to see a pixelart/emulator scaler that was temporally
aware - that is, able to use context from near or all relevant sprite frames
to get additional detail/context. This would require a compositing emulator,
as some others have discussed, but I could see it producing notably superior
results, given it would have much more source data to work with.

------
panzi
Is there no 8x or 16x version of hq4x? I thought you can use it to get full HD
resolution from 320x240 (in realtime using GLSL shader).

------
nivla
Is there a paper or code associated with this?

~~~
adamnemecek
[http://research.microsoft.com/en-
us/um/people/kopf/pixelart/](http://research.microsoft.com/en-
us/um/people/kopf/pixelart/)

~~~
mxfh
Which also has the original HN discussions

[https://news.ycombinator.com/item?id=2673271](https://news.ycombinator.com/item?id=2673271)

and links to +1000 day old threads:

[https://news.ycombinator.com/item?id=2601347](https://news.ycombinator.com/item?id=2601347)

[https://news.ycombinator.com/item?id=2578706](https://news.ycombinator.com/item?id=2578706)

yet those space invaders still haunt me.

------
aaronetz
Something related that I stumbled upon some time ago (submitted to HN, got one
upvote): [http://gpuhacks.wordpress.com/2013/07/08/signed-distance-
fie...](http://gpuhacks.wordpress.com/2013/07/08/signed-distance-field-
rendering-of-color-bit-planes/)

------
tibbon
Are there any emulators that support this?

~~~
cedel2k1
Yay, tons of.. at least with the algo's they compare their stuff to.

------
kstenerud
I needed some depixelation software for some graphics awhile back, so I threw
together a quick program that depixelates using various technologies.

[https://github.com/kstenerud/depixelate](https://github.com/kstenerud/depixelate)

So far it supports HQX, XBRZ, and Scale2x.

------
kumarrahul
Proper Video here :- [https://research.microsoft.com/en-
us/um/people/kopf/pixelart...](https://research.microsoft.com/en-
us/um/people/kopf/pixelart/supplementary/video_multi_comparison_4x_h264.mp4)

------
dangayle
Am I the only one who prefers the Vector Magic versions? They're like mini
Picassos.

------
granttimmerman
Here's the home page: [http://research.microsoft.com/en-
us/um/people/kopf/pixelart/](http://research.microsoft.com/en-
us/um/people/kopf/pixelart/)

------
virtualritz
It actually looks like as if you took the polylines from EPX and used the
points as the CVs for a biquadratic or bicubic interpolating spline (e.g.
Catmull-Rom), you'd be very close to their result already.

------
whywhywhy5
I think hq4x is the best. Microsoft's curves too much - and while it can look
good/stylized for some pieces, it's ultimately inaccurate and warps many of
the pieces from their original form.

------
asd
Ctr+f "Invader 1" and choose the Vector Magic version. Star Trek!

------
neals
Funny how I don't remember those vikings to be so low-detail and tiny.

------
sitkack
Todo: Blend hq4x and the kopf techniques in a post processing step.

------
bovermyer
The Secret of Mana ones made my eyes leak, not gonna lie. Awesome graphics
technology update not withstanding, that game was an integral part of my
teenage years.

------
pesenti
Enhance!
[http://www.youtube.com/watch?v=qHepKd38pr0](http://www.youtube.com/watch?v=qHepKd38pr0)

------
TepidSolarsoul
I have a hard time believing the use of Nintendo IP in this case (big
corporation-backed research lab) falls under Fair Use.

------
computerslol
Looks great :) Makes me a bit less tense about the appearance of older
programs in the 4k+ display future.

~~~
joosters
This would be awesome if it was available as a scaler for high-DPI screens and
old apps. Or even bundled into televisions for upscaling.

The paper does note that it is designed for hand-edited pixel art, where every
different coloured pixel is treated as significant. So I wonder how it behaves
when scaling photos?

(Edit: the paper does note that it doesn't do well on anti-aliased
graphics...)

~~~
panzi
Televisions have upscale algorithms. But nothing like this. But they don't
need anything like this because they don't scale pixle art with sharp borders
and very low resolution. They scale video footage (which one could say is
"anti-aliased") with about 640x480 resolution.

------
klh_dk
actually running the hqx algo on the art and then throwing it through ANY
vectorisation engine with a slight roundness comes on top of any of these.

And Microsoft is not really being fair to Vector Magic here, there's a ton of
settings there to improve that rendering depending on what you want...

------
josephschmoe
Doesn't work well with geometric objects. Look at the swords, shields, the
letter 'A'.

------
NicoJuicy
They forgot natural neighbour interpolation... Should show better results then
their research ;)

------
dlsym
Well I guess it's retro week again on HN... Better check my bookmarks from 3
years ago...

------
Legend
This seems pretty awesome. Can someone please tell me where this could be
useful?

------
fisherprice
The 'ours' filter (I guess it's the one from MS) is mind-blowing.

------
Udo
At first I thought this was a live JS-based demo. That would have been
amazing.

------
bart3r
Are there any ruby libraries that would depixel an image in a similar fashion?

------
actionsketch
I just learned that my favorite depixelizer is hq4x

------
digitalpacman
This is pretty amazing of a demo.

------
NKCSS
Just awesome.

------
rocky5
amazing.

------
sz4kerto
I can't believe noone linked this so far.
[https://www.youtube.com/watch?v=3uoM5kfZIQ0&hd=1](https://www.youtube.com/watch?v=3uoM5kfZIQ0&hd=1)

~~~
panzi
[https://www.youtube.com/watch?v=JMIHNiR3CP8](https://www.youtube.com/watch?v=JMIHNiR3CP8)
;)

