Hacker News new | comments | show | ask | jobs | submit login
Depixelizing Pixel Art (2011) (johanneskopf.de)
363 points by doppp 8 months ago | hide | past | web | favorite | 87 comments



I was excited about this when it was all the rage. It speaks to me as a programmer. Some years later, I stumbled upon this excellent article by an artist considering the problem from the art perspective: http://www.dinofarmgames.com/a-pixel-artist-renounces-pixel-...

It points to the problem with these upscaled versions: They aren't any good. In the end, the artist demonstrates how to get good results, draw it again: http://www.dinofarmgames.com/wp-content/uploads/2015/04/yosh...

Of course, the algorithmic approach lets you upscale any number of preexisting games. However, I have concluded that I like the original art better.


That article is fantastic, so right!

There's a ton of graphics research that has the same basic issue, I feel. The premise behind a lot of it seems to be how to do impressive art without an artist, how to automate our way to compelling imagery.

I am guilty of this too, as an occasional graphics researcher and full time tool maker, I'm implicating myself here too. I am with you, the algorithms speak to me as a programmer, and they are really fun to think about and code.

But I suspect we're missing out on some opportunities to do amazing things with an artist in the loop, that we're not yet achieving what we could if we built tools that try to enhance creative people rather than replace them.

Adobe is pretty good about taking Siggraph papers and implementing them with more artist friendly controls than the paper, but I can't help but wonder what we might have if all the Siggraph papers on average aimed to be more controllable and less automatic in the first place.


In their comparison http://johanneskopf.de/publications/pixelart/supplementary/m... the Vector Magic results appeal the most to me precisely for this reason. For most images they look more like an artists rendition than an attempt to trace the image. Which is good.


when it works it's beautiful but worse cases like mario face are ruined. hq4x isn't as pretty but seems more consistent over the whole gallery of examples http://johanneskopf.de/publications/pixelart/supplementary/m...


I had to look but there is a NES emulator that does sprite replacement which is interesting:

https://github.com/mkwong98/HDNes https://www.youtube.com/watch?v=N57ApspYHVE


>Of course, the algorithmic approach lets you upscale any number of preexisting games.

This is the point of these algorithms, for sure. Hand-crafted graphics will always beat algorithmic upscaling of lower quality hand crafted graphics, given the same amount of time to do each, but that isn't possible for most examples of games with pixel art.


Meanwhile, over in r/gamedev, someone just released a tool that does a good job of pixel-arting regular images.

http://pixelatorapp.com/

https://www.reddit.com/r/gamedev/comments/7ylvt9/pixelator_a...


Bizarrely, the pixelated stuff in the images you linked look way better. You couldn't use the depixelized whale or dolphin or whatever (our link) in a game, it looks awful. Some of the full comparison is even worse:

http://johanneskopf.de/publications/pixelart/supplementary/m...

Mario looks horrid.

But you can use literally any of the pixel art'ed images from your link - they all look truly fantastic and production-ready! They're ready to ship now.


That's a really good video; it really shows the problem. Any still from that image looks pretty great, but I agree that in motion it's no good.


(just a note that you replied to the wrong comment, in case you want to move it)


I'd be interested to see the result of iterating on an image using these tools. Like go from high res to pixel art, back to high res etc.


I remember back in like 2010, someone uploaded and ripped a YouTube video 1000 times and then edited the natural degradation of the compression. Went from perfectly watchable video to horrible static-y mess.

Here it is! https://www.youtube.com/watch?v=icruGcSsPp0


Here’s another one up the same alley that is interesting to watch: VHS generation loss.

> What happens if you make a copy of a copy of a copy (and so on) of a VHS tape? This experiment shows how the quality degrades with every generation.

> The copying was done using two PAL VCRs in SP mode.

> (The video is Fading like a flower by Roxette)

https://youtu.be/mES3CHEnVyI


Thanks for the link!

This may sound crazy, but I wonder if there's a way to "automate" this deconstruction with ffmpeg...


Update: here's a one-liner that will do exactly that

ffmpeg -i input.mp4 -strict -2 -crf 51 output.mp4 && rm input.mp4 && mv output.mp4 input.mp4

For better results, run the above command ten times:

for ((n=0;n<10;n++)); do ffmpeg -i input.mp4 -strict -2 -crf 51 output.mp4 && rm input.mp4 && mv output.mp4 input.mp4; done


Theoretically, it should be possible to recompress the video back into the compressed file with no loss, provided the same codec is used, but it would be far more expensive computationally.


Aren't most image/video compression algorithms lossy? There would surely be more than one possible "decompression".


Like translating a phrase into a foreign language and back repeatedly, I inagine you’d end up with something fun and impressionistic. I’d like to see it too.



If you're out of imagination you can copy and paste some comments from here and watch the results.

http://www.translationparty.com/bizarrely-the-pixelated-stuf...


From

> Bizarrely, the pixelated stuff in the images you linked look way better.

To

> But then I remembered I look way better in the morning light.

In one step. Lovely! This sounds like poetry.


I've never heard of this before but just wasted 20 minutes on it. Very interesting concept with hilarious results


Most things I tried were slightly changed from english to japanese just once, then converged. Maybe it's just my lack of imagination!


Most of what I tried gathered somewhat from English to Japanese. Perhaps it is my lack of imagination!


The best examples, those with more divergent results, are those with a lot of subordinate clauses, which get mixed-up and moved from different parts of the sentence. You can select complex sentences from the list below.


> Like translating a phrase into a foreign language and back repeatedly, I inagine you’d end up with something fun and impressionistic.

This exact game features in the plot of Philip K. Dick’s 1969 novel, Galactic Pot-Healer.



The authore of the depixelising algorithm actually also did something like that:

"Content-Adaptive Image Downscaling" http://johanneskopf.de/publications/downscaling/

I always thought it was sad that it didn't get as much attention. I think it could also interesting to use in thumbnail generation based on regular photos, for example.

Going to ask in that reddit thread if the author is already aware of this algorithm. Might be able to take some cool new ideas from it if they don't! :)


Huh, that's actually quite interesting - but for an entirely different purpose.

My better half enjoys a bit of cross stitch. For a cheapo little gift I've played with outputting custom cross-stitch patterns based on pixellating images and reducing the number of colors to something more manageable, but it's often a little janky in GiMP and requires a bunch of manual tweaking - for anything remotely detailed it quickly becomes more hassle than it's worth.

This, on the other hand, might just do the trick!


This was my first thought as well. I've gotten decent at using photoshop to do this with generous use of cutouts and indexed color modes to custom forced palettes, but it's a very laborious process.

Playing with this, it's very effective for solid-color graphics as it quickly settles in on a nice limited palette. The line-work also does a reasonable job, limiting the manual fixing to areas where lines converge and so forth.

I'm working on writing up a post for /r/crossstitch comparing some different methods, so check over there in a while if you're interested.


The longest discussion about this article on hacker news so far: https://news.ycombinator.com/item?id=2578706

I wonder why Microsoft payed him to do develop this kind of algorithm.


Probably as a research project to see if it's worth using in HD conversions of old games, such as the Age of Empires remake that's dropping tomorrow. The algorithm seems to work well on standalone sprites, but kinda merges in a gooey way when two sprites are next to each other. The super mario example from the previous discussion shows this slightly: http://files.jjcm.org/upscaling.mp4

Very curious to see this as it progresses however.


I'm still impressed with the super mario example, I can't personally see anything negative about it and would enjoy playing the game with that "art style."

It's not perfect but seeing the alternatives it is a huge step in the right direction.


I imagine it was also useful to see if it could be used to scale up low res artwork in older programs to hidpi displays.


It looks like the transformation in being applied to the whole image at once in your example. If each sprite was vectorized individually, you wouldn't see the gooeyness. If you are a game developer, this change should be easy enough.


I don't think it was about videogames, more likely upscaling algorithm would have saved a TON of bandwidth for videos.

a a matter of fact some of the most recents algo available for video deinterlacing like nnedi3 are perfect for that, and of course also work for pixel art, and I think they blow away the example showed in their gallery - https://www.youtube.com/watch?v=0691zsXWbhA

I wonder if now, in the tensorflow era, a better automatic approach using generative nn could be devised to reach beyond 8x


Taking time into account is missing. I guess the graph reconstruction could also depends on some kind of pixel art optical flow


waifu2x (http://waifu2x.udp.jp/) is another often-posted superresolution approach, focusing on upscaling low-res anime images instead of extremely tiny pixel art. It uses a totally different learning-based approach - however, their approach would not work with pixel art because it depends on having a set of high resolution ground truth images, and downsampling them to generate training data pairs. No such high-resolution examples exist for pixel art.


> No such high-resolution examples exist for pixel art.

How about application icons?


good point!


waifu2x seems to be based on Deep Convolutional Neural Networks, while the article looks like a pure handwritten algorithm and at least in this case the algorithm looks like a winner..

I am wondering whether in the future we may forget about writing algorithms altogether and instead rely on ANNs to do tasks that could have done better using hand written algorithms?


You mean tasks like computing the rank of a matrix? (http://oneweirdkerneltrick.com/rank.pdf) ;)

Image super-resolution is an ill-posed inverse problem: many possible high-resolution images would be reduced to the same low-resolution image. However, some of the possibilities are more realistic than others. Consider a grey pixel in a photograph from real life. It's more likely that it was downsampled from a 2x2 block of grey pixels than from a 2x2 black-and-white checkerboard pattern. We apply such knowledge by adding regularization to the problem, or using a prior distribution in a Bayesian formulation.

Deep learning is very good at memorizing those priors by looking at real world data. I think, in problems where the prior is important and complicated, data-driven approaches have a big advantage over hand-engineered approaches. That does not mean they will take over every kind of problem.


Something incredibly similar is already built into the latest versions of Snes9x.

hq4x and the #xBRZ algorithms are amazing and fast and look nearly identical to these results.


They compare it to hq4x in their supplementary materials. Theirs is better.

http://johanneskopf.de/publications/pixelart/supplementary/i...


hq4x looks a lot better in the video comparison: http://johanneskopf.de/publications/pixelart/supplementary/v...

This method still wins out though.. but interesting never the less.


I think hq4x has a slight edge because it doesn't go all the way trying to make everything smooth, still preserving some pixelation. There's actually an analog to this in denoising: if you denoise too heavily you end up losing detail. Smart denoising manages to preserve edges, giving a better result. In this case, pixels are the same as noise, undesirable, but containing useful information.



xBRZ can do wonders, especially for Link To The Past.


Video is quite neat. http://johanneskopf.de/publications/pixelart/supplementary/v...

Reminds me of some old newgrounds flash animations and games


This page is where all the fun is. But it's also very slow.

http://johanneskopf.de/publications/pixelart/supplementary/m...


Their method change too much on eyes and facial expressions in general. Also that broad sword is really blunt.

When comparing only the 4x results hq4x wins half the time. On the other half hq4x is a close second and generally looks alright. However in some cases the vectorized ones are pretty bad.


It looks...weird? Cartoony and almost Picasso-y.

Boo and Bowser look great. Mario and Yoshi look terrible. The detailed pixel art and gradients just don't carry over. The shapes of the outlines do, however.

Scanlines IMO look the best for video games, because it also tries to simulate the medium they were originally presented in, not just the graphics data behind the images

https://imgur.com/wtxyA2d


A real-world example of this technique shows up in ColorDMD, an LCD to replace DMD displays used in pinball machines from the early 90's until just recently. Most of the displays were 128x32 with 4 (and later 16) brightness levels.

ColorDMD first had artists create color overlays for the images and animations, and then emulated the display with colored dots instead of the red/orange color from the original DMD.

A later firmware release included upscaling, with some impressive results. This thread on Pinside (a pinball discussion site) shows some examples from the game "The Simpsons Pinball Party". I've linked directly to a comment that shows an original rendered DMD frame along with colorized versions with and without upscaling.

https://pinside.com/pinball/forum/topic/colordmd-unveils-new...


I like the aesthetic of hq4x in a lot of them, it’s a little less loopy.


Every time I see this paper I'm always disappointed that there doesn't seem to exist a usable (open source or otherwise) implementation of it anywhere.


My favourite all time game is Heroes of Might and Magic III. It was released in 1999, and the art is still good, but that maybe because I'm so used to it. Ubisoft did a great job, enhancing it for the HD release in 2015, there is video showing it: https://www.youtube.com/watch?v=AX_h3RIbSKk

Though, one lesson I've learned (as a game developer myself), was that enhancing may introduce more visual problems. On a port of a game I worked (Playstation1 -> PC) we hired an artist to enhance the eye textures, but once the team behind the original game saw this, they told us that since there was no EYE animation in the game, and having such high-fidelity texture (for it's time, back in 2000) made all the characters look like toys.. Now if we left (and we did) the eye textures blurry, then your own eyes can't focus, and accept this better (without eye animation). I'm not an artist (just a coder), but this taught me "less is more".


Is there an implementation of this anywhere?


The biggest advantage of pixel art is that it takes up less resources to create a solid retro type game. Animations suddenly become a lot easier as it relies on the abstract nature of putting the imaginative efforts to the users.

By removing these artifacts, you essentially defeat that purpose. Animating vectors would take a lot more time to create something that seems professional, mostly because you have to be far more detail - analogous to moving from 2d to 3d. I personally don’t see the use cases for this in gaming.


How does this compare to XBR? https://github.com/Treeki/libxbr-standalone


This site has a great page that compares various algorithms: http://johanneskopf.de/publications/pixelart/supplementary/m... While none of them really produce usable results, I love the artistic quality of the Vector Magic algorithm.


> none of them really produce usable results

I don't know about that.

In my opinion the images that aren't game characters look best with their vectoriser/upscaler. Cursor 1 and Setup, for instance.


The chest blew my mind with how it seemed to accurately capture the shading and reflectivity of everything


Many type designers deal with this as part of their workflow if they work with traditional drawing tools like pen and ink. My understanding is that for classically designed fonts its often easier to redraw vector objects over the scan than to clean up what the automatic tools produce. However, for certain types of "distressed" fonts its a kind of necessity.


Does anyone have an open source implementation of this algoritm? Or a just a demo? I would love to play around with it!


https://github.com/vvanirudh/Pixel-Art

can't vouch for it yet, but will be giving it a try later.


A quick play around and I find myself suffering from this same issue: https://github.com/vvanirudh/Pixel-Art/issues/4

A shame, I'm really keen to spruce up some old 50x50px forum icons from where I used to hang around back in the early naughties.


As long as using the pixel art as-is remains an option.

So many classic games have been destroyed in this process.


Is there something that gets a really good result going the opposite direction?


for both directions I would say a competent artist.

Clever algorithms get you some of the way, but you will still need someone with good sense to edit the result to get something usable, which is going to cost you in either time or money.

Downsizing isn't so bad now because of higher DPI displays (e.g. font hinting is obsolete now) but previously designers would need to make individual icons in a number of sizes removing elements as necessary.


In case you missed it - from another top-level comment: http://pixelatorapp.com/


It seems pretty trivial but I could be way wrong as this is in no way my area of expertise...but- pixelization filters in photoshop and other image editing softwares have been around for a very long time.

Even logically it is an easy leap as they're(the software/filters) most likely choosing the most common color in a given area and then converting that area to a square of that color..


Just pixelating an image doesn't really give high quality pixel art. No examples handy but I'm confident you'd see this if you tried to reverse the images in the post with the approach you outlined.


Gotta say, I really like those PhotoZoom 4 results, aside from some of the fringing artifacts. They seem to preserve a lot more of the detail, whereas the presented approach focuses more on eliminating noise.


For the like pixel art, click here and scroll way down: http://noirlac.tumblr.com/archive


Could you take this a step further? For example, a human knows this thing isn't supposed to be lumpy. What would it take to make this look more like it was interpreted by a human?


It would take an understanding of what the image actually represents.


Super resolution with deep neural networks has gotten some decent results that incorporate knowledge of what the image is. However, it's probably not worth it. Running this algorithm on a single image is much simpler than finding thousands of pixel art -> image pairs and training a superresolution network.


Is it included into DosBox already?

hq4x is included I guess, but unfortunately when you get your games from e.g. GOG, you have to tune DosBox manually every time.


I love this paper ! Has anyone come up with an interactive pixel art tool using this algorithm ?


Take an image, Resize 500% so you see the pixels. Save. Run it through the algorithm. Repeat.


Is this work been already used in emulators filters?


Filters have achcieved similar results, but not quite with the same technique!

Or, this algorithm wasn’t intended for real-time processing and creates an adjacency graph of similar pixels. Once it’s done that, it will try to fit/optimize splines over the regions. Emulator filters like hqx determine each pixel’s “upscaled shape” by the colours of its neighbours, usually indexed with a table.


I want to go through the research paper but am unable to load or download it. If anyone has managed to do the impossible please share a drive link. BTW which journal is this published in?


A fantastic classic paper (but ancient history)


Isnt that called "anti Aliasing" :-)

(just kidding)


(2011)


(2011)




Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: