
Screen and asset image scaling for games - speeder
http://coderofworlds.com/screen-and-asset-image-scaling-for-games/
======
jblow
Some time ago, I wrote some articles on image scaling that go into a bit more
technical depth (with a specific focus on mipmapping):

[http://number-
none.com/product/Mipmapping,%20Part%201/index....](http://number-
none.com/product/Mipmapping,%20Part%201/index.html)

[http://number-
none.com/product/Mipmapping,%20Part%202/index....](http://number-
none.com/product/Mipmapping,%20Part%202/index.html)

A problem I get into at the end of the second article is that gamma-correction
is very important for good image scaling results. However, almost nobody gamma
corrects during scaling, even today.

~~~
james4k
I might be misunderstanding, but shouldn't gamma correction happen after
drawing your scene?

~~~
jblow
You are thinking of a different gamma-correction step. The reason you gamma-
correct during rendering is because monitors expect data to already be gamma-
corrected, and it is this expectation that causes the problem I am talking
about. Data in a bitmap is not linear with the light-intensity of the colors;
it is gamma'd!

The simplest kind of mipmapping is a box filter, where you are just averaging
4 pixel values at once into a new pixel value. Thinking just of grayscale
pixels, if you add 4 pixels that are each 1.0 (if you are thinking in 8 bits,
1.0 == 255), and divide by 4, you get 1.0 again. If you add two pixels that
are 1.0, and two that are 0, you get a value of 0.5. Which would be fine if
your bitmap were stored in units that are linear with the actual intensity of
light; but they are not, because they are in a weird gamma! What you are going
to get is something like pow(0.5, 2.2) which is way too dark.

Thus when you don't gamma-correct during mipmapping, bright things lose
definition and smudge into dim things way more than actually happens in real-
life when you see objects from far away.

~~~
zokier
> Data in a bitmap is not linear with the light-intensity of the colors; it is
> gamma'd!

That is true for common sRGB images, such as what you would see on the web.
But for games which usually have tightly controlled art pipeline it would not
be unfeasible to use linear colorspace bitmaps as textures, and in such case
you would not need to have gamma-aware scaling.

~~~
jblow
It is somewhat infeasible to use linear colorspace, because you need a lot
more precision in order to do this without banding. You end up with
substantially bigger texture maps, possibly twice as big; but actually it ends
up being a lot more than twice as big, more like 8x or 12x, because the
compressed-texture formats that graphics chips are able to decompress onboard
do not support this extra precision. So if you were to try using something
like S3TC compression to reduce this bloat, the result would be really ugly.

In general, games only use light-linear texture maps when they also need HDR
in the source texture, which is not that often. Ideally it is "purest" to use
HDR for all source textures, but nobody does this because of the big impact on
data size. (And even for the few textures that are stored as HDR source, often
the data will not be integer, but some other slightly-more-complex encoding.)

[Claimer: I have been making 3D games professionally for 17 years, so this
information is accurate.]

------
city41
When offering upscaling options, it's great to offer all the things mentioned
in the article. But _please_ always offer nearest neighbor as an option too. I
want to see the original pixels as close to their original form as possible,
and so do many gamers. Pixel art is great, the "blockiness" is part of its
charm.

~~~
Narishma
That's only true for PC games, where monitors did indeed produce blocky
pixels. Arcade, console and early computer games that connected to TVs were
designed to take advantage of the "deficiencies" of those screens to produce
smoothed images, or the illusion of more colours than the systems were capable
of, among other effects. So if you want to play those games as they were meant
to be, you either output directly (without any upscaling or filters) to an old
CRT TV, or you use those NTSC filters or shaders that most emulators have that
try to mimic (to various degrees of success) the way an old TV displays
images.

~~~
city41
I disagree, and so do most/all retro gamers nowadays. The golden standard back
then were the standard res RGB monitors that arcade games used. Consumer TVs
in America at best had composite input and paled terribly in comparison. But
we all dreamed of our games looking as crisp as they do in the arcade. Even
the NES had an arcade counterpart where you could experience NES games in
crisp, beautiful RGB.

Standard res RGB monitors are not as crisp as PC monitors. But they are still
quite nice and really do let the individual pixels shine through.

Most retro gamers now typically hook up their consoles via SCART to a CRT that
accepts pure RGB, typically a Sony PVM. I've got all my consoles and my arcade
boards running on a PVM and can really easily see the pixels making up the
art.

here's a few examples I grabbed off the web:
[http://imgur.com/a/GXHyf](http://imgur.com/a/GXHyf) They look even better in
person.

Yeah most emulators provide filters that give you scan lines and all that jazz
(none of them look quite right). And of course an RGB monitor does not look at
all like an old game running in a modern emulator with no filters. But I would
still definitely say you can see and appreciate each individual pixel.

------
steventhedev
There's an upscaling algorithm he missed: [http://research.microsoft.com/en-
us/um/people/kopf/pixelart/...](http://research.microsoft.com/en-
us/um/people/kopf/pixelart/paper/pixel.pdf)

~~~
speeder
I don't missed it, I ignored it on purpose, for two reasons:

One, I did not understood it well enough to try to explain even the basics.

Two, I could not find a binary or source I could compile easily to make a
example image.

Also you can put a "three" there: it is not widely used, while even the recent
xBR has a couple of plug-in implementations on emulators for example (and has
binaries)

EDIT: Lack of binary is also why I did not made sections for 2xSal, SuperEagle
and so on... If people help me finding that kind of stuff or creating the
needed images, I will gladly update the article.

~~~
archivator
This looks like a FOSS implementation of Depixelizing Pixel Art -
[https://github.com/jerith/depixel](https://github.com/jerith/depixel)

------
zokier
One alternative upscaling method (for non-integer ratios) is to first upscale
with no interpolation a integer ratio to make a bigger image than the target
size. Then the large image is down-scaled (with some suitable
algorithm/filtering/interpolation) to final size. Afaik it is what Apple does
with it's "Retina" MacBooks and thus I have dubbed it "retina scaling", but
there is probably a proper name for the method too.

edit: comparison [http://imgur.com/a/fC8iQ#1](http://imgur.com/a/fC8iQ#1)

~~~
alanctgardner2
In a class I took on image processing, this was offered as the definitive way
to do non-integer scaling. Apple certainly didn't invent it.

------
vinkelhake
While not really general image scaling, there's a lot of work being done on
post-processing in emulators to try to emulate CRT displays (like the aperture
grille) or even the quirks of NTSC.

[http://emulation-
general.wikia.com/wiki/Shaders_and_Filters](http://emulation-
general.wikia.com/wiki/Shaders_and_Filters)

~~~
speeder
Heh, it mentions that noone can emulate yet shadow mask CRT.

Those are what I mention in my article (TVs that have triangle pixels composed
of 3 circles)

I never use current CRTs emulation because I guess I never saw TVs that could
be emulated, I think all CRT TVs I saw consoles running were of shadow mask
type, having almost no scanline (and I was a curious child that used to put
amplifying lenses in front of TVs to see their pixels...)

------
MProgrammer
A great way to deal with scaling assets for mobile is to create your art in a
vector format to begin with. For Uncle Slam on iOS we did all the artwork in
Illustrator and exported to PNG at three sizes: iPhone 1x, iPhone 2x (and
iPad), and iPad 2x. The results are sharp images with the right level of
detail and no scaling artifacts.

We even took the exact same artwork and made an 8 foot tall banner with it,
and it looks great!

~~~
shachar
You assume here that size doesn't matter, when it might. If the difference is
only initial download size in your game, and is a few dozen MBs, then yes I
agree. but for some applications, this is content that is loaded in runtime,
and since size DOES matter in this regard, some applications will need to
upscale images.

------
zackmorris
What we did for Return to Dark Castle (which ran at a fixed 640x480, 16 bit
color) is give the user the option to scale by nearest neighbor,
interpolation, or both (mixed 50% with each other). The "both" option worked
surprisingly well, especially since it was a "quick fix"!

------
darkmighty
The reason linear filters don't work for upscaling, say a Lanczos or Sinc, is
that the base supposition made for generating the filters is that the source
is "bandlimited" \-- that is, all images are samples of some source which can
at most vary once every pixel. Problem is, while that works very well for
regions that are supposed to be smooth -- textures and stuff -- it fails at
edges, since edges have infinite bandwidth. The result is a wash out as you
can see for linear/bicubic interpolation.

Of course, it does work almost perfectly for downscaling, because in this case
we're not creating any edges, just reconstructing the original edges -- an
those vary at most once every pixel.

~~~
jblow
Actually, the effect happens just as much during downscaling. It is just not
as noticeable, because your artifacts happen at subpixel scale. But they do
affect the output colors.

The math does not really know the difference between upscaling and
downscaling.

------
cageface
You run into pretty much exactly the same set of problems in the audio DSP
realm whenever you have to deal with audio of different sample rates or with
variable sample rate algorithms like tape delay emulations.

Most typically in that domain you also use windowed sinc filters and there's a
ton of literature on the tradeoffs of specific window designs, as well as very
fast fixed point implementations etc.

It's all pretty interesting stuff and trying to make it run efficiently on
modern mobile hardware is a fun challenge.

------
Lerc
I'll add my 2c with an article that I wrote some years ago.
[http://www.screamingduck.com/Article.php?ArticleID=35&Show=A...](http://www.screamingduck.com/Article.php?ArticleID=35&Show=ABCE)

It's a look at kind of a interpolation weighted by scale2x.

