

Vector vengeance: Researchers claim they can kill the pixel within five years - mrsebastian
http://www.extremetech.com/extreme/143130-vector-vengeance-british-researchers-claim-they-can-kill-the-pixel-within-five-years

======
ChuckMcM
Sigh. TL;DR version - scaling pixels makes them blocky, scaling vectors
doesn't, for some small range of scaling we can keep upscaled pixels not
blocky.

Upscaling has a lot of research behind it. Creating a 'vectorized' version of
an image (see their paper here:
<http://eprints.gla.ac.uk/47879/1/ID47879.pdf>) certainly helps avoid dumb
scaling issues. But when you blow it up 7x (like the example) it looks like a
cartoon or a bad paint by numbers painting. The issue is that you lose
information about the picture when you capture it, and without that
information you have to 'make up' what you think would have been there had you
been able to capture it. People get a lot of mileage out of the fact that for
the most part real world imagery has a lot of linear transitions but there are
limits to that as well.

I think the work is interesting but as they say in their paper: _"What we have
shown already is that there is a viable continuous image format and that it
can be used for some conventional operations which are not handled easily in
sampled formats."_ which I agree with, but the pixel is not yet in danger of
extinction :-)

------
crazygringo
Slightly side question:

If JPG and video codecs already store information in terms of wavelets,
frequencies, or whatever have you --

Are there any programs that will render JPG or videos at a higher-than-native
resolution?

I know you're not gaining any extra signal. But if you have a 320x240 video,
and you want to play it full-screen at 1280x960 -- instead of rendering at
320x240 and then naively upscaling 4x, would some kind of direct codec
rendering to 1280x960 possibly produce more accurate results?

If the codec/format is doing a good job at extracting "meaning" from the
original image for its compression, it seems like it ought to.

~~~
brigade
Basically, no.

For audio, there's spectral band replication which works well in a large part
because harmonics make audio rather correlated across multiple frequencies.
Images have much less correlation, and thus processing in the frequency domain
is less useful.

Additionally, especially with codecs with a small DCT like JPEG, it's been
shown that once you reach a certain quantizer (throwing away all higher
frequencies, effectively downscaling in the frequency domain), it becomes
better to do a traditional downscale + upscale around the lossy coding.

Or, to put it another way, you can exploit the properties of a how a specific
lossy codec discards information to get closer to the original image
(deblocking filters are the most common example). There is a significant
amount of research on this topic for JPEG. But if you could exploit the
properties of the DCT or wavelets for better upscaling, it would work equally
well as a codec-agnostic algorithm. Frequency transforms are perfectly
reversible assuming infinite precision math, after all.

------
corysama
Quick reminder for everyone: A Pixel is Not a Little Square!
<http://alvyray.com/Memos/CG/Microsoft/6_pixel.pdf>

It's a Sample. There are many ways to filter a collection of samples to
reconstruct an image. Drawing bigger squares with nearest-neighbor sampling is
only one of them. Depending on the context of your application, you might even
go as far as the linked article:
[http://www.extremetech.com/gaming/84040-depixelizing-
pixel-a...](http://www.extremetech.com/gaming/84040-depixelizing-pixel-art-
upscaling-retro-8bit-games)

------
kemiller
You're never going to kill the pixel until you have a display that doesn't use
them.

~~~
Too
<http://en.wikipedia.org/wiki/Vector_monitor>

TLDR: Imagine a CRT/Osciloscope not sweeping row by row but instead line by
line, where line doesn't necessarily mean horizontal but can be diagonal and
even turn midway.

~~~
andrewflnr
Revised: you'll never kill the pixel unless you have a display _as good as or
better than current pixel displays_ that doesn't use them. ;)

------
randallu
The method they use for vectorization is cool (vectorizing Y, U and V
independently) and also illustrates why subsampling UV yields such good
results (for standard raster-based codecs).

Ultimately however vectors are far more expensive to rasterize than bitmaps,
especially if you want antialiasing along edges. There was a bunch of noise a
while back about NVIDIA's path rendering extension -- they do antialiasing via
16x multisampling (!!!) NOT a mobile friendly approach.

~~~
asdfs
>There was a bunch of noise a while back about NVIDIA's path rendering
extension -- they do antialiasing via 16x multisampling (!!!) NOT a mobile
friendly approach.

Do you have any links? I've seen the NVIDIA talk about the extension, but I
didn't hear any of the noise about issues with it.

------
bryanlarsen
I'm struggling to understand what the point is. Video codecs don't store data
in "pixel" format either, they store wavelet coefficients, data from the
frequency domain.

From the article, the source data is in pixel format. No codec format can be
better than the source data. And the output is displayed on a screen with
pixels. Pixels in, pixels out. The intermediate format can only make things
worse, not better. It's not surprising that they're able to do better than
current formats: it sounds like they're using a huge amount of CPU for
encoding. You could get better compression if you created a new MPEG4 profile
that required more CPU too.

The intermediate format may be tweaked to provide better results while
scaling, but you could probably add scaling hints to more conventional codecs,
too. Not sure what the point would be either unless the source data had a lot
more pixels than the output format...

~~~
pretoriusB
> _From the article, the source data is in pixel format. No codec format can
> be better than the source data._

Not necessarily true. For example if I have a bitmap picture of a printed
circle, I can use software to find the shape and convert it to a vector
representation of a circle (which would be infinitely better than the source
image).

It works very well with scanned line art, for example.

------
jgeralnik
The most important line in the article is this one towards the end:

    
    
      Ultimately, though, I think it will take a lot more than a new codec to kill the pixel.

------
jws
Summary: Video Codec based on vectorization of images rather than pixel
encoding. Slow, but highly parallel. Not much info. Developers hope to have a
codec for commercial use in some number of years.

PDF of authors' previous work on still images:
<http://eprints.gla.ac.uk/47879/1/ID47879.pdf>

Still too long?: "Mona Lisa in 50 Polygons[1]" Grows Up.

␄

[1] [http://rogeralsing.com/2008/12/07/genetic-programming-
evolut...](http://rogeralsing.com/2008/12/07/genetic-programming-evolution-of-
mona-lisa/)

~~~
Florin_Andrei
Aren't most video algorithms "highly parallel"?

~~~
brigade
For certain values of "most" and "highly", and depending on how much
compression or latency you're willing to sacrifice for more parallelism.

The more data (from earlier in the bitstream/image) you can use for
prediction, the higher a compression ratio you can achieve. But the more data
you use, the more sequential you become, since you have to wait for it to
become available before proceeding.

Modern codecs generally don't require a single frame to be parallelizable for
encode or decode, and you must sacrifice either latency (encode/decode
multiple frames at once) or compression (split a single frame up into
parallelizable slices) to become more than slightly parallel.

------
arocks
Vector art is not often the holy grail as it seems. Take fonts, for instance.
Even though formats like TrueType stores the data in vector format, numerous
resolution-dependent "hints" have to be added to make it more readable.

Similarly vector icons appear more blurry than pixelled ones especially at
lower resolutions. Extremely scaled images are unimpressive due to loss of
detail.

Vector art often works well in a certain range of resolutions. The "infinite
scaling" promise should be taken with a grain of salt.

~~~
bjourne
> Vector art is not often the holy grail as it seems. Take fonts, for
> instance. Even though formats like TrueType stores the data in vector
> format, numerous resolution-dependent "hints" have to be added to make it
> more readable.

While true, font hinting is thankfully finally going the way of the dodo. Cell
phones and other handheld devices has very crisp fonts without any aliasing
and hinting thanks to their high pixel density. The magic threshold is at
about 150 dpi which when passed makes hinting redundant. Desktop displays are
getting there, albeit slowly. To slowly for my taste. OS X already doesn't use
hinting which is why if you are used to Windows fonts, os x fonts can look
blurry. But with then new retina displays os x's text rendering simply looks
amazing.

~~~
thisrod
> The magic threshold is at about 150 dpi

Is it really so low? That's 20 dots per 10 points; surely you'd notice if a
glyph was supposed to be symmetric, but it actually had 10 pixels on one side
and 9 on the other. Also, Metafont does hinting, and Knuth must have intended
to print taocp at a higher resolution than that.

------
czr80
Aren't HiDPI displays killing pixels?

~~~
jacquesm
The 'D' in 'HiDPI' stands for 'dots', a pixel is one dot.

~~~
czr80
Well, obviously. I meant in the sense of making worrying about pixels
irrelevant. To paraphrase the old koan - if a pixel is too small to be seen,
does it really exist?

~~~
gloob
By that standard, geometric population growth is killing people.

------
nnnnni
Man, Lena really is everywhere.

