
Differentiable Dithering - underanalyzer
https://www.peterstefek.me/differentiable-dithering.html
======
dgant
A great read on dithering: Lucas Pope's development blogs while working on
Return of the Obra Dinn.

It's an incredible dive into how he created the game's remarkable and unique
look, featuring a wonderful and unexpected mathematical contribution from a
forum member. If you're not familiar with the game peek at a trailer to see
what an achievement it was.

[https://forums.tigsource.com/index.php?topic=40832.msg136374...](https://forums.tigsource.com/index.php?topic=40832.msg1363742#msg1363742)

------
bane
On dithering, the original Playstation had built in support for dithering. On
CRT televisions it helped provide a better looking visual and it's a huge part
of the "look" of the system.

[https://www.youtube.com/watch?v=bi-
Wzl6BwRM&feature=emb_titl...](https://www.youtube.com/watch?v=bi-
Wzl6BwRM&feature=emb_title&ab_channel=ModernVintageGamer)

------
anilgulecha
I've recently done a few things around dithering, and found this site good to
experiement:

[https://ditherit.com/](https://ditherit.com/)

It's open source:
[https://github.com/alexharris/ditherit-v2](https://github.com/alexharris/ditherit-v2)

~~~
aharris6
Hi, thanks for using Dither it! I built it, and would gladly receive any
feedback, suggestions or further praise!

~~~
underanalyzer
Wow exciting! I tried to use ditherit to get some baseline comparison images
for my post! Is there anyway to both control the number of colors in the
palette and have it auto pick colors at the same time?

~~~
aharris6
That feature does not currently exist, but that is a great idea. I have added
it to the list (which is just the github issue tracker, and contains no other
items). I like that it auto-analyzes the palette when the images loads without
requiring any further user input, but maybe an option after it loads to "auto-
detect X colors" as a little dropdown thingie.

------
londons_explore
> A pipedream would be an entirely differentiable image compression pipeline
> where all the steps can be fine tuned together to optimize a particular
> image with respect to any differentiable loss function.

Neural Image Compression?
[https://arxiv.org/abs/1908.08988](https://arxiv.org/abs/1908.08988)

~~~
amelius
I fear that you might end up with hallucinations in your images ...

~~~
IanCal
Similar to imagining Ryan Gosling is in your background
[https://petapixel.com/2020/08/17/gigapixel-ai-
accidentally-a...](https://petapixel.com/2020/08/17/gigapixel-ai-accidentally-
added-ryan-goslings-face-to-this-photo/)

~~~
alanbernstein
Ryan Gosling? That's quite a stretch.

------
steerablesafe
It's a very interesting approach, however once you have the probability
distribution for each pixel, independent random sampling produces a poor
dither pattern compared to Floyd-Steinberg or other error diffusion
approaches.

I think once you have the target distributions then maybe you can combine the
sampling with some error diffusion approach. The idea is to make the sampling
of neighboring pixels negatively correlated, so the colors average out at
shorter length scale.

For a sledgehammer approach you can try to have a blur in your loss function
and try to sample from the combined probability distribution of all the pixels
(ie. sample whole images). It would probably make the calculation even more
expensive or possibly even infeasible.

~~~
0-_-0
A differentiable error diffusion loss would dither the image with quantisation
(like in the post), but then minimise the difference between the blurred
dithered image and the blurred original, instead of the dithered image and the
original. This would tend to distribute errors so that the average colour in
an area is the same in the dithered image as the original, similar to Floyd-
Steinberg.

------
Const-me
Last time I did dithering was for Polyjet 3D printers. The problem is
substantially different from what’s in the article.

The palette is fixed, as the colors are physically different materials. The
amount of data is huge, an image is a layer and the complete model has
thousands of layers, because 3D.

I implemented a 3D-generalization of ordered dithering
[https://en.wikipedia.org/wiki/Ordered_dithering](https://en.wikipedia.org/wiki/Ordered_dithering)
The algorithm doesn’t have any data dependencies across voxels, the result
only depends on source data, and position of the voxel. I did it on GPU with
HLSL shaders, it takes a few seconds to produce thousands of images.

------
phonebucket
Fun. I never considered differentiable dithering before.

Would be interesting to see results using a content loss function as defined
by Gatys (2015), as opposed to the L2 loss as given. That should hopefully
capture more long-distance structures in the image rather than optimising each
pixel independently.

------
dbaupp
Very interesting!

This seems somewhat similar to the recently published GIFnets[1]. However, I
believe GIFnets is training a reusable network to a predict palettes, and
pixel assignments, while this post is focusing on optimising the "weights"
(i.e. pixel values) for a single image.

I wonder if the loss functions from GIFnets could be applied to this single-
image approach to potentially solve the banding problem via something a little
more "perceptual" than the variance term mentioned.

[1]: "GIFnets: Differentiable GIF Encoding Framework"
[https://arxiv.org/abs/2006.13434](https://arxiv.org/abs/2006.13434)

~~~
underanalyzer
That's interesting! One thing I was surprised about is that they don't address
optimizing the palette and dither pattern across time (b/c most gifs are
animated). This feels to me like it would be really interesting and a hard
problem for traditional algorithms. They do mention it as a possibility for
future work at the end tho. They also seem to have separate losses for the
palette net and the dither net instead of just adjusting both to optimize a
general image quality metric (although it does look like they have some kind
of perceptual loss, it's just not the only objective)

------
SimplyUnknown
Looks cool!

Two questions:

\- Is this approach also learning the palette? It is kind presented as a given
here but it is of course very important for a good dithering.

\- The loss function might work better on spatially downsampled images. The
downsampling causes a mix of the image colors making the dithered image look
more like the original given a good dithering. This also naturally removes the
variance that is now penalized in the loss function as this is blurred away.

~~~
underanalyzer
This blew up while I was asleep so I’ll try my best to answer now!

1\. Yes the palette is being optimized for as well which is imho what makes it
different from a quantization approach 2\. That’s a good point. I cite a
reference blog post which does use blur in the loss function towards the end
of the post. Unfortunately I think pure blur would still produce a noisy image
as it would remove variance in the eyes of the loss function but not the final
image. I would guess something like the example I give with purple, red, blue
pixels would still be a problem for blurred loss

------
bufferoverflow
What are the applications for dithering these days? I understand it was needed
when we had 4 or 16 or 256 color limits. But now we have 8-bit/channel
displays, and 10-bit is becoming popular.

~~~
formerly_proven
Eight and ten bits are the full range of the image, but dark scenes only use a
fraction of the range and often suffer from banding (and comically bad
compression artifacts on certain popular streaming services). Clean, slight
gradients as a backdrop often only cover a small distance in RGB, so again,
very low resolution and banding is the result.

Dithering is vital. Just like dithering is vital for audio, even at 24 bits.

Also keep in mind that the "10 bit" you speak of is implemented by dithering
on an 8 bit panel in almost every display. Similarly many cheaper 8 bit
displays are actually 6 bit with dithering. Additionally, 10 bit is a very
rare output format [1] and rarely used by applications apart from the handful
of HDR games; even for content creation applications 10 bit support is
uncommon, and it actually being utilized even less common.

[1] Just because everything is output and composited in 8 bit, doesn't mean 10
bit display output is entirely for naught. If you are using hardware gamma
correction, which you are when you use tools like flux/redshift/... or most
ICC display profiles, then 10 bit scanout of an 8 bit framebuffer still makes
sense.

~~~
sinity
Yeah, since the day I noticed it on one YT video I can't unsee it in every
dark one. I didn't even learn _why_ it's so bad for a long time.

Horrible huge squares/rectangles of slightly different black all over the
place.

------
enriquto
A straightforward implementation of differentiable dithering consists in
applying a large support band-pass filter to the image (so that it becomes of
of zero-mean), and then thresholding it at 0. Sure, you lose the property that
the average colors over large regions are conserved, but the image is
perfectly recognizable, even with higher contrast than the original.

------
lokl
Might be better with CIELAB color space, where "difference" is closer to
"perceptual difference."

------
tlarkworthy
Seems like the gains in pallete information is wasted in precise placement of
pixels for dithering. Net loss IMHO, except for naive formats like bitmap.
Interesting nevertheless but I guess we could do better by optimizing against
the storage format. But then we are at the state-of-the-art

~~~
marcan_42
Lower bit depth/palette encoding has not been a state of the art option for
compressing natural images like this for decades, and nobody is claiming it
is.

If you're doing this, it's either because your medium is limited (retro games
or 8-bit equivalent embedded systems), because you can't afford the CPU power
to decompress something more complex (unlikely these days), because lower bit
depth _is_ ideal for the rest of your image (e.g. largely UI graphics with no
gradients, and just a few small graphics), or because you just don't care.

But given those reasons exist, there is value in researching better dithering
algorithms. Also, to some extent, these things also apply to non-palette
formats (dithering to lower bit depths), and that is still relevant today when
e.g. converting HDR content to typical 8bpc (24bpp) formats.

~~~
the8472
Also note that some cheap displays are not even true 8bpc and employ temporal
dithering to emulate 256 color steps.

