
3D scatterplot of an image - Overv
https://alexander.engineering/imagescatter/
======
thechao
In the late '90s I was asked to do image segmentation of multispectral data of
brightfield microscope data. That is, given a picture of a bunch of cells,
find all of the cells. There were two problems: image segmentation of sick
cells is really hard (they're very blobby & fragmented), and the computer I
had available took more than 20 minutes per image.

It was literally both cheaper _and faster_ to have a grad student count the
cells.

Let's be clear though: the researchers weren't interested in the _cells_ ;
they were interested in having a pretty-darn-good count of cancerous cells vs.
non-cancerous cells. Turns out, cancerous cells were a different color
(brighter blue * brighter green) than healthy cells. So, instead, I just
converted every image into RGB 3D space, counted all the pixels near 'dark
green * dark blue', all the pixels near 'bright green * bright blue' then
divided by the average area (in pixels) of each type of cell. The resulting
counts were within 1% of the value the grad students could get, and the ratio
was even more accurate.

We could get the cell count in a single linear, forward pass, as fast as the
camera could take pictures.

~~~
tovacinni
Sorry if this is a stupid question, but can't you do this without converting
the image into an RGB 3D space? (i.e. iterate through the pixels and count the
ones within a certain range of what you want)

~~~
infinite8s
How would you determine if they were within the range? That's effectively
converting it into a 3D space.

~~~
thechao
A few plane equation evaluations—then a few ands, and a sum. Plenty of
horsepower on a pentium pro to do that.

~~~
infinite8s
Those plane equations are describing the pixel in 3D space.

------
crispyambulance
This reminds me of a machine vision application for large-scale bakeries. When
you bake bread, the color of the surface changes as the bread rises and
completes the baking process.

In the application I remember, the average color of a loaf was tracked on an
RBG plot as a function of time. The idea was that the baking process followed
a trajectory in the RGB space that was dependent on the temperature of the
oven and the elapsed time. This can be used as feedback for process control--
to signal when to pull out the bread or to change the temperature.

In a similar way the color of paintings change as they age. The individual
points in the OP's plots could be parameterized in time and according to
paint's chemistry. It might be possible, after some calibration studies, to
accurately simulate aging. Or better, simulate the reversal of aging to show
what an old aged painting looked like when it was new.

~~~
zwieback
I did the same thing for browning of french fries. It turned out that RGB was
too highly correlated to be efficient so I mapped into an "application
specific" color space where the rate was pretty much along a straight line.
Kind of like HSV but more specific to the actual cucumber shape of the RGB
distribution.

[https://www.google.com/patents/US5818953](https://www.google.com/patents/US5818953)

------
neonnoodle
This is really cool! I'm a painter and have spent a long time doing gamut
analysis of paintings.

Thank you for including both HSV and HSL. Not enough people appear to
understand/care about the difference between the two, and for painters the
distinction is _everything_.

~~~
pygy_
Why is the distinction important for painters?

~~~
discreditable
I'd guess it's because mixing colors in paint is subtractive—if you mix every
color you get black. Whereas with light, mixing colors is additive—if you mix
every color you get white. In HSL, fully saturated colors have .5 brightness,
and adding white or black changes the brightness but not the saturation. My
guess is that works more similarly to paint than HSV does.

------
rijoja
Would it be possible to make a human recognizable 3D-object in the colorspace
that also is a pretty picture in 2D?

~~~
Confiks
The answer is quite obvious. You could start off with a grayscale 2D image on
one side, and a pretty dotted 3D object on the other side. You'd then read the
coordinates of the 3D object in a specific color space to obtain color values
which you can pretty randomly (or even judiciously) apply to the grayscale
image.

Now I would pose another question: would it be possible to make a human-
recognizable 3D-object in all the four color spaces that is also a pretty
picture in 2D?

~~~
rijoja
Clever to start out with a greyscale image! But it still poses some
constraints, let's say if we start with an all black image for an edge case to
demonstrate the limitations, we can only have a 2D pattern in the 3D space,
right?

However the luminance component of the image could probably be adjusted within
certain error margins and vice versa. Perhaps by some form of bruteforce!

To know for sure I suppose coding is needed!

------
MegaLeon
Funny to see this, I made more or less the same thing in Processing for the
google devart interactive competition a few years ago:
[https://www.youtube.com/watch?v=YaA3dAJz6sI&feature=youtu.be](https://www.youtube.com/watch?v=YaA3dAJz6sI&feature=youtu.be)

Has a few extra features like being able to see different visualisation
methods and how basic colour correction influences those - also animates the
visualisation while going through the image

Realluy caught me off-guard because I used both the starry sky and the
monalisa painting back in the days as well!

~~~
btbuildem
Very neat! I like how you aggregate the points with larger objects -- sort of
a 3D contour plot.

Do you have an online demo handy?

------
psyklic
There is a really cool use of clustering with these scatterplots to do color
quantization (i.e. reducing the number of colors).

If you want 16 colors, just find 16 clusters and their centroids (e.g. let
k=16 for k-means clustering). Replace each pixel with the closest centroid.
Then, your image is quantized (and can perhaps be stored more efficiently)!

~~~
ska
There is nothing specific about color here - the same approach can be used for
intensity re-quantization for example.

Effectively you are non-uniformly resampling based on distribution, so
relative values/distances go out the window but you keep "detail" in the lower
fidelity signal.

------
pavel_lishin
This is cool, but I wonder why most of the demo images - and a lot of the
images I uploaded - seem to have their colors aligned in a single plane.

~~~
akrasuski1
Hmm, I guess someone would have thought of that, but this could probably be
used for compressing images by reorienting the color axes so that one of them
is perpendicular to this color plane, and then sending less information about
that coordinate.

~~~
abainbridge
Yeah, you can use principle component analysis to figure out the transform to
make the colour space "optimal" for compression of a specific image. I tried
that once:

[https://bainbridgecode.wordpress.com/2012/04/24/beating-
png-...](https://bainbridgecode.wordpress.com/2012/04/24/beating-png-part-2/)

and

[https://bainbridgecode.wordpress.com/2012/05/07/beating-
png-...](https://bainbridgecode.wordpress.com/2012/05/07/beating-png-
part-3-remembering-a-level-maths/)

There are plots at the end of part 3 showing the image channels separately for
YCrCb and my custom PCA derived space. They clearly show that there's a lot
less duplication of data between channels with the latter. And that in turn is
obviously good for compression.

~~~
thomasahle
Finding the first primary component also appears to be a pretty good way
greyscale an image.

------
bennettfeely
Some interesting images to upload

"One Million Colors"
[https://upload.wikimedia.org/wikipedia/commons/thumb/d/d6/1M...](https://upload.wikimedia.org/wikipedia/commons/thumb/d/d6/1Mcolors.png/800px-1Mcolors.png)

"RGB HSV Image"
[https://www.gimp.org/tutorials/Digital_Black_and_White_Conve...](https://www.gimp.org/tutorials/Digital_Black_and_White_Conversion/rgb-
hsv.png)

------
JD557
Nice visualization, although an option to show HSV/HSL in a cylinder might
help to better understand the distance between colors.

------
ekianjo
So what's the insight behind such visualizations? Can it be used to separate
photographic pictures from paintings for example?

~~~
Overv
One thing it can be used for is to easily see what kind of palette a painter
is using.

------
jathu
I'm not sure if they are mapping every single pixel or using some average
pixels, but it's pretty fast.

Here is a shameless plug for one of my color extracting library:
[https://github.com/jathu/UIImageColors/](https://github.com/jathu/UIImageColors/)

It currently takes around ~0.3s on average to extract the colors. However,
with my new PR
([https://github.com/jathu/UIImageColors/pull/54](https://github.com/jathu/UIImageColors/pull/54)),
it takes around ~0.14s on average. IMO this is still slow, I would like to
bring it below 0.1s.

I tried to optimize this with k-means to reduce total number of colors, but
the result was slower and worse color choices. If anyone has methods to
improve the performance, please make a PR.

~~~
mynewtb
What takes so long in your approach? I would think extracting simple
statistical values from an image is a very easy to vectorise and parallelise
task.

------
dahart
Sounds cool, I will check it out on a laptop.

Just in case the author is here, I tried on an iPad Pro, and got "WebGL is not
supported by your browser". WebGL is in fact supported by mobile Safari, so
I'm guessing this site is using a user agent whitelist or some other fallible
method of detecting WebGL. The best way to detect WebGL support is to attempt
to create the 3d context and wait until after it actually fails. That, or
simply do not check for WebGL failure; the browsers all handle it anyway.

~~~
Overv
Author here, not sure if I can easily fix this. The WebGL check is handled by
the third-party scatter plot library.

------
dontreact
This is a good illustration of why this algorithm which reorganizes paintings
into 2d color palettes works Using these scatter plots ou can see how you
would use the principal components to lay out the pixels on a 2d surface
[https://github.com/ardila/paintingReorganize/blob/master/REA...](https://github.com/ardila/paintingReorganize/blob/master/README.md)

------
fenollp
Relevant:
[https://www.reddit.com/r/dataisbeautiful/comments/7584no/3d_...](https://www.reddit.com/r/dataisbeautiful/comments/7584no/3d_rgb_scatterplots_of_colours_used_in_famous/)

~~~
Overv
That was actually the inspiration for this project :)

------
sidhantgandhi
This is beautiful. It's interesting to see how much of the space is not
occupied — even by images that feel like they have a lot of colors.

Next up: Finding patterns in the graph for certain artists, themes, styles,
etc.

------
adrianmonk
"based on the pixel values"

I daresay technically this applies to all useful visualizations of the data. I
don't think they'd be very useful if they weren't, in some way, based on the
pixel values.

~~~
_han
Sure, but here the pixel values are directly used as spatial coordinates, so
it makes sense to mention this.

~~~
adrianmonk
I was trying to make a point about phrasing. "Based on" is wordy and nebulous.
Just saying "of" would clearer, not to mention shorter.

------
zamazingo
This is really cool! Is there a source repository we could explore?

~~~
Overv
I've now published the source of the website on GitHub as well:

[https://github.com/Overv/ColorScatterPlot](https://github.com/Overv/ColorScatterPlot)

------
xbryanx
This could be a fun add on for an art museum's online collection browser.
"Investigate the collection by scatterplot."

------
sus_007
How much JS knowledge does one need to start plotting with D3.js or Plotly.js
?

~~~
minimaxir
For simple data visualization, Plotly.js is easier to use with another data-
friendly language like Python or R, which has libraries which push the data
into a web-friendly format (including 3D WebGL charts like these)

