
A Canvas Made of Pixels - robk
http://www.claybavor.com/?p=407
======
Phemist
>> A fun (but impractical and frustrating) variant of this feature is to have
the image change as soon as the viewer looks away. So you’re looking at a
painting, glance away to another room, and look back to find a new painting
hanging on the wall.

Re this comment. There is this thing called "saccadic masking", the gist of
which is that we are effectively blind while our eyes move from focusing on
one position to the next. Depending on the "distance" (angle) traveled by the
eye, this can last up to tens of milliseconds. Enough time to do some cool
stuff!

One of the studies that conclusively showed this effect, a fairly long time
ago, had participants wear some elaborate headgear that allowed the
researchers to track where the eyes of their participants were looking at.
They had them look at a standard sentence like

"the quick brown fox jumped over the lazy dog",

but with a twist! Every word in the sentence was masked, except for the word
the participant would be focusing on at that moment. So if he focused on the
word fox, the screen would show

"xxx xxxxx xxxxx fox xxxxxx xxxx xxx xxxx xxx".

Whenever the system detected a saccade by the eyes, they would recalculate
which word the participant would be looking at (e.g. switch to dog), and
changed the display of the screen accordingly, now showing

"xxx xxxxx xxxxx xxx xxxxxx xxxx xxx xxxx dog"

Participants were asked if they noticed anything strange about the sentence,
but they reported there was nothing strange about it! (Disregarding having a
heavy set of mirrors strapped to your head).

My point: If you were able to detect saccades of the eyes, and relatively
accurately calculate their position, you could have the region of the image
that's unattended be colourful noise, the theory says that your visitors would
be none the wiser (of course this would break down with multiple people
looking at the same image)

Even cooler: procedurally re-generate parts of the image that are unattended,
so that you're looking at an ever shifting image, but wouldn't quite be able
to pin down what's happening. Similar to this video:
[https://www.youtube.com/watch?v=ubNF9QNEQLA](https://www.youtube.com/watch?v=ubNF9QNEQLA)

~~~
cocoflunchy
This is also the principle behind foveated rendering
([https://en.wikipedia.org/wiki/Foveated_imaging](https://en.wikipedia.org/wiki/Foveated_imaging))
where you only render the graphics in high detail where the viewer is looking,
and use a low-res image for the rest with the goal of saving computing power.
Big area of development right now for VR!

~~~
Phemist
That's really cool! I'm not 100% sure, but what I remember from reading a few
psych papers about this is that as long as the "mask" is similar to the
original in a few easy to calculate sampling statistics (colour distribution,
overall hue), it doesn't actually need to look anything like the original
image, even calculating low-res imagery shouldn't be necessary. Our peripheral
vision is just that bad (at least for static imagery, movement detection is
actually pretty good). The foveated part is where the magic happens, and we've
gotten pretty good at fooling ourself into believing this "foveated" part is
much larger than it actually is.

~~~
kedean
Hey, something I know about! I actually worked on one of those research
projects in college, programming the experiments. The idea is that there is a
certain radius around the focal point where you stop being able to detect
changes. I'm not sure what the final results were, but the theory was that you
can calculate how blurry an image can be and still be discernable based on how
far it is from the focal point of your vision. It was surprising how good
people are at detecting changes in a blurry picture that's way out in their
periphery.

~~~
Phemist
My prof used to joke that, whenever he'd discuss fovea and periphery, that if
we had a fovea the size of our full field of view, we'd need a brain the size
of an elephant to process all that information. It's interesting how we're
very sensitive to sudden changes (thus movement) in our periphery, but are so
bad at classifying/identifying static imagery.

I remember reading about right eye and left eye dominance. Where they'd keep
an image on the screen saccade invariant (ie, compensate for any saccades that
were made). Slowing moving a letter/character/word/whatever to the edges of
the participant's field of view and asking when the character was no longer
legible. This happened surprisingly quickly, but at different positions for
the left eye and right eye for pretty much all participants..

------
benbenolson
He says in the article that "Apple did a good job with their monitor", but
what he fails to realize is that the panel in that monitor is made by Samsung,
and is used in numerous other monitors, such as a bunch of Dell Ultrasharp
monitors. These are the exact same monitor, except with a different frame and
stand, and are available for a fraction of the cost.

~~~
guelo
Yea but the integrated quiet computer was also important for his project.
Mounting a separate computer would have added a lot of complication.

~~~
yincrash
He does not say this. In fact, at the end of the article, he writes that
having the whole iMac is fairly impractical and having a smaller Android
system on a chip would be the better solution.

------
stevewilhelm
"Visitors to Microsoft chairman Gates' new mansion in Medina will find
themselves in an electronic Disneyland where they will be able to "call up" a
favorite painting or photograph on combination TV-computer screens.

"Right now, only a billionaire could afford to try $80,000 monitors and
$150,000 computer-storage devices (the house could have several) that may be
needed to make this all work."

from "Bill Gates: 50,000 Computer Paintings On Call" (1991)

[http://community.seattletimes.nwsource.com/archive/?date=199...](http://community.seattletimes.nwsource.com/archive/?date=19910526&slug=1285524)

~~~
otakucode
I remember reading about the various technological features Gates put in when
he built his home. I would be really interested to read/see (video would be
awesome) a followup piece about whether many of these things either required
regular upgrading or else have simply become antiquated. I would guess that
his family kept upgrading the home, but details would be interesting.

------
Tepix
Nice job! If you have a camera that tracks the viewer, you can also make the
image it displays 3D as seen in this video from 2007:
[https://youtu.be/Jd3-eiid-Uw?t=2m46s](https://youtu.be/Jd3-eiid-Uw?t=2m46s)
(back then it used a Wiimote to track the head position).

~~~
rhaps0dy
Do you think a camera would be precise enough? This looks pretty awesome.

~~~
gmac
I had a quick go at this a few months back using Apple's CIDetector[1], and
even at 'high' accuracy that didn't seem precise enough — it was distractingly
jiggly.

[1] [CIDetector detectorOfType:CIDetectorTypeFace context:nil
options:@{CIDetectorAccuracy: CIDetectorAccuracyHigh, CIDetectorTracking:
@YES}]

------
chrismbarr
I'm really surprised there's no photos of the build process. It's hard to tell
if this is actually what is being described, or just a blog post around a
photo of a framed image.

~~~
ohitsdom
Build photos would have been really cool. Also, some detail on the software
would be nice. I'm assuming it's running on OSX, but I still have lots of
questions (what language, how can you change the picture, how does he manage
it remotely, etc).

~~~
claybavor
First of all, yes the frame really exists. :)

If I get around to pulling together build photos (which by the way aren't that
exciting), I'll post them here. For now, just imagine a big hole in my wall (I
had to reframe it), a photodiode sticking out of a hole on the right side of
the picture frame, and an iMac.

The software is just a simple Objective-C / Cocoa app running on Mac OS X. I
use a Kensington presenter clicker to advance the images to change them
remotely. I drop new images into a Google Drive folder from another Mac, which
the picture frame automatically picks up and puts into the image rotation. And
if I want to make major changes, I just use a wireless keyboard and mouse to
work on the frame like I would a normal iMac. It is very unsophisticated.

------
SeanDav
My first thought when I started reading was "wow what a waste of time and
money to display a stupid picture" but by the time I was near the end of the
article, my attitude had turned 180 degrees. This is amazing attention to
detail and craftsmanship in the traditional sense. I can only admire the
process - really impressed.

------
spdustin
It brings to mind the "moving paintings" in the ride wait queues for "Harry
Potter and the Forbidden Journey" (a dark ride with a moving car on an
articulated arm). I saw those things _up close_ and they had a wonderful
canvas texture laid over the screens. It really seemed _magical_ \-- though I
admit I'm one to give myself over to the magic of theme parks when I can --
and it made me wonder if this effect could be reproduced by using a matte
silicone mold of actual canvas or something like that.

As an aside, there is a lot of amazing technology in play in that ride; I read
an unauthorized treatment of how the ride works and I was blown away at the
systems/mechanical orchestration necessary to pull it off. It's no wonder it
breaks down all the time.

------
KingMob
Thisi s cool, but to _really_ reproduce art, you have to avoid RGB-based
displays. I've always assumed the reason wide-gamut/CIE L _a_ b*-influenced
displays are uncommon is that RGB is "good enough" for most purposes outside
of image-editing/printing. But they can only show a fraction of the colors
humans can see. IIRC, the standard sRGB space used in most monitors can only
show a third of the human-visible spectrum.

~~~
morsch
Yes, that seems like a huge limitation. Though the 5k iMac is wide gamut, so
quite a bit better than sRGB. Are there even (consumer?) displays that have a
significantly wider gamut?

[http://blog.conradchavez.com/2015/10/26/a-look-at-
the-p3-col...](http://blog.conradchavez.com/2015/10/26/a-look-at-the-p3-color-
gamut-of-the-imac-display-retina-late-2015/)

~~~
KingMob
Kudos to Apple for making the new iMacs wide-gamut. But I don't know that it's
too common in the consumer space yet, mostly because once you start using
multiple color spaces, you run into the problem of images created in one color
space being shown in another. If the image data isn't tagged correctly, things
will look over/under-saturated on different monitors. (E.g., is rgb(255, 0, 0)
the reddest red in sRGB or the reddest red in P3/Adobe RGB?)

I suspect that when monitor manufacturers tire of competing on pixel density,
brightness, 3D, and size, they'll eventually get around to competing on gamut.
Maybe Apple will lead the way. A lot of photo/video cameras already capture a
larger gamut, so monitors are the last piece of the puzzle.

~~~
mark-r
Unfortunately as the gamut gets larger, so does the visual difference between
adjacent colors. You need more than 8 bits per channel from end to end if
you're going to use anything wider than sRGB.

Any RGB based color space will be triangular shaped; if it's to be physically
realized in a display, all 3 vertices must lie within the CIE gamut. Since the
CIE gamut isn't perfectly triangular you will always be missing some colors
outside of the triangle no matter what you do. You could do much better by
adding a 4th primary to create a trapezoid, which would more completely fill
the space. As the entire industry is RGB based this would be completely
impractical, but it would be fun to see as a demo.

~~~
morsch
Is the CIE gamut "it"? In other words, are all colors perceivable by a typical
human represented by it? I briefly scanned the Wikipedia article but wasn't
any wiser.

~~~
mark-r
Yes, the CIE 1931 color space is generally accepted as the limit of human
perception. I don't know how much variability there is from person to person,
since it's based on averages. There are a small number of Tetrachromats that
can distinguish a greater variety of colors, but I believe those colors are
all still within the gamut.

------
anarchitect
Nicely done, though I would have loved to have seen some build pictures. There
are a number of commercial players in this market (for example
[https://frm.fm/](https://frm.fm/)) which combine hardware and marketplace,
and the geek in me would love to see these succeed.

I work in e-commerce selling art reproductions and have thought a lot about
this. I believe as soon as you make the art ephemeral or changeable, its value
(to you, rather than monetary) is diminished and it simply becomes a means of
displaying pixels.

I jumped at the chance to digitise my music collection, I just can't imagine
doing the same for visual art.

~~~
otakucode
I would expect there are people who lovingly care for their collection of
vinyl records and find the desire to digitize a music collection equally
unimaginable.

~~~
anarchitect
Granted, though I think the difference is even more pronounced when the
primary purpose of the physical object is to be appreciated visually. Not to
detract from the visual appeal of record sleeves, but the primary goal is to
house the album.

I would also argue that sound can be reproduced digitally in a way that
closely resembles the analog equivalent, but the same is not true of
displaying art on screens vs on actual paper/canvas (despite the impressive
efforts in the linked post).

~~~
TheOtherHobbes
Some issues:

Paint on any medium is a slightly sculptural 3D. You can look at a masterpiece
from four hundred years ago and _see the brushstrokes._ The reflection from
the relief map changes as you move your head.

Fine art inkjet prints are more subtly textured, because museum-grade paper is
never perfectly flat.

A lot of art is huge - many feet by many feet. The giant displays needed to
reproduce it don't exist yet.

But the biggest problem with electronic frames is working out how to power
them without trailing an obvious power lead to the floor. You can solve this
problem by plastering the display into the wall, or by building a false wall,
or by ignoring it. None of those are ideal.

I'd love to see a practical solution, but I'm not sure one is possible with
current technology - although it may get closer with the new film displays
that are appearing.

I like the light sensing here though. That's a clever solution.

------
alex_duf
A cool effect would be to have the painting _actually_ looking at you using
face recognition.

Freaky.

~~~
ddeck
An excellent idea. Perhaps combined with the look away detection mentioned in
the article, such that the painting only tracks you when you aren't looking
directly at it. Reminiscent of many a horror movie.

------
tomp
So... when will you start a Kickstarter campaign?

------
frikk
It's too bad this isn't on github. I've always wanted to make something
similar, but using computational art instead of paintings. My dream is to make
a thing that slowly adapts to the room its in, but doesn't seem like a
computer hanging on a wall.

Does anyone know where I can learn more about the lighting techniques
described here?

------
otakucode
I continue impatiently waiting for a large format true color electronic paper
display at a quasi-reasonable price. I have had a different project idea for
such a thing for several years now, but the wait continues. I imagine such a
display would be excellent for the use described in this article. No need to
mess about with matching light levels (although doing that was super cool!),
battery would only need charged every few weeks or months, etc. There are
actually a couple products that seem to be what I want, but for reasons I do
not understand the image displayed can not be changed. I have no idea why
someone would buy electronic paper signage whose image has to be provided to
the manufacturer and which can not be changed later. I would expect a simple
printed sign would be superior for any use that would be put to.

------
mtone
I'm a big fan of flux, but brightness adjustment is another story.

On my iPad, I quickly turned it off. The auto-brightness never hit the spot I
wanted, and switched too strongly and too late. Reading on the bus, the
brightness basically goes berserk at every shadow and never settles. At home
it would be more manageable, but unfortunately desktop monitors don't support
brightness adjustments as far as I know -- really wish it was standard through
HDMI.

Having to put an entire iMac computer on the wall to have access to brightness
adjustments (at least that's my understanding of the solution) makes it a lot
less practical. But kudos to the author for addressing that essential part of
the problem.

~~~
mayoff
Keep in mind that the cheapest 5K iMac is in the same price range as
standalone 5K monitors. So you might as well use the iMac unless you can get a
5K monitor at a pretty steep discount.

------
cabirum
A color hi-res e-ink display would be a much better fit for the project: it
requires power only to change the image and doesn't need light sensors. Too
bad e-ink isn't being developed anymore.

~~~
Zenst
[http://www.eink.com/](http://www.eink.com/) appear to be doing development
and a year ago releasead a colour eink display. Alas the resolution and 4096
colour range is somewhat adrift from the requirements of such a use as in
photo realism. Indeed the market that will boost this area will be supermarket
labeling upon shelfs and the like. Even recently a few bus stops in London
utilising eink for displaying a bus timetible
[http://www.gizmodo.co.uk/2015/12/tfls-trial-of-clever-
eink-l...](http://www.gizmodo.co.uk/2015/12/tfls-trial-of-clever-eink-london-
bus-timetables-is-frustratingly-limited/)

But totaly agree it is aperfect fit for such display technology, alas the
technology is still a long way off being suitable for this niche usage. Albeit
the whole refresh aspect and power usage make it not ideal for current
technology, at least the aspect of screen burn-in and ghosting are less
considerations from the old CRT days (though they could do a better colour
range technology wise).

------
dexterdog
I was having a conversation regarding these things just a few days ago. I
always loved the idea of a digital frame and have had many over the years, but
no devices really nailed it and backlit was always the wrong way to go.
Refresh time is not an issue at all as you would refresh on inactivity if you
really don't want people to see the sausage made. Feeding data to these things
is trivial.

------
Tinyyy
Wow this is incredibly cool! I love the shy feature especially, to screw with
the viewer :P

One question I have, though, is that since the aspect ratio of the screen is
fixed, does that mean that all the photos have to be rescaled? That is kinda
inelegant, I wonder if there's a way to fix it.

~~~
madaxe_again
I think the mat around the photo is also rendered, so you can have whatever
aspect you want - although a portrait image might look a bit lost in the
middle of it.

Next step - mount the frame on a motorised bearing so it can rotate 90 degrees
depending on the image aspect!

~~~
tobr
I don't think the mat is rendered. In the video with changing light
conditions, you can see how the picture lags slightly, while the mat reacts
immediately.

~~~
Zarel
The mat is also used to make the aspect ratio less extreme, so it can't be
rendered.

------
peter303
Constrast is the lagging factor in good computer displays. These are called
High Dynamic Range (HDR) displays. The claim is the eye can see a contrast
factor of about a million at any one time, and a billion switching between
sunny daytime and darkest night. An end-to-end camera-display system would
need 20 bit amplitudes represent such a dynamic range. Many current systems
are just 8 or 12 bits.

SIGGRAPH often exhibits high end experimental displays. 4K resolution is close
to what the eye can see. Four or six basis color systems improve color
capture. But I generally just see an improvement in nature scenes. The huge
win is HDR. A good HDR display is almost indistinguishable from looking
through a window.

~~~
alok-g
[https://en.wikipedia.org/wiki/Human_eye#Dynamic_range](https://en.wikipedia.org/wiki/Human_eye#Dynamic_range)

>> 4K resolution is close to what the eye can see.

Depends on the distance to the display (whose resolution is independent on
where the viewer is).

------
theon144
Awesome! I really love the usage of the photodiode, as well as the camera,
both are pretty thoughtful finishing touches that elevate this to more than "a
cool digital frame".

How much does the display cost, if I were to make a frame like that too?

~~~
morsch
The cheapest 5k iMac is $1799, other comparable displays are roughly in the
same ballpark. I imagine the other items (frame, sensors) are trivial compared
to that expense.

Replacement 5k panels for the iMac (LM270WQ1) are merely $800 on AliExpress,
but you'd have to figure out how to output to them.

------
eliaskg
Hogwarts paintings!

~~~
nkrisc
Almost! Still need to network them.

------
nathan_f77
This is amazing! I had the same idea a few years ago but never got around to
it. I was planning to just use some regular LCD screens and give them some
nice frames. Probably wouldn't have looked that great, though.

It's going to have to wait though, I'm living in an apartment now and can't
really cut huge holes in our walls. Even if everything could fit inside the
frame, it would be pretty ugly if you couldn't hide the power cable.

------
callmeed
If I pull an LCD panel out of an old laptop, how do I connect it to something
small like a raspberry pi or beaglebone?

My understanding was that many lcd panels required custom drivers/firmware
that was often embedded in the laptop/display board.

~~~
Palomides
search for the part number on ebay and you'll probably find a driver board
with vga or hdmi input, or a board that you can buy and ask the seller to
program it for a specific display. Usually less than $30, but you'll need
power, a case, etc..

------
rkalra4
Anti-"Shy mode" with all such frames connected over intranet would be really
cool!

------
hoorayimhelping
Fantastic read, and a great idea. Very inspiring, thank you for posting it.

------
m1sta_
I use a printer.

~~~
glaberficken
you tree killer :P

