
Reflected hidden faces in photographs revealed in pupil - ca98am79
http://www.kurzweilai.net/reflected-hidden-faces-in-photographs-revealed-in-pupil
======
jere
>In addition, “in accordance with Hendy’s Law (a derivative of Moore’s Law),
pixel count per dollar for digital cameras has been doubling approximately
every twelve months. This trajectory implies that mobile phones could soon
carry >39 megapixel cameras routinely.”

Ah Moore's Law. _It explains everything_. Double every twelve months, eh? Hmm,
quick math. Facebook has been around for 9 years... oh yea, that explains why
Facebook allows you to upload full resolution 60MP photos. /s

>What do your Instagram and Facebook photos reveal?

Nothing. They're about 100 times too small for this technique. And they
probably always will be.

~~~
redwood
One thing I've always thought lost in the discussion of increased digicam
pixel count is that at a point you gain no real useful information in those
pixels, because you've crossed the threshold of whether vibration allows
resolution at that level at all.

Sure more pixels usually mean slightly more information, and there's obviously
techniques: stability-enhancement, faster shots etc, that change this critical
point. But for the average person carrying a camera: we crossed the critical
point long ago and now just waste a lot of SSD space.

~~~
hahainternet
The vibration is not as significant an issue as diffraction is. Modern 18mp
APS-C sensors already hit their diffraction limits at around f/11\. Increased
sensor density will push this down to 'reasonable' f-stops, f/8 being the real
cut-off point.

~~~
josu
Could you please explain what diffraction is in simple terms, or point me to a
source where it's explained? I have a grasp on what diffraction is, but I
don't understand the role it plays in this.

~~~
hahainternet
The smaller the aperture, the more diffracted incoming light rays. Once the
'circle of confusion' or airy disc exceeds the size of a single pixel on the
sensor, you begin to lose lens sharpness. If you imagine two strips of paper,
one green, one blue, directly adjacent. Now imagine photons from either side
of the adjacent edge passing through a lens which slightly diffracts them.

This results in an obvious merging of the blue/green strips. The more the
diffraction the more they merge, the less sharp the images become.

Current APS-C sensors at 18 megapixels have pixels small enough that an
aperture of around f/11 is the minimum size before this effect begins. As
sensors get more and more dense, the required aperture size gets larger and
larger. F/8 is considered a 'normal' aperture and once sensors hit that
density then the payoff from increased resolution is significantly less and
can impact image quality adversely.

This is why science missions use 2mp CCDs that they know the characteristics
of, rather than some 40 megapixel phone sensor.

edit: I've just realised I've typed all this out with the wrong idea. You
wanted a simple explanation of diffraction. Imagine a water tank with waves
being generated from a source at one end. Put a wall with a narrow hole in the
middle of the tank. If the wavelength of the waves is significantly less than
the width of the hole, they will for the most part pass through unaltered. As
the hole gets smaller, more significant changes to the waves occur. They begin
to 'spread out' or 'diffract'. An intuitive explanation is difficult but this
is a practical one that makes sense to people.

~~~
kika
Science missions use 2mp CCDs mostly because of the sensitivity. The larger is
the individual sensor ("pixel") the more photons it can absorb thus more
electrical potential it can generate compared to the noise level.

There's also technological barriers for creating large megapixel CCDs as
opposed to CMOS. Large CCDs are usually created by "stitching" multiple
sensors together, which increase their price as a square of the size.

You can shoot fullframe (35mm) camera at ISO 25600 and get decent results, but
if you crank up signal amplification on iPhone to 25600 you'll get nothing but
random color dots :-)

~~~
hahainternet
> Science missions use 2mp CCDs mostly because of the sensitivity

It would be foolish to suggest there was a single motivator for inclusion of a
sensor. In reality it is a combination of our answers. Large sensors allows
lower gain and better noise performance, enables smaller aperture use at
acceptable sharpness, takes less resources to process, transmit etc.

You're not wrong in mentioning it though. Although FF at 25600 is going to be
piss poor even if it's Nikon / Sony.

edit: corrected stupid misspelling

~~~
kika
You've got very high standards in image quality :-)

Nikon D600 at 25600: [http://www.imaging-
resource.com/PRODS/nikon-d600/nikon-d600G...](http://www.imaging-
resource.com/PRODS/nikon-d600/nikon-d600GALLERY.HTM)

Not even close to "piss poor" in my book.

~~~
ygra
That's about the noise level and sharpness I have on my EOS 40D with ISO 1600.
I think I'm going to cry in a corner a little.

That being said, you don't go that high unless you're very desperate and the
sensor itself can go up to 6400, the rest is software.

------
ojbyrne
As someone who has taken a lot of pictures, the key to photography is not
pixels, but light. And so I had to read the paper:

"The room was flash illuminated by two Bowens DX1000 lamps with dish
reflectors, positioned side by side approximately 80 cm behind the camera, and
directed upwards to exclude catch light"

Yes, that's the kind of lighting conditions that are routinely present at
crime scenes.

~~~
vacri
Then there are what seems to be the standard-issue security cameras, where
it's hard to even tell what the main subject is wearing.

------
RokStdy
Am I the only one who thought, "so what?", after reading this? A picture of
sufficient detail taken under optimal conditions can yield reflected images
off of the pupil. Big deal. It has always been the case that in a high
resolution image you had more of a chance to see reflected images.

Not that this isn't interesting, but their hypothetical (hostage situation)
would have been a much more intriguing test case. But, since those
pictures/videos are rarely taken with super high resolution cameras with good
lighting, it's unlikely this technique would be helpful.

 _Edit:_ Just to be clear, I think anything new in the form of research and
development is generally good. Maybe this work will be built upon to create
something great. I just don't like the implied promise that pieces like this
make. It oversells what technology can actually do and is, I think,
misleading.

------
pg
[http://www.youtube.com/watch?v=qHepKd38pr0](http://www.youtube.com/watch?v=qHepKd38pr0)

~~~
pvnick
Combine with content aware fill and you get
[http://www.youtube.com/watch?v=6i3NWKbBaaU](http://www.youtube.com/watch?v=6i3NWKbBaaU)

------
jbrooksuk
I always thought that CSI was having a laugh with scenes like this:
[http://www.youtube.com/watch?v=53ilsuJOTG8](http://www.youtube.com/watch?v=53ilsuJOTG8)

~~~
thejosh
Heh, or
[https://www.youtube.com/watch?v=sp77AjBdlEc](https://www.youtube.com/watch?v=sp77AjBdlEc).

~~~
jbrooksuk
Haha! "Uncrop" amazing! I should really watch Red Dwarf.

~~~
dmd
In a photograph pasted into a Word document and then cropped within it, you
often _can_ uncrop, because the crop is non-destructive.

~~~
jbrooksuk
Did you watch the video? He uncropped a picture which had been printed as a
cropped photo.

~~~
fullsailor
Not trying to legitimize what is obviously satire, but you could possibly
uncrop a printed photo, _if_ the original photo was online, and _if_ it was
indexed by Google. Reverse image search win.

------
a_olt
This work is not entirely new. Here is another recent article, where it is
explained how input to a smartphone/tablet can be reconstructed by analysing
reflections of the screen in various nearby surfaces, or the user's pupil.

[http://dl.acm.org/citation.cfm?id=2516709](http://dl.acm.org/citation.cfm?id=2516709)

~~~
devindotcom
How cool. Careful, though, Neal Stephenson is probably lurking here collecting
items like this for his next book.

------
InclinedPlane
In maybe 20 years or so this sort of shit is going to be scary.

Imagine what happens when we have several orders of magnitude more computing
power available. A lot of things aren't possible today because they are too
computationally intensive, but computing power is just getting cheaper and
more abundant. Imagine what happens when it's feasible to process basically
all data on the entire internet. Every photo and video. Every tweet, vine, and
wall post. Every porn video and macro image.

It'll be possible to find the identity of anyone in any picture. Moreover,
every picture or video that someone has been in will be easy to correlate. Now
scale that up to every activity. Anyone with enough computing power will
basically be able to create a dossier on anyone, or on everyone.

The implications are concerning.

~~~
Gravityloss
Why 20 years?

I'd assume that most people have been facetagged in enough photos on Facebook
so that they can be found in every photo by every random person who uploaded
there, or to a few other cloud services, which any reasonably determined
organization can scrape. People photograph a lot.

And then the CC cameras as well.

Though there are easier ways to spy on people, you can get all relevant info
from their phone apps and social media services anyway.

~~~
InclinedPlane
20 years is just a SWAG for when computational resources will be so abundant
that the scariest level of analysis will be routine. Today it's simply not
practical to determine automatically everyone appearing in a given random
photo or video. Facebook vastly simplifies the problem (by narrowing the
search scope tremendously), but there's a lot of content elsewhere.

For example, consider identifying people by the way they walk, the clothes
they tend to wear, their body build, etc. That sort of thing makes witness
protection efforts practically useless, for example. The sheer amount and
detail of information that could become available will enable things that we
can't even remotely imagine today, but ultimately the biggest change will be
loss of anonymity at every level and loss of privacy.

------
jbattle
I expect big things in the coming decades coming out of computer vision &
image processing. I wouldn't be surprised if algorithms to 'zoom & enhance'
images reach the levels we laugh about in movies today.

I'm not well read on the topic so I might be talking out of my ear, but this
is the sort of thing I'm anticipating getting more and more powerful over
time:
[http://users.soe.ucsc.edu/~milanfar/talks/milanfar.pdf](http://users.soe.ucsc.edu/~milanfar/talks/milanfar.pdf)
(jump to page 23 if you want to see demos of the algorithm)

[http://www.scriptol.com/programming/graphic-
algorithms.php](http://www.scriptol.com/programming/graphic-algorithms.php)

edit: grammar

~~~
kaoD
Noise and lack of resolution are different beasts. That paper has nothing to
do with "zoom & enhance", which will never reach the level we see in movies,
since the information is just not there.

~~~
jbattle
Yet both noise-reduction and resolution-enhancement are both fundamentally
about taking whatever information you _do_ have and using that to
interpolate/infer information that has been lost (or was never captured).

Especially in certain domains such as written letters (e.g. license plates) or
faces that have a well-known structure with variations in known dimensions, I
wouldn't be surprised to see algorithms that can probabilistically infer what
information would have originally been there. Of course you'd never
reconstruct an entire face from a single pixel, but how far can we go? I dunno
- but my hunch is pretty far

~~~
thirdsight
And of course if we look at recent history we see where that technique goes
terribly wrong (Xerox document archival compression).

------
edent
The cameras used have 39 megapixels. I wonder how well it works with Nokia's
PureView cameras - or the lower quality lenses on iPhones & Androids?

------
dhughes
> Although Jenkins did the study with a high-resolution (39 megapixels)
> Hasselblad camera,...

Medium format cameras have a much greater area sensor and probably overall
better quality ($2,000 vs $20,000 or even $50,000 medium format camera)
compared to a regular digital DSLR camera. Even a quick Google seemed to
indicate photographers prefer the larger sensor size even if it has fewer
pixels.

The article mentions _" face images retrieved from eye reflections need not be
of high quality in order to be identifiable"_ but even so there must be a
massive difference in picture quality compared to a Hasselblad medium format
camera sensor.

Still interesting though it's as if the goofy CSI or Blade Runner "Enhance!"
has come true.

------
krapp
And now i'm wondering whether you could create an app which detected faces
reflected in images and scrambled them... though with eyes that might destroy
the way the original face looks in some creepy way. Of course you'd have to
strip all the EXIF data as well including any internal thumbnails which would
be of terrible quality anyway...

------
PavlovsCat
> It would be interesting to see what hidden information is buried in law-
> enforcement (and other) photo archives — some of which could even help
> exculpate innocent persons.

Yeah, it _would_ be interesting. But would you hold your breath for it? In an
age where kids "shoot themselves" while being handcuffed and whatnot, I for
one wouldn't.

------
aaron695
Given the Internet has millions/billions of photos on it yet the authors chose
not to do a simple random sample, at all, I'm guessing this is more setting up
everything exactly correct to create the answer they want.

This is not how one normally does science.

------
im3w1l
EXIF-data is a much more realistic threat.

EDIT: Oh and camera-charecterising noise signatures.

~~~
aeturnum
Do you have any resources on fingerprinting cameras based on noise? Seems hard
to do outside a controlled environment (as noise is so heavily influenced by
environmental factors).

------
evanb
This reminds me of dual photography [1], in which you trade many projectors
and one camera for one projector and many cameras using time reversal
symmetry.

The video [2] gives a good explanation, and the revelation of the playing card
is what strikes me as similar to revealing reflected hidden faces.

[1]
[http://graphics.stanford.edu/papers/dual_photography/](http://graphics.stanford.edu/papers/dual_photography/)

[2]
[https://www.youtube.com/watch?v=p5_tpq5ejFQ](https://www.youtube.com/watch?v=p5_tpq5ejFQ)

------
Houshalter
The hard part would be identifying the person in the photo, if you didn't
already have a list of suspects. Facial recognition isn't that great, but
perhaps we could make only partially automated (i.e. the human points out
where the nose is and stuff like that.) Then use some kind of bayesian
probability to narrow it down (20% of the population has this facial feature,
calculating the probability distribution of different faces/facial features
producing the image being analyzed, etc.)

------
siler
Wouldn't it be interesting if we were able to capture so much information in a
photograph that you could examine the pupils of the people that appear in
pupils, etc.?

------
pasbesoin
Shades of "Blade Runner".

And, I'm now waiting for someone to release a "Privacy Eye" Photoshop filter
(a la the ubiquitous "red eye" filter).

------
crystaln
Remember when they called it "exponential growth" instead of "a derivative of
Moore’s Law"?

------
jrockway
This is a long-time favorite effect of anime studios:
[http://tvtropes.org/pmwiki/pmwiki.php/Main/ReflectiveEyes](http://tvtropes.org/pmwiki/pmwiki.php/Main/ReflectiveEyes)

~~~
renownedmedia
It's especially effective when the dreamy music is cued and the light colors
get all blurry

------
DougN7
This is hardly new. I've been looking at the background of people's reflected
pupils in posters (think of the big posters at hair salons and stores) for
many years.

------
japhyr
Wow! I had no idea this could be done. The implications of this technique for
forensics work are intriguing.

Is this a new idea, or have people been working on this for a while?

~~~
GrantS
This 2004 paper got a lot of attention in the computer vision community doing
essentially the same thing about a decade ago:

[http://www.cs.columbia.edu/CAVE/projects/world_eye/](http://www.cs.columbia.edu/CAVE/projects/world_eye/)

"The World in an Eye," K. Nishino and S.K. Nayar, IEEE Conference on Computer
Vision and Pattern Recognition (CVPR), Vol.I, pp.444-451, Jun, 2004.

~~~
apu
There was a related paper on using the eyes as a means of capturing a full
environment map which could be used directly for relighting a scene (e.g., for
3d rendered scenes or for compositing digital objects into scenes):

[http://www.cs.columbia.edu/CAVE/projects/eyes_relight/](http://www.cs.columbia.edu/CAVE/projects/eyes_relight/)

------
shmageggy
That picture of Obama is 18x25, not 16x20.

~~~
ecocentrik
and the reflection is off the cornea not the pupil but who's paying attention
to details...

------
oakwhiz
This works on things like Christmas ornaments as well...

------
Brajeshwar
[Video] Let's Enhance.
[http://www.youtube.com/watch?v=Vxq9yj2pVWk](http://www.youtube.com/watch?v=Vxq9yj2pVWk)

