
Candy Wrappers Are Listening (2014) - wglb
https://spectrum.ieee.org/tech-talk/consumer-electronics/audiovideo/your-candy-wrappers-are-listening
======
tristanj
Fun tidbit: Three letter intelligence agencies have known about this attack
vector since before this paper was published, and are worried it could be
exploited in the wild.

[https://www.theguardian.com/world/2013/aug/20/nsa-snowden-
fi...](https://www.theguardian.com/world/2013/aug/20/nsa-snowden-files-drives-
destroyed-london)

 _[British government officials] expressed fears that foreign governments, in
particular Russia or China, could hack into the Guardian 's IT network. But
the Guardian explained the security surrounding the documents, which were held
in isolation and not stored on any Guardian system._

 _However, in a subsequent meeting, an intelligence agency expert argued that
the material was still vulnerable. He said by way of example that if there was
a plastic cup in the room where the work was being carried out foreign agents
could train a laser on it to pick up the vibrations of what was being said.
Vibrations on windows could similarly be monitored remotely by laser._

~~~
Judgmentality
I've actually done this in a lab with a laser - it's a fun trick. What the
original article talks about though is passively measuring sound from objects
using a camera - no laser. Although they're measuring the same thing (physical
vibrations of objects) the way they're doing it is actually quite different.

~~~
petertodd
A friend of mine used that technique while working in an entomology lab
studying the mating habits of cockroaches. They were trying to figure out what
kinds of vocalizations the cockroaches were making while mating, and bouncing
a laser off their shell was by far the easiest way to record those very quiet
sounds.

It's been a few years, but I _think_ this was the manufacturer:
[https://www.polytec.com/us/vibrometry/products/](https://www.polytec.com/us/vibrometry/products/)

I wouldn't be surprised if the non-laser techniques eventually also show up as
off-the-shelf products too, simply to reduce costs - a simple camera could be
cheaper than a laser, and might have advantages in being able to record sounds
off multiple points at once.

~~~
slazaro
Also, I think a big advantage for the camera-based solutions is that they're
totally passive, while laser-based need to interact with the object.

~~~
Zigurd
To make a camera-based solution practical, it seems like one would neeed a
specialized camera, for modest resolution, but very high frame rate. Without
enough frame-rate, you can't capture high frequencies. Even if speech could be
discerned from a spectrum that rolls off at 300hz, well under the peak energy
of a typical human's speech, you would still need many hundreds of frames per
second.

~~~
fisherjeff
If you watch the video at the end, you'll see that they actually exploit the
fact that CMOS pixels are read out sequentially and use the data from each row
to extract surprisingly decent high frequency data from even 60 Hz video!

------
blt
Bill Freeman, the PhD supervisor on this project, is a really creative
computer vision researcher. Read through his CV, he has a great ability to
pick interesting problems slightly outside the mainstream. I wish for that
ability.

------
titzer
Also a really cool application of this rolling shutter is video of guitar
strings:

[https://www.youtube.com/watch?v=8YGQmV3NxMI](https://www.youtube.com/watch?v=8YGQmV3NxMI)

(Note that real guitar strings don't vibrate in this way; they vibrate across
their entire length, with limited higher order harmonics; the visible
waveforms here are an artifact of the CCD capture process).

------
thedrake
This will help to solve debates like the one about the moon landing. Use this
video
([https://www.nasa.gov/multimedia/hd/apollo11_hdpage.html](https://www.nasa.gov/multimedia/hd/apollo11_hdpage.html)
\- the highest HD I could find) of the flag moving and extract sound. Then
extract the sound (if there is any)
[http://people.csail.mit.edu/mrub/vidmag/](http://people.csail.mit.edu/mrub/vidmag/)
is the site that he mentions at the end or the ted talk given that goes into
more about the workflow.

Not a programmer so not sure how to use the code to implement it. Any takers?

~~~
Sir_Cmpwn
This is going to do nothing to solve the "debate". Moon landing deniers will
find an excuse to dismiss this evidence like they have for the rest.

~~~
jwilk
The conspirators purposefully made the flag from nylon, rather than candy
wrappers, in order to foil future attempts of sound extraction.

------
larkeith
For more details, full paper can be found at
[https://dspace.mit.edu/handle/1721.1/100023](https://dspace.mit.edu/handle/1721.1/100023)

------
joecool1029
Laser vibrometers are neat. They can also be used to sniff keystrokes remotely
by pointing them at a laptop lid. Research also released before this paper.[1]

[1][https://www.youtube.com/watch?v=xKSq9efXmh8](https://www.youtube.com/watch?v=xKSq9efXmh8)

~~~
larkeith
The linked article describes a passive monitoring system achieving the same
result, no laser required.

------
spyder
Even more "scary" (if it really works): using reflected microwaves in a
windowless room in a similar way:
[https://www.eetimes.com/document.asp?doc_id=1274746](https://www.eetimes.com/document.asp?doc_id=1274746)

------
breakingcups
I wonder how much archival footage we might be able to give a second look
knowing what we know now.

------
sopooneo
I'm assuming it would need to be extremely high frame rate video to extreact
decipherable voice data. Because even at 60 frames per second, which I
understand is at the high end for normal video, that gives you a maximum
frequency of only 30Hz, which is nearing the low end of human perception.

So it's interesting and a definite risk if an attacker is supplying their own
camera, but definitely does NOT mean you can pull voice data from the
vibrating tablecloth in a YouTube video, right?

~~~
function_seven
Not from a YouTube video, due to the video compression obliterating the small
details needed to get the frequency. But the article notes that a rolling
shutter allows even a 60fps camera to capture higher frequencies. Each scan
line of the captured video is at a sub-frame time slice. If the object
occupies a hundred scan lines, then you have 100 slightly shifted 60 hz
samples that you can combine to reconstruct higher frequencies.

------
mmagin
I thought this was as old as continuous-wave lasers, albeit the original idea
was bouncing it off a window.

------
amelius
If smartphone accelerometers become more advanced, perhaps they can be used in
the same way :)

~~~
delinka
I'm trying to work this one out in my head ...

You've left a smartphone lying around and you want to read accelerometer data
to determine what was said in proximity. Just turn on the mic instead.

You've left a smartphone lying around to remotely read light bouncing off it
... accelerometer not required. Also, just turn the mic on instead.

~~~
amelius
Apps don't have mic access universally. It has to be granted by the user.

~~~
delinka
The user leaving the phone behind for the purposes of snooping on the room.

------
0xdeadbeefbabe
Well fine, but who do you listen to? Is face to face interaction still a
thing?

------
matte_black
What is a viable defense against this attack? Some kind of white noise
machine?

~~~
azernik
A private windowless room.

------
mcknz
see also
[https://news.ycombinator.com/item?id=8131785](https://news.ycombinator.com/item?id=8131785)

------
efader
creepy

