
The Daredevil Camera - zdw
http://www.ribbonfarm.com/2016/06/29/the-daredevil-camera/
======
Animats
Beam-forming microphones are well known. Here's a fancy ceiling-mounted one
for video conferencing.[1] Here's the theory in Matlab.[2] And, most
impressive is the surveillance version, the Squarehead Audioscope Acoustic
Camera.[3] This, a 1m disk with 384 microphones, can pick out a single
conversation in a noisy arena.

There's a version which records all the audio channels in real time, using a
big disk array. This allows the phased-array processing to be done during
playback, so the "aiming" can take place later. The company now has its
catalog of surveillance products password-protected, but in the past, there's
been more info about that application. The DEA has purchased two.

There are also "acoustic cameras" for locating noise sources.[4]

All this is a spinoff of submarine passive sonar technology from decades ago.

[1]
[https://www.youtube.com/watch?v=Bepqav87D40](https://www.youtube.com/watch?v=Bepqav87D40)
[2] [http://www.mathworks.com/help/phased/examples/acoustic-
beamf...](http://www.mathworks.com/help/phased/examples/acoustic-beamforming-
using-a-microphone-array.html) [3] [https://www.youtube.com/watch?v=bgz7Cx-
qSFw](https://www.youtube.com/watch?v=bgz7Cx-qSFw) [4] [http://www.acoustic-
camera.com/](http://www.acoustic-camera.com/)

~~~
dingo_bat
My laptop has a beamforming microphone array. The interesting part of the
linked project is that you can literally see objects using homemade stuff.
Well homemade except for the printed circuit board :)

~~~
nikanj
And the digital output microphones :)

------
fudged71
Artlav, this is an incredible project!

I have an idea for you. You mention that you want to use a webcam so that you
can "overlay" this information on top of an image.

What if you could send it directly to your brain?

There is a technology called Brainport[0], which is effectively a 16x16 grid
of electrodes small enough to place on your tongue. By feeding electrical
impulses through these electrodes, you stimulate the nerves on your tongue.
After two minutes of a "champagn bubbling" sensation, your brain rewires the
senses from your tongue to your visual cortex and you can "sense" any input.

This technology has been used to give sight to the blind, balance to those
with inner ear problems, and sonar to NAVY seals.

It would be amazing if you were the first person in the world to truly "see"
sound.

[0]
[https://en.wikipedia.org/wiki/Brainport](https://en.wikipedia.org/wiki/Brainport)

~~~
therein
BrainPort sounds extremely cool. How does the brain decide to interpret them
as visual input, though?

~~~
whatshisface
I'm pretty sure that would be mechanically impossible - the brain is very
plastic, but touch-into-sight seems a little far for adult learning. Instead,
you could expect this to feel more like acquiring a new sense - the experience
reported by those who have used pager vibrators to give their body a
compass[1].

[1][http://www.thinkgeek.com/product/f358/](http://www.thinkgeek.com/product/f358/)
(sorry, I couldn't find a better source - but the product description explains
it)

~~~
tess0r
one of those projects is feelSpace[1] which is a belt with vibrators giving
you either a new compass sense, or tells you where to go (as a navigation
system).

[1] [https://www.indiegogo.com/projects/feelspace-follow-your-
gut...](https://www.indiegogo.com/projects/feelspace-follow-your-gut-feeling--
3#/)

~~~
fudged71
Thanks for the link. I have wanted one of these belts for years! But I can't
believe how expensive it is.

~~~
BuildTheRobots
Whilst a commercial offering is probably quite expensive, it strikes me as the
sort of thing you could do quite cheaply and easily with an arduino.

I've actually been collecting the vibration motors from disposable electric
toothbrushes to build my own belt; as they were designed to run form a single
AA battery powering them should be easy.

~~~
fudged71
Yes I've seen more open and DIY solutions. I was just hoping for a commercial
version that I could buy off the shelf.

------
wbeaty
Very very cool!

And it's about time someone did it! Yours is the first hobbyist project I've
encountered since my own extremely crude ten-dollar version in 1981. No square
array or processing, mine was a mechanical scanner-disk: eight LEDs glued in a
spiral to an old vinyl record album, each with an electret microphone and and
op amp. Spin the disk with a motor at about 6Hz. Wind noise, so add foam to
the mics. It worked. A mechanical acoustic-phosphor raster sensor and display
screen, where sound patterns caused light patterns. The best part was to hold
some headphones near it while playing 10KHz sine wave. This produced
interference stripes on the scanning disk! I hoped to build a huge version
someday, but somebody beat me to it: Tom Zimmerman, an MIT student who
patented the dataglove, then used the money to work at Exploratorium Museum
for a year (as Artist in Residence), building a scanning-microphone LED disk
about 6ft wide.

Since then we have companies selling DIDSON full-blown video for underwater
(with sonic illumination,) also a euro company with a 3D phase-array
microphone imager positioned on a ring or a wire sphere. See:

[http://www.soundmetrics.com/Image-
Gallery/Fisheries/Habitat-...](http://www.soundmetrics.com/Image-
Gallery/Fisheries/Habitat-Observations/Divers-during-Aquarius-ocean-research-
expedition)

Heh, the divers appear to be skeletons wearing scuba tanks.

[http://www.acoustic-camera.com/en/products/microphone-
arrays...](http://www.acoustic-camera.com/en/products/microphone-
arrays/ring-72-ac-pro.html)

~~~
Artlav
Wow, that's one heck of a sonar. This is the sort of images i'm after.

~~~
wbeaty
Put one on ROV for a trip to Loch Ness! That, or scavanging old outboard
motors in muddy lakes.

Don't miss multi-pages video gallery:

[http://www.soundmetrics.com/Image-Gallery](http://www.soundmetrics.com/Image-
Gallery)

------
ProAm
This is the definition of what a 'hacker' is, great read.

~~~
visakanv
Right on. Reminds me of Richard Feynman or Steve Wozniak, the way they just
casually talk about taking things apart and putting things together while
following a curiosity on an adventure. This is representative of the sort of
ethos that drew everyone here in the first place, I think.

~~~
kamaal
Unfortunately once you touch this sort of high, the normal day job starts to
looking boring.

~~~
pizza
But you inch closer towards a reality where there are alternatives to the
boring normal day job

~~~
Artlav
Sadly, such things don't pay the bills.

(Unless i missed some sort of karma/like/wow monetization :) )

~~~
fudged71
You seriously need to consider starting a Patreon[0] campaign. I'm sure many
people would be interested to see more from you. There are individuals on
there who are making significant salaries based purely on interest from the
crowd.

[0] [https://www.patreon.com/](https://www.patreon.com/)

~~~
Artlav
Don't that require making regular content? My posts are fairly random by time
and quality, and most are just rants on various philosophical nonsense [0] or
game design [1].

Wouldn't be fair to people who signed up for something like this camera.

[0] [http://orbides.org/random.php](http://orbides.org/random.php)

[1] [http://orbides.org/randg.php](http://orbides.org/randg.php)

~~~
ErrantX
Perhaps not as a regular income; but if funding for these projects is a
limitation I bet crowdfunding would bring in funds to keep these posts coming
:)

~~~
bemmu
Random site idea.

Could there be a version of Patreon, where not the author themself but instead
the fans start the pledges. The author could choose to take the money or not,
and could prove their identity by putting their bitcoin address in a <meta>
tag on their website.

So we could create a project on the site say "support sound camera posts from
ribbonfarm.com" and money would be gathered for it, then when a new qualifying
post would be made the site owner would be contacted "hey, you have $500
waiting for you, put this meta tag in your <head> to accept it".

This way modest authors wouldn't need to think "am I good enough for Patreon?"
and people could support them. People could provide wallet addresses to which
the money would be returned if not accepted.

~~~
whamlastxmas
This would almost certainly not work due to how easily most sites can be
hacked by experienced and motivated hackers. That website would essentially
just be a bounty list for hackers. The better option would be requiring an
extra DNS entry meaning the hackers would be trying to hack into a registrar
which is probably a fair bit harder on average.

------
Bromskloss
Yay, a sound-field camera! I've been wondering if such things are made. Should
I take the existence of this project as an indication that they aren't?

The use case I imagine is finding what part of a machine is emitting a noise,
much like how you would use a thermal camera for finding what part of it is
running hot.

~~~
Artlav
Not exactly unheard of, but quite rare. I found a company that made large
arrays like that, but they seem to operate in near field, measuring only the
sound volume.

Imagine a wall of microphones next to a truck - you can tell which part of the
wheel makes the sound. That sort of stuff.

As for an actual "camera" that turns sound into an image - i've never heard of
one.

~~~
nxzero
Isn't this exactly what sonar does?

SOund Navigation And Ranging:
[https://en.m.wikipedia.org/wiki/Sonar](https://en.m.wikipedia.org/wiki/Sonar)

~~~
Artlav
Not exactly. A sonar maps a surface using sound, while my camera looks at the
sound itself.

Think laser rangefinder on a scanner platform vs a camera.

~~~
benjamincburns
Have you seen light field photography [1]? It would seem that your array is
quite similar in concept. I'd imagine that getting images similar to those
produced by various SONAR imaging technologies is a similar sort of
deconvolution problem. If you're looking for textbooks or scientific papers to
review, I think the field of tomography is also ripe with methods to explore
here.

I'd challenge the notion however that this isn't 'just' a passive SONAR
device. Keeping the camera analogy, active SONAR use spread spectrum pings,
much like the white light of a camera's flash. Your camera lacks a flash
function, and its lens is maybe not yet as in focus as it could be.

If you did want to add a flash, you're likely a $0.02 piezo device and maybe
$20 in other various parts (and heaps of numerical processing on a PC) away
from having the ability to produce a nice 3D reconstruction of the physical
scene.

If you get direct data capture going, please do post some raw data online. I'm
sure others would have heaps of fun playing around with it.

1:
[http://graphics.stanford.edu/projects/lightfield/](http://graphics.stanford.edu/projects/lightfield/)

~~~
Artlav
Hm, why not - here is the raw data (after PDM decoding) for the 16x16 video
from the article:
[http://orbides.org/etc/snd_16x16_raw.tar.gz](http://orbides.org/etc/snd_16x16_raw.tar.gz)
(4 Mb)

One file per cell, 16bit integer samples. Stored row by row, 128 slices per
frame.

Not sure where to post it, however - here it would be buried.

------
anotheryou
I made a smaller and smoother version of the final gif:
[http://malea.lacerta.uberspace.de/up/128.gif](http://malea.lacerta.uberspace.de/up/128.gif)

(downsampled back to 16x16, than upsampled with photoshops more sophisticated
resize and doubled speed for more fluent movement)

------
visakanv
This is insanely cool. I've often vaguely fantasized about stuff like this but
this guy takes it all the way. He deserves a huge budget. Can we fund this
somehow?

~~~
Artlav
Everyone is welcome to donate a bitcoin or two. :)

The address is on my Orbital Designs site -
[http://orbides.org/](http://orbides.org/)

------
xerophyte12932
While the actual curiosity-following-adventure is amazing, i think one other
aspect to all of this that deserves a lot of praise is his writing. He managed
to take us on his adventure along with him. Not every blogger pulls that off
successfully.

Kudos to that!

------
jd20
This is pretty cool, and reminded me of the far-field microphone array in an
Amazon Echo (which uses 7 spaced out microphones). Which also makes me wonder,
taking something like this (which has way more than just 7 mics) and applying
it to the problem of far field speech recognition, are there large gains to be
had? Just thinking of what new applications become possible, when you have the
ability to print hundreds or thousands of miniature mics on a single board
that will fit in a small-ish device.

------
rasz_pl
>Unfortunately, we are still talking about a total of 64 Mbps of data. That
needs a USB 2.0 sampling board that would pull the data out of the cells over
the bulk interface

you mean a $10 EZ-USB FX2LP dongle?

You might be interested in this
[https://sourceforge.net/p/manyears/wiki/Main_Page/](https://sourceforge.net/p/manyears/wiki/Main_Page/)

Im sure you know about SeeSV-S205, its basically the same thing, but polished
and packaged neatly for FAT commercial customers ($30K).

posted on slashdot in 2013 about the same thing
[https://tech.slashdot.org/comments.pl?sid=3742015&cid=437167...](https://tech.slashdot.org/comments.pl?sid=3742015&cid=43716763)

~~~
Artlav
Interesting. Can it take 16 simultaneous SPI-like channels of data going at
40-ish MHz, and pack it into a USB uplink?

~~~
rasz_pl
not 16, but 8 '40-ish MHz' should be no problem as long as you sync them.
FX2LP gives you raw 8/16bit wide datapump in GPIF Mode mode, or a nice FPGA
friendly fifo in Slave FIFO mode, you can sort your 'spi like' data streams
out on the pc side later/in real time. and since modern PCs have at least 2
usb hosts you could use two FX2LPs for all 16 streams.

Modern i7 cpu has surprising amount of flops and could handle this no problem
(not to mention modern gpus). Consider modifying one of your boards, instead,
or in addition to doing fft on fpga try pumping raw data over usb to a PC, 64
3MHz PDM streams should be 25MB/s? comfortably within usb 2.0 limits. FX2LP
breaks no sweat while pumping ~40MB/s

top ebay link $6 [http://www.ebay.com/itm/CY7C68013A-56-EZ-USB-FX2LP-
USB2-0-De...](http://www.ebay.com/itm/CY7C68013A-56-EZ-USB-FX2LP-
USB2-0-Develope-Board-Module-Logic-Analyzer-EEPROM-/200910097384) application:
[http://www.cypress.com/documentation/application-
notes/an613...](http://www.cypress.com/documentation/application-
notes/an61345-designing-ez-usb-fx2lp-slave-fifo-interface)

~~~
Artlav
> 64 3MHz PDM

That's "64 3MHz PDM" per cell, of which there should be 16. And, they have to
be synced to a sub-microsecond precision.

~~~
rasz_pl
hence 'modifying one of your boards', as an experiment. if it works to your
liking then you could move to FX3, that one can pump 400MB/s of raw bytes over
usb 3.0. and that is for processing raw stream all on PC side.

If all you want is the current setup then above is valid with 2x $6 modules.

------
rev_null
The timing issue between the microphones doesn't surprise me. Multi channel
soundcards require a single ADC that processes all of the channels in unison
(same for the DAC).

This is also why you can't just plug four stereo cards into a PC and expect it
to make an 8-channel system.

~~~
radiowave
Strictly, multiple ADCs are fine, but what matters is that they are all locked
to the same sample rate clock, connected to each card either via a digital
audio input (e.g. SPDIF) or a dedicated clock input (e.g. word clock).

~~~
extrapickles
Most of these digital mems microphones take a clock input, so you can
synchronize the internal ADCs (normally used for doing stereo).

------
anotheryou
Why is the resolution so much fixed to the number of mics?

Shouldn't 3 mics be somewhat sufficient for intermediate values as well? More
mics => more redundancy => clearer image of the waves, but why do the
pixels/blobs "snap" to a microphone?

edit: ah, he does not use timing (at 10fps obviously not) but needs continuous
sound to see the wave pattern across the array...

edit 2: so do these gunfire locators¹ only time stamp impulses? Or do they do
continuous analysis? With natural, irregular sound, it should be possible to
match and measure the delay between mics, no?

edit 3: the gunfire locators would need to analyze for gunfire characteristics
anyways, to weed out false positives, but that could be separated from the
locating.

edit x: oh cool, if you zoom out of the final picture so it's smaller on
screen, you can even see some parallax effect happening:
[http://206hwf3fj4w52u3br03fi242.wpengine.netdna-
cdn.com/wp-c...](http://206hwf3fj4w52u3br03fi242.wpengine.netdna-cdn.com/wp-
content/uploads/2016/06/4x4.gif)

¹ [http://wordpress.mrreid.org/wp-
content/uploads/2013/06/boome...](http://wordpress.mrreid.org/wp-
content/uploads/2013/06/boomerang-mounted.jpg)

------
skrebbel
A Dutch startup called Sorama has been selling this commercially for a while.
It's very cool actually, they sell mostly to companies who want to make their
hardware less noisy.

It turns out that noisy machines are often mostly noisy because a relatively
small number of parts are too flexible, not screwed on tightly enough, and so
on. Track that down and you know where your low-hanging fruit is. A daredevil
cam is exactly what you need for tracking that down.

Their site is a tad unimpressive sadly, but their tech is very cool:
[http://www.sorama.eu](http://www.sorama.eu)

~~~
yread
Pretty cool, here is a live demonstration of their 1024-microphone camera
[https://youtu.be/dJ4NKq7trTA?t=210](https://youtu.be/dJ4NKq7trTA?t=210)

------
bcherny
This would be pretty powerful if you combine it with speech recognition. If
you put this array in the middle of a crowded room, can you better tease apart
different audio sources?

~~~
Artlav
Ironically, this thing can't record sound as we know it - it only have enough
buffer space for a few hundreds of samples per mic, 30 times per second. They
are to be FFT-ed rather than continuously recorded.

While such a contraption should, in theory, be able to distinguish between
dozens of voices in a room, it would take much, much heavier signal processing
to achieve.

------
keville
Excellent narrative on top of an incredible project!

------
acjohnson55
Brilliant project! I had a similar idea several years ago, when I was in a
master's program in music technology at NYU. I had classes on acoustics and
concert recording that really blew my mind as I started to understand how
sound behaves within spaces. The idea of a soundfield camera occurred to me,
and it seemed like it could potentially demystify a lot of phenomena sound
workers have to deal with largely by intuition. Things like reverberation,
acoustic modes, propagation patterns, wave effects, etc. I hope this project
continues to mature!

~~~
Artlav
Sounds rather interesting. Sadly, there is no way anyone would let me into an
opera house with this thing...

------
jparishy
Almost, kinda related is this Invisbilia episode that's pretty interesting
about a blind man that uses directed clicking in his mouth as a way of "seeing
via echolocation". They talk about how it activates similar parts of the brain
that vision does which I think is a cool parallel to what's been created here.

[http://www.npr.org/programs/invisibilia/378577902/how-to-
bec...](http://www.npr.org/programs/invisibilia/378577902/how-to-become-
batman)

------
jvdb
Awesome project!

Nitpick; Duga 3 was never built. The one near Chernobyl is Duga 1.

~~~
Artlav
Facepalm. Yep, somehow it got stuck in my memory as Duga-3. Something about
the third prototype/iteration of the project? No idea by now...

~~~
jvdb
Only recently found out it isn't Duga 3 (as I also always thought it was),
must be that wikipedia redirect.

But I didn't mean to detract, that's really a very cool project that you did!

------
chris_va
So impressive

If you get a x86 hooked up, doing some software cleanup might make the video
look much better. Some ideas that you'll probably come up with on your anyway:

\- Assign colors based on frequency clustering, rather than a heatmap (so,
red=first freq, green=second freq, and so on).

\- Persist colors if the frequency in a region shifts slowly enough (so every
source stays mostly the same color).

\- Remove echoes by finding correlated pixels.

------
numinos1
This reminds me of the "acoustic vector sensor" developed by a Dutch acoustics
firm that can pinpoint any conversation in a crowd.

[https://www.newscientist.com/article/mg21929364-400-matchsti...](https://www.newscientist.com/article/mg21929364-400-matchstick-
sized-sensor-can-record-your-private-chats/)

------
based2
[https://www.reddit.com/r/dataisbeautiful/comments/2vuvhf/pre...](https://www.reddit.com/r/dataisbeautiful/comments/2vuvhf/precise_3d_mapping_and_visualization_of_wifi/)

------
wyager
One of the coolest and most expository posts I have seen in recent memory.
Well done.

------
humbertomn
Great post, Artlav! Great to hear from you after Exosphere in Chile last year.
Eager for the next part of the project and hopefully you get some funds for
your always interesting ideas.

------
wesleye
This is exactly what [http://www.sorama.eu/](http://www.sorama.eu/) does. They
have created a large array of cheap microphones to do an otherwise expensive
setup. They work, for example with Coolermaster to make their cases more
silent:
[https://www.youtube.com/watch?v=JLx9DsNcM6o](https://www.youtube.com/watch?v=JLx9DsNcM6o)

------
sbierwagen
Hey Artlav, when did you start posting on ribbonfarm? I used to subscribe to
it, but unsubscribed a couple years back after Rao wrote a dumb post on
information security.

~~~
Artlav
About a year ago. He invited me in after seeing the "How do you make batteries
that work on Venus" article (
[http://orbides.org/page.php?id=1005](http://orbides.org/page.php?id=1005) ) i
wrote on my own semi-dead site.

------
abcampbell
Someone please give this guy $10m of funding to invent more cool stuff.

Regardless of commercial application, this guy can do #hardtech

------
zappo2938
We can tell the location of a sound by timing of the wave hitting one ear than
the other. Sound Localization.[0]

[0]
[https://en.wikipedia.org/wiki/Sound_localization](https://en.wikipedia.org/wiki/Sound_localization)

------
tlarkworthy
The math is how underwater imaging sonars work. But there the sonar emits the
sound too.

[http://www.blueview.com/videos/2d-imaging-
sonar/](http://www.blueview.com/videos/2d-imaging-sonar/)

------
sinaa
Extremely cool!

Now is the reverse (vision->sound) also possible? Perhaps using an array of
laser sensors, so that we can reproduce the sounds in the environment.

Also, is it possible to turn the images/video back into sound? Could be a fun
experiment!

~~~
Artlav
I've seen a phone app that does this - turns the camera's video into sound.
With a bit of training, deaf people can use it to "see" the surrounding.

------
rhema
"Daredevil" as in vision through sound. Very neat project! I wonder how much
people can see in the dark through listening to sound sources by moving their
heads for motion parallax.

~~~
kuschku
> I wonder how much people can see in the dark through listening to sound
> sources by moving their heads for motion parallax.

Listen to this with headphones:
[https://www.youtube.com/watch?v=IUDTlvagjJA](https://www.youtube.com/watch?v=IUDTlvagjJA)

It’s amazing how much you can geolocate just in even a recording.

------
nitwit005
Excellent project. I appreciate listing the insights along the way.

------
SilasX
I love it! Would love to see it extended to cover a distributed network of
emitter/receivers to image an area like in _The Dark Knight_.

------
amerine
Wow.

~~~
visakanv
I wonder how many other cool-ass projects like this are going on that we have
no idea about.

~~~
DanBC
Or the pages that give tantalising hints about a project, but there doesn't
seem to be a write-up anywhere.

[https://www.flickr.com/photos/fotoopa_hs/sets/72157611107153...](https://www.flickr.com/photos/fotoopa_hs/sets/72157611107153997/with/3636467040/)

[https://www.flickr.com/photos/fotoopa_hs/4082464167/in/album...](https://www.flickr.com/photos/fotoopa_hs/4082464167/in/album-72157611107153997/)

~~~
skiningham
I think those Flickr albums _are_ the write-up for how to make a Fotoopa.

------
dubmax123
This is awesome! I'm sensing some Machine Learning applications here.

------
boxcardavin
This speaks to my heart.

------
oinsurance3
Nice

------
soheil
I think the whole thing could be done pretty much all in software. Setup a
bunch of phone numbers in Twilio, using a bunch of your friends iPhones,
strategically located, call the numbers and process the data using Twilio API.
Whatsapp or other real-time voice communication apps with an API may work too.

~~~
dmitrygr
You'll never be able to collect phase shift info from such a setup. Not only
will each phone's DSP algos delay the data, but the jitter in transmission and
reception will just kill any chance of it working properly.

~~~
tripzilch
... but I commend their optimism :-P

