
FingerIO: Using Active Sonar for Fine-Grained Finger Tracking - jonbaer
http://fingerio.cs.washington.edu/
======
parkaboy
Regarding everyone's latency concerns, as someone who has done low-latency
audio processing on Android -- in their defense I'd bet almost anything the
demo is meant to only demonstrate the math behind this. Depending on the
platform ( _Android_ cough), low latency audio processing can be almost a dark
art itself. And hey look, they're doing this on Android.

My guess is that they decided to release the demo earlier instead of spending
days/weeks getting up to speed with low-latency audio processing in the
Android JNI.

It's an academic demo/press release. Not a software release for
production/market.

~~~
rsp1984
I'm curious, besides doing things in C/C++, are there any "magic tricks" to
doing low-latency audio processing on Android? Looking at the chart here [1]
it still seems to be a good bit behind iOS.

[1] [http://www.androidpolice.com/2015/11/13/android-audio-
latenc...](http://www.androidpolice.com/2015/11/13/android-audio-latency-in-
depth-its-getting-better-especially-with-the-nexus-5x-and-6p/)

~~~
parkaboy
In the Java layer? About the only thing you can do is ensure that you're using
the devices native sampling rate: typically 48 kHz for phones, and 44.1 kHz
for tablets. Non-native sampling will induce a rather large latency hit. The
buffering + buffer size stuff, unfortunately, is really only accessible in
native IIRC.

To be completely honest, it's been a long time since I've messed with audio-
stuff on the Java-side, so not sure if/how much things have changed for the
better.

~~~
rsp1984
Thanks. I actually meant in addition to doing the computation side of things
in C/C++, are there any undocumented tricks or pitfalls to avoid?

------
magic_man
I work with sonar and the physical positioning of the sensors is important in
trying to get useful results. Why is it these academic types don't release the
apks or software? Just publications and maybe a video.

~~~
mattlutze
You should try asking these academic types for the data and source code.

A lot of researchers are more than happy to discuss their work, but a big part
of the academic industry is your research's impact and references. A way to
get a better handle on who is looking at or following on with your work is to
implicitly ask them to have a conversation with you before getting the whole
kit 'n caboodle.

~~~
MawNicker
These sorts of comments are why I come here. Thank you. Without prior
awareness that something is common practice it can be extremely difficult to
recognize it as an option. I simply lack the motivation for this behavior. If
I were to release a teaser about my work it would be _because_ I'm not ready
to release my work.

~~~
phkahler
I read an abstract for paper once (on debluring images) and could not find the
original paper for free. I emailed the author about the situation, and got a
full color hard copy in the mail shortly after. My best hope had been a .pdf
by email, but he exceeded that by far. It's still on my shelf a decade later,
while a .pdf probably would have gotten lost in the GBs.

------
jstapels
I wonder how accurate it really is. The demo video didn't match up with the
movements at all and the on-screen drawings looked like prerecorded video that
they were trying to sync to.

It's a neat idea, but without a dedicated component or an extremely high-speed
RTOS, you're not going to come close to the level accuracy that's really
needed to do the math and still allow interaction.

I don't mean to rain on the parade, but I just don't think they really have
anything usable.

~~~
noir-york
If the tech is great then wonderful, but I still remember LeapMotion...

~~~
bossx
Have you tried the Orion SDK? It's an order of magnitude improvement in
tracking accuracy, even with the older hardware.
[https://developer.leapmotion.com/orion](https://developer.leapmotion.com/orion)

~~~
alexwebb2
I tried the "massive improvement" they released before Orion and it was still
terrible.

Only so many times a company can say they've got their issues ironed out
before I stop believing them.

~~~
LesZedCB
Orion really is quite impressive. I tried the improvement as well. Orion
actually works.

------
k_bx
We need to immediately improve on PIN-code protection upon cash-withdrawal in
ATMs. The problem has been there for a while, but man, it gets easier and
easier.

~~~
monatron
Sorry, I'm not sure how this is relevant or if this sentence even makes sense.
Can you explain?

~~~
cafeoh
I suppose he means that a small microphone/speaker setup near an ATM machine's
keypad could allow you to track the position of the finger and get the
corresponding keypresses without any intrusive modification to the ATM itself.

Of course that would only be valid if you proceed to then steal the person's
card right afterward, so all in all, not that useful.

~~~
seszett
Isn't it easier to just put up a small camera? Then you can also record the
card's number if you fail to steal it afterwards.

------
IanCal
Really interesting.

Reminds me of SOLI (which is radar rather than sonar):
[https://www.youtube.com/watch?v=0QNiZfSsPc0](https://www.youtube.com/watch?v=0QNiZfSsPc0)

Is there a way of trying this out? I know it'd only be demo line drawing
applications but it'd still be interesting to try.

------
sarreph
I _love_ it when people find ways of using existing hardware with software
innovation to make new interactions such as this!

------
wyldfire
I'm looking forward to the first theremin app.

~~~
tootie
I assume it would be impossible because the sound of the music would interfere
with the sonar?

~~~
hardwaresofton
Wouldn't the music only be on the audible spectrum? Sonar works (can work?) on
the inaudible spectrum.

------
adrianN
I wonder how much power all the processing draws. Judging from the slow
movements and the delayed update on the screens in this video, it's pretty
heavy on the processor.

~~~
jerf
The question is, is the multisecond latency because of processing and code
efficiency limitations for a academic research project, or is it because the
data is unusable without two seconds of smoothing? Given what I see in the
video I could argue either way. I do note the latent signal is still a bit
noisy, but then, touch screen input isn't necessarily clean either.

But it's also worth keeping in mind this is all off-the-shelf hardware. It
seems very likely to me that if a cell phone or smart watch was designed to do
this from the get-go that several easy hardware improvements and maybe a bit
of custom DSP work would make this work much better. (By "easy hardware
improvements", I mean things like speakers intended to emit frequencies for
sonar, microphone arrays intended to receive them, etc.) From that
perspective, even if the system we saw is fundamentally limited I'd still call
it incredibly promising considering the constraints it is operating under!

If I were a smart watch manufacturer I'd be falling over myself to get one of
my best engineers and one of my best recruiters an appointment with these
people.

------
faded242
Uhh.. not the best name choice in my opinion.

~~~
shi
Yep. When I read the title I thought it was a parody on IO-names.

------
xtf
Cats will love it. Thats why the first ultrasound remote never became standard

~~~
viewer5
What do you mean?

~~~
mcculley
I suspect he means that this would be audible to cats and have unintended side
effects.

------
crudbug
Similar to SixthSense [1] work from Media Lab.

[1] [https://www.media.mit.edu/research/highlights/sixthsense-
wea...](https://www.media.mit.edu/research/highlights/sixthsense-wearable-
gestural-interface-augment-our-world)

------
4684499
This is much like Project Soli. I started dreading devices like that. Sonars
everywhere.

~~~
gr3yh47
First thing i thought of is the horrible privacy implications when this tech
becomes ubiquitous and cloud based

------
verbatim
I can understand the "I", but what's the "O" part of this?

~~~
strictnein
The images drawn on screen?

~~~
Eduard
hm, that logic would mean a computer mouse is an input-output device, not only
an input device.

------
aub3bhat
This is not same as FingerIO since it does not uses sophisticated signal
processing but still interesting. Make sure that you remove earphones before
using it.

[https://danielrapp.github.io/doppler/](https://danielrapp.github.io/doppler/)

------
fitzwatermellow
This is cool. Thinking about smartphones as sensors opens up so many
possibilities, even if their capabilities aren't nearly as accurate as
dedicated devices. Wondering if the sonar information can be combined with
images from the camera to create a close-range depth camera?

------
szczys
Latency looks like a real issue in this demo. If it can be improved this could
be important, but think about how impatient you are if your smartphone doesn't
respond to your touch _immediately_. Users have been trained to be irritated
by laggy interfaces.

------
melling
There has been a lot of recent work with gesture based computing: Intel Real
Sense, Google's Soli, Myo, Leap Motion

[https://github.com/melling/ErgonomicNotes/blob/master/README...](https://github.com/melling/ErgonomicNotes/blob/master/README.org#gesture-
computing)

Leap Motion made huge improvements a few weeks ago with their Orion SDK:

[http://venturebeat.com/2016/03/04/leap-motions-hyper-
accurat...](http://venturebeat.com/2016/03/04/leap-motions-hyper-accurate-
hand-tracking-vr-tech-shines-in-new-orion-demo/)

We must be close to actually getting something basic for our desktops.

------
halotrope
This looks interesting. Can you try it somewhere?

~~~
lucb1e
Doesn't seem like it. Until anyone other than the researchers tried it, I
don't really believe it works outside of controlled environments.

------
debarshri
How about security? How does it protect other uses from controlling your
device?

~~~
DuckyC
How does your phone protect other people from pressing your volume buttons, or
using your touchscreen? Security is pretty irrelevant for a HID.

~~~
gr3yh47
it's obviously pretty different when you don't have to be touching the phone
to interact with it

~~~
njharman
Not when it's a few inches away. It's less distance than voice interaction.

~~~
lucb1e
A lot of people are a few inches from my pocket every day. And I can easily
imagine software listening for such sonars and sending back fake responses. I
think debarshri has a valid point.

~~~
njharman
> A lot of people are a few inches from my pocket every day.

Really? And your phone is on? With the screen visible for them to see what
they are interacting with. And as they are fumbling around in your personal
space inches from your hand or pocket, no one notices.

Get real! If true you should be more worried about pickpockets than some
random gestures.

------
djsumdog
I have a feeling this is also really depending on the hardware. The demos were
probably designed around the specific brand of watch and cellphone since
they'd need to know exact distances between the microphones/speakers.

It's a really cool concept. I wish they'd open source what they have, or at
least have plans to open source it. However if this came about via University
funding, they'll probably claim IP on it. If it was a student's own
fellowship, he/she/they might decide to create a start-up out of it.

------
JamesBaxter
I wonder how well it would work in a noisy environment?

------
TTPrograms
How many microphones do cell phones typically have? I guess I assumed one,
though background noise cancelling would certainly be improved by having more.
For this kind of positioning it seems the minimum needed would be 3 - and the
Android SDK can access those audio streams separately unprocessed? Pretty
neat.

------
labithiotis
Isn't it too early for April fools ?

~~~
ape4
Exactly my thoughts. I was watching the video thinking this is a bit far
fetched. Just like when I learned about Google's moon base.

------
sehugg
I was thinking of something along these lines for a proximity sensor / motion
detector application where you don't need very much accuracy.

------
yread
I wonder if you can just run it on any smartphone or you need to configure the
positions of microphones beforehand

------
memonkey
What happens when there are multiple devices around each other emitting the
signals? Great proof of concept.

------
dandare
I don't get it, how do they track specifically the tip of a finger? Or are
they not?

~~~
kennethkl
they probably track the closest object to the phone. just a guess. then ignore
subsequent echos for a split second.

------
k__
Ultrasound tracking is always problematic, because of all the noise.

I have the feeling, every few years someone has the idea again, to use
ultrasound for something and it starts promising, but then the accuracy and
lag doesn't go away and dogs and cats go wild.

------
supergirl
cool idea but the tracking will never be good enough to be practical.

if people really want this type of interaction then phones will start to
incorporate specialized hardware for it.

------
exotiik
For a second i tought we were April 1st

------
eltronix
It might as well be alchemy

------
pizza
sonar keylogger enabled

------
xuan
very interesting!

------
leosteve78
It's impressive!

------
basicallydan
Very cool, good job :)

------
jbverschoor
Fake video.....

------
nly
My intuition tells me this just doesn't hold water with respect to information
theory.. i.e. the number of bits of useful information about a finger you can
pull from a microphone. Putting aside human digits, has anyone even
demonstrated that you can reliably detect an eighteen-wheeler rig moving
toward a phone with this technique? And what about the range of the speaker?
Complete nonsense.

~~~
candu
Your intuition is probably wrong: off-the-shelf consumer-grade microphones can
typically gather 44.1-96k samples per second at 16-24 bits per sample. That's
a lot of potential information.

Also, consider that your ear/brain apparatus estimates object positions and
occlusions from audio signals all the time.

Also also, the eighteen-wheeler problem is vastly different from the finger
problem, as the latter is smaller, slower, and closer to the microphone, each
by 1-2 orders of magnitude.

