
Keylogging on IPhone and Android Using Gyroscope Data and Machine Learning - tomasreimers
https://medium.com/@tomasreimers/axolotl-a-keylogger-for-iphone-and-android-a8b7b62cdab4
======
Unknoob
I was a bit disappointed by the lack of a real scenario implementation. While
the data suggests it might work, I wanted to see how precise it would be in a
simple implementation like the author described towards the end, where a
simple game of tapping trains the NN and then uses the information to try and
capture a typed password.

If I got that close to making something as cool as this, I would not stop
experimenting until I found out whether it worked or not.

That is, unless I tested it, found that the imprecisions made it impossible to
obtain reliable data, and decided to not write about it to not discredit all
the work I had done.

I really hope someone takes the idea and tries it out to assess how viable it
is, then I can be paranoid about 2 more sensors spying on me.

~~~
andrewguenther
The Usenix paper they referenced from 2011 did exactly this.

[https://www.usenix.org/legacy/event/hotsec11/tech/final_file...](https://www.usenix.org/legacy/event/hotsec11/tech/final_files/Cai.pdf)

They focused purely on the number keyboard to try and extract PIN numbers and
were able to achieve 70% accuracy using only a basic classifier.

~~~
Unknoob
Thank you for reminding me of the references, I had forgotten to check them
out!

I wonder how much newer smartphones with bigger screens and more precise
sensors could improve these results.

It's scary to think of how many different sensors could be used to gather even
more information to make it more precise.

------
anfractuosity
Very cool!

Also this is mindblowing, logging keystrokes from a standard keyboard using
wifi.

"we show for the first time that WiFi signals can also be exploited to
recognize keystrokes. The intuition is that while typing a certain key, the
hands and fingers of a user move in a unique formation and direction and thus
generate a unique pattern in the time-series of Channel State Information
(CSI) values"

[https://www.sigmobile.org/mobicom/2015/papers/p90-aliA.pdf](https://www.sigmobile.org/mobicom/2015/papers/p90-aliA.pdf)

Edit: I wonder as an aside how precisely you could potentially use the
accelerometer etc. for dead-reckoning.

~~~
gregorymfoster
Co-author here :) In response to dead-reckoning:
[https://medium.com/@gregorymfoster/how-any-app-could-
track-t...](https://medium.com/@gregorymfoster/how-any-app-could-track-the-
indoor-location-of-everyone-magnetic-localization-acf3707716de)

~~~
anfractuosity
Awesome thanks a lot, I'd not seen that! It looks very in-depth (which is what
I wanted :), I'll have to have a good look through it.

Edit: Wow, I'm just skimming it at the moment, I'd never heard of 'Magnetic
Particle Filtering' before, that is really amazing!

Also semi-related, I just found this
[https://en.wikipedia.org/wiki/Magnetic_anomaly_detector](https://en.wikipedia.org/wiki/Magnetic_anomaly_detector).
Apparently they degauss submarines precisely to avoid that :)

~~~
gregorymfoster
Thanks!

------
sundaysailor
Attacks like these can be easily dwarfed by employing a nonstandard virtual
keyboard which displays a randomly permutated key layout during PIN entry.
Some online banking web apps did this already ten years ago as a defense
against "mouseloggers". Using a good RNG is advised though.

~~~
sid-
Also Swype or Gboard would thwart it ?

------
jclardy
Wouldn't the model be different for each user? and different for different
typing scenarios? Like typing one handed vs two, sitting on a bus, in a car,
while walking, versus just sitting on the couch.

It seems like it could be possible, but it would require a lot of training.
Get it into a popular note-taking app to record keystroke + accelerometer
data.

Also a simple fix for iOS as a platform would just be blocking out or
filtering motion data when the secure keyboard is showing. I assume a similar
thing could be done on android.

~~~
ninju
To allow for training of the ML to differences of each users the authors
describe have the end-user first play a game

 _Unsuspecting user downloads “Evil Flappy”, an app where they have to tap on
the screen mindlessly to advance some objective. During this tapping, the app
uses transfer learning to tailor the model to the user and test its own
predictive capacity._

Very sneaky indeed...and I loved Evil Flappy :-)

------
jbob2000
TL;DR it's possible to detect where you tapped on your phone screen somewhat
reliably. Theoretically, someone could develop an app to associate screen
location taps to keys and could pick out a password. Authors suggest phone
manufacturers should limit access to accelerometers to prevent the attack.

This is extremely contrived and would take a shitload of skilled work to get
right. It's way easier to make a phishing page coupled with social engineering
to get what you want.

~~~
anfractuosity
Would you be able to access screen taps from another app in the background?

~~~
yorwba
I think step counters depend on being able to read accelerometer data. You
might be able to add enough noise to the signal to make tap localization
impossible while still keeping steps detectable, but that requires careful
tuning.

~~~
djrogers
On iOS at least, most ‘step counter’ apps don’t acually do the step counting -
they just pull the data from HealthKit, as the phone is already tracking them.

------
thinkMOAR
Neat, though i'd call it key-estimation rather then logging, as i consider
logging a very accurate, 1:1 log of what actually happened, this could have
some errors in it, not making it less 'dangerous', lack of better word.

But for now i'll have to randomly rotate my device at each input... And or get
one of those keyboards that shuffles the letters around the keyboard at each
input.

------
bllguo
Come to think of it, I can't remember the last time I typed a password in an
app, after initial setup.

------
tinus_hn
Quite surprising iOS allows that kind of sample rate for a background app.

~~~
willstrafach
I believe in iOS 8 and up (could even be earlier) Apple closed up tricks which
would have allowed this sort of thing to work.

