I was a bit disappointed by the lack of a real scenario implementation. While the data suggests it might work, I wanted to see how precise it would be in a simple implementation like the author described towards the end, where a simple game of tapping trains the NN and then uses the information to try and capture a typed password.
If I got that close to making something as cool as this, I would not stop experimenting until I found out whether it worked or not.
That is, unless I tested it, found that the imprecisions made it impossible to obtain reliable data, and decided to not write about it to not discredit all the work I had done.
I really hope someone takes the idea and tries it out to assess how viable it is, then I can be paranoid about 2 more sensors spying on me.
Also this is mindblowing, logging keystrokes from a standard keyboard using wifi.
"we show for the first time that WiFi signals can also be exploited to recognize keystrokes. The intuition
is that while typing a certain key, the hands and fingers of a user move in a unique formation and direction and thus generate a unique pattern in the time-series of Channel State Information (CSI) values"
Attacks like these can be easily dwarfed by employing a nonstandard virtual keyboard which displays a randomly permutated key layout during PIN entry. Some online banking web apps did this already ten years ago as a defense against "mouseloggers". Using a good RNG is advised though.
Wouldn't the model be different for each user? and different for different typing scenarios? Like typing one handed vs two, sitting on a bus, in a car, while walking, versus just sitting on the couch.
It seems like it could be possible, but it would require a lot of training. Get it into a popular note-taking app to record keystroke + accelerometer data.
Also a simple fix for iOS as a platform would just be blocking out or filtering motion data when the secure keyboard is showing. I assume a similar thing could be done on android.
To allow for training of the ML to differences of each users the authors describe have the end-user first play a game
Unsuspecting user downloads “Evil Flappy”, an app where they have to tap on the screen mindlessly to advance some objective. During this tapping, the app uses transfer learning to tailor the model to the user and test its own predictive capacity.
TL;DR it's possible to detect where you tapped on your phone screen somewhat reliably. Theoretically, someone could develop an app to associate screen location taps to keys and could pick out a password. Authors suggest phone manufacturers should limit access to accelerometers to prevent the attack.
This is extremely contrived and would take a shitload of skilled work to get right. It's way easier to make a phishing page coupled with social engineering to get what you want.
I think step counters depend on being able to read accelerometer data. You might be able to add enough noise to the signal to make tap localization impossible while still keeping steps detectable, but that requires careful tuning.
On iOS at least, most ‘step counter’ apps don’t acually do the step counting - they just pull the data from HealthKit, as the phone is already tracking them.
Oh my mistake, it was actually I who misread the first comment.
Regardless, I don't think a screen tap logger would be necessary in many cases, since you could just open the keyboard on the phone and manually check the bounding boxes for each key.
Making this work on N phones would be more difficult, but not unfeasible. You'd probably just need to know a few things about the phone (resolution, screen size), and it wouldn't be hard to find that information.
Would be especially easy to just target the limited iPhone line.
Is this why most password entry forms in Android do not allow use of the Swype-type keyboard? I always thought that was strange. Now it makes more sense....backdooring
Wouldn't that have more to do with passwords not being proper words? Swiping is like autocomplete, f7_4sl+lsS$@q.. can't imagine swiping something like that.
Neat, though i'd call it key-estimation rather then logging, as i consider logging a very accurate, 1:1 log of what actually happened, this could have some errors in it, not making it less 'dangerous', lack of better word.
But for now i'll have to randomly rotate my device at each input... And or get one of those keyboards that shuffles the letters around the keyboard at each input.
If I got that close to making something as cool as this, I would not stop experimenting until I found out whether it worked or not.
That is, unless I tested it, found that the imprecisions made it impossible to obtain reliable data, and decided to not write about it to not discredit all the work I had done.
I really hope someone takes the idea and tries it out to assess how viable it is, then I can be paranoid about 2 more sensors spying on me.