
A Motion-Sensing Keyboard - nivla
http://research.microsoft.com/en-us/projects/typehoverswipe/
======
algorias
This is a project of Microsoft Research. It's purely academic at this point
(done in collaboration with ETH Zürich), and at this stage it has nothing to
do with the commercial side of the company, except being funded by them.

Apple flamewars and speculation about when this will hit the market are out of
place here. Right now it's a bunch of academics showing off a cool prototype,
which they couldn't be doing if this project had been developed with immediate
commercialization in mind.

TLDR: Yay science!

------
Buge
I guess "mechanical" doesn't mean the same thing here as it usually does when
referring to keyboards.

~~~
jtokoph
They should be using a term like "physical" keyboard.

------
dchichkov
Brilliant! Not sure I like that particular implementation (with IR), but the
idea of having the whole desktop keyboard work like a giant track pad is an
enticing one.

Can a track pad surface be integrated into every keyboard key?

~~~
interpol_p
The problem with the implementation demonstrated in the video is that the
gesture is recognised, and _then_ the action happens.

There is no one-to-one mapping between the gesture in progress and action on
screen, so I suspect it will not feel anywhere near as nice as a trackpad or
touchscreen.

~~~
cbhl
I think that could be improved with a higher resolution sensor, but the
question is whether you could build such a thing while retaining space for
physical keys.

IIRC, current gesture recognition still takes a couple hundred milliseconds on
modern consumer touch screens (iPhones, etc.), especially for the "click"
action.

~~~
mcrider
Couldn't you make the keys transparent and put the IR sensors below them?

~~~
MertsA
A lot of plastic is already IR transparent but totally opaque in the visible
spectrum. You can hack a cheap camera and take out the IR filter to make a
neat "xray camera".

~~~
kleim
There are even so-called "Black-Ops Plastics" specifically designed to be
fully IR transparent: [http://qwonn.com/black-ops-
plastics.html](http://qwonn.com/black-ops-plastics.html)

------
missing_cipher
Hah, almost like SwiftKey's April Fools video:
[https://www.youtube.com/watch?v=-QfhTWJIDCM](https://www.youtube.com/watch?v=-QfhTWJIDCM)

------
viraptor
Finally! A potential for more precise cat-on-the-keyboard state detection, so
the keys can be disabled :)

------
jwr
I wonder which IR proximity sensor they used. I'm currently using the OSRAM
SFH7773 in my projects — it's the cheapest one I could find, but it is still
quite expensive ($1.20 at qty 25 from RS/Allied). They are using 64 of them.

I also wonder how they deal with interference. Each sensor emits IR pulses and
measures the light that bounced back. On the SFH7773 at least, there is no way
to synchronize the pulses to an external source. Which means you will get
light bleeding inbetween sensors.

Sadly, the publication is not (yet) available.

On a side note: I noticed that ETH Zurich seems to be doing a lot of
groundbreaking work recently in electronics and robotics related fields.

~~~
andrewfelix
I would have thought a single camera would achieve better fidelity than 64
proximity sensors.

~~~
pbhjpbhj
Where would you put the single camera to have a view of the fingers of both
hands whilst on the keys? The 64 sensors are acting as a single camera I
imagine [yeah sorry didn't RTFA].

------
pbnjay
I find it amusing that they used an Apple keyboard... Then again the chiclet
style keys seem to be a big factor in the build.

~~~
rjd
... umm.. because its white? and every thats white is Apple?

If you actually look at it you'll see its a not printed anything like an Apple
keyboard, the letters are offset for one, they are centered on Apple
keyboards, and has Windows keys.

~~~
necubi
It's definitely an Apple keyboard (look at
[https://www.youtube.com/watch?v=Y3dUeGNIX4M&feature=youtu.be...](https://www.youtube.com/watch?v=Y3dUeGNIX4M&feature=youtu.be&t=0m54s)
— the command key is clearly visible), albeit an old one.

~~~
arrrg
It’s not an Apple keyboard, it’s a keyboard made by some manufacturer other
than Apple (it seems to be some cheap Apple keyboard knock off), but with OS X
modifier keys.

However, that’s barely relevant. They built a prototype, probably with some
cheap keyboard they either had around or just quickly ordered online. The
particular keyboard used for this particular prototype is literally of no
relevance at all. It doesn’t even have anything to do with what they want to
show.

I really don’t get this whole fucking discussion. It’s some of the most stupid
stereotype pattern matching that I have every seen on HN.

------
sologoub
Not sure about this specific execution, but the idea of having a unified input
surface is very appealing. 3D gestures have also been floating around, but
seems like this is the closest I have personally seen to something that is
close the current reality of computer use.

That said, the rotation of the image gesture seems overly confusing. Mouse or
trackpad seem like they would be an easier way to go...

~~~
pestaa
I would have reached the mouse and rotated the stupid picture in fraction of
the time, not even having the need to remember application-specific gestures.

------
rch
I rigged one of these up with a theremin and a laptop back in 2007.

~~~
tomasien
Are you serious? That's amazing if you are.

~~~
rch
yup. and thanks!

------
__david__
Microsoft R&D does some cool stuff. Kudos. I love where this is going—a laptop
with this style tech and no trackpad at all would be very interesting, very
minimal. At some level it seems like the kind of thing Apple would do—they've
already removed the separate button from the trackpad, so removing the
trackpad completely is almost a logical next step.

Looks like it might need to be a little higher res though, before it becomes
productizable.

------
aidos
This looks really interesting. I was considering making something similar when
my Myo [0] arrived (and probably still will).

The thing that I thought might make it difficult is the exaggerated movements
required. All the demos I've seen seem to make the device appear a little
insensitive. This demo (of the keyboard) has the same issue. It's probably
that you need the exaggerated signals to be sure of the user's intent - or
maybe that's just for the sake of the demo?

[0] [https://www.thalmic.com/en/myo/](https://www.thalmic.com/en/myo/)

------
guelo
The idea of turning the surface of the keyboard into a trackpad is very cool,
you could almost get rid of the mouse. Except for, how do you click the mouse
pointer? Accidental clicks of keyboard keys is probably what makes it
impractical.

~~~
ndesaulniers
Using the tracpad on an Macbook Pro, I don't usually find myself clicking, but
mostly using gestures such as tapping. Even two finger tapping is recognized.
I don't see why one of the crazy gestures this cool thing can recognize
couldn't represent what we currently think of left click and another for
right.

~~~
xerophtye
well yes, most of us tap instead of clicking. question is, how would it
differentiate a "tap" from a "keypress" if trackpad and keyboard are in the
same place?

~~~
kamkazemoose
you could do some other gesture, like raising two fingers? We are trained to
tap now, but people can figure out different gestures like pinch to zoom so I
don't think the clicking has to necessarily be a tapping gesture

------
nezza-_-
Does anyone remember the FingerWorks keyboards that one could put into
PowerBooks? Reminds me quite a bit of the stuff that they did with their
multitouch keyboard. (They got acquired by Apple in 2005)

------
tomkin
Gotta say, not sure why some are shitting on MS on this one. Whether you like
the keyboard or not, it shows that MS isn't all business and fuckups.
Sometimes it's a smart idea, or one leading to an innovation.

------
mcgwiz
Very cool. Did anyone else notice that the direction of many of the gestures
were reverse of the "natural" motion. I.e. to scroll left, the hand gestures
to the left, rather than to the right as if you're dragging the viewport to
the right. Also, to zoom in, the thumb and index finger are brought together,
rather than spread apart as if you're dragging to points on the viewport away
from each other.

Obviously this would all be configurable but it's strange to work on something
to reduce HCI friction while using a design that's counterintuitive.

------
TheEzEzz
I like the direction this is going. Something I'd love to see is a touch
surface track pad on the J key that would function as a mouse.

~~~
hayksaakian
Are you thinking something like this?
[https://en.wikipedia.org/wiki/Pointing_stick](https://en.wikipedia.org/wiki/Pointing_stick)

~~~
malvim
Hah, I HATE these, but actually like parent's idea to have a key be a small
track pad. Would have to know when to turn it off, of course, unless you want
to click some random thing everytime you type 'j'...

------
nsxwolf
I can already feel the keys scraping uncomfortably against my fingers.

~~~
felideon
I actually like the feeling and I do it when I pause my typing to think.

------
mhd
I can't wait to have "Ctrl-Meta-<live long and prosper> x" in a possible
future version of Emacs...

------
runawaybottle
Cool, I had the same basic idea at one point to do this with an Arduino. I
wanted to have infrared emitters below the keys, and have an infrared sensor
somewhere slightly in front of the keyboard (but at finger height). Your
fingers should reflect infrared light back to the sensor. Not sure how
accurate it would be, but I think it would be accurate enough to capture
sweeping gestures such as your hand moving up/down for scrolling. Something
like this wouldn't be precise enough to replace a mouse, but definitely useful
enough to improve workflow (scrolling, switching between workspaces, apps,
shortcuts). If the keys were clear, I'd imagine it would be even easier for
the sensor to determine where your hand is based on how much infrared light
your hand is reflecting back from different positions.

Very intuitive stuff, I hope to see this in laptops soon.

------
alexvr
Really cool idea, but I think they have the implementation all wrong. I'm
picturing something like a long Leap Motion sensor across the top (far side)
of the keyboard. Such a keyboard needs the sensitivity of a trackpad to be
much better than simple key commands, no?

------
apunic
The Surface's Touch Cover has these gestures too.

~~~
teamonkey
The Type Cover doesn't, however it does have a neat trick where it seems to be
able to sense your fingers nearby and activate the backlight. I always
wondered what method it uses to do that and how hackable it is.

------
hemaljshah
Conceptually this is a pretty cool idea! I easily see the applications around
web browsing or reading a doc, but a few practical problems stand in the way.

Navigation and clicking with the keyboard will be a huge problem. How could I
move the mouse pad and simultaneously click? The demo video uses gaming, but
most games still require use of both clicking on keys and a mouse click. If I
tried to hold down a key on the keyboard, but wanted to click at the same
time...do I have one finger move in mid air above some random part of the
keyboard?

Hopefully there's continued refinement though...I'm surprised there hasn't
been much thought about replacing extra keys with more customized multi-touch
gestures.

------
aperture
I find this functionality is very cool, but does it have to be motion sensing?
I find physically moving your hand off the keyboard and back on to be somewhat
tiring, especially since it's doing apparently the same thing as "scroll
down". But since the hand stays on the left side, how about adding a button
underneath the left palm, where by moving the palm slightly the command is
performed? This I think takes away any inconsistencies of "motion sensing"
while still keeping your efficiency at a high (not to mention keeping your
hands comfortable!).

~~~
cheeseprocedure
I wonder if there's RSI implications when hands-off-keyboard gestures are used
more than sparingly.

------
ChuckMcM
Well this is interesting. A while back I got a motion sensor to try some of
this on. My current keyboard[1] was originally designed for the Thinkpad
laptops/computers and has the joystick between the GHB cluster. The mouse
buttons are below the space bar. That can speed somethings up by the track pad
on the Macbook is much more expressive. In terms of full disclosure though I
do tend to collect obscure HCI devices (like the Microsoft Commander if you
remember that one!)

[1] Model M13 in black

------
acgourley
While it seems like it has a long way to go in terms of fidelity and response,
the concept is fantastic. Could make laptops even smaller.

~~~
gkya
I'd be so glad to own a laptop where the touch pad became unnecessary and got
removed. I'm a laptop-only user, and the padding between the keys and the
closer edge of the laptop is a source of pain for me, as it forces me to place
my lower arm at the edge of the computer, which causes my lower forearm and
carpus to hurt. A 15-inch laptop can be built with a keyboard that begins from
the closest edge of the device's bottom, and the space between the screen
joint and the end of the keyboard can be filled with... I don't know, a couple
of large speakers and dedicated media buttons? A phone charging unit? A coffee
holder. But such layout would definitely help my wrists and arms.

------
m_mueller
MS is putting a lot of emphasis hand gesture recognition since their Kinect
'surfaced'. Meanwhile they're still quite behind Apple when it comes to
touchpads. I wonder whether it will pay off in the future - I for one would
quite like a surface type tablet with this sort of keyboard such that one
doesn't have to lift his hand for gestures.

------
edoloughlin
Very cool, but I have to wonder how this would work covered in grease/dust.
Most keyboards are very difficult to clean properly.

------
FajitaNachos
I've been messing around with a Leap Motion device lately, and while I was
initially skeptical, there are some actions which are genuinely more pleasant
than using a keyboard alone.

Sure, it may not be ideal for everyday computing, but it can be great for
specific scenarios. Think a presentation, or something that has less frequent
interactions than surfing the web.

~~~
melling
The enthusiasm for Leap Motion seems to have faded. The company had layoffs
recently. Are they still actively improving the software?

~~~
FajitaNachos
The most recent commit to leapjs on GitHub was 7 days ago, and Wikipedia says
they raised $30 million in January of 2013. Take that for what it's worth, but
as far as I know they are doing just fine.

------
owenversteeg
This looks very similar to the Leap Motion. Both use infrared sensing,
although they seem to use it in different ways. I think the setup in the video
could probably be recreated using a Leap Motion integrated into the spacebar
or other keys. I'm curious to see what they'll do with it (whether it'll
actually become a product.)

------
mildweed
Reminds me of the Haptix Multitouch.
[http://www.ractiv.com](http://www.ractiv.com)

------
vanderZwan
A few years ago Rob Nero, then studying Interaction Design at Malmö, did
something similar for his master thesis:

[http://robnero.com/interactiondesigns/trkbrd/index.htm](http://robnero.com/interactiondesigns/trkbrd/index.htm)

------
Theodores
I look forward to the Vim bindings.

------
Dwolb
They could probably double their resolution with an IR sensor under the keys,
transparent elastomer, and IR-transparent plastic for keys. But they did
mention sampling at 300Hz was an achievement so maybe they're running into
some embedded issues.

------
emsy
What about an capacitative touchscreen with increased distance? That would be
much more accurate, allowing the keyboard to be used as a trackpad (but would
probably require fancier algorithms to distinct touch input from typing).

------
bobbles
This looks like an awesome alternative to have to swipe the screen with your
hands, and would make Windows 8 much more palatable on a desktop machine.
Would be great to see it actually running though

------
weewooweewoo
I want to know whether or not this is more fatiguing than using the Thinkpad
pointing stick, and whether or not people prefer the tactility of the rubber
cap over the swiping gestures on hard keys.

------
dnlserrano
It reminded me a lot of Apple's Magic Mouse:
[https://www.apple.com/magicmouse/](https://www.apple.com/magicmouse/)

I did not enjoy working with it, one of the few times I had to use a Mac to do
something. However, I think these new forms of interaction will definitely
play an important role in the way we will use (mechanical) interfaces in the
future.

A bit off-topic, but have you seen this?
[https://www.kickstarter.com/projects/mimu/mimu-glove-for-
mus...](https://www.kickstarter.com/projects/mimu/mimu-glove-for-music/) Also
a new cool way of interacting with computers, for a much more specific
purpose.

~~~
bennyg
I absolutely love using my Magic Mouse. Whenever I get on someone else's
computer I'm trying to go back and forth on web pages and desktops with the
mouse.

~~~
xerophtye
the sideway scorlling seems like the only worthwhile addition to me, other
than that, i kinda like the feel of physical buttons and scroll wheel under my
fingertips

~~~
bennyg
I used to think the same way, and when I bought an iMac about 3 years ago I
just stuck to the Magic Mouse that came with it. I will never go back as long
as I'm using a Mac now - it just became part of my workflow so seamlessly that
using a regular mouse is out of the question. I get just about all of the
gestures that the trackpad offers with the precision of the mouse. I love it.

------
melling
When is someone going to start replacing parts of the keyboard? Remove the
function keys and replace them with quick gestures? Is there a copy/paste
gesture on the way?

~~~
m_mueller
Rather than _removing_ functionality of already crammed laptop keyboards, I
rather think this will be used to get rid of the touchpad and allow smaller
laptop form factors that still have full sized keyboards, e.g. new kinds of
surface devices.

~~~
melling
The ESC key, and all the function keys, on my Mac laptop are half height keys.
Personally, I'd rather just have an ESC gesture, for example.

~~~
m_mueller
Replace one key with a gesture - ok. Replace all the 12 F keys and their
secondary functions with 24 gestures - no thanks.

~~~
melling
Your lack of imagination is disappointing. I didn't say that we need 24 unique
gestures. I could throw out a straw man solution but let me simply say that
some sort of convention is needed.

------
hammadfauz
This concept might well be the best thing since vi, itself!

------
pecanpieyw
I might be wrong but I'm afraid it's a concept only looks nice when you watch
it. How do they even prevent the false positive?

------
btbuildem
Hm, I wonder how quickly the sensors would be covered in random kbd crud..
Cool idea, but why not leverage a Leap Motion device somehow?

~~~
lvs
Actually, I wonder if the troubled Leap Motion device is a good argument
against this whole approach.

------
finnh
Strange that the "swiping & pinch to zoom" gestures (starting at 41) are
exactly opposite of how they work in an iPad.

I can understand swiping being backward - people can disagree about "move the
camera" vs. "move the paper" \- but pinch-to-zoom-in is wrong in all contexts.

ah, microsoft. you've got this cool research, but somehow you manage to make
the usability all wrong. how did these people not notice that they implemented
pinch-to-zoom backward from how it works on their phones?

~~~
userbinator
> people can disagree about "move the camera" vs. "move the paper" \- but
> pinch-to-zoom-in is wrong in all contexts.

It's not wrong, it's just another manner of perspective: "shrink the aperture"
vs. "shrink the paper".

> how did these people not notice that they implemented pinch-to-zoom backward
> from how it works on their phones?

On a touchscreen, it feels like you're actually working with the content
directly, but when it's a keyboard, touchpad, or scrollwheel on a mouse,
you're really working with the "camera".

~~~
finnh
Sure, there is a model that can explain it. Same w/ camera-vs-paper in swipe.

By "wrong" I mean "does not work that way in any other device, and so violates
the user's by-now-fairly-well-burned-in expectation". So, in terms of the
principle of least surprise, wrong.

Wilfully so, given that (to my knowledge at least) pinch-to-zoom has only
existed for a few years and has only worked in this one way.

~~~
xerophtye
I am guessing its configurable (like inverted camera in games) So, that should
end this debate right now.

Btw, two finger scrolling on trackpads is usually opposite of what works on
the phone too, and people have no trouble with that either

But ignore that, as i said, it must be configurable. No sense on debating on
each others preferences

------
jessebushkar
Very cool, it's like the Leap Motion, a Multi touch track pad, and a keyboard
had a baby. Looks perfect for kickstarter.

------
bitwize
They said "mechanical keyboard" which got my hopes up, but nope, it's a bog-
standard rubber-dome deal.

~~~
ckannan90
you do realize it's a research paper; they're not trying to sell it to you
right now.

~~~
idlewan
That doesn't mean they are allowed to forget using the right words.
"Mechanical keyboard" is a very specific term for keyboards, it means that the
switches used are of the 'CherryMX' type or similar. Scissor-switches are
totally different, and do not fit the 'mechanical keyboard' description at
all.

------
pekk
Great idea, anything that might let us kill the rodent at the office is worth
pursuing.

------
kraemate
What can this do that cant be done with a keyboard shortcut?

~~~
evv
Positional controls, like for adjusting the location or rotation of an element
on the screen. It's hard to imagine somebody doing 3D animation or advanced
audio editing using only keyboard shortcuts, but it could be feasible with
this device.

However, their video seemed to focus on gestures only. I'm not sure why they
aren't demoing it for positional controls- maybe they haven't exactly figured
it out yet.

I'm really excited about this and from now on I'm going to have troubles not
thinking about it every time my hand moves from my keyboard to my mouse.

------
novaleaf
all well and good, but Microsoft refuses to let users of the Microsoft Natural
Ergonomics Keyboard 4000 to swap the middle "Zoom" to "Scroll"... so I don't
see how Msft PM's will let this fulfill it's promise.

PS: the Keyboard drivers ship with key remapping software, they just don't let
you remap the zoom, though if a user is willing to hack the config they can do
it manually :(

------
jcfrei
It's definitely cool, but I can't yet think of a single user interaction where
it would be faster than just using shortcuts.

~~~
nivla
I could think of a few scenarios where I wouldn't prefer using the
keyboard/mouse:

* Controlling 3D models.

* When doing presentations.

* When watching a movie/video on fullscreen.

* Cooking by following a recipe on the computer and not having clean hands.

~~~
6d0debc071
> Controlling 3D models.

You give up the precision of a mouse though. I could maybe see it being used
to rotate the viewport, but even there I think you'd want the precision.

> When doing presentations.

That's what god made the clicky devices for.

> When watching a movie/video on fullscreen.

You're going to be close enough to your keyboard to do that and adjusting the
time to just the one you want with the precision of the mouse is going to be
less convenient than scrubbing through with a gesture?

> Cooking by following a recipe on the computer and not having clean hands.

Maybe, maybe. I'm not sure how stuff dropping off your fingers might effect it
but yeah I could see something like that perhaps.

#

For most things though the precision you're giving up doesn't seem likely to
be worth it in terms of the time saved from moving a hand to the mouse.

~~~
jcfrei
I agree on the last point as well.

------
prayerslayer
Shut up and take my money!

------
adenot
Can't wait until apple put this in the market.

------
jiri
This is huge!

------
kimonos
This is cool!

