
Keyboard Ghosting Explained - taylorbuley
http://www.microsoft.com/appliedsciences/antighostingexplained.mspx
======
ChuckMcM
I am always amazed that keyboards still do this. We have scanning matrices
because back when I was a new engineer pins on a microprocessor were both
expensive (larger packages) and they required more gates in the CPU (more
expensive CPU) or chip doing the scanning. Today transistors are free and
surface mount BGA packages can put down a lot of pads pretty simply[1]. So
your typical 104 key keyboard could actually have 104 actual wires that the
CPU scanned. And even if you don't want to put down a 144 ball BGA (that is
only 12 balls by 12 balls, logic is fast enough that you could connect all of
the keys to a bunch of serial shift registers, and scan a complete bitstream
of all keys 1000 times a second. Easily matching the reaction times of humans.

So where is that keyboard?

Just to be crisp here, an STM32F429 can have 135 digital I/Os in the 208 quad
flat pack. It also has a megabyte of flash and 192K of RAM. It also has a USB
Phy built right in so you can just wire it up to a USB connector. Its $8 in 1K
quantities in that package. So parts : chip, PC board, 120 or so switches and
plastic case and key caps. Sure its more expensive than the $0.75 AVR chip
they currently use but we're talking high end keyboards here.

~~~
leoc
Indeed there are now lots of high-end keyboards with no (or effectively almost
no) keyboard-matrix ghosting problems. Actually, many of them still have USB-
related rollover limitations: it's hard to fathom how someone can design a
brand-new USB keyboard with no rollover limitations in the keyboard matrix,
price it at $100+, and still not think (or bother?) to implement the obvious
multiple-USB-keyboards workaround—looking at you, Razer!—but there you are.
Still, you can find ones (like the Sidewinder discussed in the OP) with
competent USB, or you can fall back to PS/2 (welcome to the future).

To my mind the interesting problem is how to get NKRO down into every
keyboard. I'm sure it's possible with sufficiently clever engineering, but
even with an almost-zero cost burden over bog-standard 2KRO this is almost
certainly one of those frustrating situations where mass quality is achievable
but only if you can get customers and manufacturers to know and care enough.

> Sure its more expensive than the $0.75 AVR chip they currently use but we're
> talking high end keyboards here.

Would extra PCB layers would be another significant source of increased cost
here?

~~~
tehwalrus
not an expert but:

I was amazed when my Das Keyboard came with a USB->PS2 adapter, and said to
_use the PS2_ to get N-key rollover (I think it's limited to 6 on USB.)

I am still slightly amazed every time I remember that the older format is
higher spec - although I assume it is something to do with bandwidth on a
"serial bus" (especially since most keyboards are forced to run at USB1.1
speed, for some reason).

Anyone who can explain these mysteries is welcome to reply :)

~~~
leoc
I'm not an expert either, but afaik it's not a speed issue, just an issue of
compatibility with USB's standard driver-spec-thingy for keyboards: its
authors decided that six keys (plus modifiers, iirc) should be enough for
anybody, right?

~~~
ChuckMcM
Page 59 of the USB HID Spec [1] is the offender, it states:

    
    
       The following table represents the keyboard input report (8 bytes). 
     
       Byte Description 
       0 Modifier keys 
       1 Reserved 
       2 Keycode 1 
       3 Keycode 2 
       4 Keycode 3 
       5 Keycode 4 
       6 Keycode 5 
       7 Keycode 6 
    

Which is where the 6 key roll over comes from. However this is really the
minimum implementation so that BIOS'es can use USB keyboards. If you read the
spec carefully there isn't anything that says you can't implement a different
reporting structure, as long as your HID driver is expecting it. My guess is
rather than ship "special" keyboard drivers they just use the standard one,
and they opt to use PS/2 connectors which work in the same way (they are
really just a serial port and can send as many keys as they want) for the
'corner' case of full n-key rollover.

[1]
[http://www.usb.org/developers/devclass_docs/HID1_11.pdf](http://www.usb.org/developers/devclass_docs/HID1_11.pdf)

~~~
tehwalrus
Thank you. So the issue is that Das can't be bothered with coding and
maintaining custom drivers for every OS under the sun. I guess that's fair.

Also, I've never actually run into a problem using it over USB on a Mac or PC,
so I guess it can't be that bad! (I do play games, but this keyboard goes on
my work machine in whichever office that is.)

~~~
leoc
You don't even need to code custom drivers though, you can just present the
keyboard as multiple keyboard devices to the host.

------
BoppreH
When developing Flash games this was a personal hell. Many keyboards are
unable to process Up+Left+Space, usually meant for shooting at northeast. In
this case one of the keys is simply ignored. Worse yet the other directions
work, so you get a hard to diagnose bug.

The users end blaming you for their character deaths and the overall
experience is frustrating. The best solution I found was to use a different
key for shooting, such as left Ctrl, 'a' or 'z'.

If you ever wondered why you can't use the large button on your keyboard to
trigger the main action in a game, blame your keyboard manufacturer.

~~~
simonh
Or a small, but significant number of other people's keyboard manufacturers.
Welcome to lowestcommondenominatorland.

------
dools
I wonder why they scan the keyboard as a set of columns and rows!! When I was
at University, I created a 49 note midi controller from a discarded electric
piano keyboard.

Basically I got the hardware and wrote software for a Motorola 68hc11 to
control it. The way it worked was that you would put a value on an 8 bit
register that would activate one "bit". This bit would be connected to 8
different keys on the keyboard. You would get an 8 bit value out the other end
that would tell you which keys were connected at the time.

So for example the value:

01000000

might correspond go keys 1, 6, 11, 16, 21, 26, 31 and 36. If you got the
value:

01011001

out the other end, you knew that keys 6, 16, 21 and 36 were pressed. The
68hc11 scanned so fast that you couldn't fool it. I remember my professor
standing there for ages mashing the keyboard trying to get it to break with
different key combinations.

If such an old microcontroller could handle 48 keys so easily, why the hell
would a modern microcontroller not be able to use the exact same method? ie.
instead of doing it in "rows and columns" just treat the keys as a linear
array and deal with which keys are pressed and which aren't.

~~~
dfox
Your scheme is essentially same as is used by keyboards and has the same
problem. When you press two keys that are connected to same output wire,
additional keys on these two inputs are ambiguous. There are some software
tricks related to the fact that nobody presses multiple keys at exactly same
time, but the root problem still exists.

~~~
dools
But there are no two keys connected to the same output wire at the same time.
If I put the value:

00100000

onto the input register, and I get:

01000000

on the output register, that can only correspond to one key at the time I read
the register.

EDIT: I think perhaps you're underestimating how quickly a microcontroller can
scan a keyboard. If I press 2 keys simultaneously that are both connected to
the same output pin on the output register at different times, the rate at
which I scan the input register is so fast that it would be physically
impossible for me not to detect that as 2 distinct keys being pressed.

~~~
dfox
The problem happens when you press three or more keys at once in right
combination. And is exactly the same regardless whether you scan output wires
in sequence (as is done in simple hardware based keyboards of the 70's and
80's) or if you read them all at once.

Going back to your original example, if one would press keys 1, 2 and 6 at the
same time your microcontroller would falsely detect 7 as being also pressed
(because keys 1 and 6 shorts two input wires together). This assuming that
your output register drives strong 1 and weak (preferably floating) 0. If it
has strong drive for both 0 and 1, then such combination of pressed keys would
cause some unpredictable effect, possibly including burning out the pins.

~~~
dools
Here's a dramatically simplified diagram to illustrate the principle:

[https://db.tt/Fvs0xBpd](https://db.tt/Fvs0xBpd)

If I put a voltage on pin 2, and key A is pressed, I will get a voltage at pin
1. If I put a voltage on pin 3 and key B is pressed, I will get a voltage at
pin 1.

It doesn't matter if both are pressed simultaneously because I am reading them
sequentially.

All you have to do is scan the keys quickly enough that someone can't "sneak
in" a key press in between scans which turns out is possible with even the
lowest powered hardware.

~~~
jfoks
Ghosting is with three keys pressed: If there is a key A that is in the same
row as a key B plus and also in the same column as a key C, then a fourth key
(the 'ghost') appears to be pressed in the column of key B and the row of key
C due to the shorting of the rows and columns, also in your scheme.

~~~
dools
There are no columns in this design. The behaviour you're describing can't
occur since this isn't a 2d matrix of keys. The keys are grouped such that
putting voltage on an input pin will result in an output voltage on many pins
depending on which keys are pressed at the time.

------
sheetjs
I'm surprised the term "rollover" wasn't referenced here. Is it nonstandard?

[https://en.wikipedia.org/wiki/Rollover_(key)](https://en.wikipedia.org/wiki/Rollover_\(key\))

~~~
esolyt
The explanation is for the layman. They probably didn't want to use a
technical term.

~~~
qnaal
Wouldn't want to confuse the user with references to non-microsoft-sanctioned
documentation.

------
orochi235
It looks like they ripped all those diagrams off from a three-year old Tom's
Hardware article:

[http://www.microsoft.com/appliedsciences/antighostingexplain...](http://www.microsoft.com/appliedsciences/antighostingexplained.mspx)
[http://www.tomshardware.com/reviews/mechanical-switch-
keyboa...](http://www.tomshardware.com/reviews/mechanical-switch-
keyboard,2955-6.html)

~~~
mavroprovato
> (Graphics: Microsoft)

Last words on the article

------
qnaal
Fun Fact: The standard USB keyboard protocol can't handle more than... 6? non-
modifier keys being pressed at the same time.

IIRC, to get past this, all you need to do is emulate a hub with fifteen or
twenty keyboards attached. Then you can press every key at once!

~~~
Zr40
Not true. A USB keyboard must declare the number of simultaneously pressed
keys it is able to report. It just happens that the example descriptor
included in the USB spec has this value set to 6.

~~~
qnaal
'USB spec'\- so it's up to the keyboard driver your operating system provides
to actually set the limit?

And other than that, there's no artifact of the protocol that would hinder the
development of such support?

~~~
kevingadd
Descriptors are provided by devices, not drivers.

~~~
qnaal
Sorry, I'm not very familiar with how the usb protocol works- but I meant,
that not all OS-provided keyboard drivers support devices with up to
100-whatever simultaneous keypresses, right?

I've seen keyboards with what looked like hardware 'nkro' toggles, and
overheard discussions to the effect that many implementations of nkro over usb
are sometimes 'unreliable', which sounds like inexcusable behavior on either
part of the keyboard's controller, the software driver, or the protocol
itself.

------
reidrac
I wrote a shoot 'em up for PyWeek and I got a report from one of the judges of
the challenge saying the controls didn't work very well. I didn't know what
was going on at first, because looking at the code it was "perfect", but turns
out using the arrow keys for direction and space for shooting was a bad idea
because a diagonal movement (up and left) when shooting won't work in some
keyboards.

Since then I use z for shooting and problem solved :) Avoid three keys
combinations in the same area of the keyboard and it will be fine.

~~~
couchand
Arrows and space are the same area of the keyboard?

~~~
reidrac
In my keyboard (an ultra compact keyboard; and apparently in that judge's
keyboard), they are.

I didn't know about "keyboard ghosting", but with z (or x) and the arrows it
doesn't happen, so I assumed it was because the space key is close to the
arrows and because some internals of the keyboard it can't register three keys
down at the same time in the same area (but the arrow keys and z works just
fine).

------
moron4hire
No, no, no, no. This is all wrong. This is not _why_ ghosting happens. This is
_how_ ghosting happens. The _why_ is "because stakeholders are cheap
bastards". Manufacturers are cheap. Consumers are cheap. Consumers don't want
to pay for quality and Manufacturers won't hire quality engineers.

But there is an even bigger issue: engineers are cheap, too. Once you get a
design that "works", for varying values of "work", it's like pulling teeth to
get an electrical engineer to change anything.

Some of it has good reason. It's really difficult to prototype new circuits.
We can prototype new code as fast as we can write it. Have you ever tried
using Eagle? Or LabView? Constantly fighting with a goofy UI to build circuit
designs, and then you really only have a simulation at that point. To get the
real thing (analogous to compiling?), you have to fab a prototype, which fewer
and fewer places are doing in-house anymore, which will take a day, or maybe a
week. I compile my software 15 times an hour!

And it doesn't necessarily have to be this way. LabView sucks because EEs are
shitty programmers. Every hardware shop I've walked into, I was the guy to
suggest they start using such outlandishly cutting edge technology as _source
control_. It's been even harder to convince some of the older engineers that
no, nobody uses CVS anymore, and the little you remember of it from 15 years
ago isn't going to help you anyway.

"Oh yeah, let's write our own atoi(). That sounds like a great idea. What's
that? You didn't realize that I'd be sending you a direct copy of what was on
the LCD screen? Why wouldn't you want to know that the LCD shows a space in
the 10s column between the negative sign and the value when the value is more
than -10?"

So yeah. Everyone is a bunch of cheap bastards. I'm not saying it's any better
on the software side. Software engineers couldn't be counted on to put a
florescent light tube in the right way if their life depended on it. But at
least most software engineers try to abdicate hardware responsibility to other
people. EEs think "it's just code, anyone can do it."

------
chinpokomon
In the high school computer lab, after I showed my friends how to disconnect
from the Novell network so our instructor couldn't watch what we were doing,
we used to play a very good Street Fighter 2 clone. I quickly learned all
about keyboard ghosting and used it to my advantage to block my opponent from
blocking my attacks.

When I tried the same tactic on my computer at home, I discovered it didn't
have the same problem. The chords that were unrecognizable at school worked
just fine on my keyboard at home. I surmised that the difference was in how
the keyboards detected the key presses.

Over the next few days/weeks I wrote a program that could actually inspect the
key queue in memory and mapped out the chords which would fail. I could
actually map out what appeared to be groups of keys that caused this problem.
I did the same thing for the keyboard I had at my house and while there were
some combinations that would lock up the keys, I found that it could handle
many more simultaneous keys.

Maybe it wasn't so surprising that the next year I started my EE degree?

~~~
T-hawk
Just wondering, was that SF2 clone Super Fighter, with Pho Huang among the
characters?

I did the same thing about figuring out which keys would block my opponent
from moving, both in Super Fighter and in the PC ports of the Mortal Kombat
games. Didn't go as far as automatically calculating it per keyboard, though.
We actually tried to play honestly without using that for advantage, a
gentleman's agreement to only press keys for just as long as you needed.

~~~
chinpokomon
Eventually we decided it was cheating too and called a truce to key smashing.
That wouldn't prevent us to use it on occasion as a last ditch effort. ;-)

It's been a few years, so I don't remember too much about the game. I only
remember it was such a good port that I could learn moves on the clone and
that made me a much better player in the arcades.

I also played my share of Mortal Kombat and Doom on those desktops. Mortal
Kombat was great because it was actually a port by Probe/Acclaim and as far as
I could tell it played identically. That became a valuable skill in college.

Probably my favorite misuse of equipment was from my senior year of high
school. I got myself a 486 DX2-50 monochrome laptop and my friend got a
Toshiba color laptop. We hooked up Null MODEM cables and strung them out over
a couple seat rows and columns. We played Doom death matches in the middle of
class lectures.

Having a laptop was unusual. Using one in a classroom, even more so. I don't
know if our government teacher thought we were taking notes -- we weren't --
but I don't think he suspected we were playing multiplayer games. Besides,
collectively my friends and I had the best grades in the class, so as long as
we weren't being disruptive we could pretty much do as we pleased. We were
still the only ones who would answer any questions or engage in political
debate.

------
comex
> The internal electronics on the SideWinder X4 use a variant of resistive
> multitouch technology. Each key has a screen printed resistor in series with
> its switch. This allows the internal electronics to read the state of each
> key switch independently for very large multiple-key combinations.

This is pretty vague. How exactly does this disambiguate which keys are being
pressed?

~~~
MikeAmelung
The resistors change the voltage of the resulting signal. So the Microsoft
keyboard probably checks whether the circuit is closed and the voltage to
determine which keys are actually pressed.

~~~
userbinator
The idea is that by using resistors (which are much cheaper than diodes), any
"indirect paths" that go through more than 1 switch between the rows and
columns will decrease the voltage sensed on those, while the voltage on the
ones that go through only 1 switch is higher.

They say they've patented this technology but in fact it's been used for
(smaller) keypads with microcontrollers for a long time before that; I
remember reading some application notes that showed this technique.

------
comex
(Unrelated to my other post here...)

Aside from gaming, one application that's hurt by rollover is the open source
stenotype program Plover:

[http://plover.stenoknight.com](http://plover.stenoknight.com)

I would like to try it, but I tote around a MacBook Pro and I don't care
enough about Plover to try to carry a keyboard around and awkwardly put it on
top of the built-in keyboard or something. Still, this is one of the few cases
where I feel constrained by the unreplaceable hardware.

~~~
Anderkent
You can make a keyboard pretty small nowadays:
[http://technomancy.us/173](http://technomancy.us/173)

------
gggggggg
Microsoft make some of the best keyboards there are. Just did some tests on my
Sidewinder keyboard, no issue at all there.

~~~
kabdib
Microsoft _used_ to make some really good keyboards. But about 8 years ago
they started making junk. The last MS keyboard I could actually type on was
the Wireless Multimedia one (now discontinued).

Keyboards in the last few years have had:

\- Mushy key feel. I can't tell when I've actually done a keystroke.

\- Keys that are hard to strike (need to press on them just right, or they
hang up or don't register). This is particularly bad on the wider keys.

\- Keytops too close together; I mash multiple keys.

\- Embarrassingly bad features: Function-key lock is the classic one here.
Let's not talk about the 'diamond' arrow keys that have occasionally appeared.

I'd buy half a dozen "Natural" keyboards with decent Cherry hardware running
the show, ESC in the right spot, and no wacky features. I don't even need a
numeric keypad. I'd pay more money for some remapping (though just about any
20 cent microcontroller will have enough EEPROM to do this).

Whaddya say, Microsoft? Care to make a good keyboard again?

~~~
joshka
If you like laptop style chicklet keys (you may not), Microsoft's new sculpt
ergonomic keyboard is pretty nice. I replaced my natural 4000 with it recently
and found it much nicer on the mushiness scale. It's pleasantly clicky, but
not cherry clicky. It even fits in a bag much better due to the detached
numeric keypad. The mouse seems a lot better than the one that came with the
4000, enough for me to overlook the lack of extra buttons compared to my
logitech G500. My one gripe is the positioning of the period key seems just a
little off, something that has taken a few months to get used to hitting
without miskeying.

Edit: incidentally, this keyboard, despite being fairly new, only gets 5 or 6
keys on the rollover test at the top of the article

------
ilitirit
I remember having to work around this when I created an arcade stick with the
guts of an old PS1 keyboard. I used an Excel Spreadsheet to map the keys and
to figure out which combinations of inputs would result in the least amount of
blocking/ghosting in practice.

So basically, if you have a two-stick console with 6 buttons for each player,
you had to wire it in a way that was impossible for Player 1 to ghost/block
Player 2's keys (and vice versa). So when you found a combination that
ghosted/blocked, you'd just map them to opposite directions on the same side
since it's not possible to press UP+DOWN or LEFT+RIGHT at the same time on a
joystick.

Here are some details:
[http://arcadecontrols.com/arcade_input.shtml#KeyboardGhostin...](http://arcadecontrols.com/arcade_input.shtml#KeyboardGhosting)

------
Avitas
This explanation is poor. The keyboard controller and interface are crucial.
NKRO is possible, but can it can be flaky over USB. There are many variables
involved.

Want to learn more? Hit the geekhack forums (search for NKRO) or geekhack
channel on freenode.

~~~
jonchang
I wouldn't say the explanation is "poor." The article details one of the
reasons behind the lack of n-key rollover. Note how the text explicitly limits
the scope of the article.

> Typically, ghosting is the result of one or more of following three
> limitations: the hardware can't read the given key combination, the software
> on the computer doesn't support multiple simultaneous keys, or the
> communication protocol between the hardware and software limits the maximum
> number of simultaneous keys reported. The next section discusses in more
> detail the hardware design of typical keyboards that limits the number of
> keys that can be read at the same time.

The article also links to another page that describes the issue with USB:
[http://www.microsoft.com/appliedsciences/content/projects/Si...](http://www.microsoft.com/appliedsciences/content/projects/SideWinderX4.aspx)

------
malkia
One of my favourite games - Star Control II had a DOS program to test what
keys were not blocking - so two players were able to choose 2x6 = 12 keys in
total that would never block.

And then for a while was the craze for finding this good old keyboard (IBM,
HP, whatever) that did not block to play SC2 and other games (back in the days
multiplayer was divinding the keyboard - to two - and much better fun if you
ask me)

------
beagle3
The ZX Spectrum in 1982 (and I'm quite sure its ancestors ZX-80 and ZX-81 from
their eponymous years) had a keyboard just as cheap or cheaper, with a similar
matrix arrangement), but had no ghosting at all:

Instead of having an "all hot" electrical configuration, it would cycle
through the rows with a "one hot" configuration, and read the columns. If I
recall correctly, it would be something like 20 cycles of a 4Mhz Z-80 to read
one 5-column row (you'd need 8 of them to read the entire 40-key keyboard),
and it was done every 50hz/60hz interrupt by the main CPU - modern keyboards
have an ASIC on par or 100 times faster than a Z-80 just for the keyboard.

------
Bhel
In which cases are such keyboards necessary?

I use a $3 one (which I bought to replace the one I used for over a decade)
and I've never had such issues. I use this keyboard for coding and gaming, not
just for common tasks which require low performance.

~~~
garrettgrimsley
I use a Model M and sometimes encounter ghosting. Some types of gaming are
worse than others. For example, while playing RTS or MOBA I do not encounter
ghosting while playing FPS I encounter it on a regular basis.

This is because in RTS and MOBA while you might be rapidly hitting your keys
you should be hitting them in succession, not simultaneously! In the FPS on
the other hand you might be trying to walk forward and right while crouching
and reloading, which requires four keys to do. This would cause problems for
my keyboard.

Programming is more like the RTS than the FPS in this respect. You may be
hitting a lot of keys, but the combinations are more likely to involve
modifiers such as CTRL, ALT, or shift rather than a combination of letters.

~~~
Bhel
Makes sense. I barely play FPSs anymore, but I can imagine the scenarios that
you describe, and after testing the keyboard on that site, it's clear that my
actions would fail on such scenarios.

So, I take it that anti-ghosting measures are primarily meant for gaming
keyboards.

------
User8712
I like how they mention keyboard manufacturers using the term _up to_ as a way
to deceive customers, and then in the next paragraph mention the same phrase
when referring to the Microsoft keyboard.

 _' Another marketing strategy is to state that the keyboard allows "up to"
some large number of key presses.'_

 _' Microsoft's SideWinder X4 features multitouch technology that allows it to
detect, and report ANY combination of QWERTY keys, up to 17 keys.'_

~~~
leoc
It seems to be largely-non-misleading in this case though, since the
Sidewinder can apparently do "[a]ny combination of up to 17 Alphanumeric and
Navigation keys"
[http://www.microsoft.com/appliedsciences/content/projects/Si...](http://www.microsoft.com/appliedsciences/content/projects/SideWinderX4.aspx)
. The only gotcha is that numpad keys share in that same any-17-keys pool.

------
khc
> The next div class="asContentParagraph" discusses

Placeholder fail

------
sspiff
They fail to address the way they solved the protocol problem (only being able
to encode 6 keys in one event).

I'm assuming they went the custom-protocol-extension route, meaning sending
more than 6 keys restricted to Windows-only drivers?

Or did they do something else?

~~~
kuschku
I'm using the Sidewinder X4 on linux, and the anti-ghosting-feature works even
without any custom driver, so I assume they used the "register one keyboard as
multiple keyboards"-approach.

~~~
sp332
What does `lsusb` give you?

------
eikenberry
[http://www.renoise.com/blog/crippled-chords-without-full-
n-k...](http://www.renoise.com/blog/crippled-chords-without-full-n-key-
rollover/)

------
mukeshsharma
its helpful to understand about keyboard ghosting..

