Look at PPM. Your prediction model would work better with personalised data. PPM is efficient (nb. I see you are using python - look at this https://github.com/willwade/pylm - although be warned - i think my code is not quite right..)
Layout shifting for finger movement - well its great if you didnt have to look. The time for visual processing the letters adds a significant lag (its why typical word prediction isnt used that much and when it is - not over 3 predictions (I have papers on this if you are interested). But its not all bad..
Switch users who need next letter prediction this could dramatically support their rate of input. (view https://youtu.be/Bhj5vs9P5cw?si=VnytfH_vdEUWuLok&t=73 - now note how the keyboard blocks the scan up. But imagine if it just scanned each letter first by next most likely - or heck - like this repo - actually changes button position and kept the scan pattern the same. It would be a ton more efficient)
Exactly, a key should never be in the same place twice in a row- that's just wasted movement that could be better optimized. Letters the computer predicts you probably won't use should automatically disappear, and be replaced with numerous copies of others based on the things it thinks you will type.
We should extend this concept of constant automatic optimization to all aspects of everyday life- for example your workplace location should physically relocate each day to an optimal location based on where each person coming in that day lives. An algorithm should tell you where to put away the dishes in your kitchen based on a constantly changing optimization algorithm, so that your dinner plates are in a different cabinet each day. Language itself should be radically redesigned daily to keep it optimal, with changelogs pushed out to be learned and memorized each morning before communicating with anyone.
A good part is that it will do nothing to improve security and will actively harm peoples' ability to type things correctly, and the best part is when it makes its way into scientific literature and algorithms. Good luck reproducing that which was entered incorrectly!
Presumably if something is sufficiently impossible to interact with in any capacity both for the intended audience and nefarious actors, it is also "secure." For example, if your computer keyboard layout is scrambled through a one time pad, where no copies exist except the internal one, inputting non-random data of any kind would be provably impossible.
Air gapping is a legitimate security technique. However, in this case the air gap would be between the ears of whoever designed it.
I think that this is an attempt to stop the “clean key” problem. Security system keypads (especially outdoor ones) tend not to be cleaned, so as time passes, it is easy to spot the dirty keys. Dirty keys are not being pressed and are therefore not in the passcode.
So look for the clean keys and try combinations from there. In a 4 digit (0-9) keypad, knowing the clean numbers drops the possible codes from 9999 to 24 (if my early morning math holds up).
Also helps the issue of someone looking over the shoulder of a valid person. Chances are they are just seeing the position and not the character pressed. So the keyboard changes and you actually need to know the character not just the old position.
If they aren't distinct, you wouldn't have 4 clean buttons, but just 3 - in which case we also know the repeating digit repeats exactly once and we get 12x3 (36) possible combinations. With two clean buttons, it's 6 (if both repeat) + 4 (if only one repeats) = 10 and if there's 1 that's just one, and a terrible password.
I encountered a case of this in college, where there were four clean digits - a tough task to be sure! But fortunately the digits happened to be the same set of digits that comprised the room number. It took two guesses, because there was a twist - the combination was the room number, backwards.
I guess you would be able to count the number of clean keys and thus know both the number of distinct digits and the digits (but not the order nor which digit that's repeated)
1. You have lots of spy-data samples that reveal which physical key is pressed (perhaps they sound different) and the precise timing of those strikes, but you don't know what scrambled numbers were actually being shown. (And it's always the same code.)
2. The trick is that users take longer to press a number when it's displayed far away from its "normal" position, because they had to seek longer to find it.
3. This means you can infer the true numbers based on how quickly or slowly presses happen versus which physical key is struck.
For a simple example, assume a two-digit code where there are nine keys. If the fastest first press is always the top left corner, and the fastest second press is always the middle, we can guess the code is either 15 or a 75, depending on if the user is accustomed to phones or keyboard numpads.
P.S.: On reflection, I could probably have shortened all that by describing it as a "timing attack" [0] except in meat-space.
One mitigation might be to get the user to enter digits at a consistent pace, by forcing a delay between showing the random layout versus accepting a button press. There would need to be some penalty for early presses, to keep lazy users from just tapping the desired button repeatedly until it became active.
This would be an interesting one to integrate into password entry forms... although you'd need to show the randomised keyboard layout on screen.
Or have a keyboard with oled or e-ink keys, like the Optimus Maximus [0] promised to deliver. It's kinda weird that nobody else seems to have picked up on this concept since then. Probably just impractical or too expensive.
I read that its patents expired in 2016; around 2015 there was a concept for an e-ink button keyboard, but that site is now a plain gambling ad. There's also https://www.nemeio.com/ that still works, but its buttons look like sunken screens under plastic domes.
A number of countries use this when giving your pin for a credit card or similar (I've noticed it in both Greece and India).
I can't help but feel like it's less secure than the default layout - I'm quite good at hiding my PIN and typing quickly, but when the positions of the numbers are randomised, I feel like I practically end up saying my PIN out loud as I try and remember it.
Had this at an ATM recently, and it took a couple of tries at my PIN before I looked at the keypad and realized what was going on. One more wrong PIN and I could have lost my card.
One way to avoid this element of surprise, for those who do not enjoy surprises, could be to have only a single key whose action upon clicking would simply be the most probable next action.
This project brilliantly points out that we are limited and shaped by how keys and layouts are presented to us.
What I don’t understand is why not go all the way. Why not also change the concept of what’s needed to enter a character? It would make sense that keys aren’t the ideal affordance to express a key every time.
Use a volume slider for a letter, use radio buttons, on/off toggles. What if I could draw the letter on my trackpad?
I hope this opens the door for a thriving ecosystem of expansions.
I have been proposing to adapt Shorthand* to computer tactile input many times. Touchpads and touchscreens can use it. It seems faster and easier for both the computer and the person than having to recognize slow handwriting.
Secretaries used this for many decades, because it allowed them to record speech in real-time.
There is always a dismissive comment about how this is not a good idea for one reason or another, but I still think it would be a great feature to have.
(Google's gboard offers a handwriting keyboard, but it's on the touch screen and doesn't count. Also, it's not graffiti, so a bit more error prone. It does recognize joined-up handwriting though.)
There's also some keyboards on f-droid for braille text input and morse code input.
There were a couple of aftermarket systems for the Psion as well. Normally the keyboard was better, but writing on the screen turned out to be easier in a bumpy taxi.
(In iMessage, as one easy example app, there's an entry box to send a new message. If you tap into that and get a keyboard click the keyboard icon in the bottom corner and there's a "moving finger" icon you can tap to get a trackpad to draw letters on. Older and smaller Watches won't even have the keyboard and will just have the trackpad.)
I had the idea this week to use the Rocksmith USB adapter to develop a stenography system that allows me to control my computer using my guitars with chords, notes, and riffs.
There's a Rocksmith mod that allows you to control the menu using the guitar. Also, the Rocksmith USB cable is simply an ADC (with lower latency than most)
> What about analog partial key press? There are Hall effect keyboard switches that do that, I want to try some.
That sounds fun, now I really want to try out a keyboard where half press is lowercase and full press is UPPERCASE! No more pinky strain from shift presses.
Some pipe organs have keyboards that support that. I can't figure out when it started, but my guess is the 1920s. No hall effect, they had mechanical contacts. (I've never got to play one and always wondered if you could feel when the two positions were hit or if you had to guess and rely on sound - organ pipes often have a long delays between when you press the key and when the speak)
A simple solution that would work on any keyboard (but will be much easier to do on a keyboard that supports custom layout and layers, e.g Ergodox) would be to send small letter of key is pressed under 0.5s (to be tested what value feels best) and capitalized if over this value. After a while it might feel natural to press quickly for a small letter and slightly longer for capitalized
This is true. We’ve been so formatted by big keyboard that I never considered just the tip of my fingers. I hope you’ll make good use of our opposable thumbs in this new keyboard paradigm
I agree. I blew out my left arm for a year from too much typing, so I switched to kinesis style keyboards so I could put modifier keys under my thumbs. I dunno if your suggestion was meant to be serious, but it's still great.
It wasn’t, but the reason I like to explore absurd looking ideas is because of the serendipity that lateral thinking opens up. Thanks for exploring that!
What would be a cool use case for thumb controls here? Is this something you could do with those gamer keypads?
I was pressing control and alt and shift with my pinky fingers, and that's how I blew out my arm.
I switched to a Kinesis Advantage keyboard and now I have control, alt, hyper, super, backspace, and shift under my left thumb. My right thumb has control, alt, hyper, super, enter, and space.
I don't know anything about gamer keypads, is there one you suggest I investigate?
I'd put some random keys on the bottom of my seat or wired into the TV remote or the doorbell button of my neighbours. The ordered shape and regular size, position of keys is too much of a burden to mee, constricts my wondering creative mind too much.
Took me 1 year to rewire my brain to switch from Qwerty to Dvorak (I struggled with from RSI for nearly 6 years) and nearly 2 years to gain full speed. It is hard, very hard for the brain. It is mentally tiring to rewire your brain like this and you will do your job at a much worse efficiency (forget pair programming for a while). You need new keyboard stickers also. Great job at making this idea but it is not practical.
I switched to Dvorak 20 years ago and I love it. It only took me about 2 weeks to learn and to pass my qwerty typing speed. I could actually touch type on either keyboard for the first year or two after I switched. I recommend not using stickers or dvorak keyboards at all, it helps you learn faster and your family won't get pissy because they can't use your computer anymore.
Also, it is not difficult to use other peoples computers. You can switch keyboards pretty easy these days on windows and mac. In fact, my very first computer (Apple IIc) had a mechanical switch to flip between qwerty and dvorak.
And there's also the problem of using computers that aren't yours. I haven't tried (and I will never), but I imagine switching between layouts is at least very inconvenient.
Dvorak is supported by almost every computing device under the sun today. Old Apple models (I want to say the IIc?) even had a hardware button to switch layouts. Switching between Dvorak and Qwerty is usually an easy to find User setting. (If doing it on someone else's user account, you just have to remember to be kind and switch it back when done.)
As a Colemak touch typer I envy that some days. Colemak is available easily everywhere but Windows. macOS, iOS, Android, and Linux all have it similarly out of the box with Dvorak, but Windows it still requires an install and that install still requires Admin approval because keyboard layouts are needed by kernel-level drivers. But I knew that when I switched, took a few years to realize there's no shame in hunt-and-pecking on QWERTY when using someone else's machine.
(I was also privileged when I switched to know that I didn't need to use shared machines. I was far enough along in grad school where I was allowed/encouraged to do everything on a personal laptop, and the few remaining "lab classes" with shared machines all allowed Remote Desktop. That took advantage of things like if you remote desktop back to one of your own machines, your remote machine still responds to your chosen layout once you've logged in. Just have to hunt and peck long enough to type your machine's address, username, and password.)
I switched when I was a teenager and have Thoughts (TM) about typing Dvorak, though I'm locked in at this point. The most painful experiences typing with it are:
1. Using a shared machine for something important. I've had to hand write and then hunt-and-peck essays on shared machines when I was still in college.
2. Typing something on a friend's computer when I'm inebriated. Not a huge problem nowadays with phones and being older, but in college trying to change the playlist when I was inebriated on a friend's QWERTY layout was... tough to put it mildly.
3. Piloting the Media PC with my partner. She types QWERTY and I don't. I keep Dvorak around as a second layout and switch if I need to do a lot of typing on the Media PC.
There's lots of both pros and cons of using Dvorak in my experience but at this point I'm so locked in that I'm not going to switch. But in my adult life, I never have to use a shared machine either so I'm always a few feet away from a Dvorak keyboard.
It's also interesting how #2 is shifting in recent years. These days it is more likely to be someone handing you their phone to change a Spotify playlist and while I touch type Colemak, I tend to swipe type QWERTY on phone keyboards, so it's not as tough now than when it was someone's laptop/desktop running the playlist.
Also, Apple Music's SharePlay is such an interesting modern approach too, where multiple people can control the same playlist from multiple devices.
Yeah, this is why I picked Dvorak: it seems there are better layouts but Dvorak picks up the greater part of available improvement, and it's more generally available.
I switched to Dvorak ±8 years ago, and I can still type qwerty almost as well as I could back then. I could imagine it being a problem if you're using other people's machines on a daily basis though.
thats wonderful. id like that to not change the order of the letters - but change the highlight order. Do a round 1 of frequency order first (just do first say 6 letters) then do a round 2 which is standard order..
i probably am not making much sense. Look at where I'm coming from in the world of Assistive Tech - https://docs.acecentre.org.uk/products/echo (go to around 5 min mark in the vide)
Thank you for the link! That looks fascinating. I have instantly added it to my "keyboard" bookmarks folder!
I will promptly forget it exists and keep typing on the random layout I learned to touch type on 20 years ago, all the while suffering from terrible finger acrobatics and wishing for a better layout every single day.
I've made my own layouts[0][1] and I'd say the many variants of Hands Down are probably the best example of a layout that's been developed in a thoughtful manner.
Very nice website with many solid principles! Though one big flaw in all the designs I've seen is that they rely on bad data - text corpus that is the output of human editing efforts, not the input.
So all those "0.00001%" stats aren't as precise as they seem, and you still see the frequently used backspace/enter in the awful pinky position despite the official philosophy of "Hands Down is easy on the pinkies"
Have you seen any design that is based on actual human input?
As far as I'm aware hands down layouts were primarily designed to be used on a split ergonomic keyboard with thumb cluster, the space and enter lives on the thumb cluster.
Even those better thumb variants don't seem to statistically take that into account (judging from the description/heatmaps/data sources), but then it also presents the "slab" variant...
For those who type on German keyboards or in German, there are a number of layouts that have evolved around the Neo 2 layout [1]. The main driving force is ergonomics, but since not everyone works the same way, many variants have emerged (and still do) [2]. Interesting is the craftsmanship, including customized programs to optimize the layouts [3].
I always changed to ANSI layout, independent of the layout of the actual keys. Now I carry a HHKB with me, and connect through BT to whatever computer I want to work on. No matter what human language I write in (I do some) I prefer using accidents like tildes and umlauts with modifier keys.
Yeah when I was a 20 or so and just started working I taught myself Dvorak for a while, but realized how bad it was for coding in C. So I made my own C language one as well.
Of course you still have to type text and at the time you couldn't easily switch between different layouts, so after a time I got bored of it and gave up.
Keyboards aside, I find it really frustrating when something I use frequently - such as the toolbars in excel - changes without warning and breaks my muscle memory.
I bought a keyboard last year that has a small embedded CPU that I can reprogram with QMK [1]. I designed an objective/score function for key placement, based on frequency of letters (and groups of 2 and 3 letters) and finger movement (in the case of strings of letters). Each finger was weighted by how much I didn't want to move it haha. I used this function to optimize my "perfect" keyboard and reprogramed the keyboard to match it. I'm still sometimes trying to learn it to this day, I don't use it often enough. My biggest problem with it is that it is an ortholinear keyboard... Not confortable enough. Anyways.
I had plans in the future to make a small keylogger on my PC to follow my ACTUAL usage of keys, and run the optimization again automatically. Then the software could message me what would be changed and upload the layout on the keyboard every X month (not too often, y'know).
I'm waiting to get another keyboard for this, probably a split keyboard.
This is not a bad idea. It's not dissimilar from an autocomplete engine. If reset per word and using a static probability table it will work its way into muscle memory. In practice fingers will only need to leave the home row for some initial letters.
I too am looking. I started drawing on a napkin how it would work. Basically injection mold the letter and stem only in one color, then put that in a different mold and injection mold the rest of the key around that. Seem like it should work, but I haven't done enough injection molding to know how feasible it might be (letters don't seem to bad, but 5/% probably needs a 4 part mold or something and I can't imagine keeping all the parts together). Worse, there is a lot of tiny machining operations needed and so for a one off my cost per key gets to be $15 each (mostly on tiny bits needed that will thus break often).
3d resin printers might be a good option, my background leans to injection molding so I didn't think of it until now. Here you would print two pieces and then snap them together.
Both of the above have the letter molded through the key and so whatever chemicals are in my fingers won't destroy the lettering after a year of use. Though plastic selection is also important otherwise my fingernails will destroy the key anyway in a short time. Not everyone has fingers/skin chemistry like mine and so some won't understand why I care while others are all in with me.
I’ve backed this product, the Flux Keyboard, on Kickstarter - it allows for the type of dynamic changes that the author is describing in case others are interested:
Could a LLM tailor the keyboard to intent ... as in it shifts keys around to help you write what it thinks you want to write? like a sentence starts with a if then the left keyboard becomes permutations of an future if clause on key,the right becomes else statements..
> (…) In the first year, "s" will replace the soft "c". Sertainly, this will make the sivil servants jump with joy. The hard "c" will be dropped in favour of "k". This should klear up konfusion, and keyboards kan have one less letter. (…)
That joke is many decades older than the creation of the european union: http://www.i18nguy.com/twain.html (probably not written by Mark Twain, but seemingly first appeared when he was still around?)
Repeatedly pressing the home row keys almost always turned into word end sequences like erere, I assume from words like there/were/here. I'm not sure how to go further with single character frequencies.
Look at PPM. Your prediction model would work better with personalised data. PPM is efficient (nb. I see you are using python - look at this https://github.com/willwade/pylm - although be warned - i think my code is not quite right..)
Layout shifting for finger movement - well its great if you didnt have to look. The time for visual processing the letters adds a significant lag (its why typical word prediction isnt used that much and when it is - not over 3 predictions (I have papers on this if you are interested). But its not all bad..
Switch users who need next letter prediction this could dramatically support their rate of input. (view https://youtu.be/Bhj5vs9P5cw?si=VnytfH_vdEUWuLok&t=73 - now note how the keyboard blocks the scan up. But imagine if it just scanned each letter first by next most likely - or heck - like this repo - actually changes button position and kept the scan pattern the same. It would be a ton more efficient)
(and a bit of a rabbit hole.. What if keys had word predictions on them? This is basically the end result of ACE-LP: https://discovery.dundee.ac.uk/en/publications/ace-lp-augmen...)