
Typing with a Piano [video] - leafo
https://www.youtube.com/watch?v=MAYlMcyVZ2k
======
Adam13531
Hey, I'm the guy who made this! This all started from a random idea I had, but
then I was tickled by the thought of becoming faster than the average typist
on a piano. It all culminated with a livestream today
([https://www.twitch.tv/videos/404585365](https://www.twitch.tv/videos/404585365))
where I did typing tests for about 3 hours.

I'll try to address the comments on here, so ask any questions you have!

~~~
rijoja
You beat me to it! I've been working on similar thing, so I'm a bit jealous
that you published this before me! Then again it makes me happy that I'm not
the only one with far out ideas like this!

My approach is a tad different though. My approach works by using the
microphone. And I have the code to extract notes from a microphone via a FFT.
Allowing for typing with say a guitar.

Extracting single notes works quite well. I haven't gotten around to test
chords just yet, but I think I have the fundamentals there.

I'm a bit short on musical knowledge and instruments so so far I've only
tested it with my voice (ugh), synthesized sounds and partially filled water
bottles.

Also I have a suggestion for a graphical interface that would make the
learning curve easier and making it language/alphabet independent.

Please reply to this comment if you are interested in integrating this.

~~~
jackxbritton
Been doing some pitch detection recently myself, but I went for an average
magnitude difference function over an FFT. My understanding is that FFTs are a
bit more intensive than autocorrelation-like solutions, but they work better
with polytonal signals. Typing seems suited to single-tone instruments - did
you consider using AMDF (or a similar function)?

~~~
rijoja
To be honest all of this is on the border of my knowledge horizon. No I've
never heard of AMDF but I will check it out for sure.

In the long run as I wrote I am aiming to support chords so that'll be
necessery.

Would AMDF make the response time quicker?

For what purpose are you doing pitch detection?

------
mastazi
It reminds me of the Michela Method which is a method of steno typing that
uses a short keyboard, similar to a cut down version of a piano keyboard. It
is used in the Italian Parliament to produce the official transcripts.

Video of the stenotype machine at the Italian Senate (audio in Italian)
[https://youtu.be/e-z1ZAsQjio?t=18](https://youtu.be/e-z1ZAsQjio?t=18)

Wikipedia page about the writing method (unfortunately only available in
Italian, you may try Google Translate)
[https://it.wikipedia.org/wiki/Metodo_Michela](https://it.wikipedia.org/wiki/Metodo_Michela)

How to build a Michela Stenotype machine by modifying a MIDI keyboard:
[https://www.youtube.com/watch?v=q9Y9jtOB7G0](https://www.youtube.com/watch?v=q9Y9jtOB7G0)

OpenSteno SW that supports various Stenotype machines including the Michela
machine built using a MIDI keyboard as seen above
[http://www.openstenoproject.org/](http://www.openstenoproject.org/)

~~~
mastazi
I just wanted to add that, in order to use the Michela Method with Plover (the
software from OpenSteno), you will also need the Plover-MIDI plugin:
[https://pypi.org/project/plover-midi/](https://pypi.org/project/plover-midi/)
(both "plover" and "plover-midi" are on Pip)

------
aasasd
Early printing telegraphs used keyboards like musical ones:

[https://en.wikipedia.org/wiki/File:Telegraph_Keyboard.jpg](https://en.wikipedia.org/wiki/File:Telegraph_Keyboard.jpg)

[https://en.wikipedia.org/wiki/File:Printing_Telegraph.jpg](https://en.wikipedia.org/wiki/File:Printing_Telegraph.jpg)

See the article for some words on the topic:
[https://en.wikipedia.org/wiki/Printing_telegraph](https://en.wikipedia.org/wiki/Printing_telegraph)

> _Hughes telegraph devices, which also had piano style keyboards, were very
> popular in France, where there were likely many more piano and harpsichord
> players than telegraphers._

------
Rampoina
See [http://annafeit.de/pianotext/](http://annafeit.de/pianotext/) for another
interesting take on this idea.

------
britch
Wow pressing the key harder for upper case is very good. I would take that in
a regular keyboard.

~~~
Adam13531
What's fun is that it's the only time when typing in all caps actually makes
you feel like you're shouting!

But I'm not too sure how useful it is for efficiency even after practicing so
much with it; there are a lot of false positives and negatives since keypress
velocity doesn't matter for regular typing. With enough data, you could figure
out a specific velocity threshold for each key so that you make fewer
mistakes, and then maybe it'd be more helpful.

~~~
ordu
If you make a responsive keyboard, which would give a different tactile
sensation for CAPS and for lowercase, then it would be easier. Piano gives
this kind of distinction, historically it was due to a mechanism behind
"FortePiano", in a modern electrical piano it reproduced for a convenience.

It can be made in a mechanical way, like a tristate button -- off, slighly-on,
on.

------
rafael859
I was surprised by how fast he was able to type, and despite the fact that it
seemed like a terrible/funny idea in the beginning, the end result sounded
interesting (if only space wasn't mapped to a single note, so we wouldn't hear
it so often). I'm sure that you can't get much faster, but I'd like to see how
good one can get after some more practice.

~~~
williamdclt
I find that space being always the same note does bring some more musicality
actually! It brings structure to the sound instead of sounding like a mess of
random notes

------
andrewflnr
I wonder if you could get more musical sounds by considering frequency of
pairs of letters as well, and tweaking the layout to put them on nice
intervals like fifths, fourths and major thirds.

Edit: apparently "bigram" is the word I was looking for instead of "pair of
letters".

~~~
Adam13531
Something I didn't mention in the video is that I had to tweak the positioning
of certain letters due to common digrams in English. For example, I originally
had "v" as a right-hand thumb key due to its frequency as a standalone letter,
but some common English words have the digram "ev" ("ever", "every", "never"),
and "e" was also a thumb key, so I remapped the "v" to my pinky to make those
words much more smooth.

To do that same process for musicality would be challenging since your average
English sentence contains so many different letters. I would probably do
something like assign each letter a set of sounds, then pick one of those
sounds based on the rest of the sentence.

~~~
andrewflnr
It's definitely not something you'd want to do manually. I'd want to come up
with a representation of the layout that lets you run a genetic algorithm or
similar on it, and score them on musicality when used to type some
representative corpus of text. That's a much bigger research project, though
(read: extra insane :D).

~~~
Adam13531
RE: "much bigger research" \- fully agreed! Sometimes, I daydream and wonder
what it would be like if it were my _job_ to do something like this for, say,
two whole months. Then I could try tackling issues like making this more
musical. Then again, I know that if I had two months, I'd try to break the 80
WPM barrier.

In the end, I don't think there are many practical applications to something
like this. I only had one real problem that could potentially be solved by
this—it's when I'm working in a music program with my piano and limited
physical desk space. The keyboard would be out of reach, so I could press an
escape chord on the piano to enter Keyiano Mode, type whatever I needed, then
reenter Piano Mode. The learning curve could be decreased by making on-screen
overlays to show you what each key would do at any given time.

But yeah, that's solidly in daydream territory since I've got bigger fish to
fry for now.

~~~
rijoja
I am doing exactly this. As a matter of fact I am working on something eerily
similar. I took a year of just to muck about with my own project to see what
happens. In particular the field of user input.

Seems like we're thinking very similar on this problem as my approach would be
to have a constant onscreen HUD.

Initially my target was gamepads and other things viable for Virtual Reality.
To try to demonstrate the universality of the method however I am working on a
musical input method. Where musical keys can turn into keystrokes.

When you think about it you'll find that having a HUD onscreen gives a lot of
room to implement clever algorithms that can speed up the process. Think
huffman trees and other things that'll make the necessary keystrokes
compressed.

What I have in the pipeline as of now is an FFT based algorithm that extracts
the notes from the microphone, opening up for all instruments basically. The
FFT part was surprisingly simple when dealing with one note at least, the
tricky part might be chords and noisy environments...

What's so cool about far out ideas like this that you do for fun is that
someone somewhere might find a usecase for it. Say for changing note cheats
while playing a guitar or something similar.

------
mikecsh
Reminds me of old-school chorded keyboards[1] that allow you to type via key
combinations from a small number of keys.

[1]
[https://en.wikipedia.org/wiki/Chorded_keyboard](https://en.wikipedia.org/wiki/Chorded_keyboard)

~~~
leoc
Here's the chapter on chord keybords
[http://billbuxton.com/input06.ChordKeyboards.pdf](http://billbuxton.com/input06.ChordKeyboards.pdf)
from Bill Buxton's work-in-progress book _Human Input to Computer Systems:
Theories, Techniques and Technology_
[http://billbuxton.com/inputManuscript.html](http://billbuxton.com/inputManuscript.html)
. Steno keyboards
[https://en.wikipedia.org/wiki/Stenotype](https://en.wikipedia.org/wiki/Stenotype)
are chording keyboards. The Velotype
[http://www.velotype.com/en/](http://www.velotype.com/en/) and Veyboard
[http://www.veyboard.nl/en_main.html](http://www.veyboard.nl/en_main.html) use
a similar system that would be a lot more suitable for general-purpose text
input. Then there's the important niche of one-handed chording keyboards, like
the one Doug Engelbart designed for the NLS system, and the mobile, wireless
Twiddler3 [https://twiddler.tekgear.com/](https://twiddler.tekgear.com/) .

~~~
mikecsh
Wow, thanks for the treasure trove of links!

------
illnewsthat
Besides learning how to type with a piano, the person in the video has been
live streaming his game development progress on Twitch for the past 2+ years.

Here is the twitch channel where he broadcasts daily the game development:
[https://www.twitch.tv/adam13531](https://www.twitch.tv/adam13531)

Here is the game he is developing ("An online strategy game with a focus on
automation"): [https://play.bot.land](https://play.bot.land)

------
williamdclt
This is very good! It makes me feel the same as when I started learning Vim
actually :) the movement of the fingers and what happens on screen seems so
uncorrelated, but you get used to it

------
nixpulvis
Integrate this with
[https://github.com/qmk/qmk_firmware!!](https://github.com/qmk/qmk_firmware!!)!

~~~
rijoja
404?

~~~
murkle
[https://github.com/qmk/qmk_firmware](https://github.com/qmk/qmk_firmware)

~~~
rijoja
thx

------
KyleBrandt
I imagine adding the pedals as shift and/or modifier keys could make things
more compact.

3 pedals and 2 feet gives you 6 I think.

------
nullc
The Kurzweil keyboards let you type (e.g. to enter program names) using a
simple A-Z mapping to keys, with a little practice it was easy to be quite
fast at entering text.

I briefly tried to find a youtube video of someone using it, but no joy-- I
guess it's too boring to watch someone type. :)

------
afandian
Full circle, implement text-to-speech on a piano.

[https://www.youtube.com/watch?v=sjtZxjEH6cM](https://www.youtube.com/watch?v=sjtZxjEH6cM)

(if you've not seen that video, it's a little mind-bending)

------
bartimus
Since adjacent notes don't often sound harmonious, would it be cool if you
could remap your keyboard to use more harmonious notes? For example also
utilizing same notes from different octaves?

------
zwkrt
"uttering a word is like striking note on the keyboard of the imagination".
Except this time in reverse!

------
navane
Ah, I remember playing the original starcraft broodwar with a midi keyboard.
About 1 hour of python coding.

------
t3hprofit
can't tell if april fools or awesome thing. lol. definitely don't release this
kind of thing on April Fools Day. Awesome idea though and I'd love to see the
implementation

~~~
Adam13531
The April Fool's joke was that I'd even bother spending my time trying to
learn this, but I assure you, it's all real! I linked to this elsewhere in the
thread, but here's a Twitch VOD of practically nonstop typing tests for a few
hours:
[https://www.twitch.tv/videos/404585365](https://www.twitch.tv/videos/404585365)

(it's not really worth watching if you're satisfied with the YouTube video )

~~~
t3hprofit
Yea I saw your design doc. Impressive project. I only questioned it because
looking at your YouTube it seemed that you had a history of posting on April
1st. Which, I really appreciate learning how to hack. Now I am getting
everything free from Amazon. But don’t tell the CIA

~~~
t3hprofit
Also “traceRT”, I died

------
soheil
Next make a computer keyboard sound like a piano.

~~~
betenoire
This is a thing, of course. It can be used for playing midi on your laptop,
and such.

