Neural implant lets paralyzed person type by imagining writing - https://news.ycombinator.com/item?id=27134049 - May 2021 (183 comments)
Brain-Computer Interface User Types 90 Characters per Minute with Mind - https://news.ycombinator.com/item?id=27157369 - May 2021 (40 comments)
This is pretty similar but I guess it's different work:
Human use of high-bandwidth wireless brain-computer interface - https://news.ycombinator.com/item?id=26690126 - April 2021 (210 comments)
I'd be interested in knowing how this metric changes over time as the user gains more experience with the BCI device. The article mentions that researchers recorded his neural activity while he was thinking about writing letters. Would the man eventually find that the system is more accurate or faster when he instead learns how to think "the thought that generates the letter A in my BCI device"? Fascinating stuff all around.
> electrodes implanted in his motor cortex recorded signals of his brain activity
so I'm assuming other thoughts and interactions had little to no effect.
You need to establish the initial feedback loop somehow and imagining writing is a good way to do it. But once you have it I’d suspect you could get faster doing what you’re describing.
Do we have adequate wisdom to wield the powers we are granting ourselves?
What’s the typing speed and error rates over time as the subject practices? How do these compare to the progress with the BCI?
Code and data (for replicating results offline):
Published paper (you can find its full contents online):
The real improvement, it seems would be to speed and latency, looking at the diagram, the sampling is over the course of 3 seconds, which is butt-slow. A good NN would be able to compressed patterns in the data stream and blat out more than one letter, or contextually learn letters in the word, or learn whole words, etc. But that is not, it appears, supported by the model.
I bet the next step wouldn't be to even parse words to make a sentence. It seems to me the next low hanging fruit would be to enable this machine to parse common ideas. I wonder how complex it would be to translate full sentences like "Good Morning", "I gotta take a dump", or "I'm hungry". It doesn't seem like it would be that much of a leap, since the user already has to imagine the idea of different letters. Admittedly I have no idea how different those concepts are, or how they would express themselves in the brain to be interpreted by the machine.
There is actually a path for this that's been done before, in a way. Dragon Naturally Speaking was evolved this way.
As I understand it, that evolution took decades.
In 1952 Bell Labs came up with Audrey (Automatic Digit Recognition). Voice specific, and could only recognize numbers 0-9. This is where the OP linked Brain Computer Interface (BCI) is.
In 1962 IBM revealed Shoebox at the World Fair. Shoebox could understand 16 English words. It would listen to the words and complete an instruction for example adding up numbers and providing the result.
Harpy came in 1971. Funded by Darpa and developed through a collaboration between CMU, Stanford and IBM. Harpy cold work with ordinary speech and pick out individual words, but it only had a vocabulary of around 1000 words.
In 1974, Kurzweil forms Kurzweil Computer Products (KCP) for development of pattern recognition technology.
In 1976, KCP introduces the Kurzweil Reading Machine, combining three technological firsts.
In 1982 Dr's Jim and Janet Baker launched Dragon Systems and prototyped a voice recognition system that was based around mathematical models. The Bakers were mathematicians and the system they came up with was based a hidden Markov model – using statistics to predict words, phrases and sentences.
In 1983, Kurzweil Music Systems launches a keyboard synthesizer that accurately reproduces the sounds of acoustic instruments.
In 1985, Kurzweil Applied Intelligence introduces the first speech-to-text computer program.
In 1990, Dragon Dictate was launched as the first general purpose large vocabulary speech to text dictation system. This was a groundbreaking product for Dragon, but it required users to pause between individual words.
In 1994, KurzweilVoice for Windows 1.0 is launched, bringing discrete speech command technology to the personal computer environment.
In 1995, Kurzweil Technologies is founded.
By 1997, the problem of having to pause between words had been overcome and Dragon Naturally Speaking v1 was launched, 45 years after Audrey.
In 1997, the Continuous Speech Natural Language Command and Control software is launched as Kurzweil Voice Commands; The Medical Learning Company is formed.
In 2000, Kurzweil forms FAT KAT, Inc. to develop artificial intelligence that can make decisions about buying and selling on the stock market.
Then in 2001 KTI introduced "Ramona," the virtual reality rock star.
Yes, the last two have little (maybe even nothing) to do with speech recognition, but I found them interesting, so I thought you might too.
The sources for the above are primarily:
I would imagine eventually you would end up with system where less common words or complex nuanced ideas would need spelled out and commonly used thought words or sentences could be recognized.
This says it requires an implant, but I am not sure if its true, his contributions are much older though and they might not be the same as the current Braingate research, but they were also for aiding with disabled people with controlling prosthetics, and also with signal processing and calibration to the user, they have made text input before inplants were required, so I don't see why its required, its benefit is being more convenient than setting up the probes or as a wearable. https://www.frontiersin.org/articles/10.3389/fnins.2012.0007...
edit: The researchers compared the BCI output to a prompt that T5 was supposed to restate. I was thinking that T5 was communicating without prompts in the experiment. This isn't my idea of translation accuracy, but you've got to have some baseline.
If not, there's blinking/mouth controls like Stephen Hawking.
i wonder if it's a coincidence?
The current technology which is production ready is optics using pancake lenses. For an example look at Huawei's VR glasses or HTC's Vive Flow.
What you seem to want is just a screen strapped to your head with some lenses to make it look further. Why do you want that exactly? Portability? Space saving?
In any case brain implants are the ultimate input device, which will solves a major information bottleneck for work in general.
OTOH brain implants provide for information exchange that is orders of magnitude faster, even if it is just text, as we do here.
It's Star Trek like technology (think TNG series' Geordi La Forge).
 - https://www.gboards.ca/product/ginni
Not to say that this device was probably trained on the person a lot before they could reach that accuracy. A "information extraction" device would have to be trained on it's victim first, but why would they play along with the training?
Maybe I'm wrong, but I think most people have control over what they consciously think, at least it works like that for me.
So if James Bond gets captured, all he has to do is to not imagine writing the information, he can think it all he wants.
I am looking forward to our new world.
The article alludes to this:
> the researchers say that alphabetical letters are very different from one another in shape, so the AI can decode the user's intention more rapidly as the characters are drawn, compared to other BCI systems that don't make use of dozens of different inputs in the same way
The fact that it works so well on these complex motions means it can probably work better and faster if they use an alphabet with simpler--but still distinct--motions. Probably lots of lessons to be learned from shorthand and other rapid transcription techniques.
Losing the ability to communicate scares the hell out of everyone. This is amazing progress. And it'll have plenty of applications even for able-bodied people.
“Imagine writing an A”
Then they look at what fires and record it.
Instead you’d ask “imagine typing an A” and then do the same thing.
Eventually when brought training happens to capture variation you start to get visual feedback and can train faster.
It could be amazingly effective though if this is where we’re at already. Imagine the speed and enjoyment increase for anything from typing to gaming to driving a car. You’d get completely left behind if you rejected it.
Don't get me wrong I'm sure there will be advances. But this current tech is based off reading nerve data meant to be movement data - the user needs to mentally trace each letter.
So I don't see this form of the tech at least being able to compete with qwerty let alone stenography.
Actually to that point, stenography would allow people to input data (it must be language specific and error tolerant but most BCI is as well) at ~5x the average typing speed on qwerty but that hasn't proliferated.
EDIT: On second thought I could see it matching physical movement, maybe _slightly_ outperforming it by a few % by skipping a few physical limitations. I think this should be essentially identical to any other physical motion based communication.
I just thought about tracing out a letter as to how I might write it and it took me a second or so per letter, around the speed I actually write I'd guesstimate, and I can't write anywhere near 90 characters in in minute, probably because my brain has adapted to sync with my hand speed. I'm curious whats actually meant by "tracing" for movement signals because I'm either slow at this or it means something a bit different. I can easily type 90 characters a min but in a lot of cases its rote memorized patterns for words I'm thinking of in sequence (I'm not really thinking of individual letters in words, just words as a known pattern of keystrokes), at least I think that's how I think.
Anyway, assuming they do, I'm not sure if it'd be a real advantage over physically swyping with your finger (for those that can obviously) - it seems like they'd be roughly equivalent?
Actually that'll be my second question - how is this system affected by things like tremors?
I'd suspect they originate from your brain in which case your "mental movements" should have the exact same quirks and limitations as physical movements.
The end game of course is not needing to use the movement system to interpret information.
Those that train on a BCI from an early age will "type" significantly faster and more naturally (as if at the speed of thought) than those that do not. There's a natural limit to idea-formation => symbol-formation => symbol-expression. Those that have trained on keyboards are able for this to fly from their fingers with only a slight delay; those that speak at something like "auction-speed" are mostly executing verbal macros: ie, it's just a single thought/action, highly trained so that it can manifest at high speed.
I can retype the entirety of the text above with just a few actions: cmd-A, cmd-C, ->, cmd-P. Performing a novel action, however, moves at a different speed entirely.
> (You can do this)
> (OMG, is it on already?)
> (What a nice bottom)
> (Don't think bottom you idiot)
> (She's entering the room, think something else, now, fast, bunny bunny)
> (Bunny, you got this)
I think it's related to my ADHD. Reading is very difficult because as I read, my voice just completely wanders off, CONSTANTLY. "Oh hey, 'gargantuan' that's a great word. Reminds me of a video game monster... Oh did you just read 2 whole pages without absorbing a single bit of it?"
To add a bit more commentary: this is the one time I consider my ADHD to be a terrible disability. I cannot read. I just can't. Grad school was HELL when I had to read. But my ADHD and associated strong independent inner monologue is immensely powerful when I'm trying to solve problems myself, such as doing software design.
And yeah, I can have conversations with the voice. Yeah, it can help to focus moving my thinking forwards on some problem-solving thing. But, alas, often when I'm doing that it just starts saying obviously-silly things, as if it's a Markov model. Sentences that might sound structurally reasonable but that are obviously not what I meant to say/think, and obviously not true.
But it almost never interferes with reading. Reading is too compelling. Even if my thoughts go off in some other direction while reading and I do the "just read a page but absorbed nothing" thing, there's no voice involved then, it feels like a different process.
The only time I experience the sort of word-based hijacking you're describing is with what I think of as "internal argument". I'll think of a previous or possible future discussion and structure my reply in words. Although since words are slow, often the words will sort of collapse and I'll shift to sort of a mental outline mode, where it's more a feel of structure with occasional words or phrases cropping up.
The Origin of Consciousness in the Breakdown of the Bicameral Mind Paperback –
by Julian Jaynes
I definitely experience it when reading & writing carefully and sometimes in deep thought with conscious effort, but never minute to minute and definitely not compulsively. I'd describe my minute to minute like you, more abstract and intuitive. Yet at the same time I'm very introspective, it just doesn't happen with an inner monologue.
edit: after reading the study, I think I fall into the not having an internal monologue. Fascinating.
When he asked people how they kept track of how much time had passed, he found that people who pictured a number in their heads were able to keep time and read without issue, while those who "spoke" the time to themselves found it impossible to read.
I’ve always kinda suspected people making this claim are lacking introspection to such an extreme extent that they don’t even recognize the inner voice that’s omnipresent.
Your essential point for justifying this skepticism is that you cannot imagine people are this different. In my experience people are always a little bit more different than you can imagine. After all, there are people that tirelessly work to charitable ends on one end and people that run death camps on the other.
> people making this claim are lacking introspection to such an extreme extent
It's perfectly acceptable to you to imagine that you are (essentially) fundamentally better or more complete than them, but not that perhaps they are your equals and merely experience life differently. I think it could be valuable to figure out why one is easy to you while the other is hard.
If you're having a hard time grasping that, try doing or thinking about things while doing a mantra. I think you'll find that you're still able to "think" while the only thing your inner dialogue is saying is some kind of mantra. (It may take some practice)
You can also try speed reading apps which force you to absorb information without the time for an inner monologue.
I find this helps with introspection as you can observe ideas without the (direct) bias of language. Being able to recognize and observe the thing that's making your inner monologue happen is a useful skill, I think. I can't really imagine being bound to language like you're describing, and it often takes me a while to put more complicated ideas into words.
Languages categories are never going to be accurate. Is a whale a fish or a mammal? Well technically a mammal, but if you want to put someone in charge of them it's probably better if it's the department of fisheries than whoever's in charge of buffalo. One of of them has boats. The word is just a word, a pointer at a vague collection of things with similar properties. Being able to think about and work with the things directly without the distraction of language is very important to me.
And now it sounds like a lot of meditation is training to be able to think the way you or I do, haha.
Is there any useful, productive research out there about this stuff? The only time I’ve come across any convincing or scientifically rigorous psychology was when Feynman did some for fun in his spare time and wrote about it
- Don't accidentally get caught in the fallacy of "how I experience existence must be how everyone does
- Is it possibly just a semantic distinction at that point? If you are completely consciously unaware of an internal voice that speaks your language, does it matter if it's there or not?
It’s not a semantic distinction to me, since the mechanism underlying it would then be the same and it’d just be their recognition of it that varied which is way less surprising.
If such people were not rampant, I do not think these teachers and apps would be so popular.
Instead of a voice I have a constant song playing in my head when I am not focused. The song changes multiple times per week but if I am on idle I have a song.
When I speak the song turns off, but no voice comes on, unless I consciously prepare my words.
Now when typing this comment, I have a voice (which is my own voice) say the words I am about to type milliseconds before I type them.
But I also have an inner voice . I think the music stops when the voice starts, not sure.
For the last week I’ve had “right down the line” by Gerry Raferty (70s pop song) playing.
This hypothesis would explain hearing the inner monologue when reading, as reading actually transcribes visual data directly to their phonetic counterparts.
Thinking of images is also different than thinking of words (since all of us speak the language).
I’m not saying it’s impossible, but that I suspect its more likely a lack of introspection - I’d need to be persuaded empirically somehow and don’t know how to test it.
No more of speak the language than see with our eyes. Personally while I do have some kind of inner voice, my thoughts tend much more heavily to the visual. And memories too. If I need to recall a phone number or spelling then I'll imagine it written.
Like you, I had a hard time believing people were different in this. The whole idea of an internal narrator seemed absurd to me. Why would anybody need a narrator for themselves? They're right there! But enough people claim that this is their real life that I'm willing to believe it, however tedious and exhausting that sounds to me.
Without language and semantic meaning tied to ideas, what does 'thinking' mean at all?
My point is more that there are always thoughts (typically in the form of words, but sometimes images) flowing through your mind all of the time. Meditation and 'mindfulness' is focused on recognizing them as they happen and getting control of that kind of thing (at least enough to reduce thought loops, rumination, unwanted emotional response, etc.).
I get that you have a hard time understanding thinking without words because that's your main experience. But please understand that it's different for other people.
And not just people. Animals can be very thoughtful. Watch documentaries, for example, of animal cognition and problem-solving. From crows to chimps, an awful lot of thinking happens, just not in words.
I also think that an internal voice that doesn't get conscious awareness is likely to become a process that doesn't present as voice. So it's not like someone can just pay more attention and hear something, because it stopped talking a long time ago.
If you can speak, read, and write though?
I think you can get better at quieting the voice or letting thoughts pass, but I don’t think you can really turn it off for longer than a few moments. Gurus that claim they have and have “reached enlightenment” just seem to be lying either to themselves or everyone else (or both).
Obviously nobody is claiming psychic powers here, but I wouldn’t expect brains to operate that differently - so it’s surprising to me.
Everyone's neurotransmitter balance is different. Of course that will cause brains to sometimes operate wildly differently.
Ultimately this is just a hunch though about what I suspect is more likely, I can obviously be wrong.
And with that thought, I wonder if this sort of technology might be really useful for people with intrusive thoughts or schizophrenia, etc. Being able to objectively measure how well any given medication or therapy is working feels like a win to me.
Same with my urge to break the tension and jump onto train tracks.
Same with just intensely inappropriate thoughts during formal events. Like wanting to stand up and scream the F word or something.
I created a tool for myself to avoid this pattern, which is effectively a kind of interactive map-reduce system.
Also, I really like insights like this. Is there a dedicated place where people discuss these kind of meta-cognitive topics?
Check previous HN discussions of the broad topic, nice rabbit-hole.
Not sure why people love to skip any interesting conversation and immediately run into the weeds.
Imagine if we worked as hard at solving the actual problem rather than discussing all the things that could go wrong.
I'm not convinced a device to read your internal monologue from the outside is even possible, or if it is, it may be very, very large. A device that sees you're conducting one, perhaps, but reading out the contents externally? I'm not sure it could gather enough information and training data to ever decode it. (That is, my point is more information theoretic than technological.)
It's excerpted here if you don't want to read the whole book:
once regime changes
> "Will this idiot please shut up and end the meeting"
Everyone looks at you
A long long time ago in a galaxy far far away, I was dating an insanely hot girl...
When she asked me to go get an STD test prior to us doing anything...
I was 18 - and it was all good...
So I go to the clinic and am awaiting to see what happens at said test...
This ISANELY hot nurse comes in and tells me about the procedure and such and then leaves me in the room to disrobe...
I sit there literally goin g through such thoughts in my mind "DONT GET HARD DONT GET HARD" etc...
So I spend what feels like an eternitiy trying different thought methods to keep me distracted away from that amazingly beautiful nurse...
Then the door opens.
And in steps this troll of a person to do the actual procedure.... dealing with if I am flaccid is no longer a problem.
A few years later my DAD told me about Whitehouse.com being a porn website...
I check it out (this was lit. like 1999 or so)
I find a nice video and the girl in the video was my super hot girlfriend who asked me for the STD test... and thats when I knew why.
(BTW I'm now old and still flaccid.)
Currently it requires an invasive and risky brain surgery which AFAIK would be a definite no-go. At least in the US, this would likely be considered extremely inhumane unless it could be done without such significant risk of death or brain damage - given the nature of the surgery I find that unlikely.
In the cases where an invasive brain surgery would be permitted, I imagine torture would already be an option. Keep in mind torture doesn't have nearly the same risk of death and permanent brain damage as an invasive brain surgery.
interrogator> We have been torturing you for 10 hours. If you spell "John" we will stop torturing you and torture John instead.
interrogated> [The pain is too much. We have a deal. Sorry John.] JOHN
A better interrogator can be more subtle, like
interrogator> We have some evidence  that John is the one that put the bomb, and that you are innocent, but my boss is not sure. If you confirm this info we will send you immediately to your cell.
 It's a lie! Also, John is innocent in case you are wondering.
I wonder what happens to these BCI results when the user is on psychedelics a la MK Ultra
EDIT: Wait your point was slightly different then I first read but that makes perfect sense!
The difference is on the level of hearing someone's chess moves dictated vs. reading their mind to understand how they perceive the game.
It is entirely a consciously composed and intentionally sent message.
You're assuming that everyone's motor cortex/nervous system is wired exactly the same way. Given what we know about the variability of the human body, I wouldn't expect this to hold for the entire population.
Every human is wired differently. You need to train an ML model to read signals specific to that human for this tech to work.
- Imagine, very hard, the act of moving your arm to write a message.
- Do everything that you would do to speak a word, except the actual act of articulating a word. Get as close as you can while stopping the actual muscles involved from firing.
These are not mind-readers. They hook into your normal brain circuitry. It mostly works the same way that you can type out a message on your keyboard without thinking about the keyboard: it's a brain HCI, `/dev/input`, not kernel-space.
If you connect input in a feedback loop with an output and train on a particular output, of course you’re going to get the output you’re training for. It’s just not going to bear much resemblance to the data that’s on the hard disk without cooperation from the host system.
Quite a big leap from being able to identify a letter someone is thinking of REALLY HARD with the express purpose of making the software recognize it to randomly implanting something in someones brain and being able to browse through their thoughts.