It was actually the inspiration for my undergraduate research which explored (scratched the surface, really) the use of Lojban as an "interactive metaprogramming language". The benefits of Lojban are that it is significantly more expressive than making sounds that are mapped to vim-like commands via a hacked natural language recognition engine, is isomorphic between text and speech (you can speak mathematical expressions with precision), is phonologically and grammatically unambiguous, and is standardized/has a community -- even if it's a small one.
I still think it's a paradigm shift and would be indescribably more efficient and powerful than traditional code exploration and editing. And I would love to continue, but it's such a massive project and working on something that foreign alone for long enough is really isolating because it's hard to explain. Lojban also has its fair share of problems for this and would need to be further standardized and scoped out for computers, so there's really not a clear path forward in any case.
Regardless, for those _currently_ suffering from hand and arm issues preventing them from coding and/or working, my advice is:
1) don't panic. Go on antidepressants and/or see a therapist if you need to; it's going to take a while to recover
2) rest as much as possible and make sure to get a lot of sleep
3) wear wrist braces to sleep to prevent further damage in the short term (consult a doctor and all that, you want to avoid muscle atrophy so don't use them for too long without starting some kind of muscle strengthening program)
4) invest in proper tools (standing desk + ergonomic keyboard is like magic for me, I can actually type again)
5) gain weight if you're on the low-side of normal weight -- this helped my cubital tunnel syndrome quite a bit by giving my ulnar nerve more protection
And finally, don't give up hope; I'm able to work full time and don't wear wrist braces to sleep at all anymore after a little over a year.
Although anecdotal, this is real-world evidence, that you can write something that is, highly technical, with a lot of domain-specific knowledge, to professional standard, via voice dictation.
The question then becomes, is there a market, for example, for a "C++ aware" or "Python aware" modded voice application? How big would the market be? How much would people pay? Could it be covered by health insurance in an RSI case?
The biggest issue is that most coding is not producing code; it's navigating and editing existing code in a non-linear way. I think that makes it extremely difficult because it needs to be integrated with static (or dynamic!) analysis of a codebase. And once we take that step, we should be able to turn programming from shuffling text around to a more conversational (not unlike REPL-based) development.
There are also issues with the ability to pronounce things in some programming languages and being able to abstractly refer to these (often not linear) things. Because voice recognition is slower in terms of commands per second (as far as I've seen), we'd need more powerful and expressive ways of communicating intent to reach parity with keyboard-based programming.
What I was hoping for with "interactive metaprogramming" was to be able to do project-wide queries and edits. It seemed that trying to find a voice-based equivalent to the way we program with text and keyboards was never going to be a great fit and that, really, we should have more powerful tools for interacting with code anyways. Lojban provides a lot of fertile ground for these ideas, I think. For example, Lojban has a really powerful way of talking about space, time, relative positions, anaphora, and meta-linguistic correction/negation. I can imagine that being incredibly useful for programming by voice.
And I think the conversational style is key because the biggest issue with voice-based interfaces is presentation of relevant information. Lojban is based on relationships so it would be very natural to query the language itself with datalog-like evaluation, which provides an elegant solution to the hidden commands problem.
In regards to the price, I'd be willing to pay quite a lot. Losing the ability to program was one of the darkest times in my life. Soul-crushing, really. I was completely lost, unable to work and unable to enjoy myself. I remember when I thought $300-350 for a keyboard was expensive when looking at the Kinesis Advantage 2, but today, I'd be happy to pay 10 or 100 times the price knowing what it did for me. It's the difference between being able to work and enjoy myself (not being in constant pain from trying to work anyways) and laying around feeling hopeless and worthless.
It certainly is not impractical now. I wrote my PhD thesis, along with most of the software using VoiceCode.io. These days I’m on Talon. Dictation has basically been my sole key input device for three years. Like anything, it takes practice.
1) NEVER take meds that mess with your brain! That's such an American vice.
5) yes, but building muscles is better than building fat (can be combined;)
I think my case was a bit exceptional in how bad it got. I went from having no hand problems at all to having acute pain to the point where I couldn't function in daily life in a matter of weeks. Things such as working, driving, or opening doors with doorknobs became essentially impossible for me. I couldn't even sit in most chairs or sleep without my wrists being in constant pain.
2, 4, and 5 are the most useful because they're preventative. 3 is to deal with the physical trauma and 1 is do deal with the mental trauma.
I agree with you that being on psychoactive drugs is not ideal, but when your life becomes nothing but sitting around in pain, I think that they're a reasonable short-term solution. Although non-hand based exercise is surely better if you can manage it, I found that running with my wrists taped up was fine and helped as much or more than the drugs, and I stopped the drugs after a few months when I felt more able to deal with it.
First, voice coding could provide a far more efficient interface than typing based because the inherent difference in speed between being able to type out instructions and speak them out. All programming languages to date have been designed for typing based input. However, it should not be difficult to design one for sppech based input. In other words, it would make sense to design such interfaces for all users instead of only such users that have some problems using typing based interfaces.
Second, instead of creating generic, one size fits all solutions for the voice to code interface, it would make more sense to design such interfaces on a per language and platform basis. From a coding perspective, the language and the platform or the system software you plan to use is what contributes to the palette of programming instructions. For example, Ruby on Rails. A voice interface could be designed better keeping this in mind.
- You can change the entire voice grammar from anything to anything in a couple of milliseconds.
- I'm building a scope system, kinda like syntax scopes in a text editor like Sublime, but for arbitrary systemwide stuff. So you'll be able to say "this command activates on scope 'code.ruby.method' and 'filename matches test_*.rb" (which takes advantage of the fast grammar swap feature).
Getting really specific with commands and action implementations means heavy lifting can be done for you in a lot of really cool cases, and recognition accuracy goes way up. It's also hard to remember all this stuff, one fix is to use floating UI that shows you the most relevant context-specific commands.
As far as "generally accessible", another goal of Talon is to preempt RSI and find ways to convince people who can type fine to use it anyway. Two ways are "make many things way faster/smoother than typing" and "lots of cool eye tracking features".
Sounds like you're making it easy to create the equivalent of macros to abbreviate the input of common tasks.
It would be interesting to see if you could make this system function without a screen so even a blind person could produce code.
You definitely don't understand the implications of highly dynamic grammars then. A couple instances where rapid grammar changes matter are: "I want to switch between my text editor, a terminal, and a browser, and have exactly the most specific and relevant commands available at all times with no delay", and "I want to have syntax-aware voice grammars that are very in tune with how my programming language and framework operate"
You appear to be talking about this without any significant research, sources, or knowledge of my software (which you have multiple times made fairly confident statements about that are not remotely true), please stop generalizing about what I'm doing without some examples to back it up.
Part of the benefit of designing a language specifically for speech based coding is that the design of the associated recognizert could be tuned to that language. Compared to a brute force translated version of the keyboard input based language developer workflow, a speicifcally designed language would likely have a far narrower vocabulary. This would make it possible to tune a recognizer for better recognition accuracy and speed.
You will note my comments were not about your software, but hands-free programming interfaces in general. They continue to be this way, you will note. I am addressing the subject of this topic at a high level. I also have a fairly good idea on where your software stands within that context. Don't need to do any research for that.
Personally I feel that this shouldn't even be necessary. In theory all you need to be able to vocalise is what you want the code to achieve, then that could be mapped to something similar to a snippet based on what language you are using. Most programming languages share the same core abilities, it's just the syntax to achieve the task that matters once you know what the task is. So this shouldn't need to be handled by a dedicated programming language for voice coding, it could just be handled by a wrapper (A text editor for your voice with an advanced NLP based snippet system).
Something like: "Check if A is equal to B, if it is, return A, if not, return B" would be much more efficient than saying "if parenthesis capital A equals equals captial B..." and so on.
Sort of akin to how many of today's new high level languages are actually implmented by translating to C and then using a C complier
This caused me some great stress because I am an avid gamer, and also a developer.
Knowing that there are solutions, really reassures me, at least for my day job.
Physical therapy can help a lot.
One suggestion that helped for an acquaintance: if it's RSI from typing on a standard keyboard, but isn't too severe, switch to a fancy split ergonomic keyboard (provided that's different to what you currently use), change the key layout to Dvorak, throw away your existing typing habits and slowly learn to touch type again.
Another relative was diagnosed with https://en.m.wikipedia.org/wiki/Carpal_tunnel_syndrome . Surgery can help.
1. Adjustable desk height.
2. tried several keboards before settling on the best split.
3. tried several mice.
all helped, but... symptoms were also confusingly intermingled with a pinched nerve. PT helps that. See a doctor, yes, and also keep your mind open to other/multiple confounding causes.
Apart from a handful of unspeakable words which would need to be named speakably... The bigger reason I stay away from forth is mostly the memory/paging/record system
I was disappointed in the article in that it basically amounted to little more than an endorsement of talk-to-text. Lacked details. Coding? Didn't even hardly mention navigating a document other than a few "line up", "line down" quips. Coding isn't like dictating an essay, it requires constant back and forth navigation of a document (source file). Just this week, I've probably touched +100 source files because we decided to change how common objects are allocated (yes, the source is poorly arranged and has way too much coupling and very poor cohesion).
I believe it's far easier to solve navigation than text input. I will gladly record for you (or anyone here who has an idea) a synthetic (pre-planned) demo video of some more complex task. Want to come up with a specific workflow for to test? Like maybe "find <some project> on github, fork, clone it to my machine, perform <some specific refactor> / code editing task, commit, push" all without my hands, and I upload the first take.
The underlying systems are far more powerful than the article covered, e.g. in Talon you can write an editor plugin that collects filenames and turns them into commands in realtime to switch between files. Someone did a PoC with Talon's eye tracking to specify a symbol for insertion by looking at it. There have been demos of scope-based navigation / selection.
Personally I heavily optimized my alphabet/number input case (there's a specially designed alphabet that can do about 280 letters per minute at 100% accuracy), and use Vim commands directly, but many of the users use more conventional editors.
I plan to get into the habit of live streaming hands-free coding at some point so people can get a better idea of what it looks like to use this sort of thing, and so I can go back and watch and realize that I could be doing some things much better.
Not exactly true. The word by itself would be recognized as a command, but in a sentence it’ll be treated as any other word, assuming you don’t pause for too long before it.
I just defined a command "return" in Voicecode which presses enter.
"testing return testing test test return test return test" spoken in one breath types the following:
testing test test