I would argue that working in VR is fundamentally superior to working in PCs/laptops. It's basically 10x the screens, focus, and work immersion. Simula's text quality is also very good (getting around the eye fatigue present in older VR Desktops).
On a second thought: if you would make it easier to "fix" window, such that one can focus on it and work. As when you are "placing" a new window in your video, that could make it a workable solution.
But I agree that wiggling is off putting. I think VR demos should low-pass filter the output to some degree to give a more realistic perception of what VR is like.
But I feel its mostly rotational motion that is perceptible in VR demos rather than translational motion. I suspect that's because a small rotational motion results in a large shift in perspective for objects in the world that are some distance away from the camera while small translational movements don't really result in a significant perspective shift.
Would be interested in seeing what research into human head motion while sitting/standing/walking/etc. shows though.
Yeah, both rotation and translation would be filtered in the idea I had, so that small wiggling motions aren't as visible.
Have you actually tried VR? The fact that you think there's wiggling makes me think you haven't. Yes, videos make it look that way but I can guarantee you that's not reflected in the actual experience. VR is notoriously bad to watch on video and doesn't capture the experience at all.
As for pinned windows: fixed HUD is a no-no in VR games because it burns in same spot of your eyesight. When a HUD is desired, players are usually given a virtual or symbolic helmet that allow display to lag behind head movement.
I think the wiggling is actually kind of natural -- you don't notice your head or eye movements when you do them automatically, but it might be exacerbated in VR.
The fixing is an interesting idea, we'll try it out. I'm a bit worried it might feel weird in VR.
Corollary discharge causes the brain to “subtract” out the motion of the image on the retina. It happens when our nervous system instructs the eyes to look in another direction. You can have someone else move your eyes or even do it yourself by pressing gently on the side of the eyeball with a fingertip on top of the lid. Wiggling one eye this way is not supported by the hardware of the eye and produces a very noticeable wiggly view of the real world.
I mentioned subtracting out the motion going on in the visual system. Experiments suggest that something like this is happening. If a muscle paralyzing drug is injected into the eye’s muscles it causes a wiggly world because the intent to glance somewhere causes both an ineffective command to move the eye, producing no real movement of the image in the retina while the triggered corollary discharge subtracts the intended motion from the still image and causes a perceived motion (in the opposite direction).
I think navigation needs eye tracking, which sadly no headset currently supports. Focus-follows-gaze would be a game changer.
Right now, Simula uses "dumb" eye tracking, in the sense that windows receive keyboard and cursor focus when the user's forward eye gaze intersects a window. We also have it so that users can control the cursor focus with their forward gaze (presently binded to `Super + Apostrophe`); similarly, users can drag windows around by holding `Super + Alt` and looking around. The experience adds up to something quite productive once you learn all the keyboard shortcuts (your fingers don't need to leave the keyboard).
You can have just one window at a time, and with a good headset, the wiggle is not significant. You can even adjust it so that your workspace follows you when your head turns.
I haven't used Windows' Virtual Desktop since the early days, but I suspect our text enhancements have improved upon the situation dramatically (when keeping hardware constant).
The webcam view is also interesting; if it were "inverted" so that, instead of being confined to a window, it was outside of all windows (that is, the background "wallpaper"), it would work even better as a laptop replacement (that is, the screen doesn't cover your whole view, allowing you to keep aware of your surroundings).
The AR idea is really interesting. We'll try replacing the background with "reality" as seen from the HMD cameras.
So that's the future, maybe: cubicle dwellers get their own VR offices...
So being able to fetch up windows so it's perfectly centered above the keyboard would make it easier for me to use.
But after watching your video, I feel like this really needs some tiling window manager (i3 etc) inspired way of easily creating workspaces with tiles, and effortlessly navigating between them. With relatively minimal head movement.
Also, does Simula support curved displays? It would seem very natural to arrange different curved screens around you instead of flat ones, which also potentially waste quite a bit of useful space.
I'll put curved windows onto our todo list. Not sure if they're a good idea for all applications but at least some would benefit.
I won't get to try out Simula until Sunday, but based on the video, my intuition says the apps being curved would look more natural / immersive.
For VR gaming (Oculus Rift), I start to experience mild discomfort after about 30min, which grows severe by 60min. It's hard to envision using VR for a full work day. (My eyes are fairly light-sensitive from laser-eye surgery; not sure how much is saccading, vs. bright light directly in front of the eyes.)
Many early VR Desktops (and VR games) didn't go to special effort to optimize text quality, and I think this has given people a lower than warranted impression of what is possible in VR today. There's no doubt that VR hardware is going to get exponentially lighter/sharper over time, and that these improvements will be welcome, but things are good enough now to put in 1hr+ long sessions without eye strain (at least in my experience as a daily user).
I think the Index might be OK but I couldn't comment.
I never would have even purchased a VR headset had it not been for the promise to start working on a productivity environment. But before we can start building the killer VR office apps, we have to get the basic 2D apps working crisply/perfectly, IMO.
In 2017, I had a conversation with a co-worker, where we talked about how badly we wanted a VR desktop environment with infinite screen-space. There's VRDesktop, but that isn't this. And then again the same conversation last year with another friend.
After seeing the demo video, I was compelled to go look at Vive prices (~$1,000, ouch).
I feel like my productivity would skyrocket with this, mostly due to lack of background distractions. Imagine wearing noise-cancelling headphones. I want this so bad.
I feel we are just scratching the surface of what's possible. Yes, multiple (arbitrarily sized) windows are neat. But we can probably do more.
Even without any changes though, I just wish my VR headset was lighter. I can go more than 8h straight in VR... playing games. Not sure about working.
Will try this :)
SteamVR does this, but I've found it less useful than movable/resizable windows.
> I feel we are just scratching the surface of what's possible.
Definitely. This is one of the first viable iterations, but there's lots and lots of improvement potential everywhere.
> Even without any changes though, I just wish my VR headset was lighter.
Yeah, agreed. But I think future hardware will improve on that.
Would allow you to spin the windows around without having to spin yourself, for prolonged periods of "switching over to the right" for instance.
Could use a different gesture to move around the 3D space too allowing you to navigate to different pockets of work
Have you thought about an Android/Quest version?
I'll take a look at Quest docs and see if it's doable, but I don't have high hopes for it.
Having said that, getting even a standard desktop setup on Android to work well is iffy so doing it in VR does seem like a pretty immense challenge. But it would be a breakthrough in what it could demonstrate in terms of the concept.
Now you need something like a knob or dial to rotate the background, so you don't need to move your chair or turn 360.
Holy crap, I never thought of it that way!
Anyone know if this is possible with macOS?
It can create extra displays, but you have to buy an Elite license for that.
Where does Haskell fit in with Godot and Simula?
The big win, as far as I can tell, would be to engage the user's spacial memory. (There is a small but non-zero niche for 3D visualization of complex systems: weather, large molecule, etc.) You're going to want to combine "memory palace" with "zooming" UI in a kind of pseudo-3D (I think of it as 2.618...D but the exact fractal dimension isn't important I don't think.) Then infuse with Brenda Laurel's "Computers as Theatre"...
https://en.wikipedia.org/wiki/Brenda_Laurel - https://www.goodreads.com/book/show/239018.Computers_as_Thea...
Wait until resolution improves and we break out of the "desktop" paradigm. We could have a collection of unlimited windows and tabs that exist in a continuum around us, and we could use gestures to organize and surface the contextually relevant ones.
We won't need a bulky multi-monitor setup, and we could work remotely nearly anywhere. Imagine carrying your workspace with you.
> The big win, as far as I can tell, would be to engage the user's spacial memory.
Absolutely! Physical workspaces and work benches are incredibly functional because we are spatial animals. Breaking out of the limitations of using a screen could unlock more of our senses for use in problem solving.
I'm extremely excited about this technology. It will be great for software engineers, creatives (2d and 3d artists), mechanical engineering, CAD, ... you name it.
I really hope this keeps getting pushed forward. While I'm using all of my spare cycles on a tangentially-related problem domain, I'd be more than happy to donate money and ideas. This technology will be a dream come true if it continues to mature.
You can get 90% there by just using multiple desktops IMHO, at least that's my experience.
Headsets have improved almost ~2x in resolution and have halved in price and some have become wireless with better optics. A long ways from 4-5 years ago, but still need another doubling of resolution (or maybe more) and an increase in wearing comfort (lighter, more compact) plus an improvement in wireless latency and maybe a reduction from $400 to $300, and you’re looking at something that would be useful just as a replacement for multiple monitors.
Plus probably improvement with registering where your laptop and mouse are automatically. In principle that could be done with a software update to inside-out tracking software.
Additionally, some improvement is possible with similar to current resolution but with improved subpixel rendering and RGB pixel layout.
Seeing what has been done with the Oculus Quest since I last checked out VR like 3 years ago has left me pretty impressed. A lot of this stuff with multiple windows in this demo could be done natively and wirelessly with the Quest (which runs a kind of Linux). The inside-out tracking is impressively good. If combined with an insert you can put your tracked controllers on Bluetooth mouse and keyboard (so the Quest can register their positions in3D space to allow proper rendering in-headset), it could give you a high productivity workstation experience just about anywhere with WiFi (could be through phone). Hand-tracking (Which works already) could even allow gestures, although I’m not sure how important that is. Can the Quest do subpixel text rendering like ClearType but 3D?
Eight hours+ of using gestures a day sounds like hell.
There's no reason you can't make the windows always perpendicular to the line of sight while still fixing their centers in space.
I don't know if you're still watching this thread, but it would be interesting to hear an example of when one might care to lock a window's orientation.
I've experimented with combining the kinesthetic advantages of pie menus and gesture with the method of loci and zooming interfaces, including a desktop app called MediaGraph for arranging and navigating music, and an iPhone app called iLoci for arranging notes and links and interactive web based applets.
>MediaGraph Music Navigation with Pie Menus. A prototype developed for Will Wright’s Stupid Fun Club.
>This is a demo of a user interface research prototype that I developed for Will Wright at the Stupid Fun Club. It includes pie menus, an editable map of music interconnected with roads, and cellular automata.
It uses one kind of nested hierarchical pie menu to build and edit another kind of geographic networked pie menu.
MediaGraph Demo Video:
>iPhone iLoci Memory Palace App, by Don Hopkins @ Mobile Dev Camp.
A talk about iLoci, an iPhone app and server based on the Method of Loci for constructing a Memory Palace, by Don Hopkins, presented at Mobile Dev Camp in Amsterdam, on November 28, 2008.
iLoci Demo Video:
Here's some more discussion about window managers, the Method of Loci, MediaGraph and iLoci, and pie menus:
DonHopkins 81 days ago | parent | favorite | on: Nototo – Build a unified mental map of notes
>Great idea, I totally get it! Your graphics are beautiful, and the layering and gridding look helpful.
It reminds me of some experimental user interfaces with pie menus I designed for creating and editing memory palaces: "iLoci" on the iPhone for notes and pictures and links and web browser integration in 2008, and "MediaGraph" on Unity3D for organizing and playing music in 2012, both of which I hope will inspire you for ideas to implement (like pie menus, and kissing!) or ways to explain what you've already created.
>A memory map editor can not only benefit from pie menus for editing and changing properties (like simultaneously picking a font with direction, and pulling out the font size with distance, for example), but it's also a great way for users to create their own custom bi-directionally gesture navigable pie menus by dragging and dropping and "kissing" islands together against each other to create and break links (like bridges between islands). (See the gesture navigation example at the end of the MediaGraph demo, and imagine that on an iPad or phone!) [...]
>I like the idea of moving away from hierarchal menu navigation, towards spatial map navigation. It elegantly addresses the problem of personalized user created menus, by making linking and unlinking locations as easy as dragging and dropping objects around and bumping them together to connect and disconnect them. (Compare that to the complexity of a tree or outline editor, which doesn't make the directions explicit.) And it eliminates the need to a special command to move back up in the menu hierarchy, by guaranteeing that every navigation is obviously reversible by moving in the opposite direction. I believe maps are a lot more natural and easier for people to remember than hierarchies, and the interface naturally exploits "mouse ahead" (or "swipe ahead") and is obviously self revealing.
1. A way to neatly arrange all the windows on a virtual sphere that surrounds the user, possibly arranging them automatically in a similar manner as a tiling window manager.
2. A way to rotate the before-mentioned sphere around you without forcing the user to rotate it's head. This would avoid much of the neck strain. It could be done by, for example, holding a button on the keyboard while moving the mouse or by a simple keyboard shortcut to rotate the sphere by X degrees in any direction.
This concept could also be extended to virtual desktops where each desktop is a sphere around the next, like an onion, with the ability to "zoom" in to the next desktop.
one more thing for the author to play around with (I don't have the hardware to try it myself) is experiment with speed/acceleration -- the mouse pointer movement has speed and acceleration parameters that affect how it moves, so that if you want you only have to move your mouse very little in real dimensions to move it thousands of pixels.
It might cause motion sickness, but maybe you can get away with pitching/yawing 5 degrees for every 1 degree of head movement, so to look "straight up" you only have to tilt your head up 18 degrees. Hopefully you'd still have the illusion of being oriented in a space.
That would cause extreme motion sickness.
I don't know anyone who used MS Windows when their software was all still MS-DOS programs. Windows really took off when programmers started writing programs designed to take advantage of the native GUI paradigm.
Likewise, VR is never going to be accepted as a practical user interface until it moves beyond the concept of "windows". I fully believe it's possible to make a great VR user interface, but I can't believe it's going to look anything like some rectangles floating in space.
This could be a VR-fvwm. What we really need is a VR-GTK+.
What would a non-rectangular UI for manipulating text look like?
I've always envisioned an infinite 2D plane. I would love a terminal that stretches off forever above me, like the Star Wars opening text. Or a spreadsheet that goes on forever in 2 dimensions.
You could also embed your 2D-non-euclidean spaces in 3D. Want to enforce 80 character line wrapping? write on the surface of a cylinder that literally wraps around after 80 characters.
Another idea could be to use depth instead of colours for syntax highlighting. Comments could pop out of the page.
I think there are a lot of possibilities.
Programmers work with 2D overlapping windows and will continue to do so for the foreseeable future. I'm giving a criterion for when that's going to change. They're connected. You can't beat a 2D display for displaying 2D data.
> What would a non-rectangular UI for manipulating text look like?
Looked at a DOM Inspector recently (HTML)? Or Computed Styles (CSS)? Or Network Activity (HTTP)? All of the most common text formats I use, I view through a non-text interface. These aren't inherently 2D data streams -- that's just what we do because they're being put on a 2D display. All of them would be even more useful in 3D.
That's not even counting the biggest classical use for an extra dimension: time, e.g., version control history, animation state, or database transactions or migrations.
It's still very difficult for me to imagine how I would translate interfaces I'm used to into another dimension. Even if I picture something floating in front of me, I only perceive a 2D projection of it. Perhaps with clever transparency or by rotating it around I could receive more information than I would in 2D.
I feel like a character in Flatland, I stuck in my own dimension.
I'd need probably a dozen or so channels of information that I had to cycle between quickly to convince me, given I've gotten used to multi-monitors, workspaces, and keyboard shortcuts. Maybe some futuristic high level analyst job that needs to look through lots of different kinds of information (geographical, text, videos, etc) at once.
For another interesting use case, how about a 3d debugger that lets you follow a program flow over multiple classes and stacks...
Essentially you get your vector data, and map it into a texture of distance from the boundary, with positive being outside the boundary, and negative inside. Then you apply a shader to that resulting texture to make it presentable for the user. This means crisp contours/outlines at all but the most extreme proximity, and easy implementation of other common vector effects (such as bevels, shadows, etc).
Overall it was a pretty rough road, but someone has to go first and I'd really like for Haskell graphics to take off.
I ended up writing bindings for Godot so we didn't have to deal with graphics stuff from scratch (like in the first few iterations).
While this lost us some of Haskell's benefits and it's not super functional-y, it's still nice to work with and most of the issues we encounter are related to the bindings being unpolished.
If you have any questions about doing this kind of project in Haskell, hit me up.
Nix, despite issues with everything OpenGL related, was the only thing that actually worked. And Cachix made installation really fast.
PS: if it starts building for some reason during the install process, let us know. We've tested it on various machines but due to some peculiarities specific configurations might require a rebuild even with Cachix. We can add that to the repo so that future installs and updates are faster.
We initially tried to get a static binary with AppImage and even nix-bundle, but those ended up being failed/painful paths.
Barbara Liskov lent me a book that described it in 1973 and recommended that I read about Simula 67 before starting to work on a small part of the the CLU compiler design. I believe the book was Structured Programming by Dijkstra, Hoare, and Dahl. It had a chapter by Dahl describing Simula 67.
Right now we're working on perfecting Linux VR Desktop (for 2D apps), but the long-term goal is to bring forth killer office VR apps (for productivity, not gaming/entertainment) that wouldn't otherwise be possible in normal 2D Linux. These apps will likely take advantage of VR as an amazing "Simula[ting]" environment.
Disclaimer: I worked at its company at that time
Here are a couple of links showing off Simula's text quality:
Many early VR Desktops (and VR games) didn't go to special effort to optimize text quality, and I think this has given people a lower than warranted impression of what is possible in VR today (even on old headsets).
We'd have to implement a Godot VR interface for the Quest. If they have Linux support at all, this should be doable.
Someone asked if I could watch all the Star Wars movies at the same time. I got 6 trailers running at once in 6 separate virtual monitors with the default built in OS feature.
A cross-platform possibility would be if it served up a WebVR page that could be accessed with Firefox Reality.
If you are at a desk and/or without a swivel chair, you're pretty limited to a grid of ~9 screens directly in front of you, and even then it looks like you're turning your head a TON to see the outer layer.
Some ideas that might help:
1) To use an analogy, could you increase the "mouse speed" of moving your head? Right now it's 1:1 with the virtual space and makes geometric "sense", but it might be nice to be able to focus on screens on the left/right by moving your head less and having your focus move just as fast. For example, moving your head 50% of what you do now to zoom over to the same place would reduce neck strain when looking at those windows, but also open up a bit more window space if, for example, you could still move your neck 100% of what you're doing and be able to "look over" far enough to fit another set of screens even further out. I don't know if this disconnect in pan speed would be disorienting, but it's worth trying. :) c.f. https://i.imgur.com/0KCo9hG.png
2) Keyboard gestures to "re-center" your focus might also go a long way. Perhaps instead of craning your head to look at a window that isn't "straight ahead" for a long time (as long as you're working in that window), you could do a quick glance at that window and hit a keyboard shortcut that would recenter wherever you're looking to be your "straight ahead", so you could then straighten your neck back out and continue working in that window while in a more comfortable position. c.f. https://i.imgur.com/u6SI284.png
3) Likewise, we don't all have amazing headset resolutions and there's a lot of potentially wasted space whenever focusing on a particular window that's not "full-screened". Perhaps a keyboard shortcut to expand whatever screen you're looking at to the full size of your view would be helpful for eye strain. Even something that could temporarily "pin" a window to your view (full screen and moving with your gaze) might go a long way toward minimizing unused focus/pixels. c.f. https://i.imgur.com/yA1twKf.png
My immediate thought watching this was to mount a keyboard on a swivel chair, but I think requiring end-users to get customized hardware to take full advantage of your environment might be the wrong way to go. :)
Your brain adapts, and starts to believe at a primal level that "things fall this speed when I drop them," "I move this fast when I walk", "when I point over there, it's at this angle," etc.
If anybody remembers playing GTA for too long, and later walking outside with the feeling like you ought to be able to jump in the nearest car and drive it a way, you have an inkling of how your brain gets remapped.
Remarkably quickly, moving around in the real-world can feel "off" to the same degree that it felt "off" when you first entered the VR world when you start messing with these primal feedback loops. Get ready to bump into things, and don't operate any heavy machinery if you've been in a non-1:1 VR mapping.
2) I believe we already have that (via Super-')
3) Good idea. Being able to instantly fullsize and "pin" a window sounds good, maybe even as a toggle.
It's not like Alyx is the only reason to own a VR headset.
The world of VR is so diverse it's tricky to recommend things without knowing you.
Personally I'm not much of a gamer so I tend to be interested in narrative stuff, abstract visuasation and geometry stuff or just exploring interesting environments.
And the games I do like tend to have one or more of the above qualities.
You might be an RPG fan, a strategy gamer, an art aficionado , looking for productivity or educational tools or any number of things.
PavlovVR is CS:GO in VR and done really well, hilariously good fun.
Volumetric video, spatial website navigation design guidelines, and augmented reality are high on m wish-list.
Wikipedia articles follow a format which makes it easy to find information. Perhaps site maps for a VR web could be arranged according to a convention so that users can easily figure out where to look for specific information. Early VR we web tried to do this by arranging content like rooms in a building, and streets, but that is cumbersome to use in practice. I think the wiki format suggests a much better approach can be created.
Our approach with mouse cursors has been to assign every window its own mouse cursor. This is possible in Simula since the active window is the one that the user is currently gazing at, so that -- for example -- if you gaze at one window and move its cursor, it won't affect the other windows' cursors. We also allow users to control mouse cursors with their eye gaze (presently bounded to `Super + Apostrophe`), which is a good productivity boost.
In fact, you can do everything in Simula with just the keyboard (no mouse or VR controllers needed): move windows, move the mouse cursor, click the mouse cursor, etc. Once you learn the shortcuts, it's very quick.
I do await the day my whole system is a AR/VR hybrid, but I suspect it'll be a while still.
I've been spending an hour and a half in VR everyday as part of my quarantine workout, and it seems fine, though I'm not really reading anything in that environment.
X11 apps are supported via Xwayland.
Eventually we could move on to having tubes implanted for feeding and waste. You could stay in there indefinitely, forget you ever even existed outside the simulation...
So we set a higher DPI than usual, and then use a supersampling algorithm with a suitable kernel as a lowpass filter. This allows us to avoid artifacts while maintaining sharpness.
From a practical point of view, at least for me, "it just works like normal".
As I understand the biology of it, there are really two different relevant mechanisms for "focus".
The lens that is your eyeball changes shape to focus on things closer/father away in real life. In VR this doesn't happen, there's a fixed focal plane a few meters away. I know this, but I don't notice it at all. Everything is always in focus and my brain doesn't seem to have any trouble dealing with it.
There is also the problem of "vergence", your eyes point at the thing you are looking at, which means they point "closer to eachother" when looking at something close, and in parallel lines when looking at something at infinite distance. VR headsets implement this reasonably well. If you hold a finger a few inches in front of your nose and focus on something a few meters away you should be able to see two copies of your finger (or at least I do, I'm told that other people have to try to see this). This works in VR as well.
As an aside. Trying to play shooters in VR has been surprisingly hard for me because of the vergence issue. I'm not use to aiming a real weapon, and it turns out I have trouble figuring out which near-field image corresponds to the eye I'm trying to aim with while I focus on the target in the distance. I'm not sure if I'd find it easier in real life.
There's prescription lenses for headsets, but they're fairly expensive and IMO only worth it if you use VR a lot.
Yeah, me too. As far as my eye is concerned, most things are "far away". Interestingly though, I can read things in VR "farther" than I can in real life. I am not exactly sure what the convergence looks like.
Whatever I’m editing should be frontmost and dead center, IMO.
You could reach me at:
Or if you have some ways of contacting you I'd be very happy to do so
That's the classic example of using a new technology to emulate an old technology, without taking advantage of the unique advantages of the new technology, before the grammar and language of film had been invented.
>The first decade of motion picture saw film moving from a novelty to an established mass entertainment industry. The earliest films were in black and white, under a minute long, without recorded sound and consisted of a single shot from a steady camera.
>Conventions toward a general cinematic language developed over the years with editing, camera movements and other cinematic techniques contributing specific roles in the narrative of films. [...]
>In the 1890s, films were seen mostly via temporary storefront spaces and traveling exhibitors or as acts in vaudeville programs. A film could be under a minute long and would usually present a single scene, authentic or staged, of everyday life, a public event, a sporting event or slapstick. There was little to no cinematic technique, the film was usually black and white and it was without sound. [...]
>Within eleven years of motion pictures, the films moved from a novelty show to an established large-scale entertainment industry. Films moved from a single shot, completely made by one person with a few assistants, towards films several minutes long consisting of several shots, which were made by large companies in something like industrial conditions.
>By 1900, the first motion pictures that can be considered "films" – emerged, and film-makers began to introduce basic editing techniques and film narrative.
Simply projecting desktop user interfaces designed for flat 2D screens and mice into VR is still in the "novelty show" age, like filming staged plays written for a theater, without any editing, shots, or film grammar.
VR window managers are just a stop-gap backwards-compatibility bridge, while people work on inventing a grammar and language of interactive VR and AR user interfaces, and re-implement all the desktop and mobile applications from the ground up so they're not merely usable but actually enjoyable and aesthetically pleasing to use in VR.
The current definition of "window manager," especially as it applies to X-Windows desktops, tightly constrains how we think and what we expect of user interface and application design. We need something much more flexible and extensible. Unfortunately X-Windows decades ago rejected the crucially important ideas behind NeWS and AJAX, that the window manager should be open-ended and dynamically extensible with downloadable code, which is the key to making efficient, deeply integrated user interfaces.
For example, the "Dragon Naturally Speaking" speech synthesis and recognition system has "dragonfly", a Python-based "speech manager" that is capable of hooking into existing unmodified desktop applications, and scripting custom speech based user interfaces.
Another more ambitious example is Morgan Dixon's work on Prefab, that screen-scrapes the pixels of desktop apps, and uses pattern recognition and composition to remix and modify them. This is like cinematographers finally discovering they can edit films, cut and splice shots together, overlay text and graphics and pictures-in-pictures and adjacent frames. But Prefab isn't built around a scripting language like dragonfly, NeWS or AJAX.
Here's some stuff I've written about the direction that user interfaces should take to move beyond the antique notion of "window managers", and enables much deeper integration and accessibility and alternative input and output methods.
>Imagine if every interface was open source. Any of us could modify the software we use every day. Unfortunately, we don't have the source.
>Prefab realizes this vision using only the pixels of everyday interfaces. This video shows the use of Prefab to add new functionality to Adobe Photoshop, Apple iTunes, and Microsoft Windows Media Player. Prefab represents a new approach to deploying HCI research in everyday software, and is also the first step toward a future where anybody can modify any interface.
>Here are some other interesting things related to scriptable window management and accessibility to check out:
aQuery -- Like jQuery for Accessibility
>Morgan Dixon's work with Prefab is brilliant.