The only thing I'm aware of that does something akin to this is Rutkowska's Qubes OS.
It's just a ton of system administration work. I've been playing with containers and chroots lately, and I actually can do exactly what you're asking for. Right now it's for servers (i.e. always run a mail server, DNS server, web app etc. in their own containers). But eventually I would want to do it for desktops, and sort out issues with X and sandboxing, etc.
My personal wish for furthering system's research of today is something that would kill the need for hypervisors. Android, for example, runs each app under a different UID, thus achieving isolation w/o hypervisor. Let's devise something similar for desktops/servers; bonus points if the system can run existing binaries.
This isn't really a research topic. It's mainly an implementation issue. It's possibly a standards issue, but I think giving each app its own POSIX environment with Linux namespaces gets you almost all the way there.
The main obstacle I'm having is that distros like to spew files all over file system. So although I could technically use raw .deb packages, I'm ending up making my own packages that are bettered suited to the "every application in a container" model.
So to make what you're asking for a reality, I think the things that need to happen are:
- Linux needs to get hardened more (i.e. right now it is not entirely safe to run untrusted apps unless you are adding something like seccomp, e.g. which ChromeOS contributed to the kernel for this use case)
- There needs to be the equivalent of Debian ("the universal operating system") for applications in containers. This just requires a lot of mucking with configure and build scripts. It's not hard, just tedious. Application authors have to learn to package a slightly different way.
Let's say I'd like to have a context menu where I can choose "Run in sandbox", then I would either choose a preexisting sandbox configuration or define a custom one. IMO, there's a plenty of research do be done on how users can seamlessly define allowed data-flows (what the sandboxed application can read and write), and how to implement the allowed data flows.
There's a fine line between "implementation issues" vs "research topic". Research can touch other fields than technical, e.g., HCI.
As an example, GPG is a powerful system, but all available UIs and "integrations" with existing mail clients are clunky at best. We don't know how to create a good crypto UI for GPG. There's a research topic.
Therefore, even the technically best sandbox won't be used if it entails a "ton of sysadm work". (For example, I have Linux installed in virtualbox VM, but I don't use it often because the integration with the hos OS -- e.g., clipboard sharing -- is clunky.)
You might like to look at Bromium (bromium.com), founded by a bunch of ex-Xen folks. I think they were doing something like this for the enterprise.