I'm not sure why the author is having so much pain with Qubes. Indeed the lack of GPU in guest VMs is annoying but it is possible now to assign a GPU to a HVM fairly reliably thanks to all of the VFIO/gaming-on-linux enthusiasm in the past years. Otherwise, I also find that running browsers in multiple VMs on laptop is a problem if you don't disable JS by default because modern websites have become so bloated, it's a tragedy. The LVM remark is also strange. It's very reliable for plenty of people, but there is the risk of running out of space for metadata . Thin-pools for VM storage allows for some great Time-Machine-esque incremental backups also . But for managing multiple development environments, Qubes is a blessing, not even including all the security benefits.
Another option for Xen fans is XCP-ng + a thin client machine for accessing the VMs. One can also use firejail+Xephyr to achieve graphical isolation  (not sure about Wayland).
It looks like architecture changes in Qubes future  may make KVM a reality.
This is still a very cool effort, I'll have to give the Wayland bits a close read.
: https://github.com/QubesOS/qubes-issues/issues/3243, https://listman.redhat.com/archives/linux-lvm/2018-July/msg0...
As for the OP, I feel like if somebody cares about security, they shouldn't be doing any of this. Trying to come up with some self-designed hodgepodge of things isn't really enough security-wise, even if you do use VMs, and I'd find it hard to trust something like this as a platform to do anything important on.
Security isn't a black or white issue. There are levels of security. Many tech people want something better than the (very insecure) standard setup on Linux/Windows, but they don't want the Qubes straight-jacket. This means they search or develop alternatives and that is overall a good thing.
Theoretically, all modern systems have IOMMUs, which are supposed to be able to allow an operating system (or hypervisor in this case) to restrict what a device can do.
In practice, IOMMUs can't be trusted, for two reasons.
First, the implementations (i.e., the actual hardware) are frequently full of unpatchable security holes. It's not just the CPUs themselves that need to be correct; every chip in the PCI chain has to DTRT from a security perspective or risk opening up a vulnerability.
Secondly, particularly with GPUs, many systems have 'magic backdoors' that side-step the IOMMU systems completely. Frequently this is to make certain operations "faster" or "easier".
Basically, unless you have done your own audit and testing of the hardware you own (or someone you trust has done the same thing), you have to more or less assume that a malicious guest with control of a physical device could break out of the system.
In Xen's SUPPORT.md document, we very carefully tried to balance this:
Because of hardware limitations
(affecting any operating system or hypervisor),
it is generally not safe to use [PCI passthrough]
to expose a physical device to completely untrusted guests.
However, this feature can still confer significant security benefit
when used to remove drivers and backends from domain 0
(i.e., Driver Domains).
See also: https://www.qubes-os.org/faq/#can-i-run-applications-like-ga...
But the fact is, even if you are doing GPI passthrough in Qubes, it's much more secure than running any other system.
Some links I remember going through:
> Additionally, Qubes restricts the config-space a VM may use to communicate with a PCI device. Only whitelisted registers are accessible. However, some devices or applications require full PCI access. In these cases, the whole config-space may be allowed. You’re potentially weakening the device isolation, especially if your system is not equipped with a VT-d Interrupt Remapping unit. This increases the VM’s ability to run a side channel attack and vulnerability to the same. See Xen PCI Passthrough: PV guests and PCI quirks and Software Attacks on Intel VT-d (page 7) for more details.
But I haven't seen anything slow. I even play 4k full-screen video in a VM, which does burn 3 cores. I have it maxed out at 16GB of RAM, which is only just barely enough, with ZRAM swap enabled. It allows a VM, when I ask, access to the built-in microphone and camera, as needed, and takes them back. Wifi works, audio works. I keep an external 4k monitor plugged in, and that works. The touch screen can be made to work with trivial fiddling.
I was surprised to find him presenting Wayland code that looked like bad C, but is described as C++. The only C++ish thing about it was using static_cast<T>(e) instead of C-like (T)e. It is not encouraging to find such bad code in a new-ish project (you can lead a horse to water...). Good C++ code would look a lot like his OCaml code.
The only thing I have never got to work, at all, is clipboard operations between VMs. It announces it's doing something with various clipboards, but nothing apparent ever happens.
I would rather see the dom0 running NixOS than ancient Fedora.
> The only thing I have never got to work, at all, is clipboard operations between VMs. It announces it's doing something with various clipboards, but nothing apparent ever happens.
This sounds very strange, has been working flawlessly for me since forever. Perhaps you could fill a bug or ask at Qubes forums: https://qubes-os.discourse.group.
[Edit: virtwl.ko lets a wayland client inside a VM connect to a wayland server which is on the same machine but outside the VM].
I mean without it, all the noise about Wayland being more secure than X11 is just noise, because Wayland doesn't do remote display, so you have to allow that evil app you're scared of to run on the same machine as the compositor and all your other apps, and you can't even put it in a VM. That's less secure than being able to put it on another machine and use the network. In theory you might be able to run the evil app under another userid, but in practice this often breaks in all sorts of ways because people just don't do that very often so it exposes untested codepaths in UI/widget/GUI libraries. You can't even put the evil app in a VM -- until virtwl.ko came along.
If the Wayland folks want their security claims to no longer be laughed at they need to get virtwl.ko into mainline Linux. And get it supported by more hypervisors than just the one that comes with ChromeOS (crosvm).
I've been using sway on both my desktop and laptop for over a year now, but that doesn't mean I don't get to complain about Wayland. Since leaving X11 I did finally get everything working again (except @#*&^@%@ torbrowser), but I've been underwhelmed with the benefits gained by all the effort I had to go through in order to switch to Wayland. And I still miss remote display always working all the time without special effort from each individual app developer. Every bolt-on remote display option offered for Wayland is a flaming dumpster fire of slowness compared to X11 on the LAN or xorg-xrdp over the public Internet. And the least-flaming dumpster-fire of slowness, waypipe, has turned into abandonware.
> like many Wayland specifications, it seems to be in the process of being replaced by something else.
Funny because true.
But you can run it with different UID.
> In theory you might be able to run the evil app under another userid, but in practice this often breaks in all sorts of ways because people just don't do that very often so it exposes untested codepaths in UI/widget/GUI libraries.
Regardless, privilege escalation bugs are a dime a dozen these days. Running evil things as another userid is no substitute for moving them into a VM or off of the machine completely.
It doesn't really have a better name. It was written by Google to allow Android apps to run inside a VM on ChromeOS. Although it does work with mainline Linux, Google is understandably not motivated to invest effort in branding virtwl.ko with a non-cryptic name, or marketing it as its own project independent of ChromeOS. AFIAK nobody outside of Google was using it until Alyssa (same person responsible for the panfrost awesomeness) got it working on Linux.
As for what it does?
Wayland clients and servers must (a) run on the same physical machine (b) as processes on the same kernel. Virtwl.ko removes requirement (b), allowing the client to be inside of a VM.
This way of using Wayland and VM's seems very interesting and useful. What I'm wondering though, is whether it would be realistically possible to also expose some kind of GPU acceleration to clients in VM's, for example by means of a virtual OpenGL adapter such as VirGL, which itself uses virtio on top of the host GPU driver. This would not get you anything near the performance required for games, but it should be way better than software rendering inside the clients. Do you think something like this would be possible already?
Since then I've been using proxmox, and I'm at the point where I don't use the gui anymore, I just do everything from the command line.
You can do VM things (like run macos in a vm), but I do most things in lxc containers.
It would be kind of nice if proxmox had something like a Dockerfile, but with local containers that didn't depend on going out to dockerhub to pull in and run code.
If so, that is pretty interesting...
Although, using LXD with Ubuntu is totally painless and easy as well..
I used qubes as a desktop os, but it was a little clunky. Like the copy/paste or moving stuff between the vms.
But the main problem was I couldn't ssh INTO the qubes machine, and I couldn't really run server kinds of things (though maybe it was possible to dig into the networking and the firewall vm).
When I tried proxmox, it was more set up for server kinds of things. For example, all the VMs showed up as machines on my local network. I started to make it a desktop, but then backed off. I decided to leave the debian main os vanilla. Instead I made it a server. And I got away from VMs and use containers.
I do have a macos vm that I remote into though. It's a copy of my laptop which I haven't used as much the last year.
Routed turns your virtualization host into a router for a separate network on which your VMs live (afaik). So in order to reach that network you have to define static routes that ultimately lead to the virtual interface set up by libvirt.
When you start your routed network (with net-start), libvirt should set up a static route on the host machine automatically. You can see this route with "ip route". To make it available to all hosts on your network, you can configure a static route on your (physical) network's router that covers the IP address range of the virtual routed network and has the (physical) IP address of your virtualization host set as the gateway.
For example: Assume your virtualization host has 10.0.0.10 as its address in your "real" network and your virtual routed network covers 10.0.1.0/24. You would have to set up a static route on your "real" router that says "route 10.0.1.0/24 to 10.0.0.10".
Libvirt automatically sets up a route in the local routing table of your virtualization host that says "route 10.0.1.0/24 to virbr0" (or whatever your network's virtual interface is called).
Then, if everything worked, you should be able to connect to your VMs in 10.0.1.0/24 directly.
For example, with version 3.2 which I used for a while, one has to use the default disk setup (no ZFS or other cool storage tech) with slow 2layer filesystems, no GPU acceleration for applications in VMs, mandatory encrypted backup scheme. Regarding security, Qubes makes some strange choices such giving the regular VM unix user root privileges accessible via simple passwordless sudo.
Qubes runs only on a subset of available hardware (motherboard has to be good enough) so watch out for that. Also interaction with VM manager and graphically intensive applications in VMs was sluggish and internet/firewall/audio would randomly stop working when restarting VMs and require system reboot to fix. Some of these may be better now with newest version 4.x, but I am doubtful.
It's worth mentioning that Qubes has good reason for this.
A big part of the purpose for their existence is to help protect journalists and others who would greatly benefit from enhanced security, but don't know how to get there themselves.
Serious security-seeking user needs to educate himself about how the computer and internet works, about operational security, pervasive tracking, typical attacks etc, not just use a product and call it a day. Only so educated user can decide if the Qubes with its benefits/drawbacks is worth it.
People have work to do, work that has computers as a tool, not a goal. Lowering the bar from "Learn the exploits" to "Be careful and stay in the lines" is a Good Thing, imho.
Qubes provides an effective framework to say "This is how to use it, this is what you should be aware of, don't do this".
Just as a datapoint youtube or vlc seem to work well enough. The sluggishness is definitely noticeable though. As a developer it encourages me to optimize the display performance of my web applications. If it performs ok on Qubes it's liquid smooth on my 3+ year old android phone. Take that as you will...
> internet/firewall/audio would randomly stop working when restarting VMs
I started with 4.X and I haven't noticed this. I never actually used 3.X so I can't say if it's something that was fixed or never present on my hw.
> provides a functional but very tightly-knit product that cannot be easily modified to suit your own needs.
This is a fair assessment. But I've come to realize I need the isolation Qubes provides more than I thought I would. No advanced threats or anything. Just the ability to put email, chat, password manager, and work in separate reproducible environments is really cool. Clicking a url on a chat won't open in the browser I'm signed into google or in my work browser. If you have the need for gpg or tor they offer solutions with a high degree of isolation, though I haven't looked into them much.
As another datapoint, when I installed Qubes about six months ago on a i6600k with 16GB RAM, in the default VM and browser, YouTube was basically unwatchable.
See here for details: https://www.qubes-os.org/doc/vm-sudo/
You can replace passwordless sudo with a secure user prompt: https://www.qubes-os.org/doc/vm-sudo/#replacing-passwordless...
OK, an attacker capable to attack the hypervisor when running as root may be often "game over" if it runs as regular user. But that kind of attack is so rare, why is it the base of the argument at all?
Meanwhile your browser/other big SV application/script from Github can become root whenever it wants, or by accident. It can send contents of the whole VM over the Internet to anybody. Or an honest bug in it can destroy your whole VM. This is why the unix privilege separation or even more capable MAC systems exist. Seriously, running untrusted applications with root access is a braindead idea.
Anyway, Qubes team is not against such kinds of isolation: https://qubes-os.discourse.group/t/isolation-within-the-same....
For that to be true, one would have to run each application in a dedicated VM. A great idea, but Qubes does not recommend that for efficiency reasons. It does pose performance and usability challenges which is why most people do not use it that way. So yes, you do care about applications in single VM not snooping/manipulating each other. Luckily there are standard unix and MAC mechanisms to isolate them.
Also, there are disposable VMs for truly untrusted things.
FWIW, I did the same thing and was unhappy with the result. The Librem hardware wasn't great quality and Qubes really requires a massive desktop to run well. And if you are paranoid enough to need Qubes, then you want ECC memory to defend against side channel attacks.
1. Poor hardware compatibility - due to clinging to Xen (dead/dying)
2. Poor performance (youtube @480p or xfce desktop @4k run at 5fps on a high-end system, don't even try 3D accleration) - due to not treating performance needs seriously. GUI domain and hiring a community manager are signs that they're starting to listen, but it looks like it will take years to land in full.
I pray that Qube's devs are taking this project (and others) as a wake-up call: there is a huge unfulfilled need that people are willing to do a lot to satisfy. Leveraging more modern projects (like crosvm & Wayland) can hopefully help reduce the cost ("evolve or die").
: dozens of mailing list threads reflecting a stance of "it's the users who are wrong"
: https://spectrum-os.org/ "a step towards usable secure computing" (an obvious wink to qubes), https://github.com/nrgaway/qubes-kvm-dev, etc.
Even ultra conservative debian stable (buster, right now) uses something fairly up to date, and xen 4.11. I'd be concerned about using anything older than that.
Works fine for me, except bluetooth (see below). What's the problem with it?
> bluetooth would be handy
A dedicated Audio VM is coming in 4.1: https://www.qubes-os.org/news/2020/03/18/gui-domain/#audio-d.... Related issue: https://github.com/QubesOS/qubes-issues/issues/1590.
If an app has write access to your home folder, it has root. But with flatpak and portals, its realistic that direct home dir access will no longer be a thing for most apps.
I meant that most operating systems using X as the display server are not going to have application sandboxing.
I'm pretty sure the top level comment here was just FUD, although I assume it was well intentioned, just misunderstood.