I've seen people run a Linux firewall in runlevel 0 (this is 100% possible if you disable the shutdown script) where the entire user space was disabled and only the netfilter stuff running in memory was active. This MASSIVELY limits the attack vectors to only the kernel itself and virtually nothing in user space while giving you a full Linux feature set, something I'm guessing Qubes doesn't yet do. I learned about this technique in the late 90s where a guy had 2 firewalls in active / failover mode using I believe LVS (also in kernel) with the sentry firewall cd (http://sentry.sourceforge.net). There was out of band VPN into the network which allowed serial access to reboot them but once the ACLs were burned onto the CD, it was as immutable as it gets with no serious way to harm anything. Besides it being slightly harder to manage, I can't believe no one else has tried it and written about it.
Sorry for getting side tracked, this looks absolutely fascinating and is another great use case for Qubes. The world of unikernels and containers / lightweight vms is clearly converging for immutable infra.
I've been recently looking into Javascript based Unikernels. I have found 2 that look very promising.
NodeOS:
Takes the linux kernel and strips it to its bare minimum then adds Node/V8 running in kernel space for application support. No C applications, no POSIX utilities. POSIX equivalents (written in JS) are provided for only the most necessary tools.
To build a server, you develop it as normal. Then, the build step packages your app with the kernel to be run as a standalone VM image. The OS is essentially immutable and since it contains no 'general purpose' utilities, the attack surface is limited to your application.
For example, if you need a load balancer. You could import a Node load balancer implementation as a dependency, add a custom config, and generate the VM image for deployment.
The image itself only requires a few MBs in space and can be booted up in seconds so versioning/deployment becomes trivial.
Runtime.js:
Takes things even further. Instead of using a thin linux kernel, the whole OS is implemented in JS. They're still building out the basic low-level infrastructure (ex networking protocols, filesystem adapters, etc). Regardless, it's fascinating to see somebody building an OS from scratch in a language that generally isn't known for system-level development.
The performance improvements to be gained from eliminating all of the OS-level protections negate the performance disadvantages of using a higher level language. Eliminating all of the complexity that comes with monolithic, general purpose operating systems dramatically reduces the attack surface.
That's just Node.js. There are plenty of Unikernels being developed for other languages. Since the cloud operates on virtual machine infrastructures, with mostly single-purpose applications, it makes sense to build single-purpose operating systems that can be deployed to the cloud.
runlevel 0 is not really a thing from the kernel point of view, it is a construct if init. You can do the same by just having nothing in userspace other than the commands for config.
That doesn't do it justice and there's more to it than that e.g. the readability (and 'debugablity') of the new code.
I note that the post does have a very useful summary at the end, which is a 90 sec read (there's even a ToC at the beginning that can take you straight there). I've copied it below in case that's useful.
"QubesOS provides a desktop environment made from multiple virtual machines, isolated using Xen. It runs the network drivers (which it doesn’t trust) in a Linux “NetVM”, which it assumes may be compromised, and places a “FirewallVM” between that and the VMs running user applications. This design is intended to protect users from malicious or buggy network drivers.
However, the Linux kernel code running in FirewallVM is written with the assumption that NetVM is trustworthy. It is fairly likely that a compromised NetVM could successfully attack FirewallVM. Since both FirewallVM and the client VMs all run Linux, it is likely that the same exploit would then allow the client VMs to be compromised too.
I used MirageOS to write a replacement FirewallVM in OCaml. The new virtual machine contains almost no C code (little more than malloc, printk, the OCaml GC and libm), and should therefore avoid problems such as the unchecked array bounds problem that recently affected the Qubes firewall. It also uses less than a tenth of the minimum memory of the Linux FirewallVM, boots several times faster, and when it starts handling network traffic it is already fully configured, avoiding e.g. any race setting up firewalls or DNS forwarding.
The code is around 1000 lines of OCaml, and makes it easy to follow the progress of a network frame from the point where the network driver reads it from a Xen shared memory ring, through the Ethernet handling, to the IP firewall code, to the user firewall policy, and then finally to the shared memory ring of the output interface.
The code has only been lightly tested (I’ve just started using it as the FirewallVM on my main laptop), but will hopefully prove easy to extend (and, if necessary, debug)."
Great work. That's doing it right. Another good one tried in separation kernels was safer runtimes for imperative languages like Ada. A networking stack and firewall in one like that would be an interesting experiment as well.
QubesOS uses a Linux system with 300MB of RAM for the firewall? I just installed OpenWRT 15.05 Chaos Calmer on my router, with 32MB of RAM and 4MB of flash, and almost 1MB to spare. There are some differences between QubesOS and OpenWRT, but that still seems to be excessive by an order of magnitude.
I don't think Firewall VM is trimmed in any way, compared to their standard VMs, so it's running a lot of "cruft" [1] - Xorg, systemd and friends, extra Qubes plumbing. It all adds up to ~200MB, and then the rest is used by buffers/cache.
[1] at least on 3.1-RCx I have running, where I had to create Firewall VM manually, as installer has failed to deploy those "service VMs".
Sorry for getting side tracked, this looks absolutely fascinating and is another great use case for Qubes. The world of unikernels and containers / lightweight vms is clearly converging for immutable infra.