
Ways we harden our KVM hypervisor at Google Cloud: Security in plaintext - EvgeniyZh
https://cloudplatform.googleblog.com/2017/01/7-ways-we-harden-our-KVM-hypervisor-at-Google-Cloud-security-in-plaintext.html
======
dkarapetyan
> We have built an extensive set of proprietary fuzzing tools for KVM. We also
> do a thorough code review looking specifically for security issues each time
> we adopt a new feature or version of KVM. As a result, we've found many
> vulnerabilities in KVM over the past three years. About half of our
> discoveries come from code review and about half come from our fuzzers.

Why aren't these tools being open sourced?

~~~
devoply
Maybe because they think it's a competitive advantage.

~~~
SEJeff
More likely it is just heavily integrated with internal google tooling like
chubby and borg.

I suspect they still use American Fuzzy Lop[1] for much of their fuzzing even
if they wrote some of it from scratch. The Linux Foundation's Core
Infrastructure Initiative's[2] "Fuzzing Project" is more or less continuous
integration running AFL[3]

[1] [http://lcamtuf.coredump.cx/afl/](http://lcamtuf.coredump.cx/afl/)

[2]
[https://www.coreinfrastructure.org/grants](https://www.coreinfrastructure.org/grants)

[3] [https://fuzzing-project.org/](https://fuzzing-project.org/)

------
nealmueller
The authors have noteworthy bona fides. Andy used to work for the NSA, and
Nelly has 16 patents with names like "Informed implicit enrollment and
identification".

[https://www.linkedin.com/in/andrew-
honig-82239750](https://www.linkedin.com/in/andrew-honig-82239750)
[https://www.linkedin.com/in/nelly-
porter-708a10b](https://www.linkedin.com/in/nelly-porter-708a10b)

------
Scaevolus
Google Compute Engine has live migration, so host/hypervisor updates
(including security updates!) can be applied without taking VMs offline. AWS
doesn't support live migration. Azure has a partial solution with in-place
migrations, which involves taking VMs down for ~30 seconds.

(I work on Google Container Engine.)

~~~
kim0
Can you explain the azure solution some more please, to the best of your
knowledge

~~~
Scaevolus
This links to the announcement and has some speculation about how it's
implemented. It looks like it might be basically a snapshot and a fast host
reboot? [https://www.petri.com/new-azure-iaas-features-announced-
augu...](https://www.petri.com/new-azure-iaas-features-announced-august-2016)

------
throwaway7767
> Non-QEMU implementation: Google does not use QEMU, the user-space virtual
> machine monitor and hardware emulation. Instead, we wrote our own user-space
> virtual machine monitor that has the following security advantages over
> QEMU: [...] No history of security problems. QEMU has a long track record of
> security bugs, such as VENOM, and it's unclear what vulnerabilities may
> still be lurking in the code.

Has this alternative VMM/hardware emulator been released? As far as I can
tell, the answer is no. In that light, it's more than a little weird to
congratulate yourself on not having a "long track record of security bugs" in
your internal-use-only unreleased tool compared to generally available
software in wide use.

------
Skunkleton
I see that google has open sourced a VMM written in go [1], but I doubt this
is the VMM discussed in the article. Has google open sourced this software?

[1] [https://github.com/google/novm](https://github.com/google/novm)

~~~
andyhonig
Google has not open sourced this for a variety of technical and business
reasons. I'm not going to get into all of them, but for example a lot of our
device emulation is built upon internal Google services. It wouldn't work
outside a Google data center. It's not out of the realm of possibility that
Google open sources some parts of it in the future, but I don't know if that
will ever happen.

~~~
jsolson
Adding to Andy's point here -- if you consider the services that back GCE in
terms of the features offered by our virtual networks, virtual storage
devices, etc. they necessarily have tight dependencies on "How Google is
Built" to hit the performance targets we aim for.

I, personally, am optimistic that we'll be able to do it for the core at some
point. Something not mentioned in the original article is that architecturally
our VMM allows us to easily change the set of available functionality (virtual
devices, backing implementations of those, etc.). So, for example, the fuzzing
Andy described as depending on our custom VMM isn't even linked into the VMM
run in GCE, nor are custom devices we keep around for testing and development.
I'd very much like it if we could layer the VMM in such a way that we could
release a useful core that forms the basis of the VMM we consume internally.
Today, though, the layering just isn't quite there (among other technical and
business reasons, as Andy says), and there are Googley specifics in places
that would make even the VMM core (minus devices, etc.) unviable outside of
Google.

------
nellydpa
Hey folks, thanks for great comments, keep them coming.I am one of the article
authors, AMA.

------
koolba
> Non-QEMU implementation: Google does not use QEMU, the user-space virtual
> machine monitor and hardware emulation. Instead, we wrote our own user-space
> virtual machine monitor that has the following security advantages over
> QEMU: Simple host and guest architecture support matrix. QEMU supports a
> large matrix of host and guest architectures, along with different modes and
> devices that significantly increase complexity.

I've noticed that a lot of projects that _do_ support multiple architectures,
particularly obscure ones, tend to find oddball edge cases more easily than
those that don't. For example, not assuming the endianness of the CPU arch
forces you to get network code to cooperate well.

> Because we support a single architecture and a relatively small number of
> devices, our emulator is much simpler.

No doubt it's simpler than QEMU but I wonder if adding tests to QEMU, even if
they're only for the specific architectures they're running (most likely
x86-64) would have been just as usable.

Then again when you're Google and have the resources to build a VM runtime
from the ground up it's easier to convince management that " _This is the
right decision!_ ".

~~~
luu
> I wonder if adding tests to QEMU, even if they're only for the specific
> architectures they're running (most likely x86-64) would have been just as
> usable.

> Then again when you're Google and have the resources to build a VM runtime
> from the ground up it's easier to convince management that "This is the
> right decision!".

Is it possible you're either underestimating the effort it takes to make QEMU
solid or overestimating the effort it takes to write an emulator?

I worked at a company where I hacked up QEMU as a stopgap before we switched
to an in-house solution (this wasn't Google, although I've also worked at
Google). I made literally hundreds of bug fixes to get QEMU into what was, for
us, a barely usable state and then someone else wrote a solution from scratch
in maybe a month and a half or two months. I doubt I could have gotten QEMU
into the state we needed in a couple months. And to be clear, when I say bug
fixes, I don't mean features or things that could possibly arguably be
"working as intended", I mean bugs like "instruction X does the wrong thing
instead of doing what an actual CPU does".

BTW, I don't mean to knock QEMU. It's great for what it is, but it's often the
case that a special purpose piece of software tailored for a specific usecase
is less effort than making a very general framework suitable for the same
usecase. Even for our usecase, where QEMU was a very bad fit, the existence of
QEMU let us get an MVP up in a week; I applied critical fixes to our hacked up
QEMU while someone worked on the real solution, which gave us a two month head
start over just writing something from scratch. But the effort it would have
taken to make QEMU production worthy for us didn't seem worth it.

~~~
walterbell
Are there open-source efforts to create a better emulator?

~~~
wmf
Intel is working on QEMU-lite and there are some QEMU alternatives that aren't
focused on emulating the full PC architecture like novm and kvmtool.

------
andyhonig
I'm one of the authors of this post, AMA.

~~~
mjg59
Right now there doesn't seem to have been a lot of work in exposing security
features to guests. Is there interest in supporting stuff like Secure Boot or
any form of runtime attestation?

~~~
bonzini
QEMU + KVM + OVMF supports Secure Boot (including SMM emulation for the
trusted base), or are you talking specifically of GCE?

~~~
mjg59
Yeah, specifically GCE

------
fulafel
This sounds much more secure than AWS and their use of Xen.

~~~
QUFB
How so? What leads you to believe that Amazon hasn't taken similar measures?

~~~
numbsafari
Amazon isn't confident enough in their solution such that they will only
support customers with high levels of regulatory and compliance requirements
(e.g. HIPAA) if they are using dedicated physical hosts.

[https://aws.amazon.com/ec2/dedicated-
hosts/](https://aws.amazon.com/ec2/dedicated-hosts/)

[https://aws.amazon.com/blogs/security/frequently-asked-
quest...](https://aws.amazon.com/blogs/security/frequently-asked-questions-
about-hipaa-compliance-in-the-aws-cloud-part-two/)

~~~
nnx
Does Google Cloud support HIPAA workloads on regular shared instances?

I thought HIPAA itself mandates physical dedicated host separation.

~~~
vgt
Yup:

[https://cloud.google.com/security/compliance](https://cloud.google.com/security/compliance)

------
jron
Does anyone have a better summary of the Xen vs KVM security debate? The best
I know of is the brief Qubes architecture document from 2010:
[https://www.qubes-
os.org/attachment/wiki/QubesArchitecture/a...](https://www.qubes-
os.org/attachment/wiki/QubesArchitecture/arch-spec-0.3.pdf) (Section 3).

KVM using Google's hardened device emulation may eliminate some of Qube's QEMU
concerns.

Qubes author critique on the state of Xen security:
[https://lists.xen.org/archives/html/xen-
devel/2015-11/msg006...](https://lists.xen.org/archives/html/xen-
devel/2015-11/msg00601.html)

------
neom
If there is a DigitalOcean engineer in the house, I'd be curious to know how
this compares to how they run their KVM setup.

~~~
QUFB
Given that it took _4 years_ of continuous customer requests just to get
support for booting a custom kernel on Digital Ocean, I'd be surprised if
they're doing anything this elaborate.

~~~
bitexploder
This is a great example of the advantages only a tiny number of technical
companies have. It is great technology and great that Google can do this, but
who has the resources to competently implement a hypervisor and customize it
to their infrastructure?

~~~
moreorless
Vultr supported custom ISOs for a long long time now.

~~~
regularfry
Among several others. DO are going for the 80% case, not full spectrum
capability.

------
madez
> Alerts are generated if any projects that cause an unusual number of
> Rowhammer errors.

Is that correct english?

~~~
mcinjosh
Almost; I think it parses correctly by substituting 'if' with 'for'.

~~~
nealmueller
Thanks for catching. Fixed.

------
floatboth
By the way, FreeBSD's vmm doesn't use QEMU, a fresh new userspace part (bhyve)
was written. I think the same is happening with OpenBSD's vmm. It seems kinda
weird that Linux KVM is commonly used with QEMU…

------
ascotan
Q. How do you secure your infrastructure Google? A. We hire thousands of
programmers to write custom software. There are no known vulnerabilities. No.
Known. Vulnerabilities.

~~~
aiiane
That's not even an accurate representation of what is publicly claimed; Google
runs a VRP specifically because it acknowledges that software sometimes has
bugs and it's committed to fixing them. Google also employs quite a few of the
security researchers that contribute to identifying and fixing security issues
both within and beyond Google.

