

KVM and Docker LXC Benchmarking with OpenStack - r4um
http://bodenr.blogspot.com/2014/05/kvm-and-docker-lxc-benchmarking-with.html

======
josteink
I recently tried using both Docker and LXC and I just had to ditch Docker.

If you wanted something as "esoteric" as having normal networking in your
container, you had to understand how Docker abstracted LXC in the first place,
then find out how to do things the LXC way and then push your changes through,
with some extra duct-tape with pipeworks.

And if that's the start, before I've even _done_ anything, I can't imagine
things getting better. So Docker? No thanks.

Then I'll just go ahead and use LXC directly. It involved a bit more research,
but nothing you can't do in a few hours. And when you do, you can get to
benefit from BTRFS volumes and having snapshot management tied directly to
your containers.

Maybe I'm missing something, but after fooling around with this for a few
days, I can't see why anyone would choose the leaky abstraction that is Docker
when you can just go with the real deal.

~~~
BozeWolf
Did you try (redhat's)-libvirt? I used that for a project. It _can_ use lxc,
but also qemu and kvm. All with a generic interface (which actually is not so
generic). It was quite nice to work with.

I guess my project wasn't suited for docker. All i needed was an image to run
a process in a custom environment. I didn't need the Dockerfile and versioning
stuff. Which, in my opinion, is the reason to use docker.

So instead of saying "Docker, no thanks", you probably did not find a suitable
project. Which is good! I see a lot of people using docker, just because it's
awesome and everybody on HN is talking about it.

~~~
josteink
I used plain LXC with BTRFS and for my needs, that was exactly what I was
looking for.

And then I see no need to meddle in meta-frameworks to cover for potential
use-cases I don't have, no matter how well they are engineered or tested.

------
WestCoastJustin
If you are new to the idea of containers vs traditional virtualization, I have
put together a screencast about containers, entitled: "Introduction to
Containers on Linux using LXC". Can be seen @
[http://sysadmincasts.com/episodes/24-introduction-to-
contain...](http://sysadmincasts.com/episodes/24-introduction-to-containers-
on-linux-using-lxc)

~~~
agumonkey
I like your articles. Nice pace and voice, great examples. I'll be watching
your website.

------
justinsb
Interesting benchmarks, ruined for me by a huge experimental no-no: when one
of the tests (the Docker delete) didn't produce the result the author wanted,
he investigated the underlying problem and fixed it. Sure enough, that became
the headline 49x speed-up. Where the results fit the narrative, no
optimization was done/reported.

One thing I'm very interested by is Docker's idea of single-process
containers; but whether this can be done with KVM isolation instead of (weaker
but cheaper) LXC isolation. One thing I was surprised by from this
presentation is that the memory cost of a KVM vs LXC instance was only 185MB
vs 45MB. So KVM's stronger isolation might not be as expensive after all...

~~~
elliotf
Docker does not necessarily have to be single-process containers.

See [https://phusion.github.io/baseimage-
docker/](https://phusion.github.io/baseimage-docker/) as an example of a full-
OS docker base image designed to be lightweight.

These days, you can run the full OS inside docker[1][2], but you'll likely be
running more than you need to.

[1]
[https://github.com/dotcloud/docker/issues/1024](https://github.com/dotcloud/docker/issues/1024)
[2]
[https://github.com/dotcloud/docker/issues/2276](https://github.com/dotcloud/docker/issues/2276)

~~~
justinsb
Absolutely agree, but (at least in my mind) Docker with a full OS really is
just LXC. Running a single process inside the container is something that
genuinely differentiates Docker from other approaches.

It's a powerful idea which to date has been tied to LXC containers, though I
think that's a false association: it's just as interesting with KVM or Xen
containers.

------
rdtsc
One thing to keep in mind is lightweight containers work well for deployment
(isolation), but they can't replace KVM for general virtualization tasks, as
guest kernel has to be the same as the host OS.

If you have your hosts running CentOS and want to deploy on Windows, for
example, or want to migrate some legacy server to the "cloud" this might not
work well for you.

~~~
threeseed
What you've described doesn't sound "general" at all. It sounds like edge
cases.

~~~
josteink
Would you consider it edge-cases to run Linux-instances on a Windows Hyper-V
based server? Would you consider it a edge case running a Windows VM in Linux?

I certainly wouldn't. Those are definitely what I call "general"
virtualization cases.

And as awesome as containers are, they wont be able to cover for these cases.
It's definitely worth mentioning for those who are not familiar with the
matters.

~~~
boden
Agreed -- LXC as a technology is not a silver bullet and will not cover all
use cases. However for those looking to do virt with Linux distro's IMHO the
need to use traditional VMs becomes the edge case moving forward. I provided
some details on the gaps and use cases for traditional VMs sections here:
[http://www.slideshare.net/BodenRussell/realizing-linux-
conta...](http://www.slideshare.net/BodenRussell/realizing-linux-
containerslxc)

------
spullara
Docker stopped using LXC and now uses libcontainer:

[http://www.infoq.com/news/2014/03/docker_0_9](http://www.infoq.com/news/2014/03/docker_0_9)

~~~
nickstinemates
I believe Boden uses LXC to mean 'Linux Containers' the theory, not lxc the
tool/implementation.

In speaking to him he's very well aware about libcontainer.

~~~
boden
Yes correct... As I mentioned below the tests are intended to show the
benefits of an LXC based technology in the Cloud space. Docker was chosen b/c
that's my preferred "LXC provider", but could have just as easily been
libvirt-lxc, OpenVZ, etc.

------
mrmondo
Docker doesn't even use LXC anymore...

~~~
rdtsc
What does it use, do you have a link?

~~~
kelseyhightower
libcontainer - [http://blog.docker.io/2014/03/docker-0-9-introducing-
executi...](http://blog.docker.io/2014/03/docker-0-9-introducing-execution-
drivers-and-libcontainer/)

~~~
jimmcslim
But isn't libcontainer just using the same mechanisms in the Linux kernel as
LXC anyway? It seems a little redundant to me; although I see value in the
'execution driver' approach as I assume they will ultimately deploy Docker on
Solaris and use jails there.

I'm annoyed that the new LXC execution driver appears to have broken something
in the LXC networking; previously I was able to run a Docker container using
LXC networking, give it its own MAC address and have it assigned an IP address
by my router and be visible to all other devices on my network (use case: a
container to run Plex Media Server).

------
jaequery
he should be comparing docker to xen paravirtual or openvz, NOT kvm.

~~~
boden
The intent of the benchmark was to show benefits of LXC as a technology in the
Cloudy space. It just so happened I chose docker as the LXC "provider" for
these tests -- it could have just as easily been libvirt-lxc or something
else.

Given OpenVZ is an LXC technology, comparing docker to OpenVZ would have been
more of a comparison between different LXC providers. I agree this would be
interesting, however not in line with the intent of the tests.. BTW -- I'm
aware that OpenVZ is probably the most mature LXC solution out there.

As for docker vs Xen paravirt, that would be interesting as well and the
information I presented clearly stated additional testing is warranted. I
would encourage others to try such configurations and present results.

~~~
fatherlinnux
Well I, for one, want to thank you for all of this work. I have done some
benchmarks in the past, and they take a ton of work!!! It's easy to sit back
and armchair quarterback about performance, but it's WAY more work to sit down
and do this.

I encourage anybody that has problems with your experiment to go out and
retest the results and share their work!

