
Xen and Docker: Made for Each Other - ferrantim
http://blog.xen.org/index.php/2014/09/08/xen-docker-made-for-each-other/?utm_source=rss&utm_medium=rss&utm_campaign=xen-docker-made-for-each-other&utm_source=twitterfeed&utm_medium=twitter
======
brendangregg
Xen is a great technology, and the community has accomplished much in terms of
features and performance improvements. But I doubt the long term future is
Containers/Docker inside Xen guests, instead of just bare-metal.

Reasons to pick bare-metal:

1\. Price/performance. For many CPU-bound workloads the Xen overheads are
negligible, but for I/O-bound workloads the overheads add up. The network code
path, especially at 10 GbE speeds, is sensitive to any wasted cycles. For
large sites with thousands of instances, these overheads mean less efficiency,
and needing to run more instances, costing more for the business.

Can the future of Xen eliminate (or near eliminate) I/O overheads, so that
their resource cost is near bare-metal? Put differently: will there be a
future where all I/O paths in AWS EC2 access PCI passthrough, or some similar
technology to bypass synchronous overheads? That would be great, but is this
realistic? Bare-metal provides this level of performance today.

2\. Debugability. Many environments are sensitive to latency outliers, and
measure the 95th and higher latency percentiles. Root causing these is
difficult on hardware virtualized environments, where latencies end at the
hypercalls, but easier for bare-metal environments. Even with root access on
both the host and guests, tracing I/O between them can be painstaking work, as
the host can't see and trace inside the guest. (Is the Xen going to solve this
problem? Eg, could I have a SystemTap script trace both guest application-
level I/O requests and associate them with host device level I/O? I suppose it
can be done, but will it?) On bare-metal with containers, the host can see
all, and trace I/O seamlessly from guest processes to bare-metal devices.
Currently.

3\. Density or efficiency. Resources can be shared at a finer level of
granularity with cgroups, than is really practical with virtualized devices
such as CPUs. Unless you are only running one big Xen guest on a host.

Xen (and KVM) are here to stay: there will always be the need for different
kernel support for legacy applications. But for very large sites with
thousands of instances, the future (although it may take a while) for sites
with many instances and are performance sensitive, is likely bare-metal +
Containers/Docker.

If you're a small site, or you care less about performance and debugging, then
there may well be merit in the work this article covers.

~~~
CMCDragonkai
Have you considered using Ubuntu MAAS?

~~~
wmf
He worked for Joyent, so I'm sure he's _considered_ automated bare-metal
deployment tools quite a bit. ;-)

BTW, I tried MaaS recently and it seems to work fine but I could not
successfully customize the installed OS which made it impossible to integrate
into my environment.

------
markbnj
I don't disagree with this post, but neither do I see the point of it. Anyone
who is deploying containers on EC2 is already using the two technologies side-
by-side, not that it means anything earth shattering. Maybe I'm in the
minority here but I continue to not get the constant comparisons between
containers and VMs. They're not the same thing, and they don't solve the same
problems.

I spin up a VM from Amazon and it lets me share an underlying hardware
platform in controlled ways so that I pay only for what I need and can't
affect the other people sharing it. That's the problem VMs solve.

But now that the VM is running I still have to configure it for my
applications. It's just a virtual server, with a potentially large gap between
that state and the state I want it in. That's the problem containers solve.

~~~
singlow
The confusion I see comes from the fact that many people use/used VMs for
things that containers actually work fine for. For example, developers set up
VM with tools like Vagrant and VirtualBox to isolate the dev tools and
services used by a project in a development environment. Docker does replace
some previous use cases for virtual machines, particularly ones that did not
need the security and resource control provided by VMs.

~~~
markbnj
Yep, I agree. VMs encompassed the problems that containers address, whereas
containers do not encompass the problems that VMs are designed to address at
least not in any competitive way. Containers are essentially a finer-grained
way to manage dependencies on a VM than the VM itself is. So yeah, I guess I
do understand the reason why the comparison gets made. I just don't feel it's
a very useful one.

------
PaulHoule
I think of Xen as a dying technology.

Lately IBM passed around a paper aimed at Amazon that showed how badly
paravirtual machines behave compared to bare metal.

What they didn't note is that newer Intel chips have serious support for
hardware virtualization, and that with hardware virtualizaton you get near
bare metal performance. And they didn't note that the new instance types on
AWS support hardware virtualization.

Paravirtualization was an important technology for a while because it was the
fastest sort of virtualization ten years ago, but now it is the slowest.

~~~
sciurus
Paravirtualization bad, hardware virtualization good, is not an accurate view.
You actually want a hybrid model to get the best performance. That's why KVM
developers have put so much effort into the paravirtualized virtio drivers,
for example. Xen has put lots of work into taking advantage of new hardware
virtualization technology where it makes sense and continuing
paravirtualization where it makes sense. See the two links below for great
details.

[http://blog.xen.org/index.php/2012/10/23/the-
paravirtualizat...](http://blog.xen.org/index.php/2012/10/23/the-
paravirtualization-spectrum-part-1-the-ends-of-the-spectrum/)

[http://blog.xen.org/index.php/2012/10/31/the-
paravirtualizat...](http://blog.xen.org/index.php/2012/10/31/the-
paravirtualization-spectrum-part-2-from-poles-to-a-spectrum/)

~~~
click170
Very informative, especially the second page, thanks for linking that!

If you have any other suggested reading re Xen / Virtualization I'd be very
interested in them.

~~~
brendangregg
Try [http://www.brendangregg.com/blog/2014-05-07/what-color-is-
yo...](http://www.brendangregg.com/blog/2014-05-07/what-color-is-your-
xen.html) and [http://www.brendangregg.com/blog/2014-05-09/xen-feature-
dete...](http://www.brendangregg.com/blog/2014-05-09/xen-feature-
detection.html) .

------
nickstinemates
Always a good surprise to see new companies embracing Docker. I'd love to chat
- nick@docker.com

~~~
Ramus9845
You actually think they are going to read this and think, "Oh shit! Nick wants
to chat!"

~~~
CSDude
I met Nick by this way, and my project got published in Docker blog.

------
RRRA
We run each services in an LXC container. Containers are run inside a KVM that
is connected to a bridged interface (one KVM per VLAN).

This setup is very clean and easy to partition, secure and maintain. So yes,
light vs real virtualization can work well together!

------
click170
Site down? It's not loading for me on Desktop or Mobile. From the title it
sounds like an interesting read, I'll have to check back later.

Edit: Mwuhaha.
[http://webcache.googleusercontent.com/search?q=cache:upPfoeO...](http://webcache.googleusercontent.com/search?q=cache:upPfoeOZQacJ:blog.xen.org/index.php/2014/09/08/xen-
docker-made-for-each-
other/%3Futm_source%3Drss%26utm_medium%3Drss%26utm_campaign%3Dxen-docker-made-
for-each-
other%26utm_source%3Dtwitterfeed%26utm_medium%3Dtwitter+&cd=1&hl=en&ct=clnk)

~~~
zobzu
its just really slow:

Xen & Docker: Made for Each Other!

By Sarah Conway – September 8, 2014

By Olivier Lambert

Containers and hypervisors are often seen as competing technologies – enemies
even. But in reality the two technologies are complementary and increasingly
used together by developers and admins. This recent Linux.com article talked
about this supposed battle, noting however that developers are using Docker in
traditional VMs to bolster security. Containers allow users to develop and
deploy a variety of applications with incredible efficiency, while
virtualization eliminates any constraints and/or exposure to outside attacks.

Uniting these technologies helps developers and system administrators be even
more efficient. Let’s take a closer look at how to achieve this with Docker
and Xen Project virtualization, and why we expect more and more organizations
to use them together in the near future. This will also be a key topic at the
September 15 Xen Project User Summit at the Lighthouse Executive Conference
Center in New York City. Register today to learn more about enabling Docker in
Xen environments for a truly open infrastructure.

xen-docker

Caption: Xen In Action: lifting Docker, which is lifting containers. I heard
you like boats, so I put boats on your boat :) .

Who’s Who: What is Xen Project Virtualization?

Xen Project Hypervisor is mature virtualization technology used by many of the
world’s largest cloud providers like AWS, Verizon Terremark, Rackspace and
many more. Founded in 2003, Xen Project virtualization is proven as a highly
reliable, efficient and flexible hypervisor for a range of environments,
running perfectly from x86 to ARM.

It’s now completely integrated in the Linux upstream and is hosted by the
Linux Foundation. The same big cloud users mentioned above also contribute
regularly to the project along with many of the world’s largest technology
companies, including Citrix, Cavium, Intel, Oracle and more.

Feature updates and broader community collaboration are on the upswing too:
more commits, more communication, better integration, new use cases and
simpler and more powerful modes, such as PVHVM then PVH, as outlined in this
recent blog.

The core Xen Project team takes security seriously. The technology has also
been battle-tested by many in the defense industry including the NSA. Xen
Project users have benefited from this for years, and developers building,
shipping and running distributed applications will profit as well.

XenLogoBlackGreen

What is XenServer and Xen Orchestra?

XenServer is a packaged product consisting of the Xen Project Hypervisor and
the Xen Project Management API (XAPI) toolstack within a performance tuned
CentOS distribution. It’s free and can be installed in just a few minutes;
click here to download it: [http://xenserver.org/open-source-virtualization-
download.htm...](http://xenserver.org/open-source-virtualization-
download.html).

Xen Orchestra (XO) is a simple but powerful web interface working out-of-the-
box with XenServer, or any host with Xen and XAPI (the most advanced API for
Xen). Take a look on the project website to learn more. Both of these tool are
of course free software.

What is Docker?

In its own words, Docker defines itself as an open platform for developers and
sysadmins to build, ship, and run distributed applications. Consisting of
Docker Engine, a portable, lightweight runtime and packaging tool, and Docker
Hub, a cloud service for sharing applications and automating workflows, Docker
enables apps to be quickly assembled from components and eliminates the
friction between development, QA, and production environments.

docker-logo-370x290

Main Advantages:

    
    
        fast (boot a container in milliseconds)
        simple to use, even in complex workflows
        light (same kernel)
        container density on one host
    

The other side of the coin:

    
    
        all containers rely on the same kernel (isolation, security)
        less maturity than traditional hypervisor (Docker is still young)
        containers are using the same OS on the host (less diversity than hypervisors)
        some friction between developers and admins about its usage: not Docker’s fault, more a classical friction when you bring new toys to your devs. :) We’ll see why and how to cope with just that below.
    

Best of Both Worlds

An ideal world would:

    
    
        Let admins do their admin stuff without constraints and/or exposure to dangerous things.
        Let developers do their developer stuff without constraints and/or exposure to dangerous things.
    

Fluid Workflow

In other words, they’d be able to create really cool workflows. For example:

    
    
        An admin should be able to easily create a Docker ready VM running in a hypervisor, with the exact amount of resources needed at a given point in time (he knows the total amount of resources, e.g a VM with 2 CPUs and 4GB of RAM.
        He should delegate (with the same simplicity) this Docker-ready VM to the dev team.
        Developers can use it and play with their new toy, without any chance of breaking stuff other than the VM itself. The VM is actually a sandbox, not a jail; developers can create their containers as they need in this scenario.
    

Now you can easily imagine other exciting things such as:

    
    
        An admin can delegate rollback snapshot control to a developer. If he breaks the VM, he can rollback to the “clean” snapshot — without bothering the admin staff. Live, die, repeat!
        Need to clone the same container for other tests? One click in a web interface.
        Need to extend the resources of this current VM? One click, in live.
        Ideally, let a developer create its container from the same web interface.
    

Xen Orchestra: A Bridge Between Docker and Xen Project Hypervisor

So how do we do all this without creating a brand new tool? As you may guess,
the answer is Xen Orchestra, which today achieves much of this. Updates
planned for later this year and 2015 will deliver even more efficiencies
between the two technologies.

What XO Does Today

    
    
        Adjust Resources In Live: You can reduce/raise number of CPUs, RAM, etc., while the VM is running! Doing this, you can grow or reduce the footprint of your Docker VM, without interrupting the service. Check it out in this short video.
        Snapshots and Rollback: Snapshots and rollback in XO are totally operational since XO 3.3. Check out how this works in this feature presentation. Coupled with Docker, this is very helpful. When your fresh Dockerized VM is ready, take a snapshot. Then you can rollback when you want to retrieve this clean state. All with just a few clicks and in a few seconds.
    

Coming Soon

    
    
        Docker-Ready Templates in One Click: This feature will be released this year. In a few words, you can request our template directly from your XO interface, it will be downloaded and operational in your own infrastructure with a Docker listening and ready for action,Iin the resources you choose to allocate (CPU, RAM, Disk). No installation: It works out of the box. Read more in this article.
        ACL and Delegation: The perfect workflow rest upon integration of ACLs in Xen Orchestra is our current priority. In our case, it allows VM delegation for your team using Docker; the VM can be rollbacked or rebooted without asking you. More info. here.
        Docker Control from XO: Because we can get the IP of a VM thanks to its Xen tools, we should be able to send command to the Docker API directly through XO. In this way, you’ll just have to use one interface for Docker AND Xen (at least, for simple Docker operations). And take the best of XO for both: ACLs, visualization etc. This last feature is not in our current roadmap, but will probably pop up early in 2015!
    

We-need-to-go-deeper_inception

Caption: Coming soon — deeper integration between Docker and Xen.

Conclusion

Docker is a really promising and growing technology. With Docker and Xen on
the same team, the two technologies work in tandem to create an extremely
efficient, best-of-breed infrastructure. Finally uniting them in one interface
is a big leap ahead!

Any questions or comments? Go ahead!

By Olivier Lambert, Creator of Xen Orchestra Project

------
zobzu
It seems you could do most of the resource sharing, separation and dynamic
allocation with cgroups in docker (should it be implemented)

XEN mainly provides a stronger separation (or, like, KVM does that, too)

~~~
jpgvm
It's commonly missunderstood but Xen can also do much more then KVM because it
can run systems purely para-virtualised.

This means there is no actual machine abstraction despite it using hardware
accelerated extensions like VT-x and EPT.

The difference in overhead between a fully para-virtualised system and a
process that is running in a container is going to be pretty similar other
than I/O performance (which is a complicated discussion).

If you find this interesting lookup Xen unikernels, a fairly well known one
now is Mirage OS. Which lets you easily write Ocaml applications ontop of a
Xen unikernel.

------
labianchin
Would that be simple as running a CoreOS image inside Xen? Does it make sense?

