
The new hypervisor LXD - tkubacki
http://www.ubuntu.com/cloud/tools/lxd
======
druiid
Is it me or is Ubuntu kind of like the Sony of server software? It always
seems like they are developing concurrent solutions to fit into their model of
doing things.

Where Sony pushes their own special formats like Memory Stick, Ubuntu pushes
Upstart, Juju and now LXD. I think in the end this isn't entirely helpful to
the ecosystem as a whole when you have Ubuntu attempting to push their special
formats of things while not bringing all that much more to the table. The
question of how systemd compares to upstart might be something to consider
with my upstart comparison, but essentially all I see from Ubuntu is pushing
their brand of tooling and most of it is very Ubuntu specific.

On the other hand RedHat usually releases software that can (and usually does)
make it to other distributions and ecosystems. I'm not entirely sure what LXD
is going to bring to the table beyond what Docker or similar utilities offer,
and like many Ubuntu projects it currently seems to be VERY light on
documentation. Actually, where do I even access the documentation for this
project? Oversights like this are what killed Juju or MAAS for me and yet
Ubuntu pushes those projects like crazy at every conference I've seen them at
(Gophercon for instance).

~~~
vezzy-fnord
Upstart seems like an odd example to give, given that it was the first
sysvinit alternative that actually made it into major distributions. RHEL 6
(thus CentOS 6), ChromeOS and some of the earlier Fedora versions make use of
it. This was after an attempt to port launchd failed, because of licensing
issues. systemd came several years later.

Red Hat is a different business model. Their venturing into the cloud is more
recent. Historically, they've been more into the support business, and this
necessitated having a lot of people fix bugs in the Linux ecosystem. That and
their acquisition of Cygnus Solutions means they're the _de facto_ gatekeepers
of the Linux kernel and much of userspace.

Canonical is a more Apple-like company. They care about being internally
consistent and formulating their own brand, interacting with the outside only
where necessary.

~~~
druiid
Well I'm aware it made it into CentOS but that was kind of short lived and was
never updated to newer releases of the codebase (thus making it kind of a init
system floating around in limbo with compatibility issues.. many of which I've
personally encountered). I understand what you're saying about RedHat vs
Canonical, but tooling that is as critical as a containerization or init
system don't work very well (IMHO) in a vacuum. You have to have it more
widely available, useable and used by the community or not only can you not
build community, but you end up creating a fractured ecosystem that makes it
hard to tool for among other things.

------
bcantrill
I am, like many here, totally confused. Is this OS-based virtualization, HW-
based virtualization, para-virtualization, or something completely different?
On the one hand, there are clear indicators that this is OS-based
virtualization ("there is a catch; however, LXD is only for Linux on Linux").
That's fine; that would essentially boil down to bringing the complete
containment model of FreeBSD jails and illumos zones to Linux -- which I'm
sure would be welcome by Linux folks dealing with the structural problems of
the current (piece-meal) Linux containment model.

But then there's the line about "working with silicon companies to ensure
hardware-assisted security and isolation for these containers" \-- WTF?! If
using OS-based virtualization, why would you need hardware assistance for
"security and isolation"?! And if that "hardware assistance" is being used for
something so basic as proper containment, what happens if you don't have that
assistance? Is LXD then vulnerable to privilege escalation? And who are the
"silicon companies" we're talking about (like, is this Intel or is this not
Intel?), what is the ISA, when does it tape out, how is it being validated,
etc. etc. etc.

It's very frustrating for an announcement to be so putatively technical and
yet provide so few answers; is there a deeper technical explanation of LXD
somewhere?

~~~
derefr
> WTF?! If using OS-based virtualization, why would you need hardware
> assistance for "security and isolation"?!

I would guess that Canonical is talking about getting companies to contribute
Linux kernel patches for cgroup interfaces to various northbridge-managed
hardware virtualization tech (e.g. IOMMU tech like Intel's VT-d.)

~~~
robryk
How would you use hardware virtualization tech (other than a normal MMU) to
separate processes that run on the same kernel from each other?

~~~
derefr
VT-d in particular gives you the ability to expose one piece of hardware (that
knows how to partition itself in some way) as multiple devices on the PCIe
bus. With cgroup support, a container could be assigned one of the split
devices, and act within the container as if it were the whole device. This is
what regular hypervisors do, but they require a full set of virtualized
devices (a virtual CPU, a virtual memory, etc.) while this approach allows you
to virtualize only the resources your containers actually want to contend
over.

So you could have, say, one virtual ethernet card per container (letting you
run a container as a promiscuous-mode packet filter for its own VPC subnet,
while still not being able to snoop on other VPCs' traffic) or one virtual GPU
per container (allowing you to containerize OpenCL apps), while still having
your containers acting like regular processes otherwise.

~~~
robryk
In order to get one virtual slice of device per container I just need the
device's driver to support that or I need to have a layer on top of the driver
that partitions the device. As the different cgroups have the same kernel and
thus the same set of drivers I see no advantage in splitting physical devices.
What am I missing?

~~~
justincormack
The master device has the facility to create the subdevices which look like
onrmal devices with soem config options missing.

I am not convinced that this is what they are talking about though...

------
comex
> The new hypervisor isn’t a hypervisor

> And it’s going to be a real hypervisor?

> Yes. We’re working with silicon companies to ensure hardware-assisted
> security and isolation for these containers, just like virtual machines
> today. We’re working to ensure that the kernel security cross-section for
> individual containers can be tightened up for each specific workload.

Sorry, but WTF? Is it a hypervisor or not? From a security perspective, one
kernel per container or LXC? If the latter, as the rest of the announcement
seems to imply, what is the "work with silicon companies" about? Either
compromising Linux allows you to get access to other containers on the
machine, or it doesn't. It can't be both.

------
xorcist
What? My first thought was "cool, Ubuntu backs LXC". My second was
"waitaminute, Ubuntu actually wants to compete with Docker?". Docker, as you
all know, is a farily well established LXC management solution.

They then go on to state that LXC is a "real hypervisor" with live migrations
and such. What? Did they take an established Linux household name, with
wikipedia article and everything, and name their new semi-related project
identically?

And if it's a para-virtualized solution they're pushing, are they really
competing against Xen? I'm not sure it's wiser than competing with Docker.

Anyone from Canonical here and can explain what's going on?

~~~
sp332
Docker, well-established? It may get a lot of press but it's not even two
years old. It was just merged into OpenStack about 1 year ago.

Edit: You know Docker doesn't use LXC by default anymore, right? It uses a
different container library called libcontainer.

~~~
nl
And yet oddly for something so young, it's seeing more usage than OpenStack.
Of course that may have more to do with the problems within OpenStack, but
Docker has it's own attractions too.

~~~
peterwwillis
They have waaaaaaay more marketing than OpenStack. They've probably written
and sponsored more PR fluff than code for Docker. And it's easier to deploy.

~~~
shykes
I agree that Docker is easier to deploy than Openstack :) However...

It always astounds me how some people _massively_ over-estimate the size and
influence of Docker's marketing... Why yes, of course! The way we got Google,
Microsoft, Amazon and IBM to integrate it in their products is by ghost-
writing PR fluff. That's also how we got 600 people to contribute 9,000+ pull
requests over 18 months [1] [2] [3]. Not bad for marketing monkeys!

Seriously - after seeing so many hackers work so hard to improve the project
every day, the "it's all marketing fluff" crap always gets to me. It's just
plain disrespectful. How much legitimate engineering work do you need to see
before you start respecting other people's work?

[1]
[https://github.com/docker/docker/pulse/monthly](https://github.com/docker/docker/pulse/monthly)

[2]
[https://github.com/docker/libcontainer/pulse/monthly](https://github.com/docker/libcontainer/pulse/monthly)

[3] [https://github.com/docker/docker-
registry/pulse/monthly](https://github.com/docker/docker-
registry/pulse/monthly)

~~~
peterwwillis
I have two opinions: logical, and emotional.

My logical opinion says that Docker is a useful tool which, though flawed in
many ways, provides real value to a large number of users. I've even
recommended Docker be used for new projects in my company, on the basis that
it fits in well with what we're trying to do.

My emotional opinion is that Docker trades on trends in startup-world, systems
engineering and the open source movement for the sole purpose of eventually
generating revenue. This capitalistic perversion of what were before two
idealistic and noble things (open source, engineering) is, quite honestly,
abhorrent to me.

So to answer your question: while I might eventually respect its engineering
accomplishments, I despise it on principle. I hope one day it turns into a
simple useful tool that people can decide to use or not use without being
cajoled by developer evangelists.

~~~
zenlikethat
> My emotional opinion is that Docker trades on trends in startup-world,
> systems engineering and the open source movement for the sole purpose of
> eventually generating revenue. This capitalistic perversion of what were
> before two idealistic and noble things (open source, engineering) is, quite
> honestly, abhorrent to me.

Except that the last thing anyone with a monetary stake in your business will
do is tell you to open source your main product. The Docker project has fought
damn hard, and continues to, to make sure that a carefully curated line
between business and the open source project exists (see for instance the
creation of the Docker Governance Advisory Board).

------
wernerb
LXD will be pushed upstream into LXC (to someday replace lxc) at
github.com/lxc/lxd [0]

[0]: [https://lists.linuxcontainers.org/pipermail/lxc-
users/2014-N...](https://lists.linuxcontainers.org/pipermail/lxc-
users/2014-November/007980.html)

------
hosh
I'd like to read up on the technical docs. At first, it looks like a Docker
rival. One of the concerns I have about LXC is that it isn't using hardware
paravirtualization. It _seems_ like that is what Ubuntu is trying to do with
LXD (that is, LXD provides a hypervisor backend for LXC), but I am not sure.
If this, then that's a much more compelling argument for me to use LXC over
something like VirtualBox or ESX, or whatever.

Then again, maybe I don't understand how LXC work at all.

~~~
otterley
You may be waiting a while. Ubuntu doesn't produce useful technical
documentation like Red Hat and Oracle (neé Sun) does.

------
pella
short video ( 2:07)

"LXD - the Linux Container Daemon"
[https://www.youtube.com/watch?v=U-lXf85Mhno](https://www.youtube.com/watch?v=U-lXf85Mhno)

''"Published on 4 Nov 2014 Dustin Kirkland, Product Manager at Canonical
introduces LXD (lex-dee), a new hypervisor that delivers capabilities to LXC
containers that cloud users demand in scale out infrastructure. LXD is a
persistent system daemon developed to enable the secure management and live
migration of LXC (lex-cee) containers via an easy to use command line
interface and REST API."''

~~~
sciurus
That summary makes LXD sound like a competitor to libvirt-lxc.

[http://libvirt.org/drvlxc.html](http://libvirt.org/drvlxc.html)

------
pella
more info from Stéphane Graber Ubuntu developer:

\----------

'"

...

The concept is relatively simple, it's a daemon exporting an authenticated
REST API both locally over a unix socket and over the network using https.
There are then two clients for this daemon, one is an openstack plugin, the
other a standalone command line tool. ''

The main features and I'm sure I'll be forgetting some are:

\- Secure by default (unprivileged containers, apparmor, seccomp, ...)

\- Image based workflow (no more locally built rootfs)

\- Support for online snapshotting, including running state (with CRIU)

\- Support for live migration

\- A simpler command line experience

This work will be done in Go, using the great go-lxc binding from S.Çağlar.

Now as to what this means for LXC upstream:

\- A new project will be setup at
[https://github.com/lxc/lxd](https://github.com/lxc/lxd) .

\- Code to this project will be contributed under an Apache2 license, no CLA
is required but we will require contributors to Sign-off on their commits as
always (DCO).

\- Discussions about lxd will happen on lxc-devel and lxc-users.

\- Contributions to github.com/lxc/lxd will happen through github pull
requests only and reviews will happen on github too.

This is kept separate from the main tree because at least initially, I believe
it best to have a separate release schedule for both of those and because it
tends to be easier for Go-only projects to live in theirown branch.

...

In order to be a good hypervisor, we also need to make containers feel like
they are their own system and so we'll be spending quite a bit of time
figuring out how to improve the situation. Some of the work presented at Linux
Plumbers is going to contribute to that, like cgmanagerfs to provide a
reasonable view of /proc and a fake cgroupfs, Seth's unprivileged FUSE mounts
and all the cool things mentioned in Serge's earlier post about

Now as for the next steps. We will be creating the repository on github over
the next few hours with Serge and I as the initial maintainers. Once the
project is properly started and active, we will promote some of the most
active contributors to commiters.

The first few commits in there will be text versions of the specifications we
came up with until now. This should also serve as a good todo list for people
who want to get involved.

Over the next few days/weeks, the existing code which was used for the demo at
the OpenStack summit in Paris will be submitted through pull requests,
reviewed and merge.

...

"

see more: [https://lists.linuxcontainers.org/pipermail/lxc-
devel/2014-N...](https://lists.linuxcontainers.org/pipermail/lxc-
devel/2014-November/010817.html)

and check the thread here :

[lxc-users]: [https://lists.linuxcontainers.org/pipermail/lxc-
users/2014-N...](https://lists.linuxcontainers.org/pipermail/lxc-
users/2014-November/thread.html)

[lxc-devel]: [https://lists.linuxcontainers.org/pipermail/lxc-
devel/2014-N...](https://lists.linuxcontainers.org/pipermail/lxc-
devel/2014-November/thread.html#start)

~~~
bcantrill
Thank you; that's very helpful. And the tl;dr is that this whole announcement
is describing something that doesn't really exist yet -- it's open source
vaporware.

~~~
pella
'"LXD ... will be ready for production use within six months."'

(source: [http://www.zdnet.com/ubuntu-is-working-on-a-new-secure-
conta...](http://www.zdnet.com/ubuntu-is-working-on-a-new-secure-container-
hypervisor-lxd-7000035402/) )

~~~
PedroBatista
So are they doing a code dump or will develop a production ready hypervisor in
less than 6 months?

------
teejmya
> On bare metal, these containers are just as fast as the native OS on bare
> metal.

This press release seems to have been writen in 5 minutes.

------
Rapzid
To me the most interesting things here are that the kernel(and CPU's) will be
getting hardware assisted process isolation as well as advancements being made
around CRIU to support live migrating processes(and potentially entire cgroup
trees?).

This is good for everyone. Docker doesn't even use liblxc anymore by default,
it uses libcontainer. Wonder why Ubuntu isn't getting behind libcontainer.. In
any case, the stuff being pushed to upstream projects, like the kernel, will
flow back down to docker and everybody can enjoy new awesomeness.

~~~
josteink
> Docker doesn't even use liblxc anymore by default, it uses libcontainer.
> Wonder why Ubuntu isn't getting behind libcontainer.

Because not everyone wants what docker offers. Some people prefer and want the
more VM-esque behaviour provided by LXC.

To me LXC is the real deal, while docker offers limited convenience at the
cost of flexibility and platform lock-in. And I have zero interest in that.

I hope Ubuntu continues to offer good LXC-support, and then docker (or
whatever the other hip thing of the month is) can do whatever docker does,
because it's external to whatever distro people is running.

~~~
Rapzid
> Because not everyone wants what docker offers... Yes, I can understand this,
> but more specifically? That's hardly in the way of reasons.

> To me LXC is the real deal, while docker offers limited convenience at the
> cost of flexibility and platform lock-in. I'm not really sure what makes
> liblxc the "real deal" and "libcontainer" not? Would you care to expand on
> this though? The true flexibility you are alluding to is, I believe,
> provided by the kernel itself? Are you deriding libraries that abstract
> interfacing with these features? Where is the platform lock-in coming from?
> Docker has been making inroads into many non-linux platforms, even Windows
> recently.

> and then docker (or whatever the other hip thing of the month is) Are you
> suggesting Docker is a "flavour of the month"? That's... A unique
> perspective. In any case, as a counter point, I'd like to offer up that
> RedHat itself has partnered with docker via OpenShift. If one were looking
> for linux flavour of the month, I RedHat would be the LAST place they would
> look.

------
api
Containers are far, far more efficient. The superficial benchmarks that
suggest that the difference is small are misleading.

At a data center where I worked a while back I saw thousands of VZ containers
on boxes that could only manage maybe sixty KVMs. If the issues around
security and flexibility can be fixed, there is opportunity for orders of
magnitude improvements in density and power utilization.

------
therealmarv
Sounds like the right mixture of security and usage (cmdline tools like in
Docker) and LXC. Does anybody have any recommendations on how to manage LXC
before LXD? LXD can't be the first approach to make usage of LXC and manage
connections easy. Played with LXC, liked the concept and hated the iptables
NAT thing... not really user friendly. So kudos to LXD!

------
mrbill
I've got a Debian system that I've got a couple of Debian VMs running on under
qemu-kvm. I don't want Docker-style containers for specific apps; I need "full
system image" virtualization - I wonder if this would be helpful for my use
case?

Running the same kernel/version of OS is fine across my host system and the
guests.

------
cm-t
Some additionnal info:

* [https://news.ycombinator.com/item?id=8567020](https://news.ycombinator.com/item?id=8567020)

* [http://blog.dustinkirkland.com/2014/11/where-were-going-with...](http://blog.dustinkirkland.com/2014/11/where-were-going-with-lxd.html)

------
amaks
I'm wondering if LXD is based on the ideas from Stanford's Dune project
([http://dune.scs.stanford.edu/](http://dune.scs.stanford.edu/)) to virtualize
Linux processes.

~~~
kozyraki
Dune and its follow-up (IX, see
[http://csl.stanford.edu/~christos/publications/2014.ix.osdi....](http://csl.stanford.edu/~christos/publications/2014.ix.osdi.pdf))
can be used to secure and accelerate containers using Intel virtualization
hardware. It'd be great if the community puts some energy behind these ideas.

------
sauere
> hardware-guaranteed security to ensure that those machines can’t pry or spy
> on one another

Please stop using container/virtualization as a security mechanism. It never
worked, i never will work.

------
tronical
Does anyone know where to find the source code?

~~~
pella
''"A new project will be setup at github.com/lxc/lxd."''

( source:
[http://osdir.com/ml/general/2014-11/msg06783.html](http://osdir.com/ml/general/2014-11/msg06783.html)
)

------
rinon
I'd be extremely interested to see if this brings greater security to the
"container" model. Something resembling BSD jails is sorely lacking in the
Linux world.

~~~
cowabunga
I'm a BSD guy and have pretty shallow knowledge of Linux but isn't that what
LXC is?

