

LXC Networking introduction - gyre007
http://containerops.org/2013/11/19/lxc-networking/

======
rdtsc
LXC is interesting. It has been around probably for 5 or 7 years or so. At
least I remember looking at it then. Then it just kind of made slow progress.
It has been in the kernel since probably since 2.6.30, for example. But aside
from kernel and virtualization forum discussions, it just didn't get much
attention. That was kind of odd because it is a very cool piece of technology.

Then I guess something happened in the last year or so and all of the sudden
it became very popular. dotCloud certainly has a lot to do with it. Was it
better libvirt integration, too, perhaps:
[http://libvirt.org/drvlxc.html](http://libvirt.org/drvlxc.html)

Now of course it cannot completely replace KVM because it is more of a
container than a full virtualization. So running Windows VMs on will not work.
For Linux one could probably have a farm of hosts based on various distros
(with LXC enabled) and that would provide the ability to run various Linux OS
guests, by picking the hosts that matches it.

~~~
maccam94
I think the main reason it didn't take off until recently was because the
kernel namespacing[0] work wasn't complete. Then, even when user namespacing
was mostly complete, XFS still wasn't compatible until 3.12 [1].

[0] [http://lwn.net/Articles/531114/](http://lwn.net/Articles/531114/)

[1]
[http://www.phoronix.com/scan.php?page=news_item&px=MTQ1Nzc](http://www.phoronix.com/scan.php?page=news_item&px=MTQ1Nzc)

~~~
rodgerd
Early implementations were quite buggy; I tried it in 2010 after a
presentation at linux.conf.au and found that the tools simply segfaulted with
monotonous regularity on a then up-to-date environment. It's made significant
strides since then, obviously.

------
droopybuns
Thanks for sharing this link.

LXC has gotten me very excited about testing new linux services on a home box
again. I always worry about exposing a server to the internet with any new
services. The idea that a compromise on the box could leak everything that's
on the box usually leads me to avoid exploring new services.

LXCs seem to give me hope that I could experience a compromise, but not lose
everything.

~~~
nisa
LXC is not secure at the moment. root in a LXC container can lead to root on
the host. There is unfortunately no good summary of the problems - here is my
list (take it with a grain of salt - a lot of these problems are mitigated in
docker.io and with AppArmor in Ubuntu):

\- Without CONFIG_USER_NS and a newer kernel a lot of problematic things can
happen. If /proc or /sysfs is mounted on the container DoS or escalation to
root is possible:
[http://blog.bofh.it/debian/id_413](http://blog.bofh.it/debian/id_413) \- At
the moment no stock distro kernel has CONFIG_USER_NS enabled.

\- There are some issues related to remounting filesystems rw and altering
files

\- Mounting cgroups in the container can also lead to problems - DoS and
aquiring more ressources

\- Capabilities. You stock Linux distribution won't boot without CAP_SYS_ADMIN
(see man 7 capabilities) - there are a lot of other capabilities that could be
troublesome.

\- Not sure about this one: [http://seclists.org/oss-
sec/2011/q3/385](http://seclists.org/oss-sec/2011/q3/385)

So for running services without CAP_SYS_ADMIN and with dropping a lot of other
capabilities it can be considered somewhat safe. For everything else it's
probably dangerous.

Not sure if all these issues are still a problem today but if you are running
lxc on e.g. a current Debian Wheezy you have to know about all of them.

~~~
justincormack
sure but you should not be mounting /sys and /proc in the container. just run
one application. I do not understand the trend to run a whole Linux distro in
a container. no one did that with chroots.

~~~
nisa
Why not? No virtualisation overhead, good io, memory and cpu and even network
limits with cgroups and no extra committed ram for the vm. You just give team
xyz a login and they run their favorite distribution and software without
overhead. If you have a copy on write filesystem you even save more space and
with lvm you have easy snapshots and backups. You can put a lot of users on a
moderately fast machine this way. Thanks to lxc-attach it is also dead-easy to
debug problems for them or install software. I'd love to have this possibility
in the future.

~~~
justincormack
Because 1\. right now it breaks security 2\. its a whole lot more to manage,
and thats expensive.

I see the convenience argument, which is why people like docker, but basically
adding a whole OS overhead to every process you want to run is basically
insane in my view.

~~~
shykes
Docker doesn't require you to run an entire distro in your container. It's
what a lot of people do out of the box because it's convenient and familiar.
But as far as docker is concerned your container can be a single static binary
in an otherwise empty directory.

There is a growing trend of people building micro-containers with just the
bare minimum for their application. Docker is facilitating that trend, not
preventing it. If only because it explicitly encourages thinking of containers
as application-oriented, not machine oriented.

~~~
justincormack
not saying Docker prevents it but it is hardly widespread. Unless you use Go
or a few other languages your toolchsin won't even create static binaries for
a start.

------
seldo
A minor correction -- dotCloud the PaaS does not actually use Docker in
production, as the article claims. It uses something pretty similar, which was
the inspiration for the creation of Docker, but not Docker itself.

[Source: @solomonstre, in person at a Docker meetup a few weeks ago]

~~~
shykes
That's correct (I'm the @solomonstre in question :). We do our best to not
imply the contrary by accident. Docker is a clean slate which incorporates all
our operational learnings from dotcloud - but it is a full rewrite and this
not yet production ready.

------
zobzu
10 pages to make one part of LXC understandable, is one of LXC's issues ;-)

~~~
gesman
...plus now the enterprise needs to hire more expensive sysadmins to babysit
all of it. That's exactly what enterprises are wanting to avoid.

Our sysadmin left after he deployed LXC "goodness" to make things "better" and
we are still in a recovery mode from this.

~~~
zobzu
thats an interesting point, but yes, LXC isn't all that simple/easy, it's the
main issue. I don't know why they made it so complicated. The namespacing
technology underneath is pretty straight forward.

I'm sure they'd argue for days how that's not true (plus some of the LXC folks
actually implemented the namespacing) - but at the end of the day I make my
"jail" with the "unshare" command and mount, much simpler..

------
binocarlos
Great article! For me Docker has opened up what is great about the underlying
LXC technology - a proper logical wall between apps on the same hardware. The
networking was the hardest thing (for me) to grasp in Docker - this guide
opens up a whole new level. It's cool however that if Docker won't do
something networky I can drill down a layer to LXC instead.

------
ezequiel-garzon
Somewhat related, are there any cookbooks on how to "transform a heavy-duty
server into several virtual machines"? By this I mean something that includes,
among other things, mapping N available external IP addresses to N virtual
machines. I realize a good prior understanding of networking, iptables, etc.
would make sense. But... still... any well-detailed recipes out there? (I have
tried to look them up, but everything seems to assume more knowledge than what
I have in mind!)

------
ambiate
We are currently virtualizing our infrastructure's desktops with QVD and LXC.
There have been a few challenges, but overall, it has been painless and
exciting.

------
emmelaich
A bit of a tangent, but I wonder what happened to lguest .. and how does (did)
it compare to lxc?

Looked promising, but I never had success getting it going despite the
promises on the webpage.

And it looks like little activity and still quite a low profile after all
these years.

    
    
        http://lguest.ozlabs.org/
        http://en.wikipedia.org/wiki/Lguest

------
patrickg_zill
I like LXC but for now all my virtualized stuff runs on KVM (if needed, like
an old CentOS install with Oracle), or OpenVZ. I haven't sat down yet and
compared all the features of LXC + Docker with OpenVZ yet, though.

------
hardwaresofton
Fantastic article, super in-depth, and well written. One of the most in-depth
guides to anything lxc related I've ever seen. Thanks for demystifying a
little bit of lxc

------
willvarfar
Are there nice GUI/web tools for configuring docker and LXC firewalling etc?

