
LXD 3.8 has been released - mariuz
https://discuss.linuxcontainers.org/t/lxd-3-8-has-been-released/3450
======
vikrantrathore
Good work @stgraber and team and congratulations on continuously improving and
constantly releasing new version and updates. Indeed being able to do this
when majority of the container funding is going to Kubernetes or Docker is a
great achievement.

Has been using LXD for last 6 years in production. Its a little gem hiding
with all the spotlight on Kubernetes and Docker. Indeed the way LXD works, it
naturally provides a path from physical servers and VM instances to containers
and back.

The important part is it runs containers in userspace providing a much better
security then docker, but seems developers don't care about it.

Also you can use traditional ansible, puppet or chef to create and manage
images and containers directly instead of learning shell scripts and
Dockerfile way with additional cognitive load.

The only issue I am facing is mounting a shared NFS or CIFS in userspace
container using FUSE, since the client drivers for this fs needs to be run
through a root process. Hopefully it will be resolved in future. Tried
Lizardfs instead of NFS and the system failed in the middle and my issues on
the github is there for weeks without any feedback. I will try glusterfs and
also looking and beegfs lets see if it can work.

~~~
StreamBright
Couldn’t agree more. I think the biggest problem for more adoption is the lack
of blog posts and documentation. I wish somebody would write articles about
how to use LXD/LXC in production catered towards developers. Maybe even having
a side by side comparison with Docker.

~~~
chrisper
I think the issue is also that for a long time (or still is?) it was Ubuntu
only.

~~~
simosx
There were some issues with specific kernel features not being upstreamed
timely.

Having said that, here is a distribution usage for the snap package of LXD,
[https://snapcraft.io/lxd](https://snapcraft.io/lxd) (see that the end of the
page).

~~~
stgraber
At this point, the only feature that I'm aware of which we have in the Ubuntu
kernel and hasn't been merged upstream yet is support for overlayfs inside
user namespaces.

The other big one for a long time was fuse inside user namespaces, but that
has been merged upstream in 4.18.

Some of the AppArmor features were also a sticky point for a while as upstream
was lagging behind quite a bit, but the AppArmor maintainer has since fixed
that, so recent Linux kernels have everything that we use.

------
simosx
LXD is used in Crostini
([https://www.reddit.com/r/Crostini/](https://www.reddit.com/r/Crostini/)) as
a way to be able to run Linux applications in chromebooks.

In addition, Samsung is somehow using LXD in their new phones, with "Linux on
DeX". In that way, you can get a Linux interface with Ubuntu when you plug
your phone to a monitor/TV.

If you use a Linux desktop, you can get to run GUI applications in a LXD
"system container", therefore isolating the files from your host's filesystem.

Here is how to do this by sharing the X server of the host (filesystem
separation but sharing the X for convenience), [https://blog.simos.info/how-
to-easily-run-graphics-accelerat...](https://blog.simos.info/how-to-easily-
run-graphics-accelerated-gui-apps-in-lxd-containers-on-your-ubuntu-desktop/)

You can run things like CS GO DeathZone in such a LXD container with full GPU
acceleration, and even closed-source NVidia driver.

Some people also got an LXD container to run in a separate X server, thus
having even better isolation.

~~~
zertrin
I use LXD containers to develop an app with a lot of big external dependencies
that are currently not yet managed in a "nice" way (i.e. for now it's extract
source tarball / make / make install).

I installed CLion inside the container and use the host X server to display
the IDE and the app being developed.

The containers are so useful to isolate the main system from these make
install and to be able to install the dependencies or rollback to a previous
version in case of mistake without fearing side effects on the host system.

------
OnAndOnAndOn
What has constantly bothered me about LXD is how heavily tied it is to Ubuntu.
With Docker, you do not need to worry about moving to another major distro
like Fedora, CentOS, Debian, etc. - it's officially supported and readily
available. The LXD answer to this is "snaps", which are a whole extra
packaging system and yet _another_ Ubuntu technology you have to invest in -
making you start to wonder if you're swallowing a spider to catch a fly.

What then increases my concern is that the two other major technology pushes
Ubuntu has made into the wider ecosystem - Upstart and Mir - have both failed
to take off.

Adopting a niche technology - especially for something as fundamental as a
hypervisor - is already a hard choice. The LXD team is making adoption even
harder by not offering official repos for the major distros.

~~~
simosx
As a user you should be unaffected by the packaging of software as long as it
works.

A Linux distribution can repackage LXD in their native packaging format and
some distributions are doing it already (Debian, Alpine, Fedora, etc). If you
are familiar with packaging, you can help promote those packages to the
official repositories.

Debian: [https://wiki.debian.org/LXD](https://wiki.debian.org/LXD)

Fedora:
[https://copr.fedorainfracloud.org/coprs/ganto/lxd/](https://copr.fedorainfracloud.org/coprs/ganto/lxd/)

Alpine:
[https://git.alpinelinux.org/cgit/aports/tree/testing/lxd?h=m...](https://git.alpinelinux.org/cgit/aports/tree/testing/lxd?h=master)

~~~
OnAndOnAndOn
As a user, you _should_ be unaffected, in an ideal world. However that is not
the case. You should always be concerned about the security of your supply
chain, and should always be pulling packages from official sources only
(official by the standards of the distro and/or developers you trust).

------
hardwaresofton
Possibly an unpopular opinion, but I believe one could easily build a better
platform than Kubernetes as it sits now on top of LXD. The amount of features
baked into LXC and exposed by LXD is bonkers.

LXC doesn't have the flexibility (for example you can't just add another
runtime underneath it), but the amount of features it has packed in it could
have given earlier k8s versions a run for their money. Also, a lot of the
implementation flexibility that Kubernetes is seeing now (ex. runtimeClass) is
the result of years of inflexibility and people finding ways to work around it
(frakti, etc) and build at a slower pace (which is also a good thing, slow
measured progress and all that) -- my point is that LXC might have have been
the better base, all things equal.

Moving VM workloads right into LXD without needing to port them to be "cloud
native" might have saved people a bunch of time, never having to deal with
complexities of overlay networks (as long as you could set up LXD properly)
would also likely have saved people some time.

~~~
gtirloni
I think it's debatable if adopting LXC/LXD early in the project would have
helped.

In any case, all you have to do these days is to satisfy the Container Runtime
Interface (CRI).

[https://github.com/ktsakalozos/lxdlet](https://github.com/ktsakalozos/lxdlet)

[https://github.com/enzian/kube-lxd](https://github.com/enzian/kube-lxd)

~~~
hardwaresofton
I _did not_ know these existed, thank you so much. I was going to try and
write an runtimeClass-compliant operator or a containerd v2 runtime for LXD
system containers one of these weekends.

kube-lxd and lxdlet seem pretty old though containerd has moved at breakneck
speed in the last year... Looking through the code it might need some TLC

------
artellectual
I've replaced kubernetes / docker with LXD to host
[https://www.codemy.net/posts](https://www.codemy.net/posts) and various other
services. All I have to say is overall much less moving parts, much lower
management cost (uses less time) and much more efficient workflow. For me
using LXD has been a dream. Congrats to the team.

I plan on releasing some of the work I've been doing on LXD soon, hopefully it
will push lxd into the spotlight over kubernetes and docker.

~~~
Uberphallus
How is orchestration done with LXD instead of Kubernetes?

~~~
artellectual
I'm using terraform (it has an LXD plugin) to handle the orchestration in the
CI / CD server.

So basically. I push code terraform detects if container running service
exists, if not it creates the containers, if it does it will upgrade the
service in the existing containers without destroying, so my deploys are super
fast. I will have a more detailed video series coming out soon on the site I
mentioned above.

------
simosx
LXD provides "system containers" in contrast to the "application containers"
from Docker.

You create a "system container" and it keeps running as a stock Linux
distribution (many flavors are supported). It takes about a couple of seconds
to create a new container, and a bit less to remove it.

~~~
hardwaresofton
LXC is capable of both[0], but Docker wasn't for a long time -- but it got the
marketshare IMO because of how approachable it was and the community effort..

[0]:
[https://linuxcontainers.org/lxc/introduction/](https://linuxcontainers.org/lxc/introduction/)

------
jlokier
LXD is nice. I use it for most of my services where a VM isn't required (e.g.
because of running Windows or a particular old kernel). I particularly like
that it replaces what I used to use debootstrap for, and quicker too.

Unfortunately, I've got some older LXC (v1.0) containers that fail to boot
under LXD, which means I need to have both container systems installed on my
hosts.

It's unfortunate that this means I have to have both the "dash" commands like
`lxc-ls` and the "space" commands `lxc ls` which return separate lists. That's
because one of them is the LXC project, and one of them is the LXD project,
confusingly using very similar names for commands.

The one feature of LXC that I really miss in LXD is the `lxc-console` command.
Among other things, without console it's harder to debug those LXC containers
to figure out why they aren't booting under LXD.

~~~
sciurus
Why can't you replace the older LXC containers with new ones managed via LXD?

~~~
jlokier
Because they don't boot successfully in LXD - their init doesn't like the LXD
environment.

Obviously this can be fixed but it's non-trivial to figure out why. For the
time being it's easier to keep using LXC for those.

~~~
jlokier
Oh, perhaps I misunderstood your question.

If you mean why can't I completely replace them with newly built containers,
it's because they are complete but rather old OS images with years of
configuration history in them, running specific versions of mission critical
applications.

They are older than LXC, and were originally run on real servers, then under
chroot and UML, then under KVM, before LXC or Docker were a thing.

To be tinkered with most carefully. As much as I would like to restart from
scratch, with new versions of everything, experience has taught me it would be
unwise to do it in a hurry, or without extensive testing.

So yes they can be replaced and will be eventually, but it's not a small job
and will need time scheduled to it.

For the time being it's the Right Thing To Do(tm) to keep the existing images
still running as they are.

------
nickjj
LXCs are still useful to use in a world with Docker.

For example, if you're testing your infrastructure configuration code, you
need some type of server to run it against. Using a base Ubuntu 18.04 Docker
image might work, but there's a lot of things that Docker is incapable of
allowing you to configure, where as with an LXC, you can.

In other words, using Docker as an infrastructure target gives me about 80%
confidence that things will work on a real host, but with LXCs that confidence
level goes up to 99%.

So it's common to have a snapshotted LXC image that you can use as a target
system, so you can spin up a new fresh LXC based Linux distro in a few
seconds. It's a lot faster than using something like Vagrant and fills a much
needed gap for testing how you provision servers.

------
LinuxBender
I've heard a lot about LXD from a coworker and would love to try it. From what
I have found, I have to install and run snapd on CentOS. [1]. Are there plans
to remove this requirement in the future? It looks to be in edge/testing on
Alpine so that is promising. [2]

[1] - [https://discuss.linuxcontainers.org/t/lxd-on-
centos-7/1250](https://discuss.linuxcontainers.org/t/lxd-on-centos-7/1250)

[2] -
[https://pkgs.alpinelinux.org/packages?name=lxd&branch=edge](https://pkgs.alpinelinux.org/packages?name=lxd&branch=edge)

~~~
simosx
Currently I think the official LXD packages will be snap packages for all
distros, with the exception of the Ubuntu LTS releases. That is, in Ubuntu
16.04 LTS and Ubuntu 18.04 LTS you get LXD as an official deb package.

Obviously, this does not preclude the distributions from packaging LXD in
their own packaging format.

Apart from the AlpineLinux effort to package LXD, there is also a Debian
effort, [https://wiki.debian.org/LXD](https://wiki.debian.org/LXD)

~~~
stgraber
Note that Ubuntu 20.04 will be using the snap package, so we only really need
to focus on one thing to validate for all distros.

Users upgrading from 18.04 to 18.10 are already being automatically migrated
over to the snap.

We have 3 tracks for the snap right now: \- 2.0 LTS \- 3.0 LTS \- latest

So users that want to stay on an upstream LTS release can still do even after
switching to the snap.

------
chrisper
LXD is amazing. It replaced most VMs in my homeserver setup.

------
jpeeler
Does anybody know of a recent (given how quickly this space moves) comparison
of LXD vs systemd-nspawn? I assume that the latter has less features given the
primary reason for its existence is to test systemd, but I'd love to know
more.

~~~
CameronNemo
systemd-nspawn is a very simple container runtime. LXD is a hypervisor. They
aren't great comparison points.

~~~
jpeeler
That's fair. What I really want to know is how systemd-nspawn compares to LXC.

Reading further, I see that LXD uses liblxc for container management. This
comment about LXD using lxc command naming also was helpful:
[https://news.ycombinator.com/item?id=18672510](https://news.ycombinator.com/item?id=18672510)

------
curt15
I've dabbled occasionally with Fedora's new "Silverblue" edition which
provides atomic system updates and expects users to do much of their work in
"pet" docker containers. However, I understand that docker subscribes to the
single-application-per-container philosophy which makes it a little awkward to
adapt to the pet container workflow (for instance, systemd doesn't work
inside). LXD, being closer to a full-blooded VM, would seem better suited to
such an application.

~~~
Spivak
You can absolutely run systemd in a container. Red Hat has put a lot of work
into this arrangement.

[https://developers.redhat.com/blog/2016/09/13/running-
system...](https://developers.redhat.com/blog/2016/09/13/running-systemd-in-a-
non-privileged-container/)

~~~
CameronNemo
You can, but why would you want to?

~~~
mixmastamyk
To restart a process if it crashes?

------
Jnr
I am unsure how you are supposed to use LXD since from the readme is seems
like it is only a REST interface? Why not use lxc directly? But I am a big fan
of lxc. I have most of my stuff in containers now-days and it has been a very
pleasant experience using lxc.

~~~
simosx
If you use the command "lxc" to manage your container, then you are using LXD.
The command "lxc" communicates with the LXD supervisor over REST for you.

If instead you are using commands like "lxc-start", then you are using "LXC".

There are two possible aspects of confusion:

1\. LXD uses the "lxc" command line utility for all the management of LXD
containers.

2\. LXC is both the name for the Linux kernel "Linux Containers" functionality
and the first/early implementation of tools (those "lxc-????") for Linux
Containers.

------
sigjuice
I have tried LXD casually over the years and it might seem frivolous, but I
can't get past the lxc commands making rectangles in my terminals. Is there a
way to disable that and output plain lines like most Unix commands?

~~~
mixmastamyk
There's no quiet option?

------
fasteo
I am quite happy using LXD. Is there any built-in way to expose a tcp port ? I
am using iptables in the host to do this.

thanks

~~~
artellectual
LXD has proxy devices, which allows you to map your host's port to your
containers ports.

[https://github.com/lxc/lxd/blob/master/doc/containers.md#typ...](https://github.com/lxc/lxd/blob/master/doc/containers.md#type-
proxy)

Makes life very easy, I used to have to mess with iptables too but with proxy
device things became much simpler.

~~~
indigodaddy
This would force you to https on the lxc/LXD endpoint that you proxied to
correct? I prefer to just iptables 80/443 (guess could proxy that part too
instead of iptables) to an LXD HAProxy container, and then have HAProxy do the
proxying and domain-based load-balancing/acl's, coupled with the advantage of
terminated SSL in one place (the HAproxy container) for all
(sub)domains/services.

