
LXD – next generation system container manager release 4.3 - dragonsh
https://discuss.linuxcontainers.org/t/lxd-4-3-has-been-released/8303
======
kstenerud
LXD is actually a cool technology. It's not like Docker/k8s in terms of each
"node" usually running only one thing. It's more along the lines of a "VPS in
a box", where you can launch virtual servers using a simple command line
interface.

I use it to run all of the virtual server machines in my home LAN (mostly
virtual desktops) with each machine exposed on the real LAN's DHCP server (so
I can move them to a different box if needed), and to launch ephemeral "server
boxes" to run some tests or builds or whatever on without polluting my dev
environment (think virtualenv, but for anything).

I also used it as the base for my virtual system builder scripts ("container"
VMs with LXD, "VM" VMs with libvirt since LXD didn't support VMs back then):
[https://github.com/kstenerud/virtual-
builders](https://github.com/kstenerud/virtual-builders)

~~~
cpsaltis
LXD is indeed nice, but it still isn't very widely adopted. I believe this
comes down to 3 main issues:

1) It's strongly connected to Canonical and Ubuntu. This is mostly a matter of
perception and it is an actual community project. However, I can understand
people not feeling comfortable with "snap install lxd".

2) It sits somewhere in between k8s and docker engine. Over time, it will
probably get more k8s-like features but still it is a weird position to be in.

3) It lacks a reach ecosystem of tools supporting it and a web UI. This makes
it hard for newcomers to adopt. We're working on a web UI ourselves as part of
our open source cloud management platform ([https://github.com/mistio/mist-
ce](https://github.com/mistio/mist-ce)) and love to hear your thoughts.

~~~
jaekash
> 1) It's strongly connected to Canonical and Ubuntu. This is mostly a matter
> of perception and it is an actual community project. However, I can
> understand people not feeling comfortable with "snap install lxd".

Last time I tried it on Fedora it did not work (less than 6 months ago).

Also it offers nothing I want over podman with --rootfs

~~~
djeiasbsbo
I've tried both podman and lxd with success but I'm curious, what do you use a
tool like that for, mostly?

Not to seem like a hypeman for Kubernetes and similar tools, but I actually
seem to only ever use containers combined with something like Kubernetes or
Docker Swarm. What do you do that you want to do specifically on one machine?
Hosting something? Automation à la CI/CD?

Again, I am actually asking for good use cases without orchestration
platforms, I am just curious.

~~~
pak9rabid
I use it as a replacement for situations where I used to use KVM for Linux
VMs. For example, my tvheadend and Zoneminder servers are both running inside
LXD containers so that I don't pollute my host machine's environment. It's
also a nice way to try out another distro other than what the host machine
runs with close-to-metal performance.

------
hardwaresofton
> Now with virtual machines being supported by LXD, we found ourselves needing
> to support attaching both our traditional filesystem backed volumes to
> virtual machines (which has been possible for a while and uses 9p) as well
> as allowing for additional raw disks to be attached to virtual machines.

Didn't know virtual machine support was offered by LXD -- I haven't had much
time to actually play with LXD but sure hope it gets more coverage in the
future. It's like the alternate more stable (and feature-ful in some rights
since they've had rootless containers longer) kubernetes that no one's ever
heard of that's been around just as long if not longer.

For those who want more introduction to the ecosystem --
[https://linuxcontainers.org](https://linuxcontainers.org)

A short primer -- LXD is the "kubernetes" part, LXC is the
"containerd"/"docker" part. LXD is far more "batteries-included" than LXD and
runs "system containers" (user namespace-mapped containers that offer better
security) easily.

~~~
curt15
"and feature-ful in some rights since they've had rootless containers longer"

Does LXD actually allow you to run and manage containers as a normal user like
podman does? All the official instructions involve adding one's user account
to the "lxd" group, which is equivalent to granting oneself root privileges
without a password. [1]

[1]
[https://linuxcontainers.org/lxd/docs/master/security](https://linuxcontainers.org/lxd/docs/master/security)

~~~
jaekash
In that sense docker is also "rootless". Won't be surprising if Canonical does
not understand the words they use.

EDIT: actually I find no claims that LXD/LXC supports rootless containers, I
think the person claiming it does just don't understand it, would like to see
a citation for it.

~~~
markshuttle
That's because by default ALL containers in LXD are unprivileged. They use the
term unprivileged because the term rootless wasn't around that far back ;)

LXD uses various mechanisms to map the uid-in-container to a uid-on-host. So
root-in-container is not root on host. There are a lot of details to work
through to make that work neatly, and there is still kernel work being done to
map this nicely into the filesystem, but it works, and it's the default, and
the FAQ recommends strongly not to grant real-root to your containers, for a
good reason.

~~~
hardwaresofton
Great summary -- for anyone interested in the kernel side, there's a talk from
2019 called "A year of Container Kernel Work Past, Present, and Future of
Container Kernel Features"[0] which goes into the timeline and future work
being done. I'm not sure where it is today but it's a good watch.

[0]:
[https://www.youtube.com/watch?v=TnArHYRYT3U](https://www.youtube.com/watch?v=TnArHYRYT3U)

------
moreentropy
I wonder why systemd-nspawn isn't mentioned/used more. I've been using it for
years for full system containerization and I'm really happy with how
lightweight and functional it is. No need for additinal LXC/LXD layers.

~~~
fcantournet
It's because it's terribad.

~~~
ubercow13
Care to elaborate for people who don't already know what you're talking about?

------
wrenky
I've been using LXD for over a year now, its quite nice if you can get over
the snap/ubuntu/canonical link. I think a lot of people are confused of its
purpose- We still use K8s and docker everywhere for new applications, but weve
ported all of our old VM based applications to LXD. It gives you
containerization at a system level (you legitimately cant tell you are on a
container if you log in) with a ton of extra niceness like easy snapshoting,
transfers to other LXD hosts, clean networking and fast spin ups.

I highly recommend checking it out if you have a bunch of legacy VMs that are
a pain to manage.

------
weinzierl
I use LXD for ad-hoc clean throw-away Linux instances a lot and it is pretty
good. One thing that is confusing is that the command line tool it provides is
called _lxc_ and not _lxd_. Even more so as LXC already is a pretty overloaded
term.

~~~
sizeofchar
In fact, the biggest problem I had overall with LXD is not just the lack of
documentation, but even understanding it and its terminologies.

------
champtar
If you care about security, LXD people are just wonderful to work with, fix
bug same or next day, fixed release the same week

------
thanatos519
I've been running the same containers for many years, changing hardware every
few years. I started on OpenVZ, moved to linux-vserver, then to LXC, then to
LXD ... then on my most recent hardware switch, back to LXD.

LXD's CLI-tool-driven configurations stored in sqlite might be nice if you're
starting from scratch, but I gave up trying to determine the incantations
required to port my stuff to it. It seemed like I couldn't use the CLI to get
from here to there, but I was able to do so by directly modifying the sqlite
database. When a CLI doesn't make all valid changes possible, that's
frustrating at best. Going back to LXC took an afternoon and I have complete
visibility over my configuration, because it's all simple text files.

LXD feels like it's at a point between LXC and K8s, but I'm not sure how many
real-world systems want to be there.

~~~
jaekash
> LXD feels like it's at a point between LXC and K8s, but I'm not sure how
> many real-world systems want to be there.

I would suggest give podman with --rootfs a try.

~~~
yjftsjthsd-h
How's that help? Podman sits in the same position as lxc or single-host
docker, doesn't it?

------
Aldipower
I am running all my production servers with LXD. Letting my services and
deamons doing it's job in a lx-container. If I do need to expose a service to
the public I port-forward or reverse-proxy to the internal containers ips.
This adds an additional layer of security. Emphasis on _additional_, hopefully
not the only one. :) This makes it more unlikely that a accidentally open
database port without authorization makes it to the public.

~~~
Qerub
Won't this forwarding/proxying make the internal services not see the actual
source IP addresses making access logs less useful? (Sure, if you're proxying
HTTP there's X-Forwarded-For but that's more configuration to get right and
there are other protocols.) It sounds like you would be better served by a
firewall, ideally both on the server node and in front of it.

~~~
jlokier
> Won't this forwarding/proxying make the internal services not see the actual
> source IP addresses making access logs less useful?

Not when forwarding. When port forwarding using DNAT only, the internal
container or VM service sees the remote IP address.

As you say, HTTP has X-Forwarded-For. Because it is common to have one or more
stages of HTTP reverse proxying these days for Internet-facing services for
robustness (for example NginX proxy -> Python application), you'll probably
have X-Forwarded-For configured already.

------
jeswin
I was initially very thrilled about LXD (lexdee) when they made the initial
announcement - because it specifically said [1]:

"We’re working with silicon companies to ensure hardware‐assisted security and
isolation for these containers, just like virtual machines today. We’ll ensure
that the kernel security cross‐section for individual containers can be
tightened up for each specific workload. "

That was the thing that made it special, but later this capability stopped
being mentioned and probably isn't planned anymore. In the meantime, one could
look at Kata Containers or AWS firecracker.

[1]:
[https://www.ubuntu.org.cn/cloud/lxd](https://www.ubuntu.org.cn/cloud/lxd)

------
mrweasel
One of my main complains about LXD is the documentation. It's scattered and
generally not all that good.

Logging is also a bit of an issue for me, but that might just be me not
configuring it correctly. Basically I don't think LXD provides enough logging
to correctly and easily pinpoint problems.

I know it's a Canonical project, but at time it seems like Stéphane Graber is
the only person actively involved in the project. Everything seems to point
back to him and his Github account in some way.

------
dobin
LXD is awesome. I use it via REST for exploit.courses to dynamically create
containers for users.

------
markandrewj
I'm surprised no one has mentioned that the original execution environment for
Docker was LXC.

------
tarruda
I've always liked LXD, but always used libvirt for my container needs due to
its VM support: It is nice to use the same set of commands and configuration
for managing both technologies.

Now I might have another look at LXD due to its VM support.

------
NorwegianDude
I've really enjoyed using LXD, both in prod and at home.

I used to use OpenVZ and then proxmox, but LXD seems to cover more or less all
my needs. Not only that, but stable updates isn't reserved for those with a
license.

Great software!

------
flemhans
Still haven’t found anything as good as Linux-Vserver that’s still supported.
The ease of set-up and to get networking working, is unmatched by any modern
alternatives.

I’m still running an unofficial kernel maintained by “Ben G” to keep it going.

------
Imranashraf
Good

