
Programming Inside a Container - nkurz
https://lemire.me/blog/2020/05/22/programming-inside-a-container/
======
mailslot
I’ve worked with a few developers that have been adamant about developing
inside of containers. What I’ve noticed:

Terrible performance. One of my engineers recently thought that 150ms was
terrific for a HTTP request. Break out of the container and it was <10ms.
YMMV.

Fragile everything: Because one expects a “pristine” environment, often any
slight change causes the entire stack to fall apart. This doesn’t happen at
the start, but creeps in over time, until you can’t even update base images.
I’ve seen it a lot. It ends up only adding an additional layer of
complication.

Etc.

There are _definitely_ reasons to do this... But when a pedantic developer
that needs everything to be “just right” does it, it often becomes a disaster,
leading to shortcuts and a lack of adaptability.

There’s also the developer that has no idea WTF is going on. They use a
standard Rails/PHP/NodeJS/etc container and don’t understand how it works.
Sometimes, they don’t even know that their system can run their stack
natively. I’ve been on teams that have said “Let’s just use Docker because X
doesn’t know how to install Y.”

Docker is fantastic for many things, but let’s stop throwing it at everything.

~~~
infogulch
> often any slight change causes the entire stack to fall apart

Yes, but this is true _generally_ ; it's not specific to containers. _Any_ dev
environment naturally tends disorder with unsynchronized versions, implicit
dependencies, platform-specific quirks, etc. It takes an effort to keep chaos
at bay.

At least with containers you have a _chance_ of fully capturing the complete
list of dev dependencies & installed software. I'm interested in how
CodeSpaces/Coder.com solves these issues.

~~~
Ao7bei3s
Counter-argument(?): I've seen a product where production ran in Docker, but
development was a mix of Mac, Windows and every popular Linux distribution, on
laptops, on-prem servers and in the cloud, as per each devs preference.
Components could be run separately. The product could run anywhere.

Then standardization crept into development. Two years later, it was
essentially impossible to run it outside Docker built by Bamboo, deployed by
Jenkins in on-prem OpenStack, components were tightly coupled (database wasn't
configurable anymore, filesystem had to look a certain way, etc.), and it
required very specific library versions, which largely haven't been updated
ever again, and cannot be updated easily anymore by now. No individual team
had an overview of _everything_ inside the container anymore (we ended up with
3 Redis, 1 Mongo and 1 Postgres in that container. The project to split it
apart again was cancelled after a while). Production and development were the
same container images, but in completely different environments.

If you want code paths to work, you need to exercise them regularly through
tests. Likewise, if you want a flexible codebase, you need to use that
flexibility constantly. Control what goes into production, but be flexible
during development.

~~~
boudin
The same mistake can be done outside of containers though. Any software needs
to be maintained and its dependancies kept up to date. Containers might give
the feeling that it's not a necessity anymore as it allows to spin up an
environment in one command, but in the end those dependencies are still there.

My experience is the opposite. I once had started a job with totally outdated
software that couldn't be run anywhere else than the old server it was
currently running and had never been touched since 2008. We were able in the
end to bring everything back up to date and create containers that are: \-
easy to update \- allow devs to work on their favourite os (windows, linux or
macos) \- does not require someone help devs to fix their dev environment
regularly

------
gorgoiler
LXC (not LXD) is an utter delight compared to the competition. If you have
IPv6 it works without any hacks and it’s like having a whole datacenter all
controllable at the command line.

Anything that inserts itself into iptables feels like a no no. That’s meant
I’ve not really put much effort into LXD or Docker beyond discovering they are
kind of heavy.

The latter had poor IPv6 support the last time I tried to use it (9 months
ago.). It’s there, but it felt like a second class citizen.

LXD just feels like Ubuntu to LXC’s Debian, so I also didn’t play with it
beyond the initial few hours.

LXC itself is a joy. I run Alpine, Debian, and Ubuntu depending on my needs.
Everything is disposable, with actual data either in github or a filer. I
don’t even bother changing the hostname on the VPSs I provision your use it.
Boot, install a firewall, create containers, and forget the original host OS
even exists.

I’d really recommend getting to grips with the low lever (non-LXD) LXC stuff,
especially when you are one “vagrant up” away from having LXC tools on your
non-Linux OS!

~~~
cyphar
> LXD just feels like Ubuntu to LXC’s Debian.

A bit of a funny thing to say, given that effectively the same group of
developers work on LXD and LXC -- and most are employed by Canonical. LXD was
never meant to replace LXC, it solves its own set of problems.

I personally like using LXD because it reduces the need for me to write my own
scripts to do trivial container management (and it can manage images and
containers on a ZFS pool by itself), but if you're more comfortable with LXC
then you do you. But I disagree that LXC is significantly more "low level"
than LXD -- it just requires more manual work and the configuration format is
more transparent about the container setup, but you ultimately have the same
capabilities in LXD.

~~~
gorgoiler
In your first paragraph you perfectly describe the Ubuntu/Debian relationship!

LXD certainly looks great from the documentation. I would use it if I had
dynamic container requirements, or had a lot more containers to manage.

One odd decision is that, I think, the _lxc_ command line tool — a significant
improvement over the _lxc- <thing>_ suite of commanda that come by default
with LXC — only ships with _lxd_.

That has a smell about it that’s a bit funny.

~~~
cyphar
Well, that's because the "lxc" command-line tool manages LXD -- you can't use
the "lxc" commands without using LXD.

Arguably it should've been called "lxd-client" or "lxdc" but they probably
just felt it was too wordy -- most admins that used LXC and wanted to switch
to LXD felt more comfortable typing "lxc <command>". And those who used LXC
had no need for a top-level "lxc" command because they used the individual
programs directly. Again, since the people working on the projects are the
same there's nothing wrong with borrowing the name from the other project. :D

------
silviogutierrez
It's a great idea. But on non-native Docker, like OSX, it's simply too slow.
Even when it's acceptable speeds, I find my computer running at 100% CPU usage
with the fan out of control.

Nix is a similar idea and makes the environmental identical for all
developers. Makes on-boarding trivial. And speeds are "native" on all
platform.

As a nice side effect: it normalizes all tools like sed, grep and find.

With that said, as a _consumer_ of a Nix setup, it's about as easy if not an
easier experience than using Docker[1].

But properly learning to create your own Nix packages, etc, involves a very,
_very_ steep learning curve. Far higher than learning the basics of a
Dockerfile.

[1] Catalina issues notwithstanding, though there is a 4 command solution now:
[https://github.com/NixOS/nix/issues/2925](https://github.com/NixOS/nix/issues/2925)

~~~
zapita
Anecdotically, I develop inside a container using Docker for Mac, and don't
have any performance issues. I think it depends on what exactlty you're
developing, in particular filesystem access patterns.

~~~
silviogutierrez
Definitely comes down to file access, above all.

------
0xcoffee
I program this way also.

My preferred method is simply to download the VSCode Remote Extension Pack:
[https://marketplace.visualstudio.com/items?itemName=ms-
vscod...](https://marketplace.visualstudio.com/items?itemName=ms-vscode-
remote.vscode-remote-extensionpack)

Then in a project, simply `ctrl+shift+p` -> Add container configuration files.
Then `ctrl+shift+p` -> Rebuild Container

Couldn't be easier.

The only downsides are lack of GUI, often in python it's nice to just do e.g.
plot.show() rather then export to a file and view that file. But for most of
my non-visual programming work, a docker is amazing. Once it works for me, I
can guarantee it will work for all other developers, and more importantly, my
dev environment is really similar to my prod environment.

~~~
lioeters
I've started doing this recently, using VS Code's Remote SSH to develop
projects in a virtual machine (or actually remote).

It's so comfortable now, with the advantages of containers I can spin up and
destroy, ensuring consistent environments.

For lack of GUI, I first tried VNC and X Window client/server, but a bit
awkward.

As a sibling comment mentioned, I find it simpler to put together and serve a
"web frontend" from the container.

~~~
alfiedotwtf
> ensuring consistent environments

This.

I really don’t understand why all the hate with working in containers. If you
can bring up and tear down quickly, it’s almost free like branching in git.
It’s consistency by design!

------
kitotik
This is not a good experience. I’ve tried it a few times because it sounds
great on paper: declarative, portable, consistent dev environment! Who doesn’t
want that?

In practice it’s slow, laggy, and a maintenance time sink.

I haven’t tried it yet, but something like NixOS seems far better suited to
this problem than containers.

~~~
takeda
> I haven’t tried it yet, but something like NixOS seems far better suited to
> this problem than containers.

You don't need to go whole NixOS to get those benefits. Installing Nix on your
OS can also provide it. Although experience is better on NixOS.

------
latortuga
It's handy to be able to do this with things you already know you want to be
able to do inside your container. The downside for me though, has been having
to throw away all the careful tweaks I have made to my development environment
over the years. The muscle memory of my long list of .bash_aliases. My custom
git config with shortcuts. My scripts in ~/bin. Hell even installed software
that doesn't fit in with the purpose of the project, think things like `jq`.

Also worth noting that the "permissions issues" he mentions are handled
automatically by Docker for Mac. I had the exact problem of files created
inside the container being owned by root on Linux, and all my mac-using
colleagues just stared at me like I had horns on my head. "It works on my
machine" still exists, even with Docker.

~~~
heavenlyblue
You could just remap the uids in Docker, even on Linux.

~~~
sp332
I thought Docker only recently supported user namespaces.

------
z3t4
You could probably do fine with just chroot. Docker is overkill for most use
cases, but one advantage is that it works on Windows and Mac too. Using the
same uid for your user on both the host and the container will likely fix the
weird user issue. One thing I miss when using Docker is the pipe command in
unix like shells.

~~~
capdeck
And if you need a step up from chroot, systemd-nspawn is a good next step.
Just as light, but supports many more isolated features including all kinds of
networking. No need for go runtime as with docker, the same convenience of use
via machinectl command.

------
madduci
If you use Visual Studio Code as your daily driver editor, you can use the
remote environments with SSH or Docker.

I made a couple of Docker images to work with the official VS Code plugin to
develop for C++, Rust, Go, Java, PHP and Python

[https://github.com/madduci/remote-
environments](https://github.com/madduci/remote-environments)

~~~
panpanna
I just wish vscode also supported LXC containers...

~~~
madduci
Indeed, but I've found out that only VSCode can handle remote environments,
VSCodium (the stripped version of Code) doesn't work

------
perlgeek
Regarding the fiddly permissions between host and container, at least on Linux
podman seems to be the solution.

It is command line compatible to docker (and uses the same image format), but
instead of launching containers through a demon, it launches container
directly as the calling user (needs user namespaces, Linux 3.8+).

Now you can share volumes between host and container, and don't run into those
pesky permission problems that come from different UIDs on the inside and
outside.

Very useful in CI contexts, for development environment and so on.

------
cblackthornekc
We do this very thing at my work. When you have a million and one guidelines
around what can and can't be installed on a device. Especially on one that has
access to the internet, it is easy to say here is this container that has no
access to the internet. That seems to make infosec happy. Also, makes local
testing difficult if you need access to a web based API.

The other issue you do have is the issue of configuring them for each
application, but luckily most teams have had at least one or two people take
up the role of maintaining the images and the startup scripts.

~~~
GordonS
On one of my Windows work laptops, the restrictions placed on devs are
ridiculous, they won't even grant admin rights. Instead they have some bug-
ridden software installed that let's you elevate specific apps. You need to
request access to those apps, and your request goes through a weeks or even
months long process before you get a response. If granted, you need to enter
your username and password for each elevation, every single time.

This machine has hyper-v installed, so I created a Windows 10 VM and worked in
that fullscreen, permanently.

Gods but I hate working for this mega corp!

------
dirtydroog
I am the opposite. I develop on Linux via VMWare Player running on a Windows
host. I don't want Linux as the main OS on my personal desktop and I can keep
work related stuff on its own VM. I've never noticed any non-bare metal speed
issues.

As for distro versioning... surely you only get the user-land package
versions, and the kernel is still the version of your main OS?

~~~
milesvp
When I left my previous job, I was starting to think it might be a good idea
to start using docker containers for dev. We were using chef kitchen to
maintain virtual machines, but we had just enough production systems, that
depending on what you were working on, you might need 3 virtual machines
running at once. I didn't care on my desktop with 16GB of ram, but we had a
number of devs using a laptop with only 8GB, and 3 virtual machines was just
enough to cause headaches.

It was the first use case for docker that I thought might make sense. The
second was the headache we were likely to start running into on our CI server
when dealing with different versions of Node/NPM for different codebases, a
drift that is inevitable as older codebases get less support.

------
peterwwillis
The script and Dockerfile I use are pretty long, but they set up everything
needed to map in the most common stuff like ssh and aws creds, ptrace, sudo
support, uid/gid runtime mapping, etc.

Once you start developing your shared repos from a container, you realize that
it's much easier to automate running things from it than to develop inside of
it, so it's actually easier to get people to build CI/CD pipelines now where
before they'd wait for someone else to do it for them.

And the only problem there is there's no easy + good + free self-hosted CI/CD
software out there. Yes, you probably use X just fine, but X probably doesn't
scale, or isn't enterprise compatible, etc. The biggest barrier to automation
is both a business problem _and_ a technical problem.

------
darkteflon
> These glitches come from the strange way in which Docker deals with
> permissions and security. Contrary that what you mean read, it is not a
> simple matter of setting user and group identifiers: it may be sufficient on
> some systems but not on systems supporting Security-Enhanced Linux which
> require additional care.

Does anyone have any general advice on the best way to deal with Docker file
permissions issues on recent Ubuntu LTS distros? As a new Docker user this has
proven surprisingly intractable. As mentioned in the quoted text, simply
setting UID / GUID to the same doesn’t seem to do the trick most of the time.
I see podman has been mentioned in this thread but is there a native, no-
dependency, no-new-tool way to handle this? I feel like I must be missing
something simple. Grateful to hear anyone’s experience.

~~~
zarkov99
I do not use SEL, but on regular Ubuntu the below works perfectly with respect
to permissions:

    
    
        #!/bin/bash
        docker run \
              --rm \
              --net host \
              -i \
              -v /etc/passwd:/etc/passwd \
              -v /etc/group:/etc/group \
              -v /usr/local/share/fonts:/usr/local/share/fonts \
              -v /usr/share/fonts:/usr/share/fonts \
              -v /run/user/:/run/user/ \
              -v /tmp:/tmp \
              -v /home:/home \
              -u $UID \
              -e DISPLAY=$DISPLAY \
              -t container \
              /bin/bash

------
erik_seaberg
> The image will only contain the compiler and build tools. There is no reason
> to change any of these tools.

Smacks of "you're holding it wrong" thinking about consumers. Professionals
should sharpen the saw by continually customizing their environment.

~~~
zarkov99
Sure, and now they can keep track of those customization in a reproducible
way, by keeping their docker file under source control. What is the problem?

~~~
erik_seaberg
I probably edit ~/.bash_profile every couple of days, but reimaging my machine
is a big deal. Code can live in a git remote, but what happens to my homedir
and notes and _ad hoc_ analyses if I recreate the container that often?

~~~
zarkov99
Nothing. None of those things would be kept in the container. The only thing
you would keep in your container would be the tools and libraries you need to
build your project. You mount your home file system inside the container and
just use it to build/debug. Everything else you keep outside.

------
3fe9a03ccd14ca5
> _The idea of a container approach is to always start from a pristine state.
> So you define the configuration that your database server needs to have, and
> you launch it, in this precise state each time. This makes your
> infrastructure predictable._

I’ve tried this before, but there’s still a lot of overhead maintained the
pristine state. For example troubleshooting why a python package won’t run,
you end up I installing and upgrading a lot of other packages. You’re not sure
if that helped or it was something else — now what? You’ll spend time
wondering if you want to carry your changes over or deal with the drift.

------
wicket
And now we've gone full circle. One of the proposed use cases for containers
when Solaris Containers came about (and probably also FreeBSD Jails before it)
was that you could easily give a software developer a full OS instance to play
about and break stuff or to do whatever they desire, instead of restricting
them to the confinement of their home directory.

------
p2t2p
Developing a Java app in devcontainer on Linux using VSCode. I absolutely love
it. I need to use an SDK that does ugly things to your local maven
installation so I put everything in container, VSCode start it automatically
and embedded terminal is opened inside that container, the only difference I
notices that VSCode is not Idea.

------
l31g
I have done this from time to time when debugging applications that run inside
Docker. I'd bash into the container as root and then download and install my
development tools and check out the repo where the code lives. Once the code
is functionally correct, then I can worry about performance optimizations
given Docker...

------
zubairq
If you want to program inside a container you could use something like yazz
pilot which is designed to do exactly that
[https://github.com/zubairq/pilot](https://github.com/zubairq/pilot)

------
Narigo
I like the sandboxing effect of using Docker. Having node_modules (or any kind
of dependencies) not being able to access all files on the host is a big
relief - especially since there were attacks on popular dependencies already,
stealing wallets or similar.

------
Marazan
After coming to a shuddering halt trying to get a build env setup for a
trivial python script (poetry couldn't install the requirements because
pip.... SSL... Blah blah blah) I just used our image used for the drone step
to run and test it.

It was such a relief

~~~
0xcoffee
Python was also what prompted me to start docker.

Different packages being installed on Windows vs Linux.

Different packages being installed pip vs (some other package manager)

Simple user mistakes in the requirements file (not strict version).

Going to docker fixed #1 and #2, and the constant rebuilding of the
environment meant we quickly identified issues in requirement files. Working
in docker is the equivalent of fail-fast programming imo, its just applied to
the environment.

~~~
TechBro8615
> Python was also what prompted me to start docker.

Indeed. Nothing will drive me to containers on a new project faster than
trying to get Python working on my mac.

Funnily enough, Docker can trace its roots back to frustrations with Python
packaging (dotcloud was originally an easy way to deploy Python apps).

------
benoror
I recently came across dip, which implements a similar approach:
[https://github.com/bibendi/dip](https://github.com/bibendi/dip)

------
bJGVygG7MQVF8c
Developing inside a container isn't worth the friction to me. I know my
environment well, and I'm invested in continuing to maintain that knowledge.

------
anotherevan
I gotta admit, I was expecting a picture of a guy in a shipping container with
a laptop.

------
overgard
I think the challenge with this is most our workstations are either macs or
windows, in which case your containers are running in a VM anyway (so,
sluggish). You only really get native performance if youre running it in
linux, which is fine but.. the year of the linux desktop has not quite arrived
yet.

~~~
jfkebwjsbx
You need better VM software then.

A properly configured VM has near native performance nowadays.

