
The Slippery Slope of Docker Dev Environments - mooreds
https://blog.testdouble.com/posts/2020-02-11-the-slippery-slope-of-docker-dev-environments/
======
gravypod
In my opinion development environments should be as close to production as
humanly possible. Configuration of services, network policies, build systems
that are used in CI, etc. The closer you get to sharing tooling and
configuration between your development and prod environments the better. Does
it make some things more difficult as this author says? At first, yes. As time
moves forward you'll end up with many nock on effects that you can't get any
other way.

1\. Developers can now help you debug that one issue in prod that no one can
figure out.

2\. Deployment is as easy as adding a line to a config (load dev yaml + patch)

3\. Good devs will start automating the boring stuff. Good operators will look
at, learn to use, and love these tools.

4\. Developers will make sure containers and roll out systems work lightening
fast.

5\. "We need to run this migration in production. Tested it locally" actually
means something now.

6\. Demos can be done locally with absolutely no network connection. Demoing
to the security minded? Worried about showing off your, often down, production
env? No problem! Bring a dev laptop.

7\. Developers can test QPS quotas, build dashboards for metric collection,
benchmark performance, etc

I think it's odd to get hung up on prefixing your commands with "docker-
compose exec" or putting "docker-compose up -d --build" under watchman/inotify
or investing in setting up minikube + (tilt | skaffold).

Sure it's more stuff for your developers to know but it's very powerful to see
your change as it effects the entire living system.

~~~
geofft
The problem we have with that at my workplace is that it's hard to do things
like run IDEs/set breakpoints, run gdb or strace, get a REPL (with useful REPL
tooling), etc. inside a production-style container. I understand the arguments
that you don't want to do this in _actual_ production, you want to kill the
offending container and figure out the problem from logs - but once you've got
an idea of where the problem lies, for local dev you do actually want to debug
problems this way. Even editing a random config file or editing source code to
add a printf is more difficult inside a production-style container than in a
local checkout. Have you found good approaches for this?

~~~
crooked-v
For me, the other big complication is that we do backend stuff in Java... and
God help you if you want a self-contained image that can intelligently and
quickly reload Java code for anything that's not a very specific arrangement
of frameworks.

It's night and day compared to the Javascript world, where it's nearly a given
that the bare minimum should be watchers that recompile code and restart a
server before you finish switching window focus.

~~~
geofft
Yeah, I wonder to what extent "Just put it all in Docker" works for specific
language communities, and if you're not in those communities the advice falls
flat. (My employer is pretty polyglot, but primarily Java; my team is mostly
Python; we also build our own C code as well as third-party C code and need to
debug it every so often.)

~~~
crdrost
I mean, what Docker does is, it turns a network of executables and scripts
into a single executable with no outside dependencies.

It works extremely well for language communities that need this, in particular
it caught on because you would build something and test it locally and it
would work just fine, but then you are running the wrong version of Python or
pip installed the wrong version of a library, or you have the wrong version of
PHP or Node.js relative to Prod.

For Java and Go, I believe they have already settled these problems, no? You
might shell out to other commands and have a weak dependency on your runtime
environment that way, but if you stay within your lane, then you already have
an executable with all of its dependencies bundled in. They just have a lot
less to gain from Docker. I can't see why they would have more to lose.

~~~
geofft
OK, but, even for Python that solves the problem _once you 're in the
container_, but it doesn't really address the problem of whether my local
Python installation, the one my IDE uses, is using the same version and the
same libraries - unless you either decide I don't get to use an IDE or that my
IDE goes into the container. Right?

~~~
sebastos
Yes, you are exactly right.

I mentioned it further down, but my team and I are having preliminary success
by integrating with VSCode's "Remote: Docker" extension. They've got it rigged
so that all language support can run as remote servers inside the container.
They give you a special .json file to control how you'd like to start the
image, and you can supply a project-specific dockerfile that takes any
additional IDE-specific steps that you want to add in as layers when opening
up the project's dev container.

It still has some rough edges - for some reason, you can't supply Build-time
arguments to the devcontainer Dockerfile I mentioned above, which makes it
hard to get permissions to be non-annoying if you're worried your users will
have odd UIDs. And there are a couple other random ticky tack things. But it's
by far the closest to actually "developing out of a container" as you and I
are imagining it, with all of the attendant benefits. When you want a new dev
to start working on the project, they basically do this:

1\. git clone newproject 2\. Open that project in vscode 3\. Respond to a pop-
up that yes, I do want to open this in a container 4\. <wait for the dev-
specific layers to be built> 5\. Do a normal project build 6\. Enjoy full
intellisense, syntax highlighting, import resolution, etc. in exactly the
environment that's going to run.

In addition, by including the Dockerfile that builds the base image (the one
that the dev container FROM's) into the source, developers can now change
library versions and dependencies as a git commit, and then coerce other
developers into automatically updating their environments by pushing a new
image with a new tag to your gitlab / w/e, and then updating the tag mentioned
in the FROM in your .devcontainer/Dockerfile. It's like how you would expect
development to feel in 2020.

~~~
hcrisp
I just figured out a similar workflow using PyCharm Pro. I set up a Deployment
mapping from my Windows laptop to my Linux workstation which deploys code
whenever I save. Then the Linux folder is volume-mounted into the Docker
container. I set up a Remote Debugging configuration, and then after the
Docker container is brought up, it will attach via SSH to the Python instance
running in the remote Docker container. (Your code needs only have the
pydevd_pycharm library installed.) From there you can breakpoint, watch
variables, interact at the console, etc. It was a bit of bear to set up, but
now I find it very tractable to run and immensely useful.

------
amarshall
For development, I’ve found containers to be useful for “arms-length” daemons
(that is: databases and similar) as the containers need not change often. It
makes it easier to ensure that every project uses the correct version of these
daemons, as sometimes the OS package manager may not offer the correct
version, or different projects necessitate running different versions
simultaneously.

As for everyday running of the actual software you’re writing: no way. The
tooling for integrating an editor or IDE with the container process is pretty
poor (so one often ends up just setting up everything on the host anyway),
image rebuilds are frequent and slow, and most language ecosystems already
have tools for installing and switching between different versions. They are
useful if you want to more closely replicate your CI or production environment
for specific debugging, though. Nevertheless, the belief that running a
container locally will “never create disparity between your development
environment and your production environment” is a farce. There are so many
other variables that are often still different. It helps, but it’s far from a
silver bullet.

------
tony
Disclaimer: I would consider myself one of the developers who tries docker out
every couple of years, but just can't seem to get it.

> the developer environment just doesn’t square well with whatever system is
> running your code. In the olden days of yore, this disparity was so great
> that we often had to tunnel in to an environment that was exactly like our
> production server

For web apps, I've gotten stuff working very easily across linux, freebsd, and
mac os. Thanks a lot due to package managers, up to date library packages, and
any python/ruby/node packages with c/c++ parts _generally_ being able to find
the header/.so files. Lockfiles (package-lock.json, poetry.lock) have gotten
better as well, so I feel secure knowing that there won't be hiccups there.

Thank you for your comment on docker CLI.

I find docker commands more confusing and obtuse to work with than the
commands of postgres, python, node, webpack, combined. All of them. Because
it's about having to think about copying files, proxying against the file
system, layers of containers. By the time stuff is running I feel so far away
from the roots, I'm asking myself how much of a favor is this really doing?

The other thing is deploying these images is a lot of work compared to just
running SSH commands via fabric, pulling code, and restarting services. I also
think developers benefit from jumping in and wiring in the daemons themselves
on whatever system. I think it's part of what fullstack is. It'll come in
useful should stuff go wrong in production.

I also found my experimentation with docker inconsistent, even when I cloned
projects from git repos with docker compose, I have a high failure rate. To
the point I'd be happier if they just gave instructions for me to wire in the
daemons and db myself. And when I google, I go down a rabbit hole of
configurations and github issues that don't seem to fit (perhaps docker or
docker-compose is out of date somehow? Or is it the project itself? Or me?)

~~~
sk5t
I didn't really buy wholly into containers for a while either, but the tools
have gotten better, the process makes it easier to tick the security boxes,
and the confidence brought about by immutable deployments and having good
visibility into the service state has been really valuable in a growing
environment. No more "oh yeah, we poked at ulimit on that server late Friday
night, that's why the other ones don't work as well" and so on. No fiddling
around with multiple python or java environments. Keep after the details and
you can end up not even having anything to ssh into at all--really get out of
the game of "server touching" if you will.

------
sebastos
At my company we just finished setting up a brand new docker dev container
that we work out of, and it's awesome. However, it was absolute pain to get it
to the point where it's awesome, and the only way it worked at all was because
of the fantastic new "Remote: Docker" docker extension that has been released
with vscode. Our dev environment is intimately tied to vscode and its ability
to interpret a special .devcontainer folder to automagically add in special
additional dev layers with project-specific vscode extensions, run all
relevant language tools as remote servers inside the container, and provide
full intellisense over your project.

My questions is, how the hell is anybody getting anything done _without_ all
the features I just mentioned? I've found initial, paltry support for
particular languages in JetBrains, etc. IDEs, and that's about it. None of the
other IDEs have any real solution go getting multi-language intellisense from
inside the container. Are people just like... exec'ing into the container and
then dicking around in Vim? Or are people mapping all of the source in as a
volume, running the IDE outside the container, and just forcing the user to
have all the relevant dependencies installed on their host system if they want
to see auto-completions, go-to-definition, etc. (thus completely defeating the
purpose of containerizing your environment)?

In my eyes, the only reason Docker as a dev environment makes any sense at all
is because of the uniquely good support vscode offers, but I see it mentioned
all over the docker website that it can help you make your development
environment repeatable and exactly like your production environment. But...
that doesn't actually work, like, at all...

Can somebody help me understand what I'm missing?

~~~
goblin89
Having the environment containerized, you can run the software on your
hardware in conditions equivalent to production. That’s it; whether or not
your IDE also uses the environment inside the container doesn’t detract from
that.

Personally, I don’t care for auto-completion or go-to-definition, but I want
to be as sure as possible that if my software doesn’t break in development
then it won’t break in staging and production.

~~~
sebastos
Interesting.

>Personally, I don’t care for auto-completion or go-to-definition

What about just plain old syntax highlighting to make sure your imports are
resolving? Do you just run vim from inside the container or something, and
then find out whether it builds / works by running the container?

Like, say you had a Python project that you were going to containerize. Do you
perhaps do the actual editing of the source on your dev machine, with all of
the relevant libraries installed into a virtualenv or something? If that's the
case, then it's basically a guess-and-check workflow, right? There's nothing
strongly enforcing your virtualenv to match what's in the container. And
there's nothing about having the container that saves you from having to setup
your local environment in a way that's suitable for that particular project.
That just doesn't seem like "developing out of a container" to me. More
like... using containers as a local test environment, or something like that.

Maybe you're just explaining why the Docker folk feel it's reasonable to say
that Docker improves the development experience, putting aside whether they
imagine you literally sync'ing up your IDE up with a container as I was
describing.

------
dpc_pw
Containers are great, but everything else that Docker does is very meh. Slow,
inconvenient, hard to work with.

What I'd really want is a system that takes nix files (see Nix & NixOS), and
gets that running as a container. Or something like that: Guix or whatever
else. Free package-level caching and deduplication, and so on.

~~~
zimbatm
[https://nixery.dev/](https://nixery.dev/) does some of this. `docker pull
nixery.dev/shell/git/htop` and you get an image built on the fly with bash,
git and htop in it. Each loaded in their individual layers.

For a push approach, I am maintaining this project: [https://github.com/nix-
community/docker-nixpkgs](https://github.com/nix-community/docker-nixpkgs)

Both work well for CLI-like applications. What is missing is to get something
similar for services where there is a mix of configuration coming from
environment variables and files.

~~~
dpc_pw
Thanks. I will look at this and other comments, but I don't want stuff that
involves docker in any way. Only NixOs + containers.

------
tqkxzugoaupvwqr
The past few days I tried putting a personal project with Docker containers
together consisting of nginx, certbot, static frontend files (built from
source), backend web server. My goal was to be able to call docker-compose up.
I’ve done a similar project before without Docker so I know what the end
result should look like. Integrating the Docker containers with each other and
getting the configurations right is a nightmare. Today I realized I can’t
justify the time spent as “learning experience” anymore. What I did was
premature optimization that failed terribly and cost me days.

------
bluedino
Docker is a savior when you have several different projects with several
different versions of a language or slightly different stacks.

~~~
lostapathy
Very much this. If you only have one project and one set of dependencies, I
can see how people don’t see the value in Docker.

As soon as you have multiple projects with a mix of dependencies (especially
when you’re dealing with incompatible versions or obsolete packages) it really
becomes a huge win. Doubly so when you have out of date dependencies that (for
whatever bad reason) you can’t update, but also don’t want leaking out into
your dev environment.

~~~
geggam
Now you create an interesting set of issues for the quality and security
tests.

How do you test code quality and code security as part of your pipeline for
all of those different versions on the fly ?

eg some run python 2.7 and some run python 3.0

~~~
heretoo
Let me break your points down (using python 2.7/3.0 as the example):

a. Code quality - tackled at code review time, and probably not dependent on
python 2.7/3.0 differences; if they are version dependent then again it's a
code review process. b. Code security - again, code review

Additional:

c. Third party dependency vulnerabilities: detection should be automated;
github now supports automatic checks that open pull requests for vulnerable
artifacts.

d. user acceptance tests - seems unlikely that two versions of python are used
for the same part of the system.

Docker (especially docker-compose) makes it more visible as to what will be
deployed. In fact, it may actually be better for security when those security
checks of a system can be made before it goes to production, because the whole
stack can be built, audited and executed locally.

~~~
geggam
So you dont use automation like sonarcube / qualys and other tools ?

Our corp requires this at a minimum to even be deployed to development. Let
alone staging / prod

------
tra3
I always though of containers as production-level artifacts. In development, I
prefer to run most of my environment directly (if possible). Some components
can be run as containers if necessary.

Containers provide consistency and isolation. I dont think container, as an
abstraction, is needed at the development stage.

Once development is done, then you pass around a resultant container.

For example, I run my Rails projects directly on my Mac (Rails + Native
MySQL), my coworker on Linux runs it there natively as well. That means we
have to deal with runtime platform issues, true. However when we're ready to
"release", we build a container that is then tested, and released to prod.

------
james_s_tayler
Docker is great for local dev. I can have a complex docker-compose file with
all my dependencies and It Just Works.

------
misterbrian
Nice article, I feel like I learned a lot of these lessons over the last 1.5
years. Here's a Django/Vue.js project that I have been working on that uses 11
containers for local development (postgres, redis, redis-commander, nginx,
node, django, channels, celery, beat, flower and mailhog). I think the
weirdest things I have to do for the development environment are all related
to hot-reloading, which involves tweaked versions of container start scripts.
I run an environment similar to this one daily for work and for some other
side projects.

[https://gitlab.com/verbose-equals-true/django-postgres-
vue-g...](https://gitlab.com/verbose-equals-true/django-postgres-vue-gitlab-
ecs)

The readme for this project addresses some of the issues that you might have
with integration testing. I use GitLab CI and I'm able to take advantage of
GitLab runner to simulate my CI environment in containers that run on my local
machine.

I explored a few different ways of running integration tests in my development
environment, one using docker-compose with docker in docker, and another
method using services in GitLab CI. I learned a lot from
[https://gitlab.com/gitlab-org/gitter/webapp](https://gitlab.com/gitlab-
org/gitter/webapp).

~~~
misterbrian
I have only used this development environment Ubuntu, I don't know how well it
would work on Mac, but I would be curious to try it on Mac and Windows

------
mnm1
We use docker in production and hand a dev setup. Despite months of work, I
don't use the dev setup. One still needs to know a lot about the apps and
setting them up to get things going. Rebuilding containers takes a long time
and is generally a waste of time. Running workers that loop and check ses, for
example, eats up massive amounts of cpu while essentially just looping and
doing nothing else. It's hard to know the current configuration and keep track
of logs. It's a shit experience compared to some apache vhosts. They are
useful for some dependencies and our apps that user end of life tech that
would conflict with everything else, but otherwise it's not worth the bother.
Even on bare metal Linux it's not worth the hassle. In my osx vm, the docker
daemon needs restarting every day. The disk needs to be cleaned regularly. So
much overhead for no gain. No thanks. I gave up on this years ago. Great for
production though.

------
jasonhansel
From my experience, docker dev environments are great _if_ your host OS is
Linux (i.e. if you don't need to run a Linux VM inside of another OS).
Otherwise, performance can be very bad.

------
nhoughto
We very successfully run Kubernetes locally, in CI (via kind) and in npe/prod.
There are some differences but they have been reduced over time, very strongly
recommend this approach to teams for medium+ complexity apps.

We basically had Kube in prod, so worked backwards to make everything match
that as closely as possible, solving the required problems along the way.
docker compose for example is less likely to be used in prod and thus a poor
choice.

------
oliwarner
I use docker-compose because it makes things easier and portable.

Instead of opening four terminals and running ./manage.py runserver, npm run
dev (twice) and hugo serve, and having project instances of Postgres, Redis
and Celery running all the time, I just run docker compose up.

I want to move to a new machine? docker-compose up.

docker and docker-compose address blooming system complexity very well. I
think that's more a reflection of the complexity than docker.

------
rudolph9
People conserved with the configuration concerns notes in the article should
check out [https://github.com/NixOS/nixpkgs](https://github.com/NixOS/nixpkgs)

------
muglug
Developing with Docker is pretty great for my colleagues who use Linux
laptops.

Sadly performance on Macs is woeful (due to architectural differences between
the two systems) but even so it's still a very useful tool.

------
hannofcart
Obligatory LXD reference:
[https://linuxcontainers.org/](https://linuxcontainers.org/)

