Hacker News new | comments | ask | show | jobs | submit login
Docker is a dangerous gamble (smashcompany.com)
406 points by lkrubner 8 months ago | hide | past | web | favorite | 247 comments

I don't disagree that deploying Go or Rust apps (I think the author is wrong to include JVM in that bunch) is much nicer compared to ones where you rely on shared libraries system-wide, and may even remove the need for Docker.

However, I think the author trivializes the amount of work required to make different types of Python/PHP/NodeJS/whatever apps all work in a consistent way through configuration management, saying "I can just write a bash script or makefile." or "just download roles from Ansible Galaxy". This is so painfully ignorant and irresponsible that I fail to take the article seriously as a whole.

Even if it's just a Jenkins job doing a docker pull && docker run I've still seen massive improvements in the maintainability of configuration management code because there are so many fewer edge cases you have to take care of. No K8s required for that.

The author, writing about languages like PHP and Python (and I assume NodeJS becuase it also works in a similar way, but he doesn't explicitly mention it for some reason):

> They arose from an old paradigm, which is not suited to a world of microservices and distributed computing in the cloud.

So just because they can't build a single static binary, they're not suitable for microservices and distributed computing. Got it...

> If the switch away from Ruby/Python/Perl to a newer language and eco-system allows you to achieve massive scale with less technologies and less moving parts, then you absolutely have a professional obligation to do so.

Wow it's that easy?! Be right back, re-writing all our code in Go over a weekend hackathon.

This article made me angry.

The point is: Even if you're using single, static binaries (like we do in 90% of cases with Go), the big benefit of Docker is that you don't just get a binary, you get a service interface.

I can't stress this point enough. If you don't know what's up with a failing service, any SRE can go into the Dockerfile definition (actually, we're rarely dealing with Dockerfiles anymore, these days it's more the Kubernetes definition), look at which ports are exposed, what the environment variables are, what the service is talking to, etc.

You can harden security between those services much tighter than between binaries, by dropping capabilities, seccomp, network policies, read-only file-systems.

Also, all the other advice about the drawbacks is pretty dated: While I've run into my share of super annoying Docker problems myself in the early days (and yes, especially on Ubuntu, goodbye Device Mapper...), I've yet to encounter a single, Docker related bug on Google managed GKE on Container-Optimized OS since we've switched all of our services there a year ago.

I didn't regret our choice in a very long time.

> any SRE can go into the Dockerfile/Kubernetes definition... look at which ports are exposed, what the environment variables are

Ding ding! Applications are only a part of the issue. It's the whole server stack, all internal communication, configuration, service compsition, and networking setup.

Having these things in a declarative format are a godsend for production, a godsend for mixed language environments, and a godsend for whoever gets to kill those services and replace them with better things.

Working with sensitive data: there is also the whole shit-show of secrets management... It's not like this is impossible without containerization, but having formal security boundaries around them (both in process and systems), gives you guarantees instead of well-intentioned guesses.

> It's not like this is impossible without containerization, but having formal security boundaries around them (both in process and systems), gives you guarantees instead of well-intentioned guesses.

Containers are ran on VMs. Most of those VMs are ran in a cloud operated by the same providers peddling the container is new cloud. AWS/GCP/Azure building block is not a container. It is a VM. VMs are not hard. VMs can be treated like containers. Use them. Tool around them. You would be amazed how less crufty your stack would be when you don't use abstraction on a top of abstraction.

Containers gave us a rise of super flaky software architecture. The one that theoretically should be horizontally salable but is not.

Whenever people talk about how containers did the same to the apps as to shipping i tell them they are absolutely right. There's a pile of hard science(tm) used to properly load containers on the ships based on the content of individual containers and the trip they are going to take. That's hard science is sorely missing from the app level container management

"Hard science" would seem to be what orchestration frameworks are for, and why they're so integral to value creation. Frankly the parallel holds: you can do a lot more with shipping containers, but the standardised box is mostly a thing that lets you get started on the 'real work' :)

Re: VMs vs Containers -- the implicit assumption in what you're describing, and in the article, is that you're building in AWS/GCP/Azure with engineers who are competent in that specific suppliers ecosystem and relevant server techs. Working with hybrid cloud installations and "air-gapped" systems with international partners invalidates a lot of those assumptions... Having a stable cross-section for deployment and scaling across clouds, backed with holistic answers to configuration-heavy parts of the application stack, is a total game changer. Containers don't let you ignore VMs any more then VMs let you ignore hardware, but trying to recreate Kubernetes ad-hoc in-situ is a crap-fest.

As a software engineer I gotta disagree about abstractions creating cruft. That's what poor abstractions do. Good abstractions moves complexity to it's most reasonable point of management and the lowest point of pain. "All problems in computer science can be solved by another level of indirection" ;)

> "Hard science" would seem to be what orchestration frameworks are for, and why they're so integral to value creation. Frankly the parallel holds: you can do a lot more with shipping containers, but the standardised box is mostly a thing that lets you get started on the 'real work' :)

I have spent two years watching people moving broken architectural design to containers because the consensus was "containers would solve our problem". When the containers did nothing of sort the next level of solution was "K8s would solve the problems of the containers that we have". That also did not work. We are now waiting for a better version of service discovery/up/down wiring to come to K8S.

Container is an abstraction of shared hosting. Application that works well in containers is the same application that works well in shared hosting environment, on a bare metal or on a VM. Do that. Make your application work well in a shared hosting environment. If it does, then it will work great in any kind of a container, be that a chroot, a docker container or a VM.

I absolutely agree that containers let get one started with a real work sooner, but so can rails app template generators or express application builders in nodejs. If this is the class of problem that people advocating containers try to address then I'm absolutely, positively for containers everywhere.

> Having a stable cross-section for deployment and scaling across clouds, backed with holistic answers to configuration-heavy parts of the application stack, is a total game changer. Containers don't let you ignore VMs any more then VMs let you ignore hardware, but trying to recreate Kubernetes ad-hoc in-situ is a crap-fest.

Theoretical ability to run an application across clouds because it hooks up into Kubernetes is a pie in a sky promise. If your application even marginally complex, there's no bloody way it runs on AWS and Google without an ocean of the blood (load balancers work differently, VPNs work differently, firewalls work differently), sweat and tears of the SREs and old grumpy developers who don't care about container specialness, but can look at the logs and write some additional special proxy-wiring-of-this-to-that-on-a-container-using-this-kind-of-adapter code.

> Containers don't let you ignore VMs any more then VMs let you ignore hardware, but trying to recreate Kubernetes ad-hoc in-situ is a crap-fest.

That's not what containerize all the things people advocate. Their claim is that container is a magic bullet that allow incompetent developer produce a production code. And technical management is buying it because you know... Docker.

Containers are ran on VMs.

Except for Tryton (SmartOS).


Cannot agree with this more; but (as you already hint at) Kubernetes is the actual win, and I would not be surprised if Docker would eventually be replaced with another container solution in the Kubernetes ecosystem.

cri-o is a container runtime optimized for Kubernetes.


Does Docker itself make defining service interfaces easier than configuration management? I'd imagine that, e.g. a well-written Ansible module for an application could do all the things you mention.

Docker allows you to say "Application X" talks to "Application Y" and it will do the rest with minimal configuration. The same could be accomplished with Consul or Salt/Ansible/Puppet that knows about the services.

Really? In my experience, that's been tooling on top of Docker that makes that sort of declarative networking easy.

Docker has evolved beyond just running docker containers. I don't have any experience with it, but: https://docs.docker.com/docker-cloud/apps/service-links/

That's the issue here, you don't have any experience with it. Like most features in Docker, this one doesn't work as advertised.

> However, I think the author trivializes the amount of work required to make different types of Python/PHP/NodeJS/whatever apps all work in a consistent way through configuration management, saying "I can just write a bash script or makefile." or "just download roles from Ansible Galaxy". This is so painfully ignorant and irresponsible that I fail to take the article seriously as a whole.

He also trivializes the massive cost and difficulty of switching stacks. Even if a company can afford to upend their development processes to use a different language for new development, most of IT is legacy technology, and dynamic languages have been popular for long enough that legacy Python, Ruby and PHP applications are part of IT estates, just like legacy C++ and Java applications.

He also comlpletely negates the need of rewriting all of those "scripts" in case of a change in host provider.. None of modern companies is using own hosted servers because its simply too costly. Everyone is using cloud providers - now try to switch from one cloud provider to the other - all of those "scripts" would have to be rewritten top to bottom.

I cant take this article seriously, its just some person refusing to adapt because "in my days we did it differently"

Hello, there are a great many companies that rent mare metal servers, own their own gear in a colo, or even have DIA to their own infrastructure site.

I think you paint the whole world with the bleeding edge brush. Many fortune 500 companies are a decade behind the technology stacks described in the article and in this discussion.

Arguably containerization is one of the great hopes for Enterprise devs in the Fortune 500 space and beyond: micro investments and isolated tech stacks. It reduces a lot of the risk of innovation, while also giving a clear preservation strategy for legacy apps.

Hybrid cloud installations to leverage cloud computing resources are becoming more and more "the norm". In that context having apps and services pre-packaged for deployment in heterogenous environments is strategic power.

"The cloud" tends not to be cheap, but using cloud services to carry out jobs your dedicated site never could hope touch is something with business value.

No one is using dedicated servers or colocation?

So how do Hetzner, OVH, Rackspace, and hundreds of other companies stay in business?

Hint: cloud services are in fact way more expensive on every single metric. But they are more flexible.

If you’re using a cloud provider, using docker isn’t going to magically save you when switching providers cos chances are you’re using services of the cloud provider. Your code has to be rewritten.

And in regards to scripts or docker. No you wouldn’t need to rewrite everything.

Thats why you wrap them in api or rather your architect should. You can and abolutely should abstract yourself from cloud provider. Difference with Docker is that in case of such a move (already did that in my career) the amount of boilerplate you have to review and adjust is magnitudes lower than what he (OP) proposed in his article.

And this is exactly how you end up with an overbuilt system which is difficult to comprehend because some "architect" decided you need to be able to change cloud providers on a whim. You pay for that decision every day and 99.9% of you are _never_ going to switch providers.

Web development is full of engineering astronauts who think they need 8 levels of abstraction to solve relatively simple problems. Oh, and for that 0.1% that does actually switch, they soon realize they're abstractions weren't as good as they thought they were and they have to reimplement every interface anyway.

So you write an abstraction around the provider, and when you switch provider, you still need to implement a new implementation of that abstraction for the new provider. So in this regard, docker gave absolutely 0 benefit.

In regards to setting up a new server for a new provider, or docker, you have some boiler plate:

-> create script to attach docker to provider

New provider...

-> create new script to attach docker to provider


-> create script to provision server on provider

-> run generic script to configure it how I want

New provider

-> create new script to provision server on provider

-> run generic script to configure it how I want

In this regard, moving from 1 provider to another, is identical.

Let me ask you a question:

How do I abstract this unique idea of IOPS that AWS has?

You try, fail, and end up with a leaky, useless abstraction.

There's no cloud. It is just someone else's computer.

This was funny a few years ago. Parroting it any time someone mentions the cloud is just meme crap. This isn't reddit.

Plus you can build a single static binary in Python using nuitka (http://nuitka.net/). Ok it's static against the python dependencies but dynamic against libc for the linux build, but I've never had to deploy to a server without libc.

And if you know all your servers have the same Python version and your code is pure Python, you can even forget that and do a .pex (https://github.com/pantsbuild/pex), which a the Python equivalent of Java .war : a zip that embeds all the program and dependencies. It's light, fast, and problem free basically.

Note that typically the problem isn't that a server lacks libc, but that it has too old of a version. For example, my company runs on CentOS 6, which comes with glibc 2.12. Software built against newer glibc versions will not link against it at run time, and this is frequently a problem. The easiest solution to this is to build on the oldest linux distro you plan to target

> The easiest solution to this is to build on the oldest linux distro you plan to target

No way. Just package your libraries with your executable:

  /my_path/ld-linux-x86-64.so.2 --library-path /my_path /my_path/my_binary

Apparently nuitka's author considers it a bad idea and removed this feature: http://bugs.nuitka.net/issue136

And the funny part is that currently the easiest way to build against an older/different version of the OS without being forced into the "prod env = dev env" situation is... to use docker or any other container solution.

I would think that knowing the libc limitation were a problem, running builds on a server with this older libc version would be a possibility.

I’ve done what you mention with Docker and alpine builds (for alpine Python wheels, for example) but honestly it’s a lot more painful to ssh into running containers when figuring out the build script than it is to just throw up a clean VM.

Sure, it's possible. But I don't think it's easier. You get into all kinds of path/rpath fun and the worst thing is that things may work but you end up using the wrong environment by accident. Chroot could help, but docker has it all nicely wrapped up with the tooling.

In some cases, it's even possible to link statically against musl libc. Granted, this is easiest to do with an Alpine Linux VM or container.

Or use conda and include your c libraries and your python modules in a single python environment.

I feel like the article doesn't spell out exactly what is wrong with Docker, only that its a Gamble and Dangerous, why?

I understood the point that it may be an overkill for static binary deployments, but how is that dangerous?

The hype.

Docker is an unifying interface. It's great at what it does: make a standard facade for app deployment, setup and service management.

But it also comes at a cost. And as usual in the IT industry, the people using the tech sell it like a silver bullet, and avoid to tell you that:

- docker is complex as a product and a concept

- knowing how to run docker != knowing how to use docker properly. Remember the nosql debacle where people though they could not formalize their schema because the DB didn't enforce it ? Geeks love the circle of life.

- docker is rarely alone, it will come up with an ecosystem that will creep in your stack. Like you never really do reactjs alone but soon find out you have 1000 plugins for a complex build system and you architecture depends on a flux implementation

- most things just don't need docker. Dockers solves a technical and organizational scaling problem. If you don't have the problem, you don't need docker. Just like you probably don't need a blockchain for your new client reward policy.

- docker is a LOT of new things to learn. Not only how to use it, but also how to do all the sysadmin things you used to do right, again, but with docker. Network management used to be a fun one. Not running as root again. Persisting data.

- you will need something to install docker and the infra anyway. So you will not drop Ansible/Chef/Whatever. They will work together.

- people use docker as an excuse to not do their job properly. No packaging, little doc, etc. "Just use the docker image". This one leads to bad times.

- Not everyone uses docker. You don't live in a bubble.

- Docker images are big huge binary you download from internet and run on your server. It's very painful to audit. And so easy to ignore.

- The paradox of choice. So many things in the docker world. And it changes so fast. It's really hard to know what to use, and to keep up.

I've seen many projects wasting time and resources just because of the docker setup. It's like those people setting up a distributed architecture for a job that takes 10 seconds to run as a bash script.

Bottom line, if you think you need docker, don't use it. People needing it don't think they need it. They know they need it. And it's useful for them because they use it right.

I agree with some points, however:

- No packaging, little doc, etc. "Just use the docker image".

I'll take a project with a dockerfile used for CI over a project with short setup instructions. The first one is less likely to go out of date, or be incorrect. (Ideally it would have both, and a good operations manual. But given only those two, very common choices)

- Docker images are big huge binary you download from internet and run on your server.

I agree this is how most people use it, but you're not forced to use it that way. You can deploy memcached based on 200MB Ubuntu base, or a custom images containing the lsb layout + libc + few devices, totalling ~8MB. The huge images are a documentation/community issue, not a technical one.

Oh yeah, just like leftpad was not a technical issue. But it's still a good analogy of a big problem the JS community has.

I think leftpad was a bit different. Programming in JS you often need some dependencies. You could start from scratch and do everything in your own, but that's a huge time / resources sink. Using docker you can always start from scratch and it's barely any harder than using some huge starting image.

"Geeks love the circle of life." Best thing I've read on HN today.

>However, I think the author trivializes the amount of work required to make different types of Python/PHP/NodeJS/whatever apps all work in a consistent way through configuration management, saying "I can just write a bash script or makefile." or "just download roles from Ansible Galaxy". This is so painfully ignorant and irresponsible that I fail to take the article seriously as a whole.

Well, we did it that way for 2 decades without Docker, and the world hasn't collapsed, so?

Docker is one of many attempts at reactions to just how awful it was to manage things that way. It's not perfect, but the key takeaways is that a lot of people have independently come to the realisations that you wan deployment to as much as possible deploy static copies of already test images rather than run a bunch of configuration steps; we want isolation to various degrees; we want reproducible deployments; we want to be able to easily isolate state and regularly throw away/replace the static elements to prove that the state has been confined to the documented locations and ensure any manual tweaks are removed.

All of this is possible without tools like Docker, but they're a hell of a lot harder without some of the facilities brought by tools like Docker. It doesn't have to be docker, but it does need to be something offering isolation, a mechanism to manage images, and ways of managing data volumes. You can do that in bash if you want to, but the value in these tools is not to have to reinvent it from scratch.

What it would really require is a new application model. With a simple interface with the OS to define all of its needs in term of firewall rules, ports to be opened, access to hardware, etc. And very few interactions beyond that. Storage is cheap, it really doesn't matter to transfer a couple hundred of megabytes of binaries more. You would have a single binary file. And most importantly you would achieve version independence, i.e. the binaries of the app shouldn't even care whether it is running on ubuntu or another distribution (and it could even be cross OS) and mostly what version it is running on.

I can't help thinking that Docker is building a massive legacy application headache further down the line when all these applications will not be developed anymore, the developers will have moved on, the technology stack moved on too to new shiny tools, but the underlying OS needs to be updated and these legacy apps still need to run. The compatible base image will probably not even be available anymore. Some poor guy is going to have a horrible time trying to support that mess.

> With a simple interface with the OS to define all of its needs in term of firewall rules, ports to be opened, access to hardware, etc. And very few interactions beyond that.

That is largely what Docker is providing. It doesn't do it perfectly, but it does do it.

> I am can't help thinking that Docker is building a massive legacy application headache further down the line

That "legacy application headache" existed before Docker too, and it was much worse, because you had to hunt down toolchains and underlying OS images that'd let you get the apps to compile or run, and you could often not rely on the build and installation instructions to be complete and precise enough, because you had no guarantee that nobody would manually interfere with the process. This is in my opinion the biggest thing Docker's focus on automated image builds have given us: if you have a remotely sane process, the Dockerfile serve as evidence that the build steps work unattended (yes, you can do this with CI to verify your build steps too... until they fail in production and someone applies a workaround and don't roll it back into the build scripts; it's possible to add all kinds of steps to avoid it, or you can just build to images - docker or otherwise - and prevent the problem entirely by overwriting all he static data on every deployment)

> The compatible base image will probably not even be available anymore.

If you have a running copy of the image, then Dockers layering mean you have the compatible base image. Won't necessarily help you that much when you want to upgrade to a newer version, but at least it allows you to inspect what the dependencies are.

But all you have done is to move the problem further away from the ops guys. You still need your tool chain and underlying OS images and everything else to rebuild and re-image your guest image. When the dev team will be gone, that's not going to be any easier to maintain. Think about today's legacy software, developed 10 or 15 years ago. Even if you have the source code the tools to compile it probably don't even compile on today's OS versions (if you can even get the right licenses).

A docker with no coupling between the host and guest OS would make me happier. Then you have a binary you can just deploy 10 years after it has been compiled. And then you could extend this model to all kind of apps, including client/UI apps.

You're not going to be able to escape the problem that you need to update the dependencies.

> But all you have done is to move the problem further away from the ops guys.

No, what you've done is ensured that you have the full chain available, so that you can do your upgrades, test them, and deploy them first when they're fully tested, instead of having to try to upgrade production machines without knowing if it'll work. It doesn't save you from untangling dependencies and version incompatibilities on upgrades, but it ensures that you can do that offline, and that when you're ready to upgrade, you have images you know works.

> Even if you have the source code the tools to compile it probably don't even compile on today's OS versions (if you can even get the right licenses).

You don't need to, as long as you retain the images of the versions the tools do run on. This is why I maintain most of my build environments in images too, so that I can retain not just the finished images, but the images used to build them.

> A docker with no coupling between the host and guest OS would make me happier. Then you have a binary you can just deploy 10 years after it has been compiled. And then you could extend this model to all kind of apps, including client/UI apps.

Sounds like you want a VM. Though Docker container come close in that you're depending on very narrow interfaces. I've got Docker containers running that were made from OpenVz containers that were built from machine images back in 2008. They're still running, if only because the original machine images were poorly documented enough that rebuilding them would have been a pain. But packaging them first for OpenVz and then for Docker worked just fine with minimal effort. I'm not sure what more you want.

> What it would really require is a new application model. With a simple interface with the OS to define all of its needs in term of firewall rules, ports to be opened, access to hardware, etc. And very few interactions beyond that

so you mean ... reinvent kubernetes?

No, he means ...provide a proper abstraction at the OS level (and across OSes even).

Not the clusterfuck that is kubernetes.

That's the good part of Docker. But it comes with the ugly part (tight coupling between the binaries and the version of the host OS).

What you would want is an interface that doesn't require that coupling.

More like Nixos services and modules.

He means SmartOS, just either doesn’t know it yet or shuns going out of his comfort zone.

Isn't SmartOS an hypervisor? Sure virtual machines achieve the right level of decoupling between the host and the guest but then you have to run your whole kernel, file systems, drivers, etc. You lose all the benefits of running an app/container.

POSIX is probably something closer to what I mean.

SmartOS is an illumos distribution. illumos is a fork of OpenSolaris. Thus, SmartOS uses Solaris zones for what they call "OS-based virtualization" -- the systems all share a kernel and don't virtualize hardware. These are the same resource/performance benefits you get from Docker et al, and Zones facilities are built much more robustly.

Note that Solaris Zones predate Linux cgroups by several years, and that Zones offer full process isolation and were engineered explicitly for isolation/"OS virtualization" purposes, whereas cgroups were initially an attempt to give the kernel information about how to share resources between processes, and still do not offer real security separation or process isolation. cgroups is also a moving target, as "cgroups v2" or "unified cgroups" is still just barely becoming usable (CPU controller just added in 4.15, which is only a few months old).

You can run Docker images directly on SmartOS, because Joyent implemented the Docker API, and they also implemented a Linux compatibility API. It's not always perfect, but the fact that it works at all is admirable.

SmartOS also contains the equipment for hardware virtualization, and Joyent has even ported KVM (and recently bhyve) to illumos.

You can, theoretically, manage OS and hardware-virtualized systems at once through SmartOS, similar to what libvirt tries to provide on Linux (though their LXC integration is deprecated iirc).

I am not a SmartOS fanboy and have not as yet deployed it into anything more than a lab setting, but it, along with anything else non-Linux-centric, is consistently ignored for no real reason.

SmartOS is a hypervisor, an imaging system (via imgadm), a provisioning system (via imgadm / vmadm / Triton), a geographically distributed datacenter virtualization (via Triton) and an OS virtualizer (via Solaris zones). A SmartOS with a Joyent branded zone runs a native container at bare metal speed, with CPU bursting in the kernel as an added bonus, when the kernel detects a CPU intensive process. The whole thing is backed by ZFS and runs from memory. There is no installation.

As the Joyent branded zone is the native instantiation of the OS sharing the same kernel, there is no hardware virtualization, or rather, hardware is virtualized only once and zones then share the abstraction, as they are just normal UNIX processes. It's a completely different approach than hardware virtualization. One global zone can have up to 8192 non-global zones all sharing the same kernel, and the imgadm / vmadm combination makes the Docker problem become a non-issue.

Should we do things in 2018 how we did them in 1998 just because it sort of worked in 1998? I say this as somebody who’s usually quite conservative in operations.

>Should we do things in 2018 how we did them in 1998 just because it sort of worked in 1998?

Well, if it ain't broken, don't fix it.

There’s a very fine line between painfully cumbersome and broken.

Just curious, do you still have/use a Nokia cell phone?

Funnily enough, I had to send my smartphone in for repair recently, and I popped my SIM card back in my old Nokia phone that I'd kept in a drawer -- it works great. It's a 10+ year old device and the battery has lasted all week.

Yes, helped me skip the whole "attention drainage", "mobile detox", "missing the concern while looking at your screen", "stolen privacy" thing...

No, but my old one still works, unlike my last 8 smart phones.

And still you use the smart phones daily, not the 'working' Nokia.

It's not binary. Robustness is something we lost. Yes, I am more happy (maybe? Perhaps I now just have another way to waste my time and be interrupted) with my smartphone, but I (we) lost something in the transition. That's the point; you're adding complexity and some people want to pretend that there's no trade off.

I agree, there is always a trade-off, and it is good to make it as explicit as possible.

Broken - no, tedious and resources consuming - hell yes

> because it sort of worked in 1998

Is "sort of worked" better than "only works until the next version of docker and docker's next new, new storage layer"?

If you need to add 100x the moving parts and hidden interactions to solve some hard problems then your hard problem just got harder. You've just kicked the can down to production.

I've had to work with lots of configuration management the past few years, and I'm always cleaning up the mess because we did this for 2 decades. The world may not have collapsed but I sure have lost weeks or months of productive time trying to untangle messy configuration management setups.

> If the switch away from ... a newer language and eco-system allows you to achieve massive scale with less technologies and less moving parts, then you absolutely have a professional obligation to do so.

Read this again. It make total sense.

Eventually, some path become running against the current, and then the better path is move to something else.

And what about that millons-of-lines of code? Is the same thing as "what about all the investment in money with that inneficient car that eat gasoline like is not a tomorrow, I push so much on it but the new tech is far more efficient?, and you tell me, let that in the past?"


You put that millons-of-lines in the past and rebuild. If you have the correct mindset the change will pay off GREATLY...

P.D: I do this for work several times in my life, With very drastic/radical changes.

In fact, the MORE RADICAL the BEST. "Small" changes are not worth it. The $$$ is not there. But when you do a hard change is when the benefits get more evident...

Yeah... I'm a big fan of both Go and Clojure, but they both easily lend themselves to over-engineering. When you have to consider time to market, rapid iteration, and other such annoyances, languages that optimize for developer productivity definitely have their place, and Docker is a huge win for working with interpreted languages.

Having been around that particular block in Ruby, and seen similar things happen in other languages, I'd say that the primary problem with them isn't that the interpreters or basic tech makes it difficult. It's that the community rallies round ways of doing things that don't focus on the "uberbinary" story, and the knowledge of how to do that sort of thing is not well distributed.

For instance, it's not particularly hard to wrap everything from the app down to the ruby interpreter in a single directory that you can then drop into a .deb, but because deployment stories in the Ruby community, from at least Capistrano onwards, focused on getting the app code onto the server and then working from there, with all the problems that entails, the folk knowledge of how to do it just never spread that far. You had commonplace craziness like using RVM to install Ruby on the production servers because it was convenient, and once that mindset was embedded, the community as a whole (and I'm tarring with a very broad brush here, but it's not inaccurate) ignored simpler solutions that worked better, and didn't need tooling with as many points of failure. Bundler is another case in point: you just don't need it in production. Or at least, I never found it necessary past the build pipeline, and I think a lot of folk never found out whether they did or not.

The point is, a lot of the apparent deployment difficulty in interpreted languages is entirely incidental and self-inflicted. Now, there is a real argument that Docker removes the need to learn all the ecosystem-specific folk knowledge you need to get to a point where you can do painless, sensible deployments in a new environment, but I think it's just moving the problem around.

Thank you for your comment, I'm very interested in deployments of Ruby applications that come as close as possible to a single binary. Do you have any pointers to the mentioned folk knowledge that you can share?

There are a couple of different approaches you can take. The "traditional" way to do it would be something like exerb, which is a packager for building standalone Windows executables (although I'm not sure whether it works for recent rubies), and you literally end up with a single file. Similar ideas crop up every now and again.

But that's not the only way to do it; it's in keeping with the "uberbinary" idea to have a single tarred up directory and a known entry-point, such that once that's copied to the server you don't need anything else, no post-install build steps to get running. I built something to do everything-but-the-interpreter a while ago: https://github.com/regularfry/au.git.

The tricks that let these things work almost always boil down to knowing about the existence of a very few environment variables, and when in the build process you need them set to which values. `au` relies on $GEM_HOME, $GEM_PATH, and $PATH, and knowing what to do with $DESTDIR and `ruby-install` ought to get you a ruby interpreter as well (although `ruby-install` is just a shortcut to `make` in the MRI source). Where you'd look to learn that stuff now, I don't know.

> If the switch away from Ruby/Python/Perl to a newer language and eco-system allows you to achieve massive scale with less technologies and less moving parts, then you absolutely have a professional obligation to do so.

To add: this is wrong from another perspective. There are more goals than "massive scale" in software development. Your main costs are wages, so anything that allows programmers to be much more productive (or cheap!) is good for your business. Therefore, settling on something like Python makes sense if you can leverage its ecosystem and the wide availability of the relevant skillsets. I won't be switching to Go until its ecosystem and popularity can compete with older languages.

I agree with you for the most part, but one thing I'd throw out is that Ansible is making this conversation more interesting thanks to Ansible Container.

Just from an ops perspective, if I can use Ansible to provision infrastructure, configure bare metal, configure VMs and build containers (Docker/LXC/LXD)...all of a sudden life gets easier - especially for keeping the containers themselves up to date.

Configuration management software is good to have for managing servers because 1) they are mutable, 2) you're not always building from scratch, and 3) you may have to support many different environments (different distros, distro versions, exact same distro with different packages installed...).

Containers get rid of all these problems: Your images are immutable, you can always build from scratch, and you have full control of what goes in the image. Because of this, a plain shell script or something like Dockerfile is much simpler and easier to reason about. I really don't see the point in Ansible Container.

Containers break the problem into subsets. All Ansible does is run shell scripts.

> make different types of Python/PHP/NodeJS/whatever apps all work in a consistent way through configuration management

Do you typically deploy all of these technologies into one VM in production?

I don't think I understand the scenario being suggested here. (I'm experienced with Ansible but not Docker or Kubernetes)

Whether it's one VM or multiple, you still have to write the configuration management code (Puppet modules, Ansible roles...) to do so, and some of them will need tweaking for specific applications (e.g. in Python, installing libxml or libmysqlclient for pip dependencies, in PHP installing the various dynamically loaded extensions), which is where the rabbithole of complexity begins.

> Docker doesn't require boiling the oceans.

Look, this is from a Windows perspective but I totally get this pro of Docker. I am the last line of defense at my company when stuff goes wrong and the sheer amount of variables that customer environments introduce is frightening. In one case, some HP server management solution changed a benign environment variable to something strange; completely breaking MSBuild (which we used heavily at the time). It took weeks to escalate to me, and hours in a meeting and ILSpy figuring out what was going wrong. Oceans we're boiled by burning money.

Docker is plug-and-play. We can mandate that customers can't futz with our image. Our software becomes an appliance that plugs into their network. We can upgrade from known states, with no HP BS throwing a spanner in the works. I am pushing it heavily for on-prem for this reason.

That answers the question cleanly: stupid stuff that happens on customer environments. Don't get me started on how much this lifts off of the ops team - our Docker stuff isn't yet in production but ops are salivating over it. There is no installer to run, no xcopy, you just ask Kube to give you an upgraded cluster.

I strongly suspect change aversion here. Nobody likes their cheese being moved.

I understand your desire, but do you think the stupid stuff is going to stop happening in customer environments with Docker?

Are you still upgrading from a known state when you don't know how long it's been since they upgraded Docker? What if they're on a different version of the Linux kernel than you, or a different filesystem, causing Docker to work differently? All kinds of things about the customer environment that can affect your software are still out of your control.

The problem I've encountered with Docker in my own experience is that, while it intends to abstract over fiddly details of the system environment it's running in, it leaks just as many fiddly details, at a different level. If your goal is a self-contained appliance, you still need something wrapped around Docker, like a VM, at which point a cynic can ask why you need Docker.

Now, even though I say this, the ops team at my company (who know a lot more than me) do put Docker inside the VMs instead of just running the code in them. There is presumably some value to the abstractions it provides.

A tool doesn't need to fix _all_ problems to be useful. Yes, Docker doesn't cover differences in Linux kernel versions but it will take care of basically every other difference. Now the search space has been narrowed to Kernel/FS issues. Sounds great to me.

> We can mandate that customers can't futz with our image. Our software becomes an appliance that plugs into their network.

One of the overlooked benefits of this approach is how technically aggressive it lets you be when deploying components into strange environments... More and more we're getting tools from our suppliers based on containers that use databases, proxies, heavy server tools, and such. We fire up in seconds tools that could never viably be installed in our environment and that we have 0 competence with.

Long running databases in containers is something you need to think about, but guaranteed known states for supplier-side configuration frees them up to be daring, and firing up local DBs for config data with no outgoing ports lets me not care about how other engineers see the world.

> databases in containers is something you need to think about

I think in a few cases it makes sense to move problems out of containers. A good amount of our customers would likely tell us what to do if we asked them to not worry about the SQL database in a container - they have their backup policies and so forth. This will change over time, but I'm pretty sure that there will always be scenarios where a VM or even metal is required.

> Our software becomes an appliance that plugs into their network

That statement right there is, I feel, what Docker is all about. It converts your application from a set of files and configuration into a single blob. The conversion isn't free, but the benefits are clear and compelling.

I've been tasked a couple of times trying to debug issues in 3rd party docker images. Every one of those images did something different. Placed configuration, binaries, libraries, or logs in different locations. A simple task on a standard CentOS 7 VM that would take 5 minutes turns into 25 minutes with most of that googling ("Where the hell is your working directory?!"). It's not a fault of docker per se, but it's the result (I think) of the docker "it just works!" mentality.

What's great about my examples is that two of the applications were single binary Go applications that could be downloaded and run on almost any linux box. But they were distributed with docker ... for reasons?

Docker is great when I want to run a local Redis database in one command for development. But past that- ehhhh. Each new feature seems like Docker is just digging itself deeper.

I think this is really a problem you see in every technology that tries to make things easier. JS. MongoDB. The list goes on.

When a technology democratizes access to something that was previously only accessible to those with more specialized skillsets, it will attract people who don't know enough to know how to use it right. But, well, they're still using it. Thus, left-pad.

Its uncovering something that was always there; incompetence is rampant in software engineering. Ask the average engineer to properly set up an Ubuntu box with CICD for deployment and they'll likely fail, even with all the help of the internet. Ask them to get a docker image onto a server and they can make it work.

Wider access to technology is... well, I don't know if its a "good thing" or not. We as an industry pretty strongly believe in "fail fast"; we write tests to make sure our code works, we structure our startups to recognize and pivot around failure, and if that failure comes we stop and say "somethings fucked, stop what we're doing". So I do get scared when a technology enters the stage which allows hard failure to be delayed as long as possible. Maybe we need more technology to be harder, have opinions, and be willing to say "you're doing this wrong, just stop and go get an adult." Because the alternative... Equifax? Yahoo? Ashley Madison?

My personal design philosophy (from a System Admin, SRE, DevOps, Ops- whatever) is no black boxes. I'm not talking about knowing the inns and outs of every little component in my infrastructure/system, but instead of saying "This is Graylog" being able to say "This is Graylog- a Java app that uses MongoDB to index and Elasticsearch as a backend. We have to monitor JVM usage." The former is saying we depend on a system we have no idea how it works, and the latter is saying we depend on 2 java applications and a NoSQL database.

It horrifies me that people run stuff in production not knowing how it's configured. But, this happens and is why there is just a booming market for DBaaS and other like productions. Docker is an amazing tool, but does make somethings a bit too easy.

> Its uncovering something that was always there; incompetence is rampant in software engineering.

That's what I took away from the article. Production was setup and they don't know what kernel version it has, or what JVM version it's using, or which C library, etc. Or, now we want feature X, so we need to upgrade dependancy Y. I already did it on my local machine, but Bob from DevOps doesn't like upgrading things on QA/integration/prod without a change control...

Granted, apps have thousands of dependencies now, so keeping track of it manually is impossible. Docker came along and promised that only whatever's in this one config file is what's installed. Easy!

So it eliminated lots of the server-specific configuration management but created other problems.

I think it really comes down to developer tooling. Devs, in the absence of tooling will make their own and default to the latest stuff. If the company doesn't provide a fully mirrored developer environment, then everyone is asking for trouble.

Where I work, we have a specific team dedicated to managing this tooling.

We support Python Django, Java, and Go. You can clone the base repositories which include vagrant scripts to setup everything. You develop in that and it gets sucked up into the pipeline. You want something newer? If it works locally in our environment, the SLA says it will work in production. Doesn't work locally, then tough shit.

In personal projects I do with others, I try and do the same thing. Python? 3.6 is our version target. Java is OpenJDK 1.8.0 latest. Rust & Go are compiled with the latest versions. It might not be incompetence in software engineering as a whole, but more of an issue in defining version and software targets.

I think a lot of that can also come from people mis-using Docker as if it were a VM instead of trying to do the smallest amount of stuff possible in an image. But at the same time, there's stuff I wouldn't even try to manage in containers in production. Databases and message queues in particular.

We've had great success with Docker and Kubernetes for our stateless apps.

As a developer, one thing I absolutely love is being able to run an image without having to install anything (other than Docker). If I want to play around with something new, its easy to get started and easy to clean up.

Hosting stateless apps is simple. If the underlying host fails? don't worry, just spin up a few new copies. Slap a load balancer infront and we can do a/b or blue green.


But where is that state held? maybe your lucky and your data can be hosted in a single RDS instance. Maybe your partition scheme is simple.

However, managing distributed state is hard_.

Playing with something new should be as simple as unpacking a tarball and rm -rfing the directory when done. This should be possible in the comfort of my home directory and uid. If I need a whole separate copy of a computer's files just to try out something, then something very very fundamental is broken.

How do you install dependencies in this model? As far as can tell, you end up in situations like this


What it that something depends on a different operating system and libraries that conflict with yours?

Also the world you describe as should is not the world we live in. I should be able to exist in the world without danger or undue prosection.

Docker containers primarily mean Linux userland. It doesn’t help you with different operating systems.

  Playing with something new should be as simple as unpacking a tarball and rm -rfing the directory when done.

Needing a single install script before, and uninstall script after is acceptable.

Yes I agree things are fundamentally broken with these situations, and they're at the common OS design level.

There it is... Docker is fantastic when "it just works" but there is a huge overhead when it doesn't.

I have two CI servers for testing against different database versions, and on each box I run three different versions of Postgresql, two versions of MySQL, two versions of MariaDB, Microsoft SQL Server for Linux, and Oracle express. They all spin up from dockerfiles and run in containers. Because they all see a clean linux host, I can use the standard RPM installers for all of them, they listen on their default ports, all the datafile layouts are out of the box from the package manager.

Trying to recreate this system without using containers or VMs or something is much more difficult, because you can't for example install the MySQL 5.6 and MySQL 5.7 rpm packages on one host at the same time, you have to instead use a generic binary or build from source into custom /opt/ locations, as well as make sure everyone listens on a different port, uses unique config, etc. Oracle express gets extremely upset if things are not exactly the way it wants, and Oracle only installs from an rpm or deb package. The system used to work this way before and moving everything to docker just erased all the complexity and difficulty, it's now ridiculously easy to maintain, upgrade, and extend. Postgresql released the 10.x series some months ago, to add it alongside the 9.x versions all I had to do was add "10" to an ansible file and re-run the playbook to spin it up on the two CI workers. All of the code I use to build these servers is easily understandable and reusable by anyone familiar with Docker and Ansible.

Besides that, I use docker containers for hosts where I have for example services running that need to use particular VPNs, using the containers so that each service has its own network environment. Again, I could instead get multiple OpenVPNs running all at once on the host and mess with the routing tables and hope that a new routing rule doesn't break one of the other services, but sticking everything in Docker containers again totally simplifies everything.

I seem to be good at finding use cases where Docker makes things much simpler and a lot less work. I'm also using Ansible to orchestrate everything so I suppose I shouldn't get the author of this post started on that :).

I'd say that post actually makes a bit different point: while Docker might be good for developers, it's failing on production "side" of things. And that has a completely different set of requirements than CI servers with multiple versions on SQL engines.

The author's answer would be to reduce your database dependencies to one. Good luck with that!

Well thats probably something to do regardless

Seems like a really strange article.

This person doesn't like that you need to spend time Dockerizing your app but then writes this:

> I would recommend that a company look for chances where it can consolidate the number of technologies that it uses, and perhaps use modern languages and eco-systems that can do this kind of uber binary natively.

So you want me to re-write my 50,000 line Rails app into Go so I can ship a binary instead of Docker?

That doesn't seem reasonable, especially not when it takes literally 10 minutes to Dockerize a Ruby / Python / Node / "just about any language" app once you understand how Docker works.

That and most web applications aren't just your programming run-time or big fat binary. There's nginx, Postres, Redis, Elasticsearch and whatever other service dependencies your application needs.

I've been developing and deploying web applications for 20 years and for me Docker solves a ton of real world problems and am very thankful that it exists. Nothing that I've tried prior to Docker has made it this easy to get an app up and running consistently across multiple environments.

It also makes it super easy for me to keep my development environment in order which is important because I'm always hacking away on new side projects and need to keep a bunch of freelance related projects in total isolation with their own set of dependencies. Docker is perfect for that.

> So you want me to re-write my 50,000 line Rails app into Go so I can ship a binary instead of Docker?

Why can't you compile your Rails app into a binary? Genuine question, I haven't used Ruby in ages, but I don't see anything special about the language that would prevent this.

You can. You can use this tool called "docker", which nicely bundles all your dependencies with your code, ensures every deployment is identical, and even makes it easy to use kernel-enforced resource isolation.

Too bad you have terrible disk IO and problems when your code expects that it has a normal Posix filesystem and logs into the overlay system.

> Too bad you have terrible disk IO and problems when your code expects that it has a normal Posix filesystem and logs into the overlay system.

It depends I suppose.

I run Docker for Windows on a 4 year old computer with an SSD and those 50,000 line Rails apps have plenty of speed in development.

Rails picks up code changes in about 100ms. Basically it takes longer for me to move my eyes to my browser and hit reload than it does for a Dockerized Rails app to reload the code changes.

Flask, Phoenix and Node apps are similarly fast.

In production this is a non-issue because the code isn't mounted.

Well, there's Shoes.app which you can use to bundle your application into a vendor package, but there's not really any facility for obfuscating Ruby into executable static objects that I am aware of, neither why you would want it.

A MacOS ".app" is just a folder with executables and supporting artifacts/objects in it. (So, quite a bit like a container actually!)

I think you're being downvoted because you haven't asked "why" enough times. Why is it that you want a binary?

What is it that a container image doesn't give you that you would get some other way by making a binary? Because a container image is, at least usually when translated to its simplest form, a binary (tarball) with an entrypoint (cmd). So what are you hoping to get out of a "compiled" Ruby app that you won't be getting when it's simply Dockerized?

> it takes literally 10 minutes to Dockerize a Ruby / Python / Node / "just about any language" app once you understand how Docker works.

Is there a good reference for "dockerizing" a typical hobbyist/MVP node.js+express+babel+webpack application that would make sense to someone who understands docker and what it is used for but hasn't waded into dockerfiles and the more technical parts? Just to get a simple Ubuntu webserver set up with all the ports exposed correctly and the apps running.

I have an example Rails repo set up that covers setting up a web app, background worker, Postgres and Redis with Docker.

Feel free to use the Dockerfile as inspiration for your Node app.

The source for that app template is available at https://github.com/nickjj/orats/tree/master/lib/orats/templa....

I think it's worth the effort into learning the Docker and Docker Compose specifics because those skills are what will let you create the files you want. Basically it comes to down to how you would end up setting those things up without Docker, and then transforming those steps into a Dockerfile.

If you want to get up to speed with Docker and Docker Compose quickly (but not specifically webpack) I do have an online course available at https://diveintodocker.com/.

Thanks for the links!

There's a sad truth that many of us engineers do not want to accept. The most successful solutions are ones that do not require quality or diligence from the practitioner. There are absolutely better way to do things than Docker. Pretty much everything about Docker is terrible, but it solves one problem in a very simple way that most anybody can use. That solution is that it creates a fairly portable software package that is easy to create and run.

Now there are tons of issues in the way things are created and ran with Docker, but they will all be sufficiently addressed over time. By adding enough things around the edges (better orchestration and resource management tools, better security scanners, etc) the ends goals and needs of "business" will be met. In the end you have a messy solution compensated by a series of tools. The pros outweigh the cons.

> The most successful solutions are ones that do not require quality or diligence from the practitioner.

Until it goes "Boom". In production. With real $$ on the line. And that's where the CTO/VP of Eng recognizes that his developers don't actually understand how their system works and hence can't bring stuff back up in minutes or even hours .

you're conflating the success of the technology with the success of its users.

If the measure of success for a technology is that a lot of people are using it and working on it, then Docker is tremendously successful. If the measure of success is that the company building it is profitable, I have no idea. If the measure of success is that the founders can raise a lot of money and then take some off the table ... gonna bet successful here; they've raised $248m. Whether or not it's a good way to run applications is entirely irrelevant based on those definitions of success for the technology.

>you're conflating the success of the technology with the success of its users.

They were operating on a different definition of success.

I'm annoyed by these black-or-white statements. Docker isn't terrible. There are no "better" or "worse" way to do things. It all depends on what you are doing. Sure, applying Docker with the expectation of a silver bullet will leave a lot of people disappointed, but is that a problem with Docker? I think not.

As with any abstraction layer, lower levle complexity is abstracted and replaced with a new higher level complexity. Is that good or bad? It depends. It's a boring vague answer, but it is the truth. Saying "Docker is terrible" doesn't help anyone.

Docker as in package internal services as a container then having them deployed to a cluster which handles orchestration for you great. Your internal service, microservice if you want to call it that just becomes a process deployed to a bunch of nodes and you have a common immutable deployable asset.

Docker as in downloading random images of Dockerhub and plumbing them together. Not so great, quality varies and most people don't actually check what they are really running. Often they are VMs packaged up as a container, badly.

Yup. And there are other benefits you can get out of it once you have the cluster -- for example a service mesh can handle metrics, tracing, and circuit breaking that you'd normally need to implement in each service. That's a huge pain, especially when dealing with legacy applications and multiple languages or frameworks.

I wish there were more people like you around. The real mess I see coming, or, I should say, which has already happened, is summarised below.

As the IT world loses the freedom through the "productarisation" of everything and the wish of an unreasonable, lofty simplification our jobs become un-skilled, payed less, and a bunch of inept lead the way towards the disaster.

The rub is that there is not single answer, each IT project has its own peculiarities and history, I mean, of course there are patterns but Docker won't solve all problems, nor Python nor Go. It depends.

To unravel the complexity of the current problem at hand you need expert people, not only products and buzz words seller/buyer.

I argue that sometimes by hiring good people you would probably save the money you spend an all the hassle, licenses, bug, that those "mature" products come with.

There are some wise of companies around but the reality I see is that they over-spend on their IT infrastracture as the management, who has no idea, follows the craze.

All this will come back as more and more of those companies go burst, but we need to wise up not to get hurt ourselves.

The rub is that to run a middle-big infrastructure, or to run a software company, you do need competent people and there are not too many of them around, I am afraid.

Your post seems to suggest that we shouldn't use Docker and instead we should hire... smart people? Those are two entirely separate things.

> The rub is that there is not single answer, each IT project has its own peculiarities and history, I mean, of course there are patterns but Docker won't solve all problems, nor Python nor Go. It depends.

Docker goes a long way to solving exactly that problem, though. It allows you cover any variety of peculiarities and history by tailoring each and every app's environment appropriately. You seem to dislike Docker because it is buzz-worthy. Sometimes the buzz is onto something.

We are far apart in our view of the industry and that's OK, but I feel I work in the fashion industry, not an engineering one, which is what I wanted to stress.

There is too much confusion around and way too much marketing, it all seems new and it is not, which does not mean it is good, it depends.

Some good points. One I liked was...

"However, when you consider the hoops you have to jump through to use Docker from a Windows machine, anyone who tells you that Docker simplifies development on a Windows machine is clearly joking with you."

(By the way, can anyone give me the 20 second version of how to get a container to be able to talk to an existing remote database?)

On the other hand, his comments on language seem... schizophrenic. Some other languages are not "cloud ready", but Go is? Python has always had a concurrency thing, but jumping on Go or Elixir and expecting the magic fairys to come out of the walls and fix your stuff is delusional.

I'm quite intrigued by his other essay, "Docker protects a programming paradigm that we should get rid of"[0], because I am very much embedded in that paradigm, and while I think I have a working understanding of modern concurrency frameworks from using clojure's core.async, I don't really see where docker fits into the protection of the old way.

[0] http://www.smashcompany.com/technology/docker-protects-a-pro...

On the other hand, it's been clear to me almost from when I started using it that docker is great for development, but likely to cause huge headaches in production. It occasionally behaves in weird ways I can't explain, which is fine during development, but a would potentially be showstopper at scale.

1) I didn't get the Windows thing. Maybe I'm just using it too simply?

2) Existing remote db: I haven't tried, but recently came across docker 'run --net="host"' when finding host.docker.internal as how to access services running on the host from the docker image. Would that do the trick?


Net=host basically defeats most benefits of network isolation you get with docker. It should only be used for quick hacking. Use forward and reverse args instead. And as others said, connecting out from the container is possible by default, with the only confusion that localhost might not be what you think.

Ahh, very good, appreciate that.

Using a remote database should work out of the box assuming you didn't change the default network or security.

That's what I was thinking. Docker doesn't block outgoing ports. So connect to whatever external database you want, the same way you connect with any other external server.

> Some other languages are not "cloud ready", but Go is? Python has always had a concurrency thing, but jumping on Go or Elixir and expecting the magic fairys to come out of the walls and fix your stuff is delusional.

Python's packaging story is just remarkably awful. Every few years we hear about a new packaging format or tool or database that will fix everything, for real this time, and it turns out to be just one more failure on the pile. This is what the author means about not being "cloud ready": it's just way too hard to get a python-as-it's-actually-used (i.e. with a bunch of native C extensions that rely on system shared libraries for their functionality) service packaged up so that you can deploy it to some random server and be confident that it will work right. It's not that Go or Elixir have magical fairy dust, they've just managed not to screw it up.

What about conda? Now the anaconda distribution packages up everything including c-libraries into your conda environment to isolate you from the host libraries.

Docker in itself is not that revolutionary.

Docker AND Kubernetes combined though is such a powerful pattern. They bring the best of orchestration and binary management. It becomes a single API to scale horizontally and vertically.

Kubernetes is really what will make container work.

I fully agree. Docker always felt to me like a couple of lego blocks (image repositories and a single API for managing containers on a single host), when you needed a full box like Kubernetes (auto-restarting and scaling, resource negotiation, efficient bin-packing of containers across hosts, a single API for deploying across a cluster (possibly multiple, federated), declarative description of desired runtime state with allowance for 4-eyes control of configuration, rather than a bunch of random imperative commands).

Docker as a pseudo-VM where creation is incremental and intermediate states between Dockerfile lines are cached is really nifty for development, but the end result is little different than tar-ing up a filesystem after doing a clean box deployment. I think this bit is the sugar that got a lot of people hooked on Docker, even though it's not actually the good bit in the end.

I'm a big fan of Kubernetes and that's what I use in production, but to be honest I probably wouldn't if there weren't good managed solutions available. Nomad & Consul seems much simpler and achieves many of the same things (though you'd need to do a bit of extra glue around DNS to avoid doing SRV lookups in your service clients)

Did you look into Google App Engine, Heroku, etc.? Those were pretty seamless managed experiences for me, of course you pay for the convenience though.

Not really, those solutions weren’t quite flexible enough to do what we needed to do. Google Kubernetes Engine has been great though, not much more effort than something like Heroku but way more powerful.

...Shaking head...sometimes I just do not understand what is going on in the industry anymore...

Building self-contained .exes that install with minimal dependencies has been a pretty easy task under Windows since the mid-90s. My entire company was founded on this premise, and has been developing and marketing embedded (as in, compiles directly into the .exe) database engines for Delphi/Free Pascal for 20 years. Our customers range from small shops to very large corporations, and they all need one thing: easy packaging, branding, and distribution. Some of our customers distribute their application to thousands of machines, and most of the time the entire thing is comprised of one (or just a few) executable(s). And these are machines that the vendor has zero control over, so the product has to work in some of the most "hostile" environments one can imagine.

Under Windows, it's simply:

1) Use a standard installer to package your application executables/DLLs.

2) Make sure that you install all binaries into \Program Files.

3) Store your configuration information under the user's application data directory (local or roaming, your choice) in an .ini, .json file, etc.

In most cases, that will get you an installation (and application) that works on any Windows machine back to Windows XP. The only exception is if you need a Vista+ or Windows 8+ API, but you can code your application to fail gracefully in such environments, such as falling back to a different API or just trying to dynamically load it and display a decent error message if the API isn't available.

So, after all that: why is there such a reluctance to do the same thing on Linux ? Why does everyone want to over-complicate things ?

Getting an arbitrary binary to run successfully on Linux is trivial and IMO much easier than on Windows. The problem comes in when you're trying to deploy a service that is composed of multiple sub-services e.g. an application server, a database server, a caching server, and an http server. It's simply false to suggest that coordinating the deployment of these types of systems into a production Windows environment is easier than Linux, even without containerization... with docker there is simply no comparison. There is a reason why the vast majority of production software systems run on some linux/unix variant.

You're simply making a statement without any proof. I specifically showed how easy it can be to deploy any application on Windows going back over 15 years. Can the same be said of Linux distributions over the same time period ?

Ok. I'll give you the same steps for linux.

1) Install dependencies via package manager.

2) Execute binary.

That's it. I don't need a "standard installer", I don't have to install application files to an arbitrary location blessed by the operating system, I can store configuration information in the home-directory or in the application directory or anywhere that makes sense for the application. This works in almost every linux system "going back over 15 years".

Now explain to me how I'd deploy an application to Windows with a dependency on, e.g., two specific version of IIS, a MSSQL Database and a node application server to any Windows machine in the last 15 years.

That's certainly not been my experience with Linux. Simply trying to get something to work over multiple distributions is a very frustrating exercise, especially if we're talking anything with a UI, and backwards-compatibility with prior versions is a big issue:


Edit: also just found this, which does a way better job than I could of describing the issues:


Re: Standard installers: you aren't required to use an installer, but it makes things easier. You could just copy the .exe to a directory and run it. Most utilities work that way.

Re: \Program Files: you aren't required to install your application there, it's just good practice.

Re: installing other products with dependencies: you would install them just like any other product and would use their installer. It's up to them to make sure that they keep their dependencies in order. I, for one, certainly won't defend MS in terms of how they distribute their applications. I personally think they're a rat's nest of overly-complicated dependencies, but that is not determined/caused by Windows itself.

There is a big difference between deploying code you control and deploying other people's code you do not control. There are plenty of windows binaries that fail to run on one version of Windows or another and obviously DLL hell is a thing so I just presumed you were referring to the resources at a developer's disposal for packaging their own software for ease of deployment into a production system and not suggesting that any ol' windows binary "just works" on any version of Windows, because that is definitely not true. Regardless of the OS you have to support your target platforms.

I was referring to deploying one's own application.

Obviously, anyone can screw up anything, so the fact that an application installation won't work on a particular OS instance/version can very well be an issue with the application, and not the OS. But, that's basically my point: it's okay if the application screws something up, but the OS should present consistent and backward-compatible APIs for application binaries, and any application-specific libraries should be bundled with the application and installed into application or user-specific locations.

You mention DLL hell: this really stopped being an issue in Windows XP because of two things:

1) MS made it so that you cannot very easily drop DLLs into system directories anymore, and strongly discouraged anyone from doing so going forward.

2) MS made an effort to add features like assemblies to allow versioning, etc. to be used in the case where you absolutely, positively needed to do the above:


However, almost no one besides MS uses assemblies (.NET uses them extensively) because they're complicated to manage and they're not necessary (this is the lesson that Docker advocates are not learning). Global, shared user libraries are a feature for a past that no longer exists where disk space was at a premium.

Linux distributions need to a) figure out what a standard Linux API consists of, and b) make the changes necessary to keep these standard APIs in place across all distributions (with backward-compatibility). The browser vendors were able to do this pretty well, and JS in the browser wouldn't work at all without it.

Finally, my motivation here isn't to bash Linux because "yay Windows !", rather it's my frustration of watching this go on year after year with Linux, each year hoping that this kind of thing would get resolved and that I might be able to start targeting Linux wholesale. I just cannot understand why this is not a priority...

You forgot a few steps..

0.1) Convince every single library author to open source their code.

0.2) Convince them to submit (and maintain) the package across all Linux distributions.

And many more..

Oh, and hope you don't depend on a specific package version, because you might run into dependency hell on Linux..

Everything is "easy" when you re-define the problems as non-problems for your particular use case.

Huh? If the project is forced to rely on a closed-source library then that's just an engineering reality, it has nothing to do with the OS you're using. Ironically, this is a non-issue with docker since you can base your image off of whatever distro you need.

I'm not sure why you were confused. I just showed you a simple and common usecase where Linux deployment is not "trivial" as you boldly claimed.

>Ironically, this is a non-issue with docker since you can base your image off of whatever distro you need.

Maybe docker would help, but I do think suggesting docker in the comment thread of an article detailing how broken docker is is doubly ironical :^)

Docker is just "write code and throw it over the wall to ops" 20 years after J2EE. It was a bad idea then. It will be instructive to see if the differences in how the seams have been cut are enough to prevent it from being a bad idea now.

Someone could have easily written the same article with a slightly different title "fat binaries are a dangerous gamble" and go on to swap the opposing ideas in every sentence. And the icing on the cake; in fat binaries you need to write and maintain wrappers for services, embed files, stream resources and do a lot of other attractions as soon as your system is more complex than a game of tipcat.

I suppose Docker is good as a software packaging solution because the OS packaging systems "failed" to be simple enough.

If there was an easy way to create a software project -> Build -> distribute the result to several OS -> Expect it to run the same way config wise, then there would not be a need for a tool like Docker.

But, as with all new technologies, there are always idiots who misuse the new power. Building Docker images for deploying a single Go binary is about as idiotic as it gets.

Many Docker use-cases can be solved by a basic script that sets up the software, runs it, and cleans up after.

Yes, Docker is mainly a consequence of Linux package management practices, and in particular shared libs and dependeny management, being generally misunderstood and perceived unfit for the job. Docker is perceived as simpler, because it doesn't solve the central problem at all: that of maintaing shared libs for security updates. Using Docker is equivalent to using statically built apps (up to the fact that Docker needlessly runs as demon and with root permissions).

*) That, and tens of millions of investor money/ad spend to implant the idea into people's heads that you need Docker

Docker is a bad solution to a real problem: Software engineering is still immature and we are not capable of maintaining stable interfaces/API.

As a consequence, it's a nightmare to update a system without some kind of regressions, it's also a nightmare to make sure environments are close enough to be representative of production.

Docker kind of solve this issue by bundling every dependencies (a bit like a big java .war file, a python virtualenv, ruby bundler, or even some LD_LIBRARY_PATH trickery or static compilation with C/C++).

But this approach is wrong. Very soon, you have 69 frameworks in production each in 42 versions at least, with 13 installation patterns, so you cannot really scan the containers, and you start giving up on maintaining this huge matrix.

And at one point, you get some old (CentOS 6) containers failing to boot anyway because you updated the underlying OS (the last bit you're able to manage), and it disabled something this old container was relying upon (like its libc needing the old and somewhat dangerous vsyscall).

Stable APIs are not strictly an engineering problem, it's also a business problem. In fact, most of the engineering that occurs is due to business problems. Most of the instability around software engineering comes from business requirements - they pay engineers to do things faster/less errors/better. So we ditch the old version and build new ones. And the world goes round.

Very good point about Docker giving you a false idea about being to able to run old images on new OSs, which only real VMs can do. CentOS 6 isn't even old - it's used all over the place, and people hang on to it because it's systemd-free.

Yeah that can become a real problems and even if it runs you still end up with a lot of security issues. But this is true for any old server and unmaintained code

I think the most value in docker is "infrastructure as code", basically the way to configure a unix system in a consistent language, but that was a side-effect of docker, not it's main purpose.

I see two things missing:

Systemd can already apply cgroups to a process. If could also do a process firewall and/or use vnet interfaces for a process that would be incredible.

The next thing is a cross-platform definition language for installing and initializing a process: "I need a ldap server with this init script setup". Basically bring the IOC, DI, and Hollywood Principles to IAC.

> If could also do a process firewall and/or use vnet interfaces for a process that would be incredible.

Please, for the love of FSM, don’t give Poettering any more ideas for responsibilities to stuff into systemd.

Tools like guix and nix are better conceived over the long run.

I came to say the same. While they are not exactly competing solutions, for the most typical use-case (i.e. packaging, configuring and deploying software without dependency hells) Nix is simply so much better and less wasteful than docker. Pure bliss.

Docker has been a huge improvement for me over the past 2 years or so. Both at a relatively larger startup that made the transition from using some of the tools mentioned in this article, and more recently on a small team that doesn't even have anyone dedicated to doing operations. We've all pitched in where we saw opportunities for improvement, but pretty much right after adopting GKE we had a pretty nice way of managing our deployments.

In 5 years I think that we will have a container ecosystem that is just as stable as the Cloud ecosystem. Cloud platforms were revolutionary because they made setting up VMs, virtual networks, load balancing, etc possible to do in an automated and on-demand fashion. Container orchestration allows for many of the same concepts with a smaller footprint.

The ecosystem is still maturing, but many organizations are adopting containers as a catalyst for advancing their DevOps efforts. Some may want to wait longer than others for the ecosystem to mature, some may want to abstain altogether. It's not a radical concept though, so I disagree that most will "regret" containers being mainstream in 5 years just like most don't regret Cloud platforms being mainstream today.

This article ignores the fact that 95% of the budgets of most software companies goes towards writing code, not dev ops. Assuming Docker is indeed a leaky abstraction that is painful to use in production, this still doesn't matter as long as it makes software developers sufficiently more productive, by lowering the friction of developing new software. The development use case for Docker is so undeniably strong that it cannot be ignored. Even if you're running Chef scripts for production servers and have no intention of switching to something else, Docker can still be a fantastic investment for dev teams who have to deal with lots of dependencies. It does seem to me like dev ops is a dying art, for most companies at least.

> It does seem to me like dev ops is a dying art, for most companies at least.

It's funny. I work for a company who has a fully Dockerized pipeline, and a relatively small operations/devops team. We're looking to expand the devops team while staying rather steady for the product development team.


Maintaining services, even ones installed over the top of PaaS and IaaS requires manpower and time. And the larger we grow, the more apps we produce, the more external services we consume... the more expertise and time is needed in operations. What could at one time be handled by AWS and Google on our behalf doesn't scale as we grow.

Someone needs to understand the AWS API and all of its lovely discrepancies. Someone needs to own CloudFormation template standards and IAM roles and secrets management. Someone needs to set up and maintain security packages and vulnerabilities and prop up the pipelines. Someone needs to own monitoring and the logging pipeline and be on call 24x7 to respond to issues that aren't, can't be automated.

Certainly not what management expects or even wants (especially since it did, at one time, just work), but production is still not free, even in this day and age.

> 95% of the budgets of most software companies goes towards writing code, not dev ops.

As a company grows, expect this number to be slashed dramatically. Not because of operations, but for sales and marketing and administration and legal and HR and CS and infosec and...

Got a source for that 95% figure?

Haha I do not, sorry for the random stat. I think my point still remains though. For most companies, more value is created by software developers than by dev ops engineers. You disagree?

People have told me before that maintenance costs 10x as much as development. That was based on some studies IIRC. My experience has backed this up. But it depends what you mean by 'dev' and 'devops' also, I assume.

Most of the time its keeping crap running in my experience, which is a bit of everything.

I respect the author's argument, and I am persuaded a bit, but my team is already using Swarm in production and it's stable. Would the author, or proponents of the author's argument, suggest we backtrack and if so, how? What's the migration path?

This may be a little off topic, but has anybody watched Casey Muratori's proposal on SoC ISAs[0]? The video is long, but it's worth watching all of it. It sounds to me like a SoC ISA could make docker obsolete. If it ever happens, anyway.

[0]: https://www.youtube.com/watch?v=kZRE7HIO3vk

Docker on its own is fairly innocuous.

The problem is docker on its own is a gloriously complicated chroot with a cgroup wrapper. If it was just that I think it'd be ok. However the horror around storage (overlayfs, the mangling of DM, and the avoidance of a real filesystem designed for snapshots) is annoying as hell ()

However what people think of docker, isn't, and thats the orchestration layer. The problem is, there is no one orchestration system that fits all. For example AWS batch is a reasonable orchestration system if you just want to fire off a bunch of vaguely related tasks. However its terribly limited compared to Pixar's alfred or tractor(1/2)

K8s is a mess of config, instantiation, orchestration and logging, its a complex beast, which is difficult to tame cheaply (unless you use GKE, but then you still need to program _for_ it.)

I am currently trying to tame AWS batch to perform actual batch jobs, and write the support tools needed to make it useful. Of all the things that docker does, the only _useful_ thing it provides is the chroot+tar wrapper. However thats not worth the massive penalty of using AWS bollocks lvm+dm horror for storage.

Which leads me onto storage.

Docker, dm+lvm is the worst of both worlds, it gives you the appearance of thin provisioning, but doesn't actually give it. You have the penalty of IO redirection, but no gain. I now have to waste money by creating an AMI that uses ECS and ZFS

I think that it is telling that Red Hat have been working for quite a while to get Docker out of the stack: containers are a great idea, Docker is not great technology. The last piece is Buildah, a tool which builds and runs containers without needing a constantly running service.

There's a recent presentation on Kubernetes without Docker, Buildah and the rest of this work here:


I think the author should take a look at Convox [1]. Convox is really the "holy grail" PaaS that I've spent many years looking for. It's a self-hosted Heroku on your own AWS account, and it uses Docker. You can set up a production-ready deployment in about 10-15 minutes, including RDS, SSL, logging, high availability, and auto-scaling. The cost starts at $67 per month: 3x t2.small instances for your highly-available cluster, and 1x db.t2.small RDS instance. (convox/rack [2] is a free and open-source project.)

I've worked with OpsWorks (Chef), Saltstack, Puppet, Ansible, and Capistrano, and Convox is a breath of fresh air. I hate Chef with a passion, since I've had to maintain a cookbook for a client that is using OpsWorks. It's ugly, painful, and extremely slow.

Docker is a great tool, but it's not a PaaS. You need a layer on top to make it useful for production. But I do love having a CI image that is identical to production.

[1] https://convox.com

[2] https://github.com/convox/rack

I'm still trying to wrap my head around what Docker really is, and how/when I should use it.

Say, I have a JVM-based web application along with a database, like Play/Scala with some MySQL or H2. Is there any good reason to dockerize anything, or am I better off running them in bare form on a server?

The applications themselves (or the JVM as platform) should already act as a good enough abstraction from then underlying platform, right?

It's occasionally useful when you have what I call "misbehaved" applications that have certain picky requirements regarding environment - system libraries (e.g. specific oracle version), absolute path locations, etc. and fulfilling those environmental requirements may conflict with other applications if you installed them both on the same box.

Better behaved applications (statically compiled golang apps being one i guess) will not be so dependent upon a specially configured environment.

I think it gained popularity originally because this is a big problem on development environments and docker was the first (possibly only) technology to really try to solve it. It doesn't solve the "works on my machine" problem and it doesn't do what it does particularly elegantly, but it gets part of the way there.

I think the reason people then ported it on to production was because they liked consistency between environments, not because it is especially well suited to production environments. The creators obviously encouraged people to use it in prod (so now it's "best practice") because there's more money in your prod environment than there is in your dev environment.

I still think the whole ecosystem is somewhat shoddily put together and the culture is cargo cultish, so I try to avoid it where possible.

If you are going to be the only maintainer of your stack, in a single machine, and you are not going to pause your work for a few months, you don't need Docker.

If you are planning to bring more developers, or there is a risk that you will have to leave the project for long enough to get some form of dependency rot, then Docker will eventually make your life much easier. Though Docker or any other container technology requires a non-trivial time investment.

I'll give an example: I maintain a personal clojure/angular app for my girlfriend's students. One in a blue moon she asks me to fix things. Last time I reinstalled node, and the app was written against node 0.x, now node LTS is 6 or whatever, guess what, I couldn't even build it locally. Besides, I moved the app from a couple of VPSs (linode, DO, downsized it to cut costs, etc). But because I have my build dockerized, and it's running on docker too, I was able to quickly have an environment to run it.

Although docker images are not exactly reproducible ( you do apt-get update and you can get different results), its good enough. For me, it boils down to isolation and reproducible systems.

I think I'm missing something here. What is it that makes docker complicated or dangerous for deploying applications?

I don't understand how the argument goes from docker unnecessary and a little bit silly, to docker being actively dangerous.

Because it actively promoted "it works on my machine" and "snowflake" installs.

Supporting a legacy docker app (ie something that was shipped two years ago) is a massive nightmare, because the build system used to make it is almost certainly not going to work.

This means keeping the software patched becomes a massive massive pain. (mind you the same argument can be said for statically compiled binaries)

The assumptions, the wrapper, and the plain horror that is the storage system is just such an arse.

Well it can be dangerous if the image maintainer were to put something malicious in the build, like log all requests and send it somewhere else.

Although the same can be said for any package management libraries (npm, apt, etc), because Docker works at a server level, it opens up a whole new ballpark of exploitations on your app.

Running untrustworthy code can be risky? Nothing new here. This is not Docker's fault. Anyone can download and run untrusted code, even without Docker.

Yea, and that was exactly my point

In the middle of this ramble:

"I wish developers were more willing to consider the possibility that their favorite computer programming may not be ideal for a world of distributed computing in the cloud. Apparently I’m shouting into the wind on this issue."

Maybe understand someone's job before telling them what tools they should be using? Sure, simpler is usually better engineering. But results are important. Docker is helping a lot of developers get results, and the author is clearly not one of them.

And in conclusion:

"Docker strikes me as a direction that one day will be seen as a mistake."

Name one technical movement that wasn't! I'm not sure why it's such a shock that programmers are "fashion-conscious". It matters more that we put our collective effort behind something with a few years vision, not necessarily what that something is.

(I speak as a member of the minority waiting for the phone industry to get over its obsession with black glass rectangles and get back to making proper phones for grown-ups that come with proper keyboards, rather than pandering to the teenage dem~~increasingly loud static hiss~~)

The article starts off with the standard right tool for the right job mantra, but I think the author severely misjudges the number of scenarios where container orchestration is a superior tool.

The author also seems to downplay the simplicity and ease that container orchestration brings to the deployment and management of distributed infrastructure.

This article is very long, and the arguments very thin. I m against Docker in production. Still I think it's great/quick localy like vagrant.

I'm wondering if I don't get your point because I'm too dumb or too smart.

I may be too dumb[1] because I'm unable to infer for myself all the reasons Docker is bad that you haven't bothered to include in your article.

Or I may be too smart[2], because I found setting up Docker for Windows incredibly easy (I downloaded an installer and ran it), I find Dockerfiles easy to write (most fit on one screen), and I find running Kubernetes in production (on Azure AKS) to be straightforward (write YAML files, Google stuff occasionally).

I honestly had to scroll back to the top of your post from halfway down because I thought maybe it was written three years ago.

[1]I'm not.

[2]I'm not this either.

"Docker. The source of, and the solution to, all of our platform architecture problems."

I don't think I have much to agree with here. I do agree that one should be aware of the trade-offs of any given solution. And I agree that there is no such thing as a free lunch.

I do disagree that "devops" is a job role and the sole responsibility of particular engineers on your team.

So what does Docker bring to my team? Well it means repository maintainers have control over how their software is packaged and our deployment systems don't have to worry about configuration management on the host systems we deploy too. That saves our business time and money. It has instead allowed us to focus on high-level A/B canary deployments.

Though Docker today is much better than Docker just a few years ago.

I've used those third-generation configuration management systems to deploy services across hundreds of machines across two data centers. Usually without a hitch. It works great. But we were deploying a homogeneous suite of services that were all written in the same language ecosystem. Never had a problem.

This is why I think it's important to think of devops as an approach to structuring teams. When the people who run and operate the software are on the same team as the people who write it then the trade offs that make sense become much clearer.

None of these problems would exist if developers packaged their software and the configuration in operating system packages. Then it’s just a matter of plugging those into their OS’s respective provisioning technology. I suspect the issue isn’t that, but lack of shell scripting expertise. In the end one has to shell script anyway, even inside of systems like Docker, so prolonging the inevitable is irrational.

If you're talking about Docker vs. fat binaries, you're missing the best part of Docker; container scheduling.

If it were just about isolation you could always tarball up some chroots in a very similar fashion to Docker. But it's not just about application delivery - it's about the whole process from development to production, code to ops.

Beyond that, I don't think I'm going to "regret" Docker. Yes it's overly complicated, has a bad track record, and so forth. So? It's the best option we have right now. Something better will probably emerge and we'll move to it. I was banking on rkt myself.

But until then, I'd rather get the best we have even if I know it could be better.

Docker doesn't have a scheduler. K8s does, ECS does, Fargate and GCE does. Docker does not. (swarm counts, but its horribly limited like ECS)

Well, Docker does have a scheduler, which you mentioned. Swarm is in the same binary as Docker. But I'm talking about the ability to _use_ a scheduler, something a fat binary doesn't give you.

The author mentioned bash scripts and make as an alternative to Docker in production environments. Does anybody have experience with these techniques, (or can perhaps link us to articles / tutorials on how to implement them) Thanks in advance.

> You can have thousands of CentOS servers, and you can use an old technology, such as Puppet, to be sure those servers are identical.

I surely do feel old reading this. I actually remember being excited about Puppet as the new thing.

This essay is excellent, and to me it clearly highlights the difference between an experienced architect that has been there, done that, and got the battle-scars, and those that have not. The attitudes and behaviours Lawrence describes are depressingly common in the industry - where the bandwagon is the most common form of transport - to the detriment of us all.

I recently interviewed a candidate. When discussing languages in use, I mentioned a language this person probably didn't like all that much, or didn't have that much proficiency in. His comment wasn't "why did you chose to use that" or something along those lines, but "when are you getting rid of it". The interview came to a close pretty quickly after that. There was little interest in why these choices were made, or why perhaps that language was deemed to be the best tool for that particular job. I really wish this was an outlier, but it is the prevalent attitude. Every day I work with developers that have little regard as to what tool is the best for the job at hand, and approach problems with the idea of "Well, I have this hammer here, and I really know how to use it well, I'm pretty sure I can ram this screw into the board with it" and subsequently proceed on a mission to convince everyone that the hammer is the only tool worth considering. I suspect that the deeper issue is one of either supreme overconfidence and arrogance, or masks a deep insecurity.

Whichever one it is, the issue remains. I have had devs insist that "docker is the only way in which can deploy my app, we need Docker RIGHT NOW". Container orchestration is being deployed at $work and we have published a set of guidelines to make sure that developers have an idea on how to ensure their apps will function properly in this environment - it is much along the lines of 12-factor apps, with some tweaks. The pushback I have received from some is that "with docker it shouldn't matter, and whatever I do on my PC you can just pick up and drop in place", with a complete lack of regard for the fact that their PC is not that same as our production environment. After discussion it becomes clear that this is rooted in a combination of ignorance ("I don't know how to do that") and laziness ("I made this app, it works on my PC, it is not my problem anymore").

Doing battle with the tech-fashionistas is a regular thing. No, we are not going to re-write all out applications in C# because we have an intern that can only use C# (an actual discussion). No, we cannot randomly deploy everything on Docker because that is the current hype du-jour. An now, we are not going to back up Petabytes of data to the cloud because everyone is doing it - laugh all you want about tape, but it more cost effective, fits our use-case better, and is orders of magnitude more cost-effective.

One of the things that Lawrence mentions really resonates with me:

"The guiding rule should be “What is the simplest way to do what we need to do?” If the older technology gets the job done, and is the simpler approach, then it should be preferred. But if there is a new technology that allows us to simplify our systems, then we should use the new technology."

I wish we could all get behind that....

> The pushback I have received from some is that "with docker it shouldn't matter, and whatever I do on my PC you can just pick up and drop in place", with a complete lack of regard for the fact that their PC is not that same as our production environment.

Yes. This misconception is the main promise made in Docker's marketing, so it's no surprise that people who've gobbled it up are unhappy when confronted with the fact that the pixie dust and the lands of eternal rainbow-sunshines are fantasies. (Sidenote: it's sad how non-admin-capable developers interpret the promise of demoting ops to mindless button-pushers as fantasy fulfillment, but that's an issue for another time).

Docker had a massive bonfire of VC money to perpetuate these false beliefs, and the message has been magnified by BigCos like Google, who've placed it at the center of their corporate strategy to retake cloud.

Unfortunately, in most cases, the Direction-Setter is not going to take the credit for being misled and needing to double back and fix it. They just lob it over the wall to RealOps, courtesy of DevOps(TM), and expect them to be grateful for it.

Meanwhile, RealOps's job is much harder than before, because you have the ignoramus shouting "What are you talking about?! You just don't know how to use this revolutionary new technology from DOCKER and GOOGLE! Are you smarter than Google, Bob? I coded up a whole Dockerfile over lunch!"

We have a lot of naive, self-important dilettantes in tech these days, trying to act like the 4 minutes they spent skimming the Docker manual make them smarter than the greybeard who just doesn't have the time for every painfully empty fad on the block.

We should talk about how we can stop that infiltration.

Yes, we should. Docker is just a red herring. This is the real issue, and the root cause of Docker.

> This essay is excellent, and to me it clearly highlights the difference between an experienced architect that has been there, done that, and got the battle-scars, and those that have not. The attitudes and behaviours Lawrence describes are depressingly common in the industry - where the bandwagon is the most common form of transport - to the detriment of us all.

As someone who doesn't like docker (the software) or particularly trust Docker (the company) to be good stewards of docker: I disagree. To me, the author sounds like someone with a severe case of NIH.

Sure, you can write ad hoc scripts to manage deployment and testing (which is what docker essentially started as — ad hoc sugar on top of lxc). Ad hoc does not scale. At Megacorp we had apps written in Java, Scala, .NET, Ruby, Python, JavaScript, and things targeting those VMs like Clojure/CLJS. At some point the value of providing a standardized interface (docker or not) for both the developers and the operations folks outweighs whatever NIH might be kicking around. Operations for each app might be the responsibility of one or two people. If someone got sick or went on vacation all hell would invariably break loose. Being able to share a standard interface means it's much easier to distribute the work. Having a standardized interface means that once a problem is figured out in a staging environment, the fix can be rolled out with far more confidence elsewhere. I've seen the logical conclusion of doing everything ad hoc. And trust me, when you have dozens of products across at least four environments, with decades old legacy cruft in various places along the way, it ain't a pretty sight.

In fact my experience has led me to the ideal of: every environment should be identical aside from its scale. I'm well aware that docker doesn't accomplish this unless everyone's running Linux. However, the argument of just slap together some ad hoc scripts for the dev environment and then again some new scripts for the production environment makes my skin crawl (and it was a bit of an ongoing battle with ops folks at Megacorp outside of their docker shenanigans). Having the same interface for a dev environment that I do in production means that I can practice and test things well before they hit customers. Change requests become less scary. And, of course, my ubergoal: ad hoc environments become possible.

I wish there were more people with your attitude in the industry.

People would actually learn to do things well instead of chasing the latest crap every year so and only learning to use a technology superficially.

I wish more people in IT had your attitude. Most other professions find what works best and stick to it, or incrementally improve, but never re-invent the same thing over and over again, nuking themselves 20 to 40 years back like we do.

I work in facial recognition, where the application is treated as a sensor, and it must interface with an unknown infrastructure. Faced with this uncertainty, we chose what the author here calls "fat binaries" - our single application is pure C++, fully self contained with a complete HTTP/REST Server included. No dependencies. We publish both Linux & Windows versions, and find superior flexibility and performance over those using Docker / standardized VM technical strategies. Our running application sits in 500K of RAM, runs performance circles around anyone we are compared, and easily can be added into containers or VMs soas to be compatible with clients Docket/Container/Standardization strategy. However, we can just as easily be run on a Raspberry PI, an Intel Compute Stick, or even right on the paltry 1MB hardware inside an IP camera. Try that with Docker.

Them: “This app requires Nginx and PostGres and Redis and a bunch of environment variables. How are you going to set all that up without Docker?”

Honestly if the developers are thinking at this level and management doesn't understand pushing back and separating things out such that the app developer should only worry about "redis" or "postgres" or maybe both, but they should be environment variables to that part of the app...The company isn't doing a good job of understanding it's stack.

Keep it in mind it was an example privided by the OP

This article seems to totally go against my personal experience, that maintaining a suite of applications of varying languages in prod and dev is made a lot easier through containerization and standardizing on top of that. The same can be done with VMs but the container orchestrators (kube/etc) are usually closer to the level of abstraction the developers need (docker -> puppet)

My comment from the earlier discussion was quoted:

"Let’s say I’m working on an existing code base that has been built in the old-style scripting paradigm using a scripting language like Ruby, PHP, or (god help us) node.js.

…I can just about see how we can package up all our existing code into docker containers, sprinkle some magic orchestration all over the top, and ship that.

I can also see, as per the article, an argument that we’d be much better off with [uber] binaries. But here’s the thing: You can dockerise a PHP app. How am I meant to make a [uber] binary out of one? And if your answer is “rewrite your entire codebase in golang”, then you clearly don’t understand the question; we don’t have the resources to do that, we wouldn’t want to spend them on a big bang rewrite even if we did, and in any case, we don’t really like golang."

And the reply was:

"In this example a company has had a PHP app for a long time, and now it needs to Dockerize that app. Why is this? What has changed such that the app needs to be Dockerized now? This feels like an artificially constrained example. How did the app work before you Dockerized it? What was the problem with the old approach, that you feel that Docker will solve?"

But I think that totally missed my point. Lawrence has written a compelling argument about how wonderful what he's now calling uber binaries are. I'm sold! I want them! But I cannot make an uber binary for a PHP app (as far as I'm aware). I can dockerise it, and sure, an uber binary is much much better than a container, but maybe a container is still slightly better than a traditional app. Dockerisation is possible (for most people); uber-binarification is impossible (for many people). If you're writing an article about how uber binaries are better than containers and you miss out on the biggest advantage containers have over uber binaries for many people, your article is not going to engage with people the way you're hoping.

And for the record: We don't think we need to Dockerize our app, we haven't dockerized it, and and we're not dockerizing it. We have zero containers of any type in production; we rely on some Puppet templates and a small handful of hand-rolled shell scripts, and it works great. We're docker skeptics and agree 110% with everything Lawrence is saying here, including the bit where he talks about how great uber binaries make deployments.

"What was the problem with the old approach, that you feel that Docker will solve?"

Man, don't ask me, you're the one who wrote an entire blog post about it! (Admittedly, the post was about how uber binaries are even better than containers as solving these problems, but as above, I can't use uber binaries. I can use containers. And the more you talk about how uber binaries are like magical super-containers, the more it makes me wonder if we're missing out on something with our repo of Puppet templates and bash scripts.)

I just took his point to be, much simplified:

Moving to docker is not free and the team should honestly evaluate whether those resources would be better spent making fixes to the underlying problem. Instead of using docker to sweep the problem under the rug under rug while getting credit for being modern and utilizing the best and latest tools available.

His final response was not to you, but to all the people dockerizing php apps without honestly tallying up the real cost/benefit due to a biased affinity and comfort with languages no longer suited to the changing world.

That's just my tl;dr of his argument, not an endorsement.

Yes, and I agree with that point, but my original comment was made in the context of an article that (in my view) was making a very different point, and I don't think my comment makes sense out of that context, nor do I think the author really engaged with what I was saying. Certainly the rhetorical questions which immediately followed his quotation of my comment were totally irrelevant to anything I believe I actually said.

Definitely agree that we should move towards languages that allow you to bundle/package/binary the whole thing.

Damn docker is so good for local development at least. Was showcasing a script for putting metrics into InfluxDB. The guy looked at me like I was a freaking sorcerer when I had it running in seconds with one command: `docker run influxdb`

I think of docker as a learning tool, it is a repository of systems others have created that you can learn from. I agree you can use any build system or write scripts by hand and get similar results but docker has created a standardized ecosystem you can quickly learn how to do just about anything from.

TFA: "when you consider the hoops you have to jump through to use Docker from a Windows machine"

What are those hoops? I'm curious / slightly worried as to what I missed. Surely author didn't mean enabling Hyper V / virtualisation? Thanks.

I've been using Docker on Windows for well over 2 years. It was certainly difficult setting it up on Windows in the old Docker Toolbox days (circa 2016), so perhaps the authors experience is from then.

Since the Docker for Windows client was released I've had no problems at all. (In the interest of full disclosure there were a couple of early beta issues that were quickly fixed, however that's to be expected).

Ah, cheers, so it was probably that - running "Docker Toolbox" in Virtualbox was not a very good experience indeed (the mappings were a royal pain), but since this was a 2018 article I assumed we'd all be talking about Docker for Windows. Author lost me with that, then.

It's been very smooth for me as well, especially after figuring out how to do write docker compose files. Which is arguably not hard at all, once you get used to them. :)

Regarding "uber binaries" vs containers. Doesn’t containers like docker provide other license options? i.e. you can mix GPL and proprietary licences? I'm no expert on this, so please correct me if I'm wrong!

What matters from a copyright law point of view is whether you've formed a derivative work or not. There are a wide range of legal views on when exactly that happens. But almost everyone agrees that the technical details of how your program talks to some other program that it makes use of are irrelevant. If you're linking into a single executable but the parts only invoke each other using a simple, generic interface, there is most likely no derivative work and license compatibility doesn't matter. If you're running two separate processes communicating via pipes, but this is a deep integration where you pass complex datastructures back and forth, then most likely they form a derivative work and you do need to think about license compatibility.

Docker is why I have lost interest in working in software development.

There’s reasonable money showing small/medium companies how their early/mid-2000s dev/deployment processes can be modernised without all the cruft of “containerise all the things” or even “aws/gcloud/azure/etc all the things”

It’s not glamourous but it’s also not chasing fads or cool-kid flavours of the week.

Docker is what gave me courage to expand my skill set with devops!

Previously I was terrified of touching any server for fear of messing something up and then having to twiddle with tens of config files and apt-get. Now I just rebuild the image and deploy. Its 100 times easier.

I missed the ansible train, but my understanding is that ansible is fundamentally just doing the same stuff, just automated and better documented. If you make a mistake, you can mess up many servers and it will be a pain to fix.


I don’t like docker, it’s a complex kludge, but it now pervades every aspect of development and operations. I just don’t want to work with it, which means I don’t want to work in most computing environments.

There are plenty of non-Docker environments still out there. Gee wiz, are there ever. I have no idea where you'd get the idea that "most computing environments" are Dockerized.

Have you read any job ads lately?

Job ads in general are very depressing if you're somewhat older and more experienced. There are about 6 bandwagons rolling along, with various ropes tied between them. If you're not a fan (or you even dislike) one or two of those wagons, chances are one of the other wagons you can accept is tied to it.

Node.js, Docker, DevOps, TDD, Agile, blah blah blah. Yes each of those can be quite useful in the right situation, but it is a symptom of lack of experience (particularly where management and tech leadership is concerned). A good many of the actual jobs and work environments is probably being held back or otherwise limited by the companies' need to have some of the more common popular tools in their list. But they don't know that, and there's no point arguing it. If the job still attractive, one's best hope is to get in there, get some seniority, and quietly start moving things in the right direction. Or maybe you find out that their buzzword-soup is actually seasoned right for their situation...

But to be fair, part of our problem industry-wide is the so many choices we have. You could argue that choice is good, but it creates fragmentation which eventually self-organizes into bandwagons. The huge question is, "In the end, are we better off?" I greatly suspect the answer is, "Not much..."

I sympathize with you and have also found this frustrating, but note that job ads are first, speculative, and second, seen as marketing material. Many ads will list fads because they think it makes the company seem like it's on the cutting edge and they believe it will attract candidates. Most such companies will end up not doing anything significant with the tech.

"In the cloud" has become such a thing these days that companies that explicitly avoid the cloud on the technical side end up writing job ads that say things like "help us transition to the cloud!" I don't know if this is now considered standard marketing/HR mumbo jumbo or what, but just like a resume, a job ad should not necessarily be taken at face value.

Those very definitely depend on where you're planning to take a job, and what part of the field you plan to work in. Upon your suggestion, I went and scrolled through some job ads and was satisfied that "most" were still not yet Dockerized.

Here's what I tried: https://www.indeed.com/jobs?q=software+engineer&l=New+York%2...

Have your read any job advertisements lately?

Did you feel the same way about Solaris zones?

Zones are backed by ZFS, so thats 90% of my headaches gone.

Not that I want to manage solaris though...

"I don't like technology, it's a complex kludge, but now it pervades every aspect of development and operations" See how silly this sounds?

All new technologies are inherently more complex than the last ones, solving some problems and introducing others...

I totally disagree.

I think that technology is driven mainly by waves of simplification, not added complexity.

Indeed this is exactly why I don’t like docker because it introduces fresh new deep complexity.

Surface simplicity often comes at the cost of hidden complexity underneath. As long as users are not exposed to the underlying complexity, everything is rosy. But for things like Docker, I don't think that's a reasonable expectation.

No, surface simplicity should not mean sweeping the complexity under the rug. It should mean exposing the bare minimum in the level of abstraction being used, and doing same for the level beneath it, etc. This is how you get a scalable and maintainable structure which can do very complex things.

I only know Docker from a high level (the concepts and the problem it tries to solve) so I'm not in a position to give an opinion about it's implementation, but I think a reasonable expectation from every technology is that it should be as simple to use as the use case requires.

Of course, sometimes the problem isn't the technology but the way it it used. You won't try to make a commercial airliner suitable as a quick means of transportation to your local convenience store. It's not even overkill - it's just the wrong tool for the job. The problems start when that's considered the cool thing to do. Unfortunately, technology isn't always about what's good, but many times about what is trendy, but things will eventually converge to what works best, even if they have to go through the longer path.

This is well stated. But as anyone who has used a framework or other higher level tool knows, at some point you need something it can't do... and you're forced to dive under the pretty cover and get dirty. Sometimes, often even, the time you save with the simple exterior is totally offset by the time spent dealing with the edge case. Then special kludges are built to deal with those issues, until eventually you have 10x as much complexity in your kludges than the original fancy tool saved you in effort. Then you ask yourself, "Why did we do this?"

So there is the challenge - finding the tools which give enough advantage without imposing too many future burdens. I'm not one of these people, but there are people who still use C and Make and bash and build huge, impressive systems. Sometimes narrow and deep means capable and happy.

I don't want to come across as disagreeing with you, but can someone explain the complexity they see in docker/containerization?

It's always seemed to me as nothing more than a way to package a (mostly) preconfigured piece of software.

“Storage containers”, “network port mapping”, “layers”, I don’t even know the terms for actual images versus instructions to build images.

Containers aren’t even encapsulated like virtual machines, they are tightly bound to the kernel of the host. Ugh, nightmare.

Then management of all this. Ugh.

Complex, complicated and has necessitated effectively building operating system infrastructure within an already complex operating system to facilitate all this complexity.

Which bit of this is simple?

I've never felt like any of this was "added" complexity though, just different complexity. Managing a full operating system inside a virtual machine or packaging an application for installation with dpkg/rpm/etc is not any simpler, it's just the complexity we have suffered with to date so we are more familiar with it.

I more or less decide whether I dislike working with some specific technology by how much time I have to spend fucking about with configuration and troubleshooting instead of actually writing code.

I'm currently developing Sitecore websites and working with the CMS is pretty demoralising.

I actually agree with that statement. Tech is such good money though.

But if you knew, going in, that you likely would end up hating your career and wishing you had chosen otherwise, then why not aim for a career that you KNOW will pay incredibly well, also knowing that it may be soulless... but that you'll be able to retire much earlier and then do something useful? Aiming to be a hedge fund manager would be a much wiser choice, I think.

I make six figures and work remotely in a low cost of living area that I happen to really love. My work may be fairly pointless on net, but it's interesting and challenging.

Then you're doing it right!

If not docker then what? I heard about kubernetes but it show less polished.

My use case: I wish to deploy easier Postgres and python/.net core apps. That is.

Docker is messy, containerization is full of s##t and bugs. But I'll take the gamble.

The world moving toward standardization is a BIG deal. Docker doesn't add complexity. Docker reveals the real complexity of the problem when you require a stable solution.

Let's take this simple example. Let's say you setup shop with a Virtual Server that has Ubuntu xx.xx and you write a simple PHP code that sends an email:

   mail( poor@soul.com, "spam" );
It works. It works fine not because it is simple, but because it hides the complexity.

-> It is not clear what it sending the email here.

-> It is not clear what PHP is using to send the email.

-> It is not clear how your Virtual Server is relying the email.

-> It is not clear what will happen if it fails.

-> It is not easy to change after you have coded it.

That single line is very simple and works. But it is a mess as you keep progressing. The reason I keep investing in Docker is this: I have been burned to the core by complexity that just start pooping out on my face as my day moves.

Imagine you have this line of code instead:

   mail_service->send( poor@soul.com, "Still spamming" );
You have the following:

-> Consul managing your nodes.

-> Mail nodes that expose a mail_service

-> The Mail service needs configuration. Tricky. Can setup and configure an SMTP server, load up and build some template docker file made by some other guy, or pull the MailChimp docker container and give it my API keys.

-> Maybe I'll have three instances. Prioritize the most efficient one and leave the others just in case.

-> Mailchimp goes down? Consul moves traffic to my other nodes.

-> Mail failure? Report to my logging node. Maybe we need to standardize this. So I just load my mail node that integrates with my logging node.

-> Moving from MailChimp to MailXYZ doesn't require any code modifications. No updates for the code. No downtime.

So does it work now? Uh, tough call. Yes, it is a gamble. It is a big gamble. But it is a big gamble that will make us all better off. So let's invest in it. Remember that the tech sector was mostly made out of technological gambles.

TL;DR: Docker is messy. But not because Docker or containerization sucks. It is because they expose the messiness of the real world. Your development WAMP hides a ton of this complexity that only reveals itself once you start scaling and going through practical tests.

I think we should look into why we need all this garbage in the first place. Sane, reproducible package and configuration management would go a long way, I've been meaning to see whether NixOS fits the bill.

Until then, static binaries seems like the least bad option, and for the containerization you can just use the native OS features. And I don't understand how people take this to mean they have to write all their software in Go or Rust, any language can be compiled into a static binary.

I disagree with this article because I see successful Docker deployments every day.

> If you are using Docker and Kubernetes for an ordinary website, then please stop. There are simpler ways to run a website. We’ve been building websites for 25 years now, and we didn’t need Docker.

Sure, if that's the only kind of workload you ever plan on running. No one is really doing this though so this is just useless outrage. This is like Fox News for developers.

"[blank++] isn't necessary for ordinary websites because you really only need [blank]" is always such a generalized context-less statement that it barely deserves being written at all. :-)

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact