
The Docker Compose Specification - alexellisuk
https://www.docker.com/blog/announcing-the-compose-specification/
======
justincormack
Previously Compose did not have a specification, only the implementation, and
it was tied to Docker's products. The aim here is to open it up, and let the
specification evolve in new directions, such as better supporting deployment
on Kubernetes, as an example. It is supported with some other implementations
but we want to bring these closer together.

Compose has a lot of users, and hundreds of thousands of Compose files on
GitHub, so we wanted to open up development as much as possible.

(I work at Docker)

~~~
e12e
> better supporting deployment

Is there any "supported deployment" for docker? I had the impression that
swarm has been sidelined, and docker-compose was discouraged. Is there a path
to deployment on kubernetes that uses only compose? (I have a hard time seeing
how that'll work with ingress, and "compose" well with other kubernetes
deployments?)

We have some on-prem stuff that would work on single-node swarm - but I've
felt that compose has been in a bit of limbo this past year or so... Is some
kind of production style deployment comming back?

~~~
ensignavenger
It was announced a ~month ago that Mirantis (Aquirer of Swam and Docker
Enterprise) will continue to develop Swarm with no implied death date.
[https://www.mirantis.com/blog/mirantis-will-continue-to-
supp...](https://www.mirantis.com/blog/mirantis-will-continue-to-support-and-
develop-docker-swarm/)

It is also an open source project, so even if the corporate overlords abandon
it, the code could live on!

~~~
GordonS
I certainly hope so - Swarm is a _wonderful_ orchestration system for small
deployments (at least, I haven't tried to use it for large ones), and is
really easy to use. It's also _vastly_ simpler to comprehend and operate that
Kubernetes, and doesn't eat CPU & memory for breakfast. I'd wager that Swarm
would be a better fit than Kubernetes for most systems.

~~~
rumanator
> I certainly hope so - Swarm is a wonderful orchestration system for small
> deployments (at least, I haven't tried to use it for large ones), and is
> really easy to use.

I would argue that Docker Swarm is a wonderful orchestration system for all
practical real-world applications, with the notable exception of really global
and gigantic multi-region deployments.

Docker Swarm works superbly out of the box and it's trivial to setup a multi-
node heterogeneous cluster of COTS hardware and/or VM instances.

~~~
Spivak
I think there's a more specific statement about k8s vs swam. Swam is much
better at running other people's applications that you're treating largely as
a black box -- k8s excels at running your in-house app. The simpler deployment
model means there's far less environment specific weirdness you need to
contend to to make an arbitrary app run.

~~~
GordonS
The systems I've deployed with Swarm have generally been a mix of both - not
sure why it would make a difference if you were running in-house or off-the-
shelf images?

------
andybak
I've been mostly avoiding Docker, Docker-compose etc. (We got stuck at the
"Ansible > Digital Ocean works fine for us" stage. Docker was too new at the
time and never really had a reason to re-examine things)

But just this past week after a ton of pain previously trying to get a working
Geodjango stack running on my Mac locally for experimentation I discovered a
docker-compose script that gave me everything I wanted in about 5 minutes. A
sample app. Django, nginx, redis, PostGIS and all the complex dependencies -
all working together. I literally typed "docker-compose up" and I had a
working web GIS-enabled web app running on localhost. And the fact that the
deployment to staging is hopefully just as simple makes me smile.

Of course - production is a different story as I then have to worry that all
the magic pieces aren't full of backdoors and security holes but I guess
that's a job that needs doing anyway.

~~~
0x138d5
There are some things that have to be made clear to the team when moving to
containers (don't use joerandom/alpine as a baseimage, build via digest
instead of tag etc) but the speed at which you can onboard people is amazing.

I couldn't imagine deploying a Python application without Docker nowadays.

~~~
jwbwater
How is this better than ansible, salt, chef, puppet, etc.? Once it's automated
it's automated, no?

~~~
0x138d5
I wouldn't use the term "better" \- I still rely heavily on puppet and
ansible.

But the containerization aspect gives the option to pull up and destroy
services without worrying about polluting the host you're in.

~~~
u801e
But couldn't you get the same thing using the virtual environment provided by
python? Unless you're using a lot of modules with C extensions, you wouldn't
need to install anything outside the virtual environment.

~~~
m_mueller
> Unless you're using a lot of modules with C extensions

in my experience only the simplemost applications actually don't need anything
outside of vanilla python pip. suddenly the client wants PDF reports and
you're setting up e.g. some tex-based printing system with a bunch of host
packages needed. only containers give you the peace-of-mind that all
dependencies, current and future, can be described as part of the codebase.

~~~
u801e
> only containers give you the peace-of-mind that all dependencies, current
> and future, can be described as part of the codebase.

Couldn't the same thing be done via the package manager and a RPM spec or deb
file where all the necessary dependencies are listed and installed as part of
the package? It could be done on a VM or could be done on a machine by keeping
track of what dependencies are installed when installing the application and
handling uninstallation by removing all newly installed dependencies along
with the application.

~~~
m_mueller
but why polute host _or_ go back to VM overhead per application if we have
already a solution that is easier to handle? even just in terms of build &
launch time it's quite a bit more performant to use containers vs. VMs.

~~~
u801e
> but why polute host

The package manager can handle removing pretty much any file it installs when
uninstalling, so the host really doesn't get "polluted".

> or go back to VM overhead per application

In a development environment hosted on a VM, several applications can be
installed on the VM (rather than having one per VM) to reduce overhead. Then
testing and making code changes could be done by running a pip -e (editable)
install and modifying code in the working directory, or making the change,
repackaging and re-installing, and restarting the daemon.

With a container, at least in my experience, you need to re-build it each time
you make a change to the code, which actually takes longer than modifying an
editable install or re-building the wheel/RPM and reinstalling it.

~~~
m_mueller
typically on a dev PC you'll have multiple apps roled out for development /
maybe feature testing. if this is done directly on the host it's always
possible for these environments to influence each other. very common is to
forget to script a dependency, which only may get recognized once someone
wants to deploy on a fresh installation.

~~~
u801e
Yes, but dependencies are handled by the package manager. So if one is
missing, it's a simple matter of adding it to the list of dependencies.

In any case, the point I was trying to make is that the development cycle with
containers, in my experience, is slower because you have to go through the
build step every time you make a change. For an intpreted language like
python, that shouldn't be necessary until close to the end where you test a
fresh build before submitting the changes for review.

~~~
m_mueller
The setup is a bit more involved, but this can be mitigated in several ways.
One is docker or SSH plug-ins in editors like VS code coupled with an SSH
server included in the develop build phase, like you would use to develop on a
remote server. The other approach is adding the source tree as a volume. You
can also do a mixed approach where development is done on host, but testing
and deployment is done in docker.

------
rgovostes
Tangentially, it is a little irksome how the docker-compose tool is released
completely independently from Docker itself. There are binaries published on
GitHub for amd64, but if you want it on an ARM platform, you need to install
with pip. But pip can't install system packages, so you also need to know the
incantation to install those (python3 python3-pip libffi-dev libssl-dev).

I'm sure there's a historical reason for it not being `docker compose ...`,
but it sounds like it's a good time to promote it.

~~~
bdcravens
It was an external project acquired by Docker (originally called "fig")

[https://www.docker.com/blog/welcoming-the-orchard-and-fig-
te...](https://www.docker.com/blog/welcoming-the-orchard-and-fig-team/)

Of course that was six years ago, so it is a bit surprising they haven't done
a deeper integration since then.

~~~
fapjacks
Remember the outcry when Swarm was shipped as part of the core Docker binary?
I wouldn't be surprised if this caused some kind of cooldown with respect to
the binary assimilating features.

------
L_226
Sweet, looks good. I reckon docker and docker-compose have been probably some
of the greatest enablers for my development process, and how I think about
desiging software systems.

~~~
daveguy
Agreed. I was worried docker-compose might be abandoned for swarm-only at the
kubernetes level. Compose is a much easier level to develop and work with.
Doesn't do all the balancing and HA, but for most projects you don't need
that, just good organization of the pieces.

~~~
kalev
This is exactly why we still use compose as well. Everybody is talking about
kubernetes, but I honestly don’t see the point in adding a huge dependency
only for balancing and restarting failed containers..

I just wish there was some SaaS platform which allows you to upload a docker-
compose file and run the containers for you. That way, you don’t have to touch
the underlying OS.

~~~
larrybud
Azure App service lets you do this. [https://docs.microsoft.com/en-
us/azure/app-service/container...](https://docs.microsoft.com/en-us/azure/app-
service/containers/quickstart-multi-container)

------
crucialfelix
Good news! Everybody wants to just write a docker compose file and use that in
production. Deploy on ECS or k8s or heroku, it would be awesome. That's the
simplicity most people want for small projects.

------
pmichaud
I'm not a devops person at all, but this pile of abstractions seems like it's
going the wrong way. Like, this seems almost like a parody?

You have this problem where you want to set up infrastructure in an abstract
way, so you come up with a way to write some infrastructure requirements and
now you have a program that does all the actions necessary to take your setup
from wherever it is, to what you specified.

Then you have multiple of those systems, and it gets confusing. So you come up
with a generic way to specify the abstractions that can be sort of compiled
into any of the flavors, so that you can set up infrastructure using any of
the sub-abstractions available, instead of only a specific one.

This has the smell of a FactoryFactoryFactory sort of thing. Like, if this
whole cottage industry were my architecture project I would be saying to
myself "This is too much, a rickety, foot-gun-laden tower, destined to
crumble. I've obviously chosen a poor abstraction, lemme back way the hell
up."

Am I just totally wrong here? I feel like I'm taking crazy pills.

~~~
jaredklewis
Sometimes the proof is in the pudding.

I’ve had similarly adverse reactions to the seemingly “unnecessary” complexity
of many things: docker, webpack, autotools, react, AWS, just to name a few.

But I’ve found that, upon closer inspection, the software is usually
complicated because the problem domain is complicated. Further, as an
inviolable rule, the most popular tools for any problem are the ones that
don’t make you rewrite all your code; popular tools build on top of the mess
we already have. This can give the illusion that the tool is somehow
responsible for all the preexisting complexity. Tools which attempt a “clean
slate” approach are mostly doomed for obscurity.

In the case of docker-compose, I can say after years of resistance, I tried
docker compose and was an immediate convert. Setting up, documenting, and
sharing development environments has never been simpler.

~~~
mumblemumble
I would add a caveat: Docker compose is nice for setting up development
environments _as long as everyone 's developing on the same platform._

I work in a shop where various developers work on Linux, OS X and Windows
(useful for dogfooding - we support deploying to multiple platforms), and,
there, Docker Compose has ironically turned out to be our chief source of
cross-platform headaches, due to annoying differences in how it works on each
host platform. Particularly with the networking subsystem.

~~~
williamdclt
> Docker Compose has ironically turned out to be our chief source of cross-
> platform headaches

Which is the whole point of it. If it's not, it's just a bad abstraction.
Docker-compose itself becomes the only variable

~~~
mumblemumble
Perhaps I should rephrase:

Since moving to Docker, we now have _more_ cross-platform problems than we
used to.

We continue to use it anyway, because there are some other benefits that we
feel outweigh shortcomings like these. But still, it's annoying.

------
mfer
This is pretty exiting for 2 reasons...

1\. Writing all the YAML to deploy an application into Kubernetes is a lot of
work. If one does everything fully filled out it's pretty normal to have over
a thousand lines of YAML... or much more. With lots of repeat yourself. Tools
built from a simpler spec that can "compile" to that would be so great. Like
writing in a higher level language instead of machine code or assembly.

2\. There are a variety of platforms now. I first thought of Kubernetes vs
Nomad. But, Kubernetes using Istio is a different platform from Kubernetes
where applications use Ingress. There is more diversity showing up. Being able
to define an application in one spec and "compile" for different situations
would be useful.

Just my 2 cents. As a Kubernetes user I'm happy to see the higher level tool
bar moving. It's desperately needed.

~~~
znpy
> But, Kubernetes using Istio is a different platform from Kubernetes where
> applications use Ingress

This. After almost a year working with kubernetes I came to the same
conclusion. I am/was studying for the kubernetes certification (cka) but in
some way it's useless.

I mean, it would prove basic proficiency, but the whole ecosystem is so varied
that wouldn't necessarily mean actual and practical proficiency.

It would probably help passing through vogon recruiters but a technical
interviewer that knows his way around kubernetes wouldn't necessarily buy
that.

------
clSTophEjUdRanu
This is cool- hopefully this allows standardized podman-compose.

~~~
jabl
There is [https://github.com/abalage/pods-
compose](https://github.com/abalage/pods-compose)

------
tln
My two wishes are:

1\. Bring back extends

2\. Specify services that are only intended to be run

Maybe #2 is possible now and I just don't know it?

I wrote a wrapper at previous job around docker-compose that made it easy to
define aliases within the compose file for complete run commands... Eg to run
migrate in your rails container.

------
markl42
Nice! I wonder if there will be a mapping between compose and the cli
maintained as part of this?

(Disclaimer, I maintain [https://composerize.com/](https://composerize.com/),
hoping this helps us track parity)

------
move-on-by
This is great news. Docker really screwed up v2->v3 and have continually
suppressed Compose’s potential because it wasn’t inline with swarm. Looking
forward to Compose getting some new life put into it.

~~~
cpuguy83
Suppressed is not quite the right word. Too many inputs (from the community)
and not enough help managing and maintaining, and curating these inputs is
really what this is.

That said, I don't personally believe there exists a single specification that
can work (optimally) for both local-dev and production deployments.

Disclaimer: ex-Docker engineer, still an active maintainer on the main docker
project.

------
wwarner
To read the spec: [https://github.com/compose-spec/compose-
spec/blob/master/spe...](https://github.com/compose-spec/compose-
spec/blob/master/spec.md)

------
earthboundkid
This is good. My main annoyance with Docker in the past was that I would make
a decent Compose file for local dev but then have to rewrite everything in a
proprietary YAML format for use in CI and prod.

~~~
peterwwillis
You'll still be doing that with this new spec. It's still just a "how would
you like me to run multiple containers" spec, it's not defining the
relationships of arbitrary infrastructure and services on arbitrary
networks/providers.

What's going to happen is, you (or the dev) will fill this thing out to the
best of your ability. Then someone's going to run it in one env; it won't
work, but Ops will modify it to make it work. So then you'll commit that as
your one Compose file. Then someone will try to run it in a different env, and
it won't work; they'll fix it to make it run. Now you have two Compose files
to maintain, with 3 different use cases. So to make it easier to manage, you
go back to one for dev, and one for each env. All this because the different
components could not be defined independently and loosely coupled, because the
spec (and tools) don't support that.

I like Compose for what it's designed for: let a dev mock something up
locally. But it's going to become a pain in any environment that scales and
will need a different solution.

------
nicksantos
Docker Compose has always been an awesome file format, built on top of an
idiosyncratic orchestration engine that doesn't interoperate well with
anything else.

Writing a spec is a good first step! If work is underway to make this a shared
multi-service dev environment that works across Kubernetes, ECS, etc, that
would be even better.

When I read the spec, there are ominous words about partial implementations.
That makes me nervous. But I'm cautiously optimistic.

~~~
justincormack
There are partial implementations because some of the things in the existing
spec are just not a good match for ECS or Kubernetes, for example. The aim is
to fix those things in the spec. There are conformance tests so we can
hopefully start checking implementations soon.

~~~
nicksantos
Yay for conformance tests! What's the strategy for handling things that don't
make sense for other platforms? Do you expect that they'll be removed from the
spec eventually, or fit in as some kind of platform-specific extension
mechanism?

~~~
justincormack
We are expecting to have to have some platform specific extension mechanism
for some things eventually, but yes I think some things can be removed too.

------
lxe
Just diving into Docker, and I'm already a bit confused... There's a docker
stack (or swarm) that allows me to define containers (services?) and their
networking... but then there's also docker compose, which does the same thing?
Even the yaml config specs are similar.

~~~
peterwwillis
Docker is a tool that includes a daemon that runs in the background and a
client app that communicates with the daemon. The daemon runs a packaged-up
program and its dependencies ("image") as basically a _chroot_ environment
using an overlay filesystem and isolated network, in a custom cgroup and
namespace (basically, process space/virtual filesystem paths different from
your normal user process space/filesystem).

Swarm is a (sadly) soon-to-be-abandoned way to spin up a cluster of multiple
hosts with Docker daemons; don't waste your time on it. The comparable
solutions are AWS ECS or [to a lesser extent] HashiCorp Nomad.

Docker Compose is a console app that takes a config file and starts multiple
Docker containers, creates volumes, and configures Docker networks - on a
single Docker instance, so it's only really useful for testing stuff locally
that requires multiple containers running at once.

~~~
cholmon
> Swarm is a (sadly) soon-to-be-abandoned way to spin up a cluster of multiple
> hosts with Docker daemons; don't waste your time on it.

It seems this inevitable abandonment has been abandoned as of Feb 24, 2020...

[https://www.mirantis.com/blog/mirantis-will-continue-to-
supp...](https://www.mirantis.com/blog/mirantis-will-continue-to-support-and-
develop-docker-swarm/)

------
nixpulvis
Maybe they'll address
[https://github.com/docker/compose/issues/745](https://github.com/docker/compose/issues/745)
now...

------
stevebmark
This is great. Docker sometimes feels like a tangle of hacks, for example
docker-compose build and docker build diverge and don't use the same layers.
More standardization is better

------
AskTheRightQs
This is great. I might use it in my web app
[http://www.trackmycouriers.com](http://www.trackmycouriers.com)

------
oehpr
ok, so... what? The notion here is to take the compose file, which is
basically saying "this is what I want the state of docker to look like", and
turn it into a generic application specification that multiple orchestration
engines can contribute to and use... And... How would this work? Are we gonna
have this smorgasbord of definitions in compose now with a subset of each
working on specific orchestration systems, or something? Because that would be
_completely_ useless.

So unless there genuinely is a commitment to being able to have each
orchestration engine just _unreservedly guaranteed_ take any definition
possible in docker-compose and deploy it, then there _is no value_ to be
gained here. None. Without that guarantee, then _for every system_ you attempt
to deploy a compose file to, you must DEEP DIVE into its internals to
understand its limitations and costs and tradeoffs.

This gigantic massive timesink of a process is something I have always had to
do. And I always discover all kinds of things that would have absolutely
blocked practical deployments. This is the real thing that's bothering me, not
that k8s and azure and docker all can sorta read some subset of a
configuration. I don't care at all about that. It's not useful.

~~~
hinkley
I'm not saying this will happen, but one way I might like to see this play out
is a bit like the following.

I have a bunch of apps. Some of them talk to each other, some do not. In
general, if I deploy 5 copies of something, you should assume I mean "never on
the same box" by default. (I'm looking at you, Kubernetes. WITAF.)

You could, as a vendor, do a lot of sophisticated work on the spanning tree to
reduce the amount of network traffic versus loopback traffic. You might try to
spread out unshared volumes across as many motherboards as possible. You could
differentiate on how to most efficiently and/or stably migrate from previous
config to new config, or vice versa. You could do a bunch of knapsack work,
including (and this is a pet peeve of mine) pre-scaling for cyclical traffic
patterns.

If you've ever looked at the Nest thermostat, one of several defining features
was that it figures out the thermal lag in your heating and cooling system and
it can make the house be the correct temperature at a particular time, rather
than waiting until the appointed time to do anything. If a hockey puck on my
wall can do shit like this then why doesn't every cloud provider do this day
1?

Tack onto this some capacity planning infographics, and a system to schedule
bulk, low priority operations around resource contention, and I could probably
help get you a meeting with my last boss, my current boss, and probably at
least the one after that.

------
crb
> Docker is working with Amazon Web Services (AWS), Microsoft and others in
> the open source community

Google?

