For orchestration, offhand the most active projects seem to be Kubernetes , Swarm , Deis  and Mesos. Kubernetes is built primarily by Google, Swarm by Docker and Deis by EngineYard, with each team having experience in different areas (orchestration, containers and full-tier solutions, respectively).
Kubernetes, Swarm and Mesos handle the orchestration portions only, while Deis is a more feature-complete solution that handles the CI and registry portions as well.
Delivering updates to these solutions and doing so with zero downtime is still very early as well. Kubernetes has a rolling update mechanism, but it can still (occasionally) result in downtime if not setup correctly. Deis handles updates via git-push and will ensure that new containers are in place before the old ones are taken out of service. As for Swarm, my personal knowledge is limited in regards to rolling update, so I'll leave that for someone else to fill in.
For building and delivering images, there are as well multiple solutions. The common solutions are to use a Docker-compatible registry such as Quay  (Disclaimer: I'm a lead engineer on the Quay team) or the DockerHub . In addition to supporting simple image pushes, both registries as well support building images in response to GitHub or BitBucket, so they can also be used as an integrated CI, of sorts. Both these services are paid for private repositories. Docker, as well, has an open source registry  which can be run on your own hardware or a cloud provider.
Registries are secured by running under HTTPS at all times (unless explicitly overridden in Docker via an env flag), and having user credentials for pushing and (if necessary) pulling images. Registries typically offer organizations and teams support as well, to allow for finer-grained permissions. Finally, some registries (such as Quay) offer robot credentials or named tokens for pulls that occur on production machines as an alternative to using a password.
In terms of how servers know when updates are available, it all depends on which orchestration system is being using. For Kubernetes, we at CoreOS has been experimenting with a small service call krud  which reacts to a Quay (or DockerHub) image-push webhook and automatically calls Kubernetes to perform a rolling update. Other orchestration systems have their own means and methods for either pushing or pulling the fact that the image to deploy has changed.
Hope this information helps! (and if I forgot anything, I apologize)
It's hard to know whether to wait for Docker to provide a solution or to use something that already has momentum. Take networking for example. Solutions have been bandied about for the last year or so and only now do we have something that's production ready. Do I rip out what I already have for something that is docker native or do I continue with the community based solution.
Storage (Data Locality) also follows a similar path. Kubernetes provides a way for making network based storage devices available to your containers. But now, with the announcement of Docker v1.9 do I go with their native solution or something that has been around for ~6months longer?
I've been working with these technologies for the past year and it has not been easy building something that is stable with a reasonable amount of future-proofness baked in.
Here are a couple guides that walk you through your first Docker cloud deployment:
This gives you a private build and registry service that are secured in your own VPC and accessable only through authenticated API calls.
The software that sets this all up is open source and free, but you do pay for your AWS usage (EC2, ELB and S3).
Servers know how to fetch the new version by issuing one `release` command that triggers zero-downtown rollout on the EC2 Container Service (ECS).
Will it work with existing service discovery systems? What happens if a container I'm linked to goes down and comes up on another host? Do I get transparently reconnected?
There's so much involved with the abstraction they're making that I'm getting a suspicion that it's probably an unwieldy beast of an implementation that probably leaks abstractions everywhere. I'd love to be proven otherwise, but their lack of details should make any developer nervous to bet their product on docker.
vxlan (overlay) and bridge, depending:
"Docker ships with 2 drivers:
bridge -- This driver provides the same sort of networking via veth bridge devices that prior versions of docker use, it is the default.
overlay -- Not to be confused with the "overlay" storage driver (thanks overlayfs), this driver provides native multi-host networking for docker clusters. When using swarm, this is the default driver."
Basically integrating OVS APIs into Docker so it could use more mature networking code as well as VXLAN forwarding. VXLAN is basically IP encapsulation (a 16-bit ID) that the networking industry has standardized on. It more or less allows for L2 over L3 links. I like to think of it as the next Spanning Tree.
So the unwieldy part is the weight OVS brings as well as the VXLAN encapsulation in software - both of which have momentum towards being more lightweight.
I feel like I'm just not understanding the "docker approved" approach. Which is surprising because docker itself is so great.
The networking stuff seems interesting though, I'm very curious how the rest of the ecosystem will evolve to take advantage of it or not.
Brian Grant has explained it recently here: https://github.com/kubernetes/kubernetes/blob/master/docs/wh...
"The technical definition of "orchestration" is execution of a defined workflow: do A, then B, then C. In contrast, Kubernetes is comprised of a set of control processes that continuously drive current state towards the provided desired state. It shouldn't matter how you get from A to C: make it so. This results in a system that is easier to use and more powerful, robust, and resilient."
Look into PaaSes that are built on top of K8s, like Red Hat's OpenShift v3 or Deis.
Me and a team are working on another open-source project, Convox, that offers this on AWS.
One differentiator is that we use "pure" AWS for everything.
Load balancing comes from configuring ECS and an ELB the right way. Logging is based on Kinesis and Lambda. We're seeing great reliability and manageability with this approach.
EDIT the missing content..:
I mean currently it mostly is for the big users. There aren't too much things for "small" users. The big things like kubernetes, etc are really hard to configure / maintain, etc.
I mean it's easier to maintain ansible / puppet / chef / etc... - scripts than maintaining a real "docker" environment. even looking at deis, flynn, openshift its not just run "this, upgrade with this".
after you setup the hole thing you need to create huge buildpack scripts or Dockerfiles or kubernetes configs or whatever.
you just needed process isolation, now you build a infrastructure on top of a infrastructure.
The direction where it is going is to create an "internet operating system" you upload your application and don't care on which server it is running.
Currently that above problem has not been solved yet, and what we have is just bunch of tools to put them together and DIY.
The real power of this will come, once cloud providers will allow you to simply upload the images without having to build that infrastructure yourself.
I think you're right to fall back to ansible / puppet / chef, because this technology is simply not ready yet.
This is especially true if you're using public cloud (and while you can make it work in AWS, it will cost you more; both due to amount of effort, but also due to overhead imposed by it. Remember, AWS still charges you by VM).
There might be some benefits of using it right now when you have your own datacenter with physical machines. It could provide costs savings, compared to running your apps on dedicated physical servers or even using VMs.
as said the only thing gain for docker would be process isolation, so it should be really really awkward simple and useful on low end hardware. (as the other solution already does) and getting process isolation with cgroups isn't too hard on newer kernels. (# systemctl set-property httpd.service CPUShares=500 MemoryLimit=500M)
so what the product needs to have:
- process isolation
- easy configuration
- configure the os/software and update it easily
- nothing more than a bunch of lines per project (no dockerfile frickling)
- binary / git rollouts
If you're well invested in puppet, using puppet is going to be easier because you know it. You can happily use docker with puppet. Stop puppet from installing $APP and instead use it to docker pull && docker run $APP.
This means the logic for building your application has obviously moved to the Dockerfile. You cannot currently get rid of this logic, only hide it in abstractions. I prefer it living in the apps repo as it's a nice seperation of concerns, but you obviously would prefer it to be magic, which you can have but at the price of versatility.
If you are able write code to build and deploy apps, then moving over to using docker should be pretty trivial for you. However, I actually see docker as a replaceable part, whilst kubernetes might actually be here to stay.
* shared folders to persist necessary files. and thus volumes. and a few years later distributed volumes.
* not just isolate a process, but all its dependencies as well. no point in having a shared .so file which everybody can change, while just a single process is isolated. and thus a whole sandboxed container.
* and then deal with the size of a full sandbox, until you need some way to share unchanged files. and thus images and layers.
* and so on and so on.
big things always start small. at least in docker's case they did start small, and they're still small and lean individual projects. feel free not to use docker compose or anything else.
The project is only month and a half long so if you want I can talk you through privately
My favourite is Cloud Foundry, because I've worked on it and I trust the way it's built.
Here's how I deploy an app:
- Build time variables: https://docs.docker.com/engine/reference/builder/#arg
- docker volume subcommand, finally we have a sane way to clean up left over volumes which are left by default.
Thanks for the shoutout to flocker in the docker volume example.
$ docker volume create -d flocker --name=myvolume
$ docker run -v myvolume:/data busybox sh -c "echo hello > /data/file.txt"
$ docker run -v myvolume:/data busybox sh -c "cat /data/file.txt"
on a related note we've been have been having a great time seeing volumes get mounted and remounted with swarm 1.0/flocker.
That's really, really exciting. Been looking forward to the GA release of Swarm for quite some time, glad to see it here.
On a side note, is there any authentication / security for public-ip docker multi-networking?
feedback welcome @ianmiell
The point is that both are platforms. Application developers shouldn't really need to care about the internals of a PaaS, for the same reason that I don't really care about the internals of the Linux kernel.
One answer is to stick with whatever AWS offers. The EC2 Container Service (ECS) offers a practical solution to running your app as containers on a cluster of multiple servers without adding any other software for orchestration.
It has a lot of deficiencies, but these are solved with other AWS services. For example you still use an ELB for load balancing across your servers and containers.
The rest of the ecosystem is pushing for a more radical container future that doesn't rely on AWS.
Joyent has a true "container-native" infrastructure with Triton. You can run your app as multiple containers without considering anything about servers.
Tutum (recently acquired by Docker) operates the same way. I'm excited to see what this platform offers if it evolves in lockstep with Docker core for networking, logging and data.
Kubernetes and Swarm are projects that solve the low level challenges of orchestrating containers in a cluster. But you almost certainly need to build a lot more around these systems to get logs, load balancing, etc.
When working with multiple images, coordinating ports, volumes, environment variables, links, and other things gets very troublesome very quickly as you get into using a mish-mash of deployment and management scripts.
Compose aims to solve this problem by declaratively defining ports, volumes, links, and other things.
Compose does allow you to scale up and down easily but it doesn't do auto-scaling, load balancing, or automated crash recovery -- this is where Swarm comes in.
Kubernetes does what both compose and swarm do, but in a single product.
Both Swarm and Kubernetes are designed to accommodate provisioning of resources across multiple hosts and automatically deciding which containers go where.
Compose, Swarm, and Kubernetes are all things you can install yourself.
Tutum is far bigger and the scope of its usage falls well outside of what Kubernetes and the other's do, but suffice to say that it's more of a PaaS than anything else.
Someone please correct me if I'm wrong, I'm not very familiar with Swarm, Kubernetes, or Tutum.
Compose is a tool that issues commands to docker engines and will e.g. spin up containers and link them together in the right order. It makes rote docker commands a little less painful. It can talk to a single engine or, apparently, many engines via Swarm.
When it comes to putting providing a production "service" based on containers you need to be able to add and remove docker engines, to, for example, deploy new code via rolling update. Google Container Engine (GKE) and Amazon ECS marry docker concepts with front-to-back implementations of hosted infrastructure like server instances and network load balancers. Over-simplified, each has an agent that runs on a docker engine and does work similar to Compose and Swarm against AWS and GCE. Google's daemon is called Kubernetes.
Compose: Multi-container orchestration. You define an application stack via a compose file including links between containers (web front end links to a database). When you run docker compose up on that file, compose stands up the containers in the right order to deal with dependencies.
Swarm: Represents a cluster of Docker Hosts as a single entity. Will orchestrate the placement of containers on the hosts based on different criteria and constraints. You interact with the swarm the same way you would any single docker host. You can plug in different discovery services (Consul, Etcd, Zookeeper) as well as different schedulers (Mesos, K8s, etc).
Tutum: SaaS-based graphical front end for Docker Machine, Swarm, and Compose (although it's Stackfiles not Compose files w/ Tutum). The stuff described above is handled via Web GUI
You didn't ask but Machine is automated provisioning of Docker hosts across different platforms (AWS, GCE, Digital Ocean, vSphere, Fusion, etc).
Docker Hub is a gui on top of the Docker registry.
GitHub is a gui on top of git.
And so on.
Swarm is a container manager that automatically starts and stops containers in a cluster using a scheduling algorithm. It implements the Docker API, so it actually acts as a facade that aggregates all the hosts in the pool. So you talk to it just like you would with a single-host Docker install, but when you tell Swarm to start a given container, it will schedule it somewhere in the cluster. Asking Swarm to list the running instances, for example, would list everything running on all the machines.
Kubernetes is also a container manager. The biggest difference is perhaps that it abstracts containers into a few high-level concepts — it's not tightly coupled with Docker and apparently Google plans to support other backends — that map more directly to how applications are deployed in practice. For example, it comes with first-class support for exposing containers as "services" which it can then route traffic to. Kubernetes has a good design, but for various reasons the design feels overly complicated, which is not helped by some of the terminology they've invented (like replication controllers, which aren't program, but a kind of declaration), nor by its somewhat enterprisy documentation.
Kubernetes is also complicated by the fact that every pod must be allocated a public (or at least routable) IP. If you're in a private data center that already has a DHCP server set up, that's a non-issue, but in this day and age, most people probably will need an overlay network. While there are tons of such solutions — Open vSwitch (aka OVS), GRE tunnels, IPsec meshes, OpenVPN, Tinc, Flannel (formerly Rudder), VXLAN, L2TP, etc. — none of them can be called simple. Of course, plain Docker doesn't solve this in any satisfactory way, either, but at least you can be productive with Docker without jumping into the deep end like Kubernetes forces you to do.
Docker Networking is a stab at solving the issue by creating an overlay network through VXLAN, which gives you a Layer 2 overlay network. VXLAN has historically been problematic because it has required multicast UDP, something few cloud providers implement, and I didn't know VXLAN was a mature contender; but apparently the kernel has supported unicast (which cloud providers to support) since at least 2013. If so, that's probably the simplest overlay solution of all the aforementioned.
As for Compose, it's a small tool that can start a bunch of Docker containers listed in a YAML file. It's unrelated to Swarm, but can work with it. It was designed for development and testing, to make it easy to get a multi-container app running; there's no "master" daemon that does any provisioning or anything like that. You just use the "compose" tool with that one config file, and it will start all the containers mentioned in the file. While its usefulness is limited right now (for example, you can't ensure that two containers run on the same host, unlike Kubernetes with its pods), the Docker guys are working on making it more mature for production use.
(I work on Weave)
Weave Net also lets you create a Docker overlay network using VXLAN, without insisting that you configure a distributed KV store (etcd, consul, etc.). So I would argue Weave Net is the simplest :-)
More detail here: http://blog.weave.works/2015/11/03/docker-networking-1-9-wea...
There are a variety of hosted, virtualized and bare metal solutions available.
The CoreOS folks have some nice all-in-one VMs if you want to experiment.
Google's hosted Container engine is about as simple as it gets - and very inexpensive (I have been playing with it for a few weeks and have spent about $20).
The API is backwards compatible, so you if you don't want any of the new features, there is no breakage! E.g. if you're using Docker 1.8 (API version 1.20), prefix your API calls with /1.20/ and you're safe to upgrade to Docker 1.9.
Also, is it possible to make Swarm answer to the same client IP with the same backend server during a 'session'? This is very important for database applications where the sync after each write may take some time and you don't want the UI to show different states from different servers until the system has settled. Hipache AFAIK doesn't offer this, IMO the biggest downside of this.