The only thing that matters then between services is their bindings, which gives you the ability to use any programming language for any service. Deployment with ECS has been going well so far for me. My flow:
1.) Push code to GitHub
2.) GitHub tells Docker, which builds a private image
3.) Docker tells my build server
4.) Build server tells ECS to update the given service
5.) ECS pulls from DockerHub, stops service, starts service
The only thing missing is that DockerHub doesn't tell my build server what tag it just built! It builds tags like dev / staging for the given branches, but doesn't relay that info over its webhook. There's a ticket open about this already and I'm sure they'll get to it soon.
Nevertheless, I'm able to administer any system -- things like Elasticsearch, db, api -- from code on a branch. This is powerful to me because I have to administer environments for everything. Rather than do all this work with Puppet, Chef, or even Ansible, I can just declare what the systems ought to be and cluster them within code branches.
With ECS coming into picture, you're encouraged to forget you even have physical boxes in the first place. If you think of the power at your finger tips that results from this development workflow, I believe it's a no brainer for everyone to jump on board and get this as good as it can be. It's going to be a huge boon to the software community and enable more services sharing.
How do you change network configuration, load balancers or other external configuration in lock step with your application? Scripting by hand?
And deployments are a just small part of configuration management. How do you find out which of your applications use libz 1.2.3 (to measure CVE applicability for example)? By looking in each and every one of them?
How do you find out which application run a cron job? And which application connect to backend x without going through the load balancer?
Which application has a client certificate about to expire? How do you guarantee two applications have the same shared secret, and change it in lock step?
With configuration management all this is just a look up away. And best of all, most of this information is authoritative.
When I come somewhere new, the use of these tools instead of a directory of miscellaneous scripts is like night and day. It literally turns a two day job into a ten minute one when the environments are complex.
Docker is great for a lot of things. It enables to do daring things to your environment even if it's so complex you don't fully understand it all because you can just shuffle containers around. But I would never use it instead of configuration management.
I would argue I accomplish most configuration management via the Dockerfile. In Docker 1.7, even if I want to use a different filesystem like ZFS, I can.
I agree with you that I probably have missed many important features in my architecture. However, it's simple and works for what I got. It sounds like in the future, I'll need a combination of Puppet / Chef & Docker rather than relying purely on ECS. For now though, it's quick & easy. Thanks for the thoughts, they all make me think.
It sounds like the poster is doing something similar to a 'Gold System Image' though with Container technologies.
Configuration Management is great, but I'm with the view that you should start with a combination of gold images/system builds and then stack on Cfg Mgmt on top of that.
Theory is you will have a known base and identical builds etc, otherwise you would get subtle drift in your configurations, especially for things which are not explicitly under control of Config Mgmt (shared library versions, packages).
Of course, this is not always feasible but if I were to start from scratch, I would probably try to do it this way.
You want to have your "Gold system image" available and this is most certainly what you should be deploying from; however your configuration management - whether it's Chef, Puppet, whatever - should be able to take a base operating system and get it setup completely from scratch.
This then solves the problem of ensure that you can completely reproduce your application from scratch; but also removes the possibly horrendous slow scale up time by using a "Gold image" that has already done this.
My current process is: Jenkins runs puppet/chef, verifies that the application is healthy after the run and everything is happy, calls AWS and images the machine, then it iterates over all instances in my load balancer 33% at a time and replaces them with the new image resulting in zero-downtime. Of course another solution is to pull out those instances and apply the update, and then put them back in.
And I'm sure someone else will have their $0.02 on their own process, which actually I'd love to hear :-)
I have an Ansible repo with an init task, which configures new boxen (makes sure proper files, dependencies etc. exist). Then to deploy, I have another task that ships a Dockerfile to the target boxen group (dev, staging, or prod) and has them build the new image, then restart. This happens more-or-less in lockstep across the whole group, and scaling up is relatively easy - just provision more boxen from AWS and add the IPs to the Ansible inventory file. Config is loaded from a secrets server, each deploy uses a unique lease token that's good for 5 minutes and exactly one use.
I'd love to hear how to improve this process, since I'm dev before ops. My next TODO is to move Docker image building locally and deploy the resulting tarball instead (though that complicates the interaction with the secrets server).
Do you end up having to update multiple containers with the same patch?
Context: I've looked at Docker, but not used any containerization yet.
One potential solution that came to mind was that if there was a standard way of deploying an application into containers, and Google/Amazon/Microsoft provided auto-updating containers, the maintenance of a secure container would be in the hands of companies who (hopefully) have the resources necessary to keep the entire stack up-to-date.
All containers running code are based on these images, so the updates are picked up on the next build/deploy.
This in the sense, i guess, that if they have a security flaw in their php that gives disk access, all the attacker will see is the content of the php container as the database will be on the next container over.
Then again the containerization seems to have come alongside devops, where the mantra seems to be "update early, update often, to hell with stable branches".
If a security flaw exists in one container due to the stack not being updated, isn't there a pretty good chance that it also exists in the other containers?
Also, for any given container, there probably still is a way for an attacker to do immense amounts of damage. With the database container you can steal customer data. With the PHP container you can remotely instruct the database to do whatever you want, or just point the code at your own database.
Pull a new base image, have your build system rebuild and deploy the relevant containers.
The same way you do in any application, container or not - you look up your documentation ;)
Love it for quick scripting containers (like netcat) too since it downloads in line 2 seconds.
Certainly can be worked around, but it's a little annoying.
1.) Create a new task and update the service with the task. From the documentation (http://docs.aws.amazon.com/AmazonECS/latest/developerguide/u...):
> If you have updated the Docker image of your application, you can create a new task definition with that image and deploy it to your service, one task at a time. The service scheduler creates a task with the new task definition (provided there is an available container instance to place it on), and after it reaches the RUNNING state, a task that is using the old task definition is drained and stopped. This process continues until all of the desired tasks in your service are using the new task definition.
2.) (way I'm doing it) Update the desired count to 0 and utilize wait_until (http://docs.aws.amazon.com/sdkforruby/api/Aws/ECS/Client.htm...) services are stable. After this, update desired count to 1.
You're going to get some downtime with #2. I'm not sure about #1 because I don't understand how something like an API service could be run without error'ing out that the port it's trying to bind to and forward is not available. So I'm not sure how it would ever reach the running state.. but I haven't experimented going about it that way. I can handle the small amount of downtime.
I think the key thing is that users shouldn't have to have a knowledge of Sysops or Devops to be able to take advantage of the benefits of containers. Most developer we've spoken to don't have time to learn a completely new paradigm, they just want the benefits.
ECS is pretty exciting, but isn't the same level of abstraction accomplished with the GCE/Fleet/k8s? Even the Kubernetes or Fleet install process on EC2 lets you quickly forget about the actual boxes.
The evolution of CaaS would be sth. like hyper.sh. Due to its hypervisor nature, the whole cluster+scheduling thing can be made transparent to developers, who simply "compose" the app spec and submit to run.
Either way, thanks.
I'm unclear what value this adds in the end
Yes container images would become portable between systems, but if you hide the underlying system enough under abstraction layers what makes me choose between CoreOS or Docker or the future thing? What's the value difference?
Containers are useful if you have the build systems in source control, but if you don't, you don't know how to rebuild them or what is in them - they become dangerous in that case. They become scary "golden images".
Docker files were already very easy to regenerate things -- and I think interface wise, one of the more compelling wins. If there were other systems it's still likely they would have different provisioners.
It seems the (excuse me for the buzzword) value add then quickly becomes in the people providing management software for Docker, rather than in Docker, and Docker becomes more or less a subcommitee of a standards body.
I'm sure that's NOT true, but it's confusing why they wouldn't want to seek differentiation to me and what this means for valuation purposes.
From the Google/Amazon/Microsoft POV, that's part of the point.
From the CoreOS/Docker POV, the fact that the container technology that the major cloud hosts support is going to define their market whether they participate or not is probably why they have an incentive to participate.
> It seems the (excuse me for the buzzword) value add then quickly becomes in the people providing management software for Docker, rather than in Docker
Docker-the-company is probably going to be one of the "people" providing management support for the new standard container platform. From the advantage of also being one of the players defining the standard, and one of the key implementers of the basic software, which will give them an advantage in implementing management software.
> I'm sure that's NOT true, but it's confusing why they wouldn't want to seek differentiation to me
Sure, they want to seek differentiation. But they probably don't want differentiate themselves as "the vendor of the container technology that isn't backed by any of the major cloud hosts", when there is a container technology that is backed by those hosts.
You would think, but it seems Kubernetes, CoreOS, ECS, and Mesos are the obvious picks right now for management stacks. Docker could drop the shiny UI on top of swarm/other soon, sure, and then they could build off the name-recognition, but they are definitely letting everyone have a running start. There's something said for last-mover advantage, but these types of apps are also non-trivial in storage/networking/other areas and somewhat hard to displace.
- Today AWS announced they would support Docker Swarm and Compose natively on ECS.
- Last month devops.com surveyed development and ops teams about their usage of containers. Their conclusion was: "While this is still a nascent market, some early leaders did emerge from the responses. Docker Swarm, the orchestration technology from Docker itself, was the clear winner, with nearly 50% of respondents indicating that they planned to investigate Swarm. Close behind were Kubernetes and Mesos."
Personally I think the important part of that quote is "this is still a nascent market". We are the very, very beginning of these sorts of platforms, and nobody really knows what these platforms will evolve to be in a few years. So I really don't think it matters who has the most "market share" of the 0.00001% of IT organizations actually aware of these tools. The innovation is only just getting started.
Obviously tooling and support?
(It's analogous to asking what makes you chose between programming editors since they all edit text files? Or what makes you chose between distros, since they all run Linux and the usual userland)
I choose a distro based on the package manager - I'm used to Debian so I generally prefer apt based distros to play with.
I prefer BSD over Linux and plan9 over all of them but the are not fungible.
I do pay my supplier a monthly fee to host my container, and for that the metrics is (bandwidth + speed + storage space) / price
What they offer is the whole "value added" services they provide.
What kind of container runs on them wasn't ever their enticing quality.
Sorry for crossing streams.
That standardization makes it easier for the orchestration companies and clouds is obvious. I'm just legitametly curious what this means for Docker, Inc and the business model, since it seems to be seeding the lower end, and they haven't invested in the upper end as much -- Docker itself not being a tremendously large amount of plumbing, and all OSS, it's easy to replicate. So what they have is basically support and the leadership of that community.
As right now, this reads like I'll be able to use everything on Mesos/CoreOS/ECS and just swap out a backend, it's unclear why I would want to pick things from Docker Inc. It's like I get pluggable tooling where all the frontends can speak to backends and the image format is the same -- so it seems differentiation would have to happen at the top in the tooling, which is weird seeing efforts have gone into the bottom end and other companies have done a lot on the top.
Perhaps some messaging to address. Perhaps there's enough funding that this isn't a concern even for the next five years. I don't know, but I'm curious. It's useful to know this to tell where container-land is going, and it's an uncertain time in which management orchestration software to pick for running Docker clouds. (We can probably guess ECS is going to be around, roadmaps of others subject to speculation)
Mostly because I find the evolution of tools in this space interesting.
A few highlights:
00:28:00 keynote intro
00:42:48 Docker growth stats
00:52:54 I take our positioning from the top
01:01:22 I talk about what you call "the top of the tooling"
01:46:13 I talk about the plumbing (including runC
01:12:25 I talk about open standards (including open container project announcement)
Docker has fundamentally changed the way they think of the way they fit in the tech eco-system. Instead of selling a set of containers that only work with their tools they've opened up the platform strengthening their position as the go-to solution for management. Prudent move on their part. It limits their potential market cap but solidifies them as an entrenched member for the foreseeable future.
The company I worked for in early 2000 was using HP-UX containers.
So I guess we're going to have libcontainer support for AIX Workload Partitions and OS/400 LPARs? It's gonna be interesting to see just how big the Docker libs become.
It’s confusing why the App Container (appc) spec which is written down  and has maintainers from RedHat, Twitter, Google, Apcera, CoreOS  is not being promoted - what's the new OCP standard offering that isn't in the appc spec?
OpenContainers addresses one of the fundamental issues already with the initial "runc" tool - namely the issue of playing nicely with Systemd and not requiring a separate daemon.
It also appears to be headed towards addressing a second one: A spec.
It would also seem that what's there so far with 'runc' could easily be used as part of an appc implementation easily enough, and conversely, there seems to be little reason why this couldn't be supported by appc implementations (or others, like LXC).
So what we're (hopefully) getting is a compromise that lets people compete on value adds rather than fighting over the basics.
It was just announced.
Since AppC is a completely different format from what Docker uses, starting from that would have defeated the purpose. However it made a lot of sense to invite the people behind AppC to join, so that we could all build a better spec and implementation together, instead of arguing on technical details that don't matter.
Deploying new code (happens weekely) is just as simple as clicking one deploy button in our version control system (which does a "git pull" on the web servers). DB changes (which are very rare, once or twice a year) we run manually. The cache servers never change. All of the server run automated security updates on the OS. Otherwise we upgrade non-essential packages every few months.
Is there a way that using Docker could make things better for us? I feeling the "you should be using Docker" coming at me from every angle. Our deployment is certainly not very sexy but it is simple and doesn't take a major amount of effort. Is there a use case for a company like mine?
If your server configurations change for some reason in the future (sounds unlikely) it might be easier to start from scratch with Docker than manually adjust the servers; but that all depends on your situation.
As it stands, docker represents a very clean level of abstraction for application/data containers... combined with any number of management layers it works... but the container layer itself is simple to use in practice. Bringing windows into the mix will only muddle things a lot.
I do appreciate Joyent's work on bringing linux compatibility to SmartOS for Docker support, but my hope is that MS doesn't really screw things over here... which I really see happening given how alien the OS is compared to linux. Unless MS creates its' own common-core userspace, which combined with their efforts on opening .Net up could happen, I don't see windows applications fitting in with the rest, and it would create too divergent paths for continued success.
Who knows how this will shake down, I've been a fan of lxc/docker from early on, and I use windows on my desktop, but this just scares me.
With some luck coreOS' super quick relase model can rub off on it and the nice bits can spread and bad bits can die.
Seems to me that VMWare's VM management technology is still needed, but the clock is now running on how long it will be before their part of the stack is irrelevant, as all the smarts move into the container-management layer.
It also means you have to tag twice using the full tag if you want build numbers: this build is "x/y:latest" and "x/y:v1.2.3", when you could just do "latest" and "v1.2.3". Similarly, when you pull an image, it should pull all tags associated with that layer, so you pull 'latest' and it also brings the tag 'v1.2.3'. This seemed to be the case with the docker v1.5, but it seems inconsistent in 1.6. I haven't had time to nail down that suspicion, though.
There are other bits that could do with polish (like being able to do multiple tags at once, rather than push afresh for each one), but the main problem is the single tag field. Given the amount of metadata they already store for an image, this seems a strange behaviour to suffer.
We have been working together for a very long time, and publicly announced in October last week that Docker was coming to Windows Server.
Also, Microsoft has made some announcements recently to get on the containerization bandwagon: http://azure.microsoft.com/blog/2015/04/08/microsoft-unveils...
"Last October, Microsoft and Docker, Inc. jointly announced plans to bring containers to developers across the Docker and Windows ecosystems via Windows Server Containers, available in the next version of Windows Server."
In the short term, what Docker uses with will matter more than the output of any such committee. In the long term, well, it depends on Docker-the-company's fate.
How is that true? I'm already excited to kill dockerd just to run containers.
I see nothing wrong in using imagery as shorthand to express an idea.
And LXC doesn't define a container format either, it's more of a lower-level API, which is why Docker itself can use it to run their own containers.
A per-container configuration file, stored in the container directory. Similar to the "tarball with metadata" of Docker, but you'll have to roll your own layers (or do it the old fashioned way & hardlink common files between containers).
You're right on the container interface front, my apologies. However:
specifies an image format (basically a bootable ISO or some such). It's not as self-contained as a docker image (still need to specify a bunch of parameters), but it's available.
LXC is not 'low level' capabilities as Docker erroneously refers to it on its website or a 'low level api'. 
The LXC project in development since 2009 on which Docker was based, and now Systemd-nspawn give you pretty advanced OS containers with mature management tools, full stack linux networking, multiple storage options including support for btrfs, zfs, LVM, Overlayfs, cloning, snapshotting and a wide choice of container OS templates.
It also offers unprivileged containers that let's non root users run containers, which is a big step forward for container security, and is currently working on live migration of containers.
Docker takes that as a base, limits the container OS template to a single app, builds the containers with layers of aufs and enforces storage separation.  It's not rocket science, you can do this yourself with overlayfs and LXC in minutes. 
Docker is an opinionated way to use containers. You don't need to adopt Docker to get the benefits of containers. You adopt Docker to get the benefits of Docker. A lot of the messaging, hype and marketing conflates the 2 and it suits the docker ecosystem to do that but it does not benefit informed discussion.
Nspawn is a systemd container project, which is similar to LXC in that it gives you OS containers, not layered single app containers. Rkt is based on Nspawn if I am not mistaken.
It's curious that LXC project largely responsible for the development of most of the container technology available today is not part of this. I feel most folks who can move beyond the wild misconceptions floating around to try LXC will find it's significantly simpler to use.
We provide a lightweight VM image with a complete LXC environment based on Alpine Linux for those who are interested. It's 80 MB and available for Virtualbox, VMWare and KVM at https://www.flockport.com/flockbox
Disclosure I run flockport.com which provides an app store for servers based on LXC.
As for nspawnd, it's certainly a good tool, but as far as I know it doesn't define a container format, which is what we were discussing.
The LXC project does not refer to itself as low level, but ironically Docker which knows exactly what LXC is, does. And the results are these needless misconceptions, which is a tad unfair to the developers of the LXC project. Why would anyone try LXC if they think its 'low level'?
LXD extends LXC to add a REST api, multi-host container management and soon live migration.
LXC is a full fledged userland project and has been since 2009 when it began development. It is also responsible for driving the development of a lot of the kernel capabilities mainly 'cgroups' and 'namespaces' needed to support containers.
That's the 'low level' capabilities that LXC, Docker. Nspawn use to give you containers, but it would be as inaccurate to refer to Docker as 'low level' because it uses these, as it would LXC.
A container is simply a file system in a folder that gets booted and works on any underlying Linux filesystem so I am not sure about how conceptually a 'container format' fits there? That's one of the great things about containers. You do not need to think about storage. Simply zip the container folder and move across servers. Containers are completely portable across any Linux system today. But if you are going to use aufs or overlayfs layers to build single apps containers with constrained container OS templates then perhaps there is a need for a format.
"Instead, LXD is our opportunity to start fresh. We’re keeping LXC as the great low level container manager that it is. And build LXD on top of it, using LXC’s API to do all the low level work. That achieves the best of both worlds, we keep our low level container manager with its API and bindings but skip using its tools and templates, instead replacing those by the new experience that LXD provides."
> A container is simply a file system in a folder that gets booted and works on any underlying Linux filesystem so I am not sure about how conceptually a 'container format' fits there?
Well, then maybe you should read TFA? It's all about the creation of a standard 'container format'.
The quote is in a context and says LXD uses LXCs low level api. It does not say 'LXC is a low level api'.
How does one conclude that from the quote and take it out of context without misleading readers? Any one with a remote awareness of LXC will know that's erroneous. Do you still think your comment was accurate?
Have you used LXC, are you familiar with LXC beyond the quote, are you interested in having an informed discussion on LXC and containers? There has be a basis for informed discussion.
In any case, my point in the original post was more that LXC didn't define a container format, unlike other container tools that built upon it, like LXD: https://github.com/lxc/lxd/blob/master/specs/image-handling....
Should that read:
"a way for Docker et all to launch systemd units inside their own containers."
To allow systemd (and other code) to identify that it is executed within a container, please set the $container= environment variable for PID 1 in the container to a short lowercase string identifying your implementation. With this in place the ConditionVirtualization= setting in unit files will work properly. Example: "container=lxc-libvirt"