The virtualization and containerization of computing is a long term net win, but there isn't a lot of huge benefits financially or otherwise for the scale most companies operate at.
For example, I witnessed a co-worker spend days messing around with docker to host incredibly simple PHP/MySQL and Ruby apps. Instead of renting a few VM's on say Digital Ocean or Amazon and spending probably $50 a month, my co-worker spent at least 50-100 hours of time to put everything on Docker and Puppet.
Let's say the net cost of that orchestration was around $10,000 in terms of billable time. You could rent a separate VM for every project and still come out ahead without trying very hard and still spend less time doing the project.
The total cost on that was way out of whack AND docker continued to be a source of bugs and ongoing overhead costs in terms of time.
My point is that these tools are great when they save you time, energy, and money, but a lot of people are overinvesting in tools like docker too early and they will never get a good ROI on their investment.
I don't know at what scale these tools start to make a lot of sense, but I imagine it is somewhere over 20 or 30 machines or projects before containerization shows itself to be a good investment. I'd be curious to see someone run the numbers on that.
> You could rent a separate VM for every project and still come out ahead without trying very hard and still spend less time doing the project.
Doing the project? Yes. Running it - no!
These tools make sense at the scale of a single project with two hosts in my opinion. Here's what happens in my experience when developer with little ops experience takes care of a few VMs with small apps: (so a majority of services out there)
- where are the logs and what's the available space for them? depends on which VM, some don't log at all, some log debug because they had some work done weeks ago
- which hosts are running which app version and what's the upgrade procedure? likely every one has different manual steps
- what's your system/libraries/application upgrade story? likely missing / different for each host
- what's your DR story? likely missing
- what's the state of monitoring and are all hosts covered to the same extent? unlikely
- is the development and test environment at all similar to production? unlikely
For each of those items you're likely to go back and introduce something that docker/puppet would've either made easier or provided from the beginning. Knowing that you can consistently build the same host/app every time and that it's got the same options, utilities, tools, etc. available is a huge gain even on 2 VMs.
Yes, the huge benefits start at 1X hosts, but I've seen the benefits and I'm never deploying even one production machine without automation anymore. (salt in my case though) Not doing so is actually more work in a long run. And as soon as you deployed one app that way, you have your library of scripts - another app will likely look very similar and anything around it (monitoring, logging, backups, database deployments, ...) will likely be exactly the same.
So I think that you either pay that 10k for orchestration from the start, or you end up paying it bit by bit, every time something needs to be added or goes wrong.
What the OP was saying is that spending all that time to make docker work was a bit pointless. Especially as a VM makes no real difference to the end user at this stage.
Your points about DR and the like are valid, however if you are going to use puppet, the difference between deploying n docker instances (and keeping the state in the correct way) and VM instances are negligible. (yes, yes, docker is a chroot, so inherits the host's state, but you'll still need to control the docker environment.)
I use cgroups everyday, to avoid memory exhaustion, (25k CPU render farm) and for that its grand. Running it on a public host? madness I say. For things that require high IO(even then its questionable, AUFS...), or low security perhaps.
He said "time to put everything on Docker and Puppet" - there are a few ways to understand it. Puppet could be used to only manage containers, or it could be used to deploy bare app containers and configure them. Docker work required would be different depending on where runtime and when prepare-time configs end, but it ties in with all of the points in my opinion and so I don't think it was pointless. If it's a part of a stable setup - that's great.
(especially stuff like logging, backups, monitoring will usually affect your docker build to some degree and can be reused in other apps)
> What the OP was saying is that spending all that time to make docker work was a bit pointless. Especially as a VM makes no real difference to the end user at this stage.
I disagree. Given that you operate a 25k CPU render farm, perhaps you are no longer aware of how poorly devops tends to be managed in small shops. I'd argue Docker buys you more in those small environments, because in large environments you can build tools that works for your environment. In small environment, the traditional "solution" has been to brush these concerns under the carpet.
Just getting repeatable builds of an environment so that you can ensure the dev, test, staging and live environments continue to bear a resemblance to each other is a massive improvement for many of this type of setups. That is, assuming that there are separate dev, test, staging and live environments - often you'll find all of it on a single server. Yes, really.
Basically, people that think like OP terrify me, because too often we've taken over operations of applications from people who think like that, to find that the operational cost is staggering. We have one customer, for example, that have over the last few years easily paid several times what it would cost to properly re-engineer their application from scratch in time spent for us on operational issues because they've refused to accept that it's necessary. (EDIT: Said customer pays for this time, so on paper it works out well for us; in practice it's bad for us as it leads to lots of tension with the customer over all the issues, and we'd much rather have spent that time making their app better, and help them grow their revenue from it - long term that'd be far better for our customer relationship). This is a "simple PHP and MySQL" application.
VMs aren't really a solution by themselves. It's not app-level isolation that "makes" Docker. It's the entire structure that encourages repeatable builds, a repository, and the light weight nature that encourages breaking things into smaller components. I agree that just Docker as isolation in a multi-tenant system is still scary.
I'm currently working on deploying an application in VMs. With Docker in each VM. The VMs gives us the isolation we need in a multi-tenant system. Docker lets us ensure that dependencies between each component is well documented (or the components won't have access...), and that we have a decent image repository, and that we have ways of managing the entire lifecycle of each component. The two are complementary technologies.
In a large enough environment you could solve the issues that Docker solves with process. In a small environment that process is usually non-existent or weak.
> AUFS
AUFS is not necessary. Btrfs or device mapper backends have been supported for a long time. And even then, for any data volums that needs high throughput you'd use a bind mount from the host.
I have taken over "incredibly simple PHP/MySQL" apps that ran on a single server, and let me tell you: Over a year or two of hosting it can easily happen that a poorly set up environment (it was running on a single server - it's by definition a poor setup, lacking HA...) will end up costing the business more than 50-100 hours in mitigating the consequences of poor configuration management.
You just won't notice, because it will be 10 minutes here, 20 minutes there, and/or "emergencies" where the time spent will rarely be attributed back to the same original project, so most people don't see how much it costs them to have skimped on the original investment.
To me, these tools make sense from 1 machine, and you start seeing serious benefits once you have two (all the investment you did for one machine? You've now mostly halved the investment per machine, since you have a set of containers you know how works ready to go; you have some additional complexities as your cluster grows, such as orchestration across machine boundaries, but amortised per machine the cost seems to keep dropping)
If anything, what we are seeing is that these tools get people to do things that were easy to ignore before, causing people to think they were done setting up their environment because they had a server running the production code.
Never mind how often these setups have been unreproducible, with poor isolation, no failover/ha setup. They're cheap until they're not. And most often they're not - just that people don't really notice how expensive they get because the time is never fully attributed back the right places.
What you're arguing for here is a configuration management tool, not docker specifically. Puppet is massive overkill for a couple of servers, Ansible is easy and ideal for it, and Docker is overkill as well. Docker only makes sense "for 1 machine" if you already know Docker and all its warts.
Comparing Docker and Puppet to VMs is not really fair. How do you configure them at scale? What about failures? That would need a lot more getting into cloud configuration/scripting.
Having started with devops only recently, I find the concept of CoreOS + Docker Containers/Images/Dockerfiles (that are close enough to Bash scripting) easier to get into than learning custom UI/API of custom company to deploy to their cloud.
> The virtualization and containerization of computing is a long term net win, but there isn't a lot of huge benefits financially or otherwise for the scale most companies operate at.
I guess I have skipped that sentence. It makes all the difference for scaling.
You could say much the same about learning how to setup X stack on Linux vs just using a service like Heroku.
Most developers probably shouldn't be messing with Docker. If they aren't directly creating value for someone, then their efforts are probably best directed elsewhere. There is value to yourself for learning and scratching your own itch, but that effort needs to be treated like a side project. It doesn't cross over into your real work.
Docker is low level plumbing on which more "user friendly" workflows are being built. These workflows are what developers should be using. Docker is still young, so there is still a lot more attention on the plumbing than there is in any tools which have been built on top of it. That will change as the ecosystem develops.
But this is a common developer problem. Sometimes developers will spend time doing anything other than what they need to be doing to actually ship things. We don't get paid to write code, we get paid to deliver on things with the greatest impact on that bottom line.
In my experience, automation and devops 'best practices' are always an investment.
But there's a big recruiting and retention issue also... kinda hard to find and retain good IT/devops people these days if you do everything manually :-)
So, your co-worker got a customer to pay $10k for him to play with new tech that looked good on his resumé. And you got to fix the bugs. Many work environments make those guys come out on top...
More seriously, we tend to have an issue of over engineering. I have to pull myself up all the times to just focus on getting the job done and not on building the perfect architecture.
For example, I witnessed a co-worker spend days messing around with docker to host incredibly simple PHP/MySQL and Ruby apps. Instead of renting a few VM's on say Digital Ocean or Amazon and spending probably $50 a month, my co-worker spent at least 50-100 hours of time to put everything on Docker and Puppet.
Let's say the net cost of that orchestration was around $10,000 in terms of billable time. You could rent a separate VM for every project and still come out ahead without trying very hard and still spend less time doing the project.
The total cost on that was way out of whack AND docker continued to be a source of bugs and ongoing overhead costs in terms of time.
My point is that these tools are great when they save you time, energy, and money, but a lot of people are overinvesting in tools like docker too early and they will never get a good ROI on their investment.
I don't know at what scale these tools start to make a lot of sense, but I imagine it is somewhere over 20 or 30 machines or projects before containerization shows itself to be a good investment. I'd be curious to see someone run the numbers on that.