Docker is great, and I love Docker. I have a couple of specialized non-HTTP apps I'd love to deploy to a Docker cloud, and I'm waiting for a convincing Docker PaaS vendor to come along.
But:
1. A single, full-time sysadmin has a total employer cost of $200K per year in most tech markets.
2. A single sysadmin won't provide 24x365 coverage. They sleep, they take weekends off, they go on vacation.
3. I know people who run tech startups with 80+ employees and heavy server load who pay Heroku a few grand per month, and they just don't care. It works great, and it's a rounding error compared to their programmer salaries.
4. As I explained to one consulting client, if Heroku goes down (they do, every couple of years), you spend your time reloading https://status.heroku.com/ and apologizing to your clients on the phone. And eventually the problem goes away by itself. If you run your own servers, it's all fun and games until you lose 2 RAID drives and the RAID controller board in one night over Christmas vacation—one month after your sysadmin quit, leaving the backup system broken.
In practical business terms, you need to be either huge or completely broke before Heroku looks like a bad deal. $1000/month will buy you a lot of dynos, and for most actual business, $1000/month is a rounding error.
An excellent way of putting it ekidd. Exactly what I came here to say. I work for a small startup and I am the one wearing the sysadmin hat most of the time, and every single day I imagine these disaster modes where it could cost us almost triple what we would have payed Heroku and others (Granted, we have technical reasons why we are using bare servers, nothing to do with money). Icing on the cake is that I am going on vacation this coming Tuesday. :D
Always reason about stuff like this in terms of disasters. They are far more lethal to our company than the occasional $1000 you will be giving a dedicated hosting service.
In practical business terms, you need to be either huge or completely broke before Heroku looks like a bad deal. $1000/month will buy you a lot of dynos, and for most actual business, $1000/month is a rounding error.
I'd dispute that, because of the following:
* Heroku is in no way equivalent to a single full-time sysadmin, not even close
* The choice is not between managing your own hardware and Heroku, it is typically between Heroku, VPS (managed hardware), AWS, and then possibly your own hardware. The same answer does not apply to everyone.
* Many small companies don't have a large wage bill (outside SV), and don't have lots of devs or a sysadmin, instead they tend to have devs who do the sysadmin too.
* Heroku can be significantly more expensive than other options like VPS which also don't require managing hardware (your RAID example) - for some businesses not burning VC money, that makes a big difference. In your example of $1000, you could probably serve the same customers with $100 a month using VPS servers.
* Dynos are not the only cost, you'll be charged for storage too, and $1000/month is not a rounding error for many businesses starting out
* As your business scales Heroku would become more and more costly - there have been reports of clients being charged things like $20000 a month when they start to hit scale.
* They've had more than a few major incidents in this year alone: https://status.heroku.com/uptime - no service is perfect, but that's not once every couple of years
Heroku can look like a bad deal for all sorts of reasons, not least of which cost, and it is not comparable to hiring your own sysadmin, more like having a freelance sysadmin on call who is also responsible for thousands of other sites which are liable to be down at the same time. If you are medium/high traffic website with few other tech costs, heroku can be very expensive compared to the alternatives. Additionally, as documented by some of their customers, they are an extra level of indirection between your app and the user and that make troubleshooting issues or optimising harder -
I would disagree anyway though that Docker (in its current form) is going go to kill services like heroku, because it requires significant expertise and setup time to get right, and doesn't fully address the same concerns.
> Many small companies don't have a large wage bill (outside SV), and don't have lots of devs or a sysadmin, instead they tend to have devs who do the sysadmin too.
It's exactly those kinds of companies that Heroku is aimed at. I can hack being a devops. I don't want to:
— I don't know what I don't know, which has huge ramifications for both security and uptime
— I have plenty of opportunities available to me and am not interested in being on call all the time. Your business could not reasonably pay me enough to answer the phone at 3 am.
— One day of heads-down product development can easily create several thousand dollars in value (or more) for a company. When I am interrupted with server patches, setting up new servers, or debugging a failed deploy process, and you throw a meeting or two in the mix, I'm downgraded to shipping a few bug fixes that day.
Any competent executive wants his key developers heads-down shipping features and should not expect them to manage devops anymore than he would expect them to interrupt their day to balance the accounting books. It's a context switch. Throwing money at Heroku makes perfect sense here.
What if you can automate your AWS infrastructure, so there will NoOps, like using Heroku: CD/CI, AutoScaling, AutoHealing, just battery included?
The point is that you have the 100% control of the infra, but the orchestration service can get you covered. And of the orchestration service went down, your instances will continue to run. Not like in Heroku, their downtime is your downtime!
> They've had more than a few major incidents in this year alone: https://status.heroku.com/uptime - no service is perfect, but that's not once every couple of years
This is a frequently misunderstood metric. Whilst there is a number of incidents on status.heroku.com, the average platform uptime for a single app is still 99.995% [1]
> In practical business terms, you need to be either huge or completely broke before Heroku looks like a bad deal. $1000/month will buy you a lot of dynos, and for most actual business, $1000/month is a rounding error.
Heroku is a great deal. Still, let's not forget about the cost of production tier Heroku addons, such as databases & logging.
People don't talk about it much, but it's very possible to mix Heroku with other services running on AWS. We use heroku for our application tier and host our own DB on EC2.
(For us) Docker is not about security or scalability but about (good enough) isolation, separation of concerns and reproducability. Let me elaborate.
* Isolation: Docker enables us to pack any software and let it run with close to none side effects. Different versions of packages, libs and gems needed by apps don't interfere. It's like bundler but for all kind of dependencies not only gems.
* Separation of concerns: For our operations it doesn't matter what's inside a Docker container. We have mid sized Ruby web apps, small Go demons, NGINX with Lua support compiled in, legacy PHP apps neatly packed in Docker containers. They have a well defined interface:
The build script which consistently sums up dependencies and the build process.
`docker run` wrappers which clearly state the interface of the running container like exposed ports and mounted volumes.
* Reproducability: We are able to run the same containers in development, staging and production. A dozen containers will easily run on a developers laptop.
As a side effect the Docker architecture makes us think harder about app architecture like which services are stateless and which are not and for what reason.
The fact that containers share a kernel and thus are not 100% isolated or reproducable as with virtualization hasn't been an issue for us (so far).
There are still issues and features we're missing. For example private Docker repos are PITA and building instead of pulling from a repo means you might get fooled by the build cache. And we'd love to have build in support (or at least a common standard or best practices) for orchestration. But all together for our needs it's already pretty useful.
> And we'd love to have build in support (or at least a common standard or best practices) for orchestration.
Look into BOSH[0][1]. It's a IaaS orchestrator that works for multiple cloud backends -- AWS, openstack, warden and vsphere out of the box. I use it in my day job.
It's already been applied to working with Docker containers.[2]
My problem with every single one of the orchestration solutions I've seen is that they tend to be overcomplicated for small deployments, and make a lot of decisions that I often don't agree with.
Looking at BOSH, my immediate reaction is that they have a huge page just explaining terminology [1]. The fact that they need one (and they do, judging from what's on it) is already a red flag for me.
We run "only" about 150 containers at the moment, so maybe I'd appreciate it more if we had thousands, but to me it seems horribly overcomplicated. And I have an immediate dislike to anything that requires constantly running processes in every VM. That may be necessary for full VM's, but it's one of the reasons I don't think VM focused orchestration solutions are a good fit for container based systems. Our existing (homegrown, on top of OpenVz) system makes heavy use of the fact that every resource in the containers are accessible from the host, and that's one of the thing I like abut Docker too.
It's complicated because orchestration is complicated. You're looking at essential complexity that can't be made to go away.
What you can do is wrangle the complexity into a repeatable, declarative form. That's what BOSH and other orchestration tools do.
Everything on that terminology page is what has been found, in various systems of various sizes, to be necessary to ensure some semblance of robustness and operability as you scale from a single box to thousands of VMs and containers.
In my dayjob I've written BOSH release and deploy manifests that stand up a virtual machine containing 2 containers. I've used other deploy manifests that start a virtual machine with a small cloud of containers. My employers use BOSH to orchestrate multiple clouds on multiple backends with thousands of VMs.
Running a process inside the box is necessary to be agnostic of the substrate. BOSH can't assume that you'll run everything in Docker, or AWS, or vSphere, or any other such system. You can run heterogenous combinations per your requirements. Again, some people absolutely require that capability.
I really don't understand how all these "instead of Heroku just use X"-people do not understand that one of the main benefits of Heroku is not managing servers. If your app on Heroku has an issue Heroku will fix it (not your app of course). If your app on Docker has an issue. Who you gonna call?
You don't understand. A big barrier to entry for competing with Heroku for any other startup is creating a similar virtual environment that abstracts away from metal machines for people who just want to write apps. Docker provides that to startups. So we will see more startups competing in this space, you are going to use them and call them when shit hits the fan.
Recently, Google App Engine allows deploying apps in any language using Docker. So, with Docker, App Engine is suddenly a competition to Heroku for Ruby programmers.
I can't deploy Kafka to Heroku. I can technically deploy Akka but 1X/2X instances aren't going to cut it and PX dynos are hilariously expensive, plus clustering isn't possible.
Yeah, if your app can fit on Heroku, leave it there. But Heroku is not a silver bullet for all types of apps. Containerizing them makes it much easier to add and remove instances, easily deploy new services, move instances around as underlying resources die, etc. Docker doesn't provide everything to make this happen but it's the key.
I've spent the past two weeks moving an Akka cluster into Docker with CoreOS. I'm about to deploy Kafka in the same way and so far it's been fantastic.
Fair enough. My point is that Docker makes packaging up the various services I need to deploy much easier and more predictable. Now, rather than spin up three EC2 nodes to be Kafka instances, I just use fleet to submit three Kafka units and it finds available resources among my existing CoreOS cluster. If I need more capacity, I spin up more generic CoreOS instances on EC2 and fleet starts using them.
The groundwork is being laid and I think we're going to see a lot of competition in Heroku's space. Given the price difference between Heroku and EC2 (much less DigitalOcean) I think there's a lot of profit margin for competitors to attack. That's Heroku's weakness: it's easier than ever to build the equivalent functionality.
This may be true, but it also makes the common assumption that your time (or the time of your team) is somehow worth $0 dollars. But, I imagine that you would be the first to say that your time is worth much more than $0 dollars.
Put another way, hardware stores sell paint and paint supplies all day. Buying those and painting a room yourself is certainly cheaper, but it assumes you have the extra time to paint and don't mind you being the laborer. Hiring a painter costs more, but saves you a ton of time, energy, and elevates the level of expertise you are bringing to the job.
Getting started is one thing. The ease of how you maintain and grow is another.
I'm not sure ops time is even the one to worry about. Most things I've learned I've learned the hard way. So a server going berserk isn't fun, but it's not like I just wasted either time or money on it. I learned something. I better understand my execution environment.
Downtime, on the other hand, is just lost revenue opportunity.
I certainly do, and I can tell you the amount of time I've spent managing servers over the past year has cost us less than running our entire production stack on Heroku would have done. So has the odd bit of downtime we've encountered because I've not got the same level of experience as Heroku's ops team.
If your heroku costs significantly cut into your profit margins, you should certainly start paying attention to those.
For most well designed services, that is simply not the case. If my revenue from $1000 of heroku bills is $20,000, cutting $500/mo off of my server costs in exchange for any amount of hours or risk is simply not worthwhile.
The actual drawbacks (the author has chosen to ignore them):
- less secure
- you still have to update the system images and redeploy a lot (its made for that so its fine). you cant just spin an image and stop caring for it. libraries will update and potentially breaking ur app because u know, performance, sec fixes, the usual
- docker itself is not very nicely made, hard to debug etc. hopefully will get fixed over time.
what is actually does better?
its faster.. not 26:1 but it is faster obviously. mainly, you dont have to preprovision vms since there is no boot time. so you can deploy fast.
its also providing a much needed API/glue for all the things.
Very interesting thanks! I started outlining/working on a similar docker orchestration too a couple weeks ago here: https://github.com/pnegahdar/sporedock
Longshoreman looks interesting. I have two questions I couldn't find an answer to. How does port allocation work? And can you specify rules such as: service a should be on all hosts.
Port allocation is quite simple. We randomly select an unused port between 8000-8999.
The most recent version has a --count flag which allows you to specify the number of application instances to launch. I believe the Heroku equivalent is --scale.
Just saw the spike in traffic this morning from my post being added here, thanks!
I'm very much a "DevOps" guy. I like running my own servers. When I first took over a Heroku based application it was extremely frustrating to me that I couldn't tweak nginx, Apache, or some type of on-dyno cache without adding network overhead to each call. I was also dealing with a system that I took over after a buggy relaunch and at that time, Heroku seemed to have fairly regular outages which was a problem for our customers who were already very sensitive to the quality issues of the relaunch. Amplified the lack of control significantly because I was constantly aware of what I couldn't do to fix the problems.
I'm still actively using Heroku and the work they've done with their PostgreSQL offering has been really impressive. It has been significantly more stable too. From a development efficiency standpoint, it really is hard to beat. They've silently implemented "Shadowing" which is supposed to provided redundancy across availability zones (but not regions). The additional Dyno size options have been great too. It is a great company. I just wish Heroku had an option similar to RightScale to deploy within other data centers or at least AWS regions - but that becomes complicated because of their network of add-on providers.
What I was getting at with the article was exactly what arihant mentioned below. Docker open sources the core piece that makes Dyno-like functionality possible which opens the door for disruption in the PaaS market.
Heroku has always seemed like more "sysadmin as a service" than just pure hosting, case in point: their response to Shellshock -> https://status.heroku.com/incidents/665
One of the biggest issues with Docker is security more than anything else. You can't really rely on container separation in a shared environment if someone somehow plans on selling Docker containers.
I was surprised i didn't see it mentioned in the post. VM-style isolation is there for a reason. Dunno if Docker is the wave of the future, but as a Heroku(AWS?) killer would be nice to learn how sharing all this performance goodness on the same system is resolved isolation-wise.
Docker and Heroku use the exact same sandboxing technology under the hood (Linux namespaces and cgroups). This is not surprising since Docker came out of Dotcloud, a direct Heroku competitor. The pros and cons of linux namespaces have been extensively discussed, it's no silver bullet but when used properly is quite robust and is rapidly gaining industry recognition.
you have to be careful how you define "secure" here. you are correct that selinux is a requirement if you are worried at all about x-container interaction. but even with selinux, you are still exposed since you are sharing the kernel and would have to do work for top, ps, and others to not share what is going on across the system.
It's cute to be bombastic, but no one's getting killed. The dotCloud people have always been my favourite PaaS people, and competition in deployment UX is sorely friggin' needed.
I know this is a dumb/basic question, but it's one I've wondered and don't fully understand. How is Docker that much easier/better than just writing some other kind of script that configures a machine for your app to run in and deploys it?
Docker like a package manager, except its better than that, its like a package manager that works across distributions. And there is already a full stack for just about everything. And we don't need to go through the normal package management red tape.
Docker also provides a level of separation and isolation that makes it easier to set up configuration and also increases security.
It provides a type of building block with a standard interface for connecting to other pieces.
It allows me to easily customize a build off of an existing stack.
It provides a binary distribution of an application, so you know the whole system that you are deploying is exactly the same one you tested.
Not to mention the fact that because intermediary containers are imaged, you don't have to worry about making a mistake in your script.
Sometimes (okay, every day) I find myself trying a bunch of different things to get something to work. With Docker, you've got a series of "save points" along the way. You can continue building an image from the last good point, and end up with a really clean image that represents the shortest path from bare install to "what I need to get done."
I was playing around with a Perl script about a month ago that would strip all the RUN statements out of a Dockerfile, convert them into a shell script and use that to bootstrap a Vagrant box. So what I'm saying is, if you don't know what you're doing (or just like to experiment with OSes), Docker is really interesting.
I work on a sorta-kinda Heroku competitor (Cloud Foundry), so my views are suspect and of course are mine alone etc etc.
However, asking "which should I use for my app, Docker or #{PaaS}?" is a bit like asking "which should I use to build my house, a brick or a general contractor?"
They are different things, at different levels of abstraction.
People have already written Docker deployment systems for Cloud Foundry[0]. Plus you get all the other stuff you'd have to write yourself.
I don't think docker is accessible enough right now to kill Heroku. You have to put on your devops hat when you use it. With heroku, you push and it works.
I'm not trying to bash Docker. In fact, I use it quite a bit, and I never use Heroku. I just think that compared to Heroku, it's much more of a low-level tool. With Heroku, you can just paste a few really simple commands and your webapp is now up.
The tooling isn't there, yet, but will come. Someone - I hope Heroku - will make something really good, that developers rather than ops people can use to get something production ready with Docker easily.
I agree with the article. However, I am also a fan of PaaS like Heroku, IBM's BlueMix, etc. because they save labor costs. I also always run a beefed up VPS and I have my own "Heroku like" git push deployment set up.
Docker in particular, and containerization in general are the future especially for very large shops like Google, Facebook, etc.
But:
1. A single, full-time sysadmin has a total employer cost of $200K per year in most tech markets.
2. A single sysadmin won't provide 24x365 coverage. They sleep, they take weekends off, they go on vacation.
3. I know people who run tech startups with 80+ employees and heavy server load who pay Heroku a few grand per month, and they just don't care. It works great, and it's a rounding error compared to their programmer salaries.
4. As I explained to one consulting client, if Heroku goes down (they do, every couple of years), you spend your time reloading https://status.heroku.com/ and apologizing to your clients on the phone. And eventually the problem goes away by itself. If you run your own servers, it's all fun and games until you lose 2 RAID drives and the RAID controller board in one night over Christmas vacation—one month after your sysadmin quit, leaving the backup system broken.
In practical business terms, you need to be either huge or completely broke before Heroku looks like a bad deal. $1000/month will buy you a lot of dynos, and for most actual business, $1000/month is a rounding error.