
Docker is the Heroku Killer - joeyespo
http://www.brightball.com/devops/docker-is-the-heroku-killer
======
ekidd
Docker is great, and I love Docker. I have a couple of specialized non-HTTP
apps I'd love to deploy to a Docker cloud, and I'm waiting for a convincing
Docker PaaS vendor to come along.

But:

1\. A single, full-time sysadmin has a total _employer_ cost of $200K per year
in most tech markets.

2\. A single sysadmin won't provide 24x365 coverage. They sleep, they take
weekends off, they go on vacation.

3\. I know people who run tech startups with 80+ employees and heavy server
load who pay Heroku a few grand per month, and _they just don 't care_. It
works great, and it's a rounding error compared to their programmer salaries.

4\. As I explained to one consulting client, if Heroku goes down (they do,
every couple of years), you spend your time reloading
[https://status.heroku.com/](https://status.heroku.com/) and apologizing to
your clients on the phone. And eventually the problem goes away by itself. If
you run your own servers, it's all fun and games until you lose 2 RAID drives
and the RAID controller board in one night over Christmas vacation—one month
after your sysadmin quit, leaving the backup system broken.

In practical business terms, you need to be either _huge_ or completely broke
before Heroku looks like a bad deal. $1000/month will buy you a lot of dynos,
and for most actual business, $1000/month is a rounding error.

~~~
grey-area
_In practical business terms, you need to be either huge or completely broke
before Heroku looks like a bad deal. $1000 /month will buy you a lot of dynos,
and for most actual business, $1000/month is a rounding error._

I'd dispute that, because of the following:

* Heroku is in no way equivalent to a single full-time sysadmin, not even close

* The choice is not between managing your own hardware and Heroku, it is typically between Heroku, VPS (managed hardware), AWS, and then possibly your own hardware. The same answer does _not_ apply to everyone.

* Many small companies don't have a large wage bill (outside SV), and don't have lots of devs or a sysadmin, instead they tend to have devs who do the sysadmin too.

* Heroku can be significantly more expensive than other options like VPS which also don't require managing hardware (your RAID example) - for some businesses not burning VC money, that makes a big difference. In your example of $1000, you could probably serve the same customers with $100 a month using VPS servers.

* Dynos are not the only cost, you'll be charged for storage too, and $1000/month is not a rounding error for many businesses starting out

* As your business scales Heroku would become more and more costly - there have been reports of clients being charged things like $20000 a month when they start to hit scale.

* They've had more than a few major incidents in this year alone: [https://status.heroku.com/uptime](https://status.heroku.com/uptime) \- no service is perfect, but that's not once every couple of years

Heroku can look like a bad deal for all sorts of reasons, not least of which
cost, and it is not comparable to hiring your own sysadmin, more like having a
freelance sysadmin on call who is also responsible for thousands of other
sites which are liable to be down at the same time. If you are medium/high
traffic website with few other tech costs, heroku can be very expensive
compared to the alternatives. Additionally, as documented by some of their
customers, they are an extra level of indirection between your app and the
user and that make troubleshooting issues or optimising harder -

[http://news.genius.com/James-somers-herokus-ugly-secret-
anno...](http://news.genius.com/James-somers-herokus-ugly-secret-annotated)

I would disagree anyway though that Docker (in its current form) is going go
to kill services like heroku, because it requires significant expertise and
setup time to get right, and doesn't fully address the same concerns.

~~~
nthj
> Many small companies don't have a large wage bill (outside SV), and don't
> have lots of devs or a sysadmin, instead they tend to have devs who do the
> sysadmin too.

It's exactly those kinds of companies that Heroku is aimed at. I can hack
being a devops. I don't want to:

— I don't know what I don't know, which has huge ramifications for both
security and uptime

— I have plenty of opportunities available to me and am not interested in
being on call all the time. Your business could not reasonably pay me enough
to answer the phone at 3 am.

— One day of heads-down product development can easily create several thousand
dollars in value (or more) for a company. When I am interrupted with server
patches, setting up new servers, or debugging a failed deploy process, and you
throw a meeting or two in the mix, I'm downgraded to shipping a few bug fixes
that day.

Any competent executive wants his key developers heads-down shipping features
and should not expect them to manage devops anymore than he would expect them
to interrupt their day to balance the accounting books. It's a context switch.
Throwing money at Heroku makes perfect sense here.

~~~
gnepzhao
What if you can automate your AWS infrastructure, so there will NoOps, like
using Heroku: CD/CI, AutoScaling, AutoHealing, just battery included?

The point is that you have the 100% control of the infra, but the
orchestration service can get you covered. And of the orchestration service
went down, your instances will continue to run. Not like in Heroku, their
downtime is your downtime!

------
ende42
(For us) Docker is not about security or scalability but about (good enough)
isolation, separation of concerns and reproducability. Let me elaborate.

* Isolation: Docker enables us to pack any software and let it run with close to none side effects. Different versions of packages, libs and gems needed by apps don't interfere. It's like bundler but for all kind of dependencies not only gems.

* Separation of concerns: For our operations it doesn't matter what's inside a Docker container. We have mid sized Ruby web apps, small Go demons, NGINX with Lua support compiled in, legacy PHP apps neatly packed in Docker containers. They have a well defined interface:

 __The build script which consistently sums up dependencies and the build
process. __`docker run` wrappers which clearly state the interface of the
running container like exposed ports and mounted volumes.

* Reproducability: We are able to run the same containers in development, staging and production. A dozen containers will easily run on a developers laptop.

As a side effect the Docker architecture makes us think harder about app
architecture like which services are stateless and which are not and for what
reason.

The fact that containers share a kernel and thus are not 100% isolated or
reproducable as with virtualization hasn't been an issue for us (so far).

There are still issues and features we're missing. For example private Docker
repos are PITA and building instead of pulling from a repo means you might get
fooled by the build cache. And we'd love to have build in support (or at least
a common standard or best practices) for orchestration. But all together for
our needs it's already pretty useful.

~~~
jacques_chester
> _And we 'd love to have build in support (or at least a common standard or
> best practices) for orchestration._

Look into BOSH[0][1]. It's a IaaS orchestrator that works for multiple cloud
backends -- AWS, openstack, warden and vsphere out of the box. I use it in my
day job.

It's already been applied to working with Docker containers.[2]

[0]
[https://github.com/cloudfoundry/bosh](https://github.com/cloudfoundry/bosh)

[1] [http://docs.cloudfoundry.org/bosh/](http://docs.cloudfoundry.org/bosh/)

[1] [http://blog.pivotal.io/cloud-foundry-
pivotal/products/managi...](http://blog.pivotal.io/cloud-foundry-
pivotal/products/managing-stateful-docker-containers-with-cloud-foundry-bosh)

~~~
vidarh
My problem with every single one of the orchestration solutions I've seen is
that they tend to be overcomplicated for small deployments, and make a lot of
decisions that I often don't agree with.

Looking at BOSH, my immediate reaction is that they have a huge page just
explaining terminology [1]. The fact that they need one (and they do, judging
from what's on it) is already a red flag for me.

We run "only" about 150 containers at the moment, so maybe I'd appreciate it
more if we had thousands, but to me it seems horribly overcomplicated. And I
have an immediate dislike to anything that requires constantly running
processes in every VM. That may be necessary for full VM's, but it's one of
the reasons I don't think VM focused orchestration solutions are a good fit
for container based systems. Our existing (homegrown, on top of OpenVz) system
makes heavy use of the fact that every resource in the containers are
accessible from the host, and that's one of the thing I like abut Docker too.

[1]
[http://docs.cloudfoundry.org/bosh/terminology.html](http://docs.cloudfoundry.org/bosh/terminology.html)

~~~
jacques_chester
It's complicated because orchestration is complicated. You're looking at
essential complexity that can't be made to go away. What you _can_ do is
wrangle the complexity into a repeatable, declarative form. That's what BOSH
and other orchestration tools do.

Everything on that terminology page is what has been found, in various systems
of various sizes, to be necessary to ensure some semblance of robustness and
operability as you scale from a single box to thousands of VMs and containers.

In my dayjob I've written BOSH release and deploy manifests that stand up a
virtual machine containing 2 containers. I've used other deploy manifests that
start a virtual machine with a small cloud of containers. My employers use
BOSH to orchestrate multiple clouds on multiple backends with thousands of
VMs.

Running a process inside the box is necessary to be agnostic of the substrate.
BOSH can't assume that you'll run everything in Docker, or AWS, or vSphere, or
any other such system. You can run heterogenous combinations per your
requirements. Again, some people absolutely require that capability.

------
sleepyhead
Until shit hits the fan.

I really don't understand how all these "instead of Heroku just use X"-people
do not understand that one of the main benefits of Heroku is not managing
servers. If your app on Heroku has an issue Heroku will fix it (not your app
of course). If your app on Docker has an issue. Who you gonna call?

~~~
klochner
Docker + {aws, digital ocean, rackspace} is substantially cheaper than heroku.

Docker makes these services relatively comparable to Heroku in complexity, so
the heroku value prop has been diluted.

~~~
jasonmccay
This may be true, but it also makes the common assumption that your time (or
the time of your team) is somehow worth $0 dollars. But, I imagine that you
would be the first to say that your time is worth much more than $0 dollars.

Put another way, hardware stores sell paint and paint supplies all day. Buying
those and painting a room yourself is certainly cheaper, but it assumes you
have the extra time to paint and don't mind you being the laborer. Hiring a
painter costs more, but saves you a ton of time, energy, and elevates the
level of expertise you are bringing to the job.

Getting started is one thing. The ease of how you maintain and grow is
another.

~~~
ukd1
This. No one ever factors in Ops time when I hear them complaining about
Heroku.

~~~
nirvdrum
I'm not sure ops time is even the one to worry about. Most things I've learned
I've learned the hard way. So a server going berserk isn't fun, but it's not
like I just wasted either time or money on it. I learned something. I better
understand my execution environment.

Downtime, on the other hand, is just lost revenue opportunity.

------
zobzu
The actual drawbacks (the author has chosen to ignore them):

\- less secure

\- you still have to update the system images and redeploy a lot (its made for
that so its fine). you cant just spin an image and stop caring for it.
libraries will update and potentially breaking ur app because u know,
performance, sec fixes, the usual

\- docker itself is not very nicely made, hard to debug etc. hopefully will
get fixed over time.

what is actually does better? its faster.. not 26:1 but it is faster
obviously. mainly, you dont have to preprovision vms since there is no boot
time. so you can deploy fast. its also providing a much needed API/glue for
all the things.

------
AdrianRossouw
We couldn't find exactly what we wanted to replace heroku with in our setup,
so we ended up building longshoreman :

[http://longshoreman.io](http://longshoreman.io)

[http://mikejholly.com/introducing-
longshoreman/](http://mikejholly.com/introducing-longshoreman/)

~~~
coenhyde
Hi Adrian,

Longshoreman looks interesting. I have two questions I couldn't find an answer
to. How does port allocation work? And can you specify rules such as: service
a should be on all hosts.

~~~
mikejholly
Port allocation is quite simple. We randomly select an unused port between
8000-8999.

The most recent version has a --count flag which allows you to specify the
number of application instances to launch. I believe the Heroku equivalent is
--scale.

------
brightball
Just saw the spike in traffic this morning from my post being added here,
thanks!

I'm very much a "DevOps" guy. I like running my own servers. When I first took
over a Heroku based application it was extremely frustrating to me that I
couldn't tweak nginx, Apache, or some type of on-dyno cache without adding
network overhead to each call. I was also dealing with a system that I took
over after a buggy relaunch and at that time, Heroku seemed to have fairly
regular outages which was a problem for our customers who were already very
sensitive to the quality issues of the relaunch. Amplified the lack of control
significantly because I was constantly aware of what I couldn't do to fix the
problems.

I'm still actively using Heroku and the work they've done with their
PostgreSQL offering has been really impressive. It has been significantly more
stable too. From a development efficiency standpoint, it really is hard to
beat. They've silently implemented "Shadowing" which is supposed to provided
redundancy across availability zones (but not regions). The additional Dyno
size options have been great too. It is a great company. I just wish Heroku
had an option similar to RightScale to deploy within other data centers or at
least AWS regions - but that becomes complicated because of their network of
add-on providers.

What I was getting at with the article was exactly what arihant mentioned
below. Docker open sources the core piece that makes Dyno-like functionality
possible which opens the door for disruption in the PaaS market.

For what it's worth I also posted a followup a couple of days later called
Tempering My Docker Enthusiam ([http://www.brightball.com/devops/tempering-my-
docker-enthusi...](http://www.brightball.com/devops/tempering-my-docker-
enthusiasm-retracted)).

------
michaelbuckbee
Heroku has always seemed like more "sysadmin as a service" than just pure
hosting, case in point: their response to Shellshock ->
[https://status.heroku.com/incidents/665](https://status.heroku.com/incidents/665)

~~~
kmfrk
I'd say it's more that they aim to be more full-stack than just the upper
sysadmin layer.

------
kbar13
One of the biggest issues with Docker is security more than anything else. You
can't really rely on container separation in a shared environment if someone
somehow plans on selling Docker containers.

~~~
twelve40
I was surprised i didn't see it mentioned in the post. VM-style isolation is
there for a reason. Dunno if Docker is the wave of the future, but as a
Heroku(AWS?) killer would be nice to learn how sharing all this performance
goodness on the same system is resolved isolation-wise.

~~~
shykes
Docker and Heroku use the exact same sandboxing technology under the hood
(Linux namespaces and cgroups). This is not surprising since Docker came out
of Dotcloud, a direct Heroku competitor. The pros and cons of linux namespaces
have been extensively discussed, it's no silver bullet but when used properly
is quite robust and is rapidly gaining industry recognition.

------
kmfrk
Counterpoint: Heroku Button is the Docker killer
[[https://news.ycombinator.com/item?id=8148794](https://news.ycombinator.com/item?id=8148794)].
:)

It's cute to be bombastic, but no one's getting killed. The dotCloud people
have always been my favourite PaaS people, and competition in deployment UX is
sorely friggin' needed.

------
freditup
I know this is a dumb/basic question, but it's one I've wondered and don't
fully understand. How is Docker that much easier/better than just writing some
other kind of script that configures a machine for your app to run in and
deploys it?

~~~
korzun
Docker is an application run time, basically a small package that is build to
run software of your choice.

It's much faster (and smaller) to build such small images .vs building out the
whole OS with Memcached installed (for example).

And you don't have to deal with OS configuration, etc.

~~~
pit
Not to mention the fact that because intermediary containers are imaged, you
don't have to worry about making a mistake in your script.

Sometimes (okay, every day) I find myself trying a bunch of different things
to get something to work. With Docker, you've got a series of "save points"
along the way. You can continue building an image from the last good point,
and end up with a really clean image that represents the shortest path from
bare install to "what I need to get done."

I was playing around with a Perl script about a month ago that would strip all
the RUN statements out of a Dockerfile, convert them into a shell script and
use that to bootstrap a Vagrant box. So what I'm saying is, if you don't know
what you're doing (or just like to experiment with OSes), Docker is really
interesting.

~~~
pjmlp
AWS or Azure, right mouse click "save image snapshot".

------
jacques_chester
I work on a sorta-kinda Heroku competitor (Cloud Foundry), so my views are
suspect and of course are mine alone etc etc.

However, asking "which should I use for my app, Docker or #{PaaS}?" is a bit
like asking "which should I use to build my house, a brick or a general
contractor?"

They are different things, at different levels of abstraction.

People have already written Docker deployment systems for Cloud Foundry[0].
Plus you get all the other stuff you'd have to write yourself.

[0] [http://www.cloudcredo.com/decker-docker-cloud-
foundry/](http://www.cloudcredo.com/decker-docker-cloud-foundry/)

------
egdelwonk
Check out Dokku for deploying docker containers. It's like a mini-heroku in
100 lines of bash.

[https://github.com/progrium/dokku](https://github.com/progrium/dokku)

~~~
IbJacked
Also worth checking out is dokku-alt ([https://github.com/dokku-alt/dokku-
alt](https://github.com/dokku-alt/dokku-alt)), a pretty active fork of dokku
with added features.

------
nickporter
I don't think docker is accessible enough right now to kill Heroku. You have
to put on your devops hat when you use it. With heroku, you push and it works.

I'm not trying to bash Docker. In fact, I use it quite a bit, and I never use
Heroku. I just think that compared to Heroku, it's much more of a low-level
tool. With Heroku, you can just paste a few really simple commands and your
webapp is now up.

~~~
riffraff
in addition to that, heroku comes with an ecosystem of things that work with
it.

I.e. In any new app, even a toy one, that I deploy to heroku, I am going to
get a basic new_relic account for free.

~~~
brightball
For what it's worth, you do get that anywhere now.
[http://newrelic.com/application-
monitoring/pricing](http://newrelic.com/application-monitoring/pricing)

------
pjmlp
For those us using JVM and .NET languages in AWS and Azure, the real value are
the runtimes running on top of hypervisors without a needless OS layer.

I would rather see more investment into exokernels.

------
mark_l_watson
I agree with the article. However, I am also a fan of PaaS like Heroku, IBM's
BlueMix, etc. because they save labor costs. I also always run a beefed up VPS
and I have my own "Heroku like" git push deployment set up.

Docker in particular, and containerization in general are the future
especially for very large shops like Google, Facebook, etc.

------
ioulaum
> I'm waiting for a convincing Docker PaaS vendor to come along.

I believe Google Cloud services support that now.

