
Tell HN: I accidentally ran up a $1000 Heroku bill - erikpukinskis
This is pretty typical.  Every few years something slips my mind and I end up stuck with a "Erik, you're a moron tax".<p>I was experiencing a spike of traffic when my site was on national news a few weeks ago, and I turned my dynos and workers way up on Heroku to handle the traffic.  The spike went down, but <i>I forgot to turn off the dynos</i>.<p>Flash forward a few weeks, and I am stuck with $1300 bill on a site that I'm not making any money on.  I asked Heroku if they could help me out a little... I'm a good customer, and even though I'm a cash-strapped in-the-red bootstrapper, I gladly pay them hundreds of dollars a month to run my sites, bills payed on time and in full.  I was hoping they'd maybe give me a little discount... not pay for my servers, but maybe not take a profit off of my stupid mistake.<p>They shot me down. No refunds available, even though these were virtual servers, and didn't contribute any real load to any of their machines.  Although obviously they provisioned machines for me, so they had real costs.<p>Part of me wants to be upset, but obviously I was the one who screwed up. Still, it leaves a bad taste in my mouth.  I guess a smaller host might offer more personal care, but I wouldn't get Heroku's world-class engineering baked into their platform, which I really, really appreciate.<p>What do other folks think? Am I just a dumbass who lost a chunk of paycheck, or is this poor customer care?
======
patio11
Heroku runs on top of EC2, and they get charged the same amount of money by
Amazon whether your instances are idle or on fire. That doesn't matter,
though, because businesses do not do cost-based pricing.

There comes a time in a young man's life when "haha, whoopsie, I'm a kid who
is not responsible for his own decisions" is no longer sufficient for a do-
over. I am not sure when that is, but if you're running a business, well,
welcome to being an adult.

~~~
staunch
How horribly condescending and wrong. Any company that lets their customers go
from huge fan to hater in a week has a major bug. The important question is
why did their product turn a fan into a hater, and how can they prevent it?

Heroku should almost certainly have user defined cut off points. They should
require every new user to set a maximum monthly bill to something they can
afford (or "unlimited").

I did exactly that for a similar type of service I worked on. One of the
biggest anxieties customers had was that they'd get a massive unexpected bill,
just like this guy did.

Heroku's best move would be to cut his bill in half and institute a cut-off
feature. Every service like it should follow suit.

~~~
patio11
This is the Ruby versus Java debate: should we treat professional programmers
like they know what they're doing, or should we straightjacket them?

To which I answer: Yes. It is reasonable to run a Enterprise-oriented cloud
scaling company which second-guesses your customers' decisions at every point.
It is also reasonable to run an Enterprise-oriented cloud scaling company
where a professional engineer engaged in capacity planning clicking OK to an
estimate presented in 60 pt Arial is assumed to know what they're doing.

~~~
docgnome
I don't agree. It's not an issue of straight jacketing programmers. It's an
issue of reminding a user they are incurring an enormous bill for no reason. A
simple "Hi, we see you're using X widgets but your traffic would allow you to
reduce this to Y widgets. If you'd like to do that, here's a link. If not just
ignore this email." That would solve the problem with very little effort. It
would keep people forgetting and having a heart attack, getting pissed and
leaving. It really doesn't matter if it is the customer's fault if it causes
them to leave with a bad taste in their mouth.

------
alextgordon
I'm going to go against the grain here and say it's partly Heroku's fault. If
a customer is paying hundreds of dollars a week for a feature they're not
using, that's the time to send a friendly automated email saying "Is
everything okay?".

This is _especially_ important with a service such as Heroku, where users will
be changing these settings all the time and so there's a high probability of
human error.

~~~
dotBen
I can think of loads of reasons why you might be legitimately in that
situation...

Perhaps you intentionally provisioned a large amount of headroom for a
forthcoming expected/possible traffic spike.

In that example it would be annoying to get "are you sure you want to have
this extra resource running?" emails.

 _(sure, you should be building the system to auto-provision but that would
have required a better programmer and you just got the interns to build it,
etc)_

~~~
alextgordon
I get tons of emails saying "Would you like to buy viagra?". Getting an
occasional email saying "You might be paying us too much money" is low on my
list of annoyances.

~~~
sliverstorm
Not only that, but I will almost never be annoyed by someone who says "wait,
are you sure you really want to pay us that much?"

------
StavrosK
This reminds me of something my friend did. We had an assignment to try out
EC2 with Hadoop and we needed 8 machines, and we were warned in class to turn
them off or they'd keep charging.

Guess who didn't turn them off.

So he casually mentions it to me and tells me that he racked up a 500 pound
bill, but his CC didn't have any money in it so it was fine. I asked "so did
you turn them off?" He said "no, not really, but it's okay because they can't
charge me".

So he racks up another 500 pounds, and only turns the instances off when he
finds out that he didn't use the inactive CC but another, active one. Long
story short, he's 500 pounds out (Amazon were kind enough to refund the first
500, which disappeared as soon as it hit his account, since he owed another
500).

~~~
dotBen
reminds me of a friend who was studying for his MSc Physics at Warwick... the
students had to do a lab experiment with a radioactive isotope and were warned
to only have the box containing the isotope open for a few minutes in order to
reduce exposure to safe limits.

Guess who forgot to close this box and after 2 hrs of exposure now has to
worry about cancer and/or infertility in later life. True story.

Look at it this way: at least your friend took CS and not Physics and thus is
only £500 in the hole.

~~~
ams6110
Borderline negligent, though, on the part of the instructor. Lab assistant or
supervisor should have been double-checking safe handling of something that
dangerous.

~~~
sigstoat
he can't be expected to keep track of something for 2 minutes without someone
watching over him, but we can let the fellow operate a car, have mains power
in his home, purchase cleaning chemicals, etc?

loads of graduate students in the physical sciences are regularly given access
to stuff that dangerous or worse. if we've got to have somebody watching over
each and every one of them, graduate education will grind to a complete halt.

~~~
radu_floricica
You are supposed to get training when dealing with anything different and
dangerous. Say you rock climb for the first time... and the instructor talks
to you for 10 minutes about various stuff and somewhere in the middle he says:
"and make sure to check that you can't see 'danger' written on the clasp of
the harness, you may fall if it's not secured properly". You have _lots_ of
stuff to think about... your posture, the grip, the path you want to take...
in a group of 10 people, is it really that unlikely that one will simply
forget?

That's why the first couple of times somebody more experienced will check
everything about your harness.

~~~
sigstoat
and what will the training for remembering to close a box consist of? your
posture, the grip, the path the lid will take?

~~~
erikpukinskis
checking that they closed the box.

------
spooneybarger
I'm going to go with dumbass. Nothing personal there, but you turned up the
level of service and continued to use it even though you didn't want/need it
anymore. The fault is entirely on you. Think of it this way, they provisioned
load to you which in turn even if you didn't use, couldnt be provisioned to
others w/o potentially over provisioning.

~~~
deno
Making money on people's mistakes seems like a bad business model. Give him at
least discount and you'll have an evangelist.

~~~
sr3d
You wrote as if Heroku were taking advantage of their customers. They don't,
you pay for what you use. The poster should not even post such a self-centered
post on HN to tell people to sympathize with him. He screwed up and he is a
whiner. I don't want someone like that becomes an "evangelist" because all
he'd try to do is to get everything his way.

------
fierarul
What you are doing is distasteful because you made a mistake and now you are
using bad publicity against Heroku to force them to lower your bill (or just
out of spite).

Heroku might optimize for this usecase by detecting spikes somehow and
notifying you that you still have machines running, but other than that, I
don't see how any of this is their fault.

------
100k
I wish Heroku officially supported autoscaling. I've been using Google App
Engine lately and the autoscaling and pay for what you _really_ use is very
compelling.

~~~
arfrank
They do for their background workers, but not the dynos.
<http://github.com/pedro/delayed_job/tree/autoscaling>

------
collint
Hmm, you might be out of luck, but it does bring up the question of why Heroku
doesn't have load-triggered scaling, or some sort of dead-man's switch option.

~~~
jmonegro
There are third-party solutions for that, like this one:
<http://github.com/ddollar/heroku-autoscale>

~~~
andrewacove
Having been a Heroku user for a while, and having looked into autoscaling for
dynos and workers a few times (but not having really needed it yet), I really
think they could do a better job of publicizing that these solutions exist.

I imagine that tons of their user base (especially for small hobby sites that
only occasionally, if ever, see high traffic or run background tasks) look for
this, and yet there's nothing pointing to it other than old posts on the
mailing list. At least put information and resources for autoscaling in the
Docs section of the site.

Acquiring resources only as long as you need them, and then releasing them
when you're done, should be the standard service - complete user control
should be the advanced feature set. They already do this for websites on a
single dyno: they spin them down when they're idle for a while. But they don't
do it with workers (what effect does the latency of starting a backup job up
really have? it's already running in the background).

~~~
tfe
It's because they need to keep a process running all the time to poll the
queue for new jobs. So basically, they keep an instance of your app booted at
all times for job processing (if I understand how DJ workers work). And that's
pretty much like a dyno, except it doesn't serve traffic. And there are no web
requests to trigger it to spin up like there are for dynos.

~~~
aaronbrethorst
One of Heroku's employees has a fork of delayed_job that handles autoscaling
workers in the background. I use it in conjunction with several hourly cron
jobs and it works like a charm.
<http://github.com/pedro/delayed_job/tree/autoscaling>

~~~
andrewacove
Yes, I was aware of this, though I haven't had the need to use it yet.

But my question stands: why isn't this stuff very visible - i.e., on Heroku's
DJ doc page: <http://docs.heroku.com/delayed-job> , or at least linked from
there? Or published as an official piece of code on the Heroku github page?

~~~
pts101
Because it would decimate their bottom line if everyone stopped running extra
instances.

~~~
andrewacove
Really? I would certainly hope that's not the case. Relying on customers
overspending due to technical deficiency seems like a pretty poor business
model.

I'd certainly be happier paying more per pro-rated unit of time that I
actually use than I am feeling as though I'm in an arms race to not be
fleeced.

Is there really that much revenue to skim off the top? Either way, they're
paying Amazon out of those charges.

But I'm not so cynical as to think that their success is reliant on this
income.

~~~
pts101
_Either way, they're paying Amazon out of those charges._

I'd be extremely surprised if Heroku actually spools up EC2 instances the
moment you move the slider. It'd be a far more intelligent (and profitable)
design to simply make sure there are sufficient resources to maintain the
desired service levels with a buffer to deal with spikes in demand.

It should be relatively straightforward to provide service levels that are
functionally identical to actually starting each "dyno" or "worker" with a
greatly reduced cost structure.

 _Relying on customers overspending due to technical deficiency seems like a
pretty poor business model._

I didn't mean to say they're relying on it, simply that it's adding to the
bottom line and it would be greatly missed.

------
chr15
I am not surprised that they didn't give a refund; most businesses wouldn't.
However, I think businesses these days should start integrating a one-time
"whoopsie" allowance as part of their customer service (within reason, of
course. It may or may not be feasible depending on your business model).
You're probably going to have people that try to take advantage of this, but
there are some people who make honest mistakes, and I would give them the
benefit of the doubt. Just treat people well.

I believe if you take care of your customers, they'll take care of you. In
general, the one thing that people will remember for life is how you make them
feel.

The benefits of providing awesome customer service outweighs the cost IMO (I
haven't done the math, but that's my gut feeling). I remember reading a story
on reddit yesterday where a gamer experienced the almost exact same situation
with Steam:

[http://www.reddit.com/r/gaming/comments/dcnp7/one_of_many_re...](http://www.reddit.com/r/gaming/comments/dcnp7/one_of_many_reasons_i_love_steam_customer_support/)

that provides a good example of how news of awesome customer service can be
spread, and possibly grow sales (Zappos).

For example, I have several sites hosted with WebFaction.com and I'm
unbelievably happy with their customer service. They respond within 10-30
minutes on any support tickets. I'll continue to use them in the future. (see
how that was a good example of what awesome customer service can do?)

~~~
cglee
Why would you say most businesses wouldn't? I'd think most business would do
something to help a loyal customer stay loyal. I agree with you that it's
absolutely good business practice to help customers recover from their own
mistakes.

On many many occasions, businesses have literally given me money even when it
was my stupid mistake. Some examples just over the past few years:

\- Sprint credited me $100 when I racked up a $250 bill by not watching my
usage

\- Bank of America credited me $100 when I left my card in the ATM and someone
withdrew $100 from my account

\- Hyatt Place gave me a free room when I made a reservation on the wrong date
(and never showed, or called)

\- Southwest Airlines gives you full credit if you cancel your flight any time

\- ATT gave me all my rollover minutes when I changed plans (by rule, you lose
them - i had over 2700)

Those are just some examples that I can remember...I'm not sure it's a healthy
mentality to expect this, but it certainly leaves a very positive impression.
Leaving your customers in a state of feeling ripped off when you can obviously
help is not the right business decision.

~~~
jacquesm
And you prove them in the right simply by recounting those instances here.

------
sad
It's tough to say. You really sound like a reasonable person and if you are
the model customer you appear to be then I would probably side with you.

I like a very simple rule: Customer retention is far more important than
customer acquisition. It may be a flawed rule, I'm not sure since I'm not a
successful business person. But it sounds good to me. It seems vitally
important to retain your good customers.

~~~
ams6110
Yes, the very fact the the OP is now considering (however causally) the idea
of finding a smaller host with more personal care is something I'd be
concerned about if I were his current host. Who knows, maybe he'll go out and
find an awesome small host he never knew existed. Customer gone, and he'll
probably start recommending the new host to all his friends and associates
too.

If it were me, and a good customer made a mistake, I'd eat the cost to keep
the goodwill.

------
pclark
Not sure why anyone but yourself is to blame, and I think it would have been
nice for Heroku to give you a discount then, after this post they should go
quiet and let you pay. I think this post is bad form.

------
sr3d
I'd go with "Dumbass" and add "whiner" to the list. Why is it poor customer
care when all Heroku did was saying no to an unjust request? You can move your
site to another hosting service, or host it yourself.

"I am stuck with $1300 bill on a site that I'm not making any money on." So
that means if you make money on the site, then you'd pay? What's the
difference here? You're saying that because you're poor now, you don't have
enough integrity to say "oops I did something wrong" and take the
responsibility?

I'm myself a boostrapper burning thru my savings and to tell you the truth, I
do everything myself to cut on expenses. I'd love to use Heroku, but they are
too expensive for me to run. So if you don't want to make another mistake,
then start to accept responsibility and learn to take care of problems
yourself.

------
gfodor
Happened to me on EC2 once. They cut me a break and gave me 20-30% discount.
They can though, because the idle machines were not costing them any money.

~~~
amock
They weren't costing Amazon any money if they had excess capacity, but if
those machines being in use prevented them from being sold to someone else
then they did cost it.

------
audyyy
If you could get away with turning up your dynos and workers in anticipation
of a traffic spike and then not have to pay; it wouldn't be fun any more.

~~~
jacquesm
Why? It could be fun for everybody, just set a baseline and a peak level and
use as much as required with a nice little graph of how close you came to your
maximum allowance so you can plan your future hosting expenses.

------
niico
I happened something similar to me. I forgot I was running a recurring billing
adwords campaign. I realized I had that campaign when the bill came with a
lovely 3.700$ total. The worst part is I didnt got to sell anything.

------
thibaut_barrere
I don't find this poor customer care, really: I consider this normal.

You're talking about "a few weeks": did you check your expenses in the
meantime ?

I would put some reminders (google calendar etc) or habits in place to
regularly check your expenses.

------
checoivan
It's human error. Simple as that. Happens to absolutly everyone even the
brightest individuals. I've seen in "the livesite" that this kind of spike
problems are more common than you think, and the ideal solution is to automate
an alarm that tells you both, when a spike is happening, and when X throughput
passes over a threshold. This kind of automation is easy and spares lots of
trouble.Could be as simple as scheduled task parsing some logs. Some more
fancy option is always good though :)

------
sliverstorm
At this point, I'd probably just be thanking my lucky stars it was only $1.3k.
You'll make it back.

------
akshayubhat
Something like this wont happen to Google App Engine, thankfully.

------
dh
I can see the problem from both sides but really Heroku had some more more to
gain by just crediting it or most of it in what you would say about them.
Clearly by the rules, you should have to pay, but that does not mean that a
business should not do the right thing and work with a customer.

Imagine how big a raving fan you would have been if they had credited the
entire thing. That is a tiny cost compared to acquiring a new customer and
turning them into a raving fan.

------
bdmac97
I got burned by this on Heroku recently as well but to a much lesser extent
$$$-wise. My app typically runs on 1 dyno because it's basically not supported
anymore. At some point I must have been messing around in the interface and
accidently bumped my dynos. Oops.

I personally think they should have a "WTF your load is like zero, you
normally have X dynos, but you're using Y dynos for no real reason... dumbass"
alert email... I certainly would have appreciated it!

------
JJMalina
Reading this makes me laugh because today I got a $1400 bill from National
Grid for gas and heating. The normal bill is around $40 usually. An
interesting phone call awaits tomorrow. In all seriousness though Heroku
should strike a deal since this is your first offense, just like banks will
waive fees for you when you overdraft the first time.

------
thrashr888
I got stuck with a large, unexpected bill at my auto dealer. They comped the
next large bill. They're awesome and I'll be a customer for life. They're also
the highest (Yelp) rated shop in the area so I bet I'm not the only one.

I suppose it depends on what sort of shop Heroku wants to run. It just seems
like customer service is not a big deal to them.

------
jschuur
I'm kind of disappointed that Heroku didn't at least chime in to comment on
the situation in this thread. It's one thing to play hardball and stick to
their guns, but when there's an obvious, popular thread about it, they should
at least own up to the decision and chime in.

------
theDoug
In my experience a smaller host would have likely been sunk by your lack of
attention requiring them to allocate more services (used or not), not be able
to provide more personal care. You're lucky you're on a big host.

------
djb_hackernews
This happened to me, but not an such a large scale with an EC2 instance. I
emailed amazon and they severely discounted the bill, with a warning to be
more careful because they might not be able to do it in the future.

------
daemin
Well the lesson that I've learnt here is that when adjusting something like
this up temporarily it is a wise idea to add an event to your calendar to
switch the thing back down to normal levels.

------
bfung
does Heroku have an api? if so, think of it as investment towards writing a
traffic spike script/cron job =)

------
thepumpkin1979
First: If I were Heroku I won't give you any discount, not after this.

Secondly: Heroku is high class hosting, they charge you for being able to host
yours "spikes of traffic".

Third: I'm pretty sure you enjoyed their free hosting plan for a long time,
you should be thankful.

