
Twilio is erroneously over-billing and suspending accounts - amduser29
http://status.twilio.com/services/account-portal
======
RobSpectre
Rob from Twilio here. We are still actively working on this incident and aware
that erroneous auto-recharge billing and account suspensions have occurred for
a number of customers.

If this has happened to your account, please send an email to help@twilio.com
where we have our support team working on making this right for everyone
affected. We are spinning up additional resources to make sure every customer
issue gets resolved as quickly as we can.

Link for this article goes to our status board which serves as the
authoritative source of information on this incident. We will be delivering
updates every half hour or as new information becomes available.

We are very sorry. This is far short of our commitment to you. We'll have more
on the incident and what we will do to make this right later today.

~~~
WillieBKevin
Rob, best of luck to you and the Twilio team while fixing and recovering from
this incident.

We've been with Twilio for years and have received excellent service. Billing
issues are always serious, but its nothing compared to a call routing issue,
and when it comes to call routing Twilio has been flawless. I'd much rather
call my bank and discuss an overdraft fee then call a client and explain why
their calls were not routed properly.

In the years we've been with Twilio, we have not experienced any significant
downtime. Compare that to one of their main competitors that experienced eight
consecutive hours of downtime during business hours earlier this year, and I'm
confident we're in good hands.

~~~
RobSpectre
Thank you very much for that.

We came up way short for you this morning - committed to making it right.

~~~
benjamincburns
Seconding Willie's comments. I'm not your customer today, but seeing how y'all
are handling this would make me feel confident to use Twilio should the need
arise.

When you get some time (I'm sure you have none right now), read some of the
Dreamhost discussion below. You folks appear to be doing an _excellent_ job
responding to this.

~~~
gknoy
I completely agree. I've seen several public instances like this one over the
past year here where Rob and the Twilio team have done what appears to be the
Right Thing (from a customer service perspective). It's inspiring to see them
do this, and to see that it is indeed possible to execute very well with a
focus on customer happiness and "making things right".

------
benjamincburns
For people commenting about overdraft charges, Dreamhost went through this a
while back. I was in college at the time, and didn't realize they'd drafted a
second charge to my account. A series of small transactions later, and I was
$300 in the red with very little income to fix it. They offered to reimburse
overdraft fees for those affected, but I never was reimbursed.

This is also great as a "how not to tell customers that you just double billed
them." At least Twilio is getting this right.

[http://dreamhost.com/dreamscape/2008/01/15/um-
whoops/](http://dreamhost.com/dreamscape/2008/01/15/um-whoops/)

Step 1: Don't complain to the customer how inconvenient your massive mistake
was for _you._

~~~
phaemon
What's wrong with the Dreamhost post? He specifically says: >"The end to this
story is that of course, I’m very very sorry, we’re very very sorry, and I’m
sure you’re very very sorry this happened. I really am. I understand the sort
of problems that an unexpected large charge to your credit card (or worse yet,
your debit card) can cause. If the tone of this blog post seemed a little
light, I apologize I don’t mean to offend and I realize how serious an issue
this is."

I can see why you'd still be annoyed about not getting your charges refunded
though. What happened there? Frankly, if your bank didn't refund them when you
told them what happened, they seem pretty scummy. I hope you dumped them as
soon as you could!

~~~
benjamincburns
The bank was SunTrust. They _were_ very scummy.

As for what's wrong with the post, a ton. This was e-mailed to every single
customer. For many people, myself included, this was their first notice that
something was wrong with their finances. And it was the author's fault. The
line you quoted was at the end of a very long post. The apology should have
been right in the first paragraph. Instead in the first two lines he's
complaining about how bad this is for _him._

Then he makes a bunch of jokes. Then he explains why it was just tiny mistake.
Don't feel bad if _I_ missed the problem? I'm not the one being trusted with
millions of dollars of your customer's money... [Edit: to me that bit reads as
"See? You missed it too! Now why should I feel bad?"]

After all this, finally, he apologizes.

For me, this guy's mistake meant that I now had to worry about how I was going
to pay bills/rent, and how I was going to eat. My situation was probably quite
unique, but that didn't make it any less impactful for me.

To me the whole thing reads as "I just accidentally removed a few _million_
dollars from my customers banks/credit lines - so now I'm gonna have a laugh
with them!"

I can say with great confidence that I'm not immune to the mistakes that
caused this to happen. But what I can say is that were I in his shoes, I'd
lead with an apology, be very succinct about what happened, and finally list
details as to how I'm going to make it better.

Edit: To his credit, if you read the next post he did a much, much better job
there.

Edit 2: And also for what it's worth - I still use Dreamhost. Aside from this
SNAFU they've ran an excellent service, and (again, except in this case) their
customer service has been excellent in responding to any issues I've had.
However now I always pay with credit rather than debit, and I don't store my
card info.

~~~
phaemon
Fair enough. It's probably a bad first contact on the issue. I read it as him
fully accepting responsibility for the error and explaining exactly what had
happened.

But then, it's easy enough for me to view it in a generous light, when it had
no personal effect on me. I would no doubt have felt very different had it
been my account that was wiped out!

Oh, and yeah banks suck, especially when you're poor. Been there...

------
eksith
This sort of thing has happened before to another company, but that was a case
of human error compounded by the fact that there were no software safeguards
in place.

[http://dreamhost.com/dreamscape/2008/01/15/um-
whoops/](http://dreamhost.com/dreamscape/2008/01/15/um-whoops/)

At the very least, it would be wise for any company to implement billing
features that will under bill rather than overbill. Under billing at least
gives you the option of notifying users later (or you just eat it).
Overbilling can sour a professional relationship to the point of ending it.

~~~
aculver
Here's a choice piece of code we have in production along the lines of what
you suggest:

    
    
        if prorated_amount > (cost_per_month * 2)
          raise "critical! we were about to prorate-charge someone #{prorated_amount}"
        end
    

It's never been triggered, but this is probably the only place in our system
where we're actually responsible for an exact amount that ends up on an
invoice, so having a reasonable upper limit seems like a no-brainer.

~~~
hartleybrody
What if this code is called multiple times in quick succession? Based on the
fact that I got a dozen separate emails over the span of 10 minutes, I'm
guessing Twilio's overcharging happened in a series of transactions, not all
at once.

This code -- while good for catching bad single calculations -- wouldn't catch
that.

~~~
wizzard
Well I don't think he was saying "cut and paste this exact code into your
codebase" but rather that it is possible to write safeguards such as this...
in this case, perhaps checking whether an account has already been billed this
month, or whether there is already a transaction pending, or...

We can't prevent every mistake but we can make sure the same one doesn't
happen again.

------
dazbradbury
We've been charged about 10k and going up...

Looks like their auto-recharge balance test keeps returning 0 or less than 0,
it charges your card, and loop.

EDIT: Looks like they're doing something - the UI at least now displays $500,
rather than $10k before. The only real confidence I'll get is when I can see
todays account transactions from the bank.

~~~
jerguismi
I had like $240 billed, when there was nothing used... fuck.

~~~
RobSpectre
I think that reaction is entirely appropriate in this case.

We just unsuspended the accounts affected by this incident and are currently
working on the balances.

If you haven't already, please send an email to help@twilio.com indicating you
are affected by the incident so we can communicate directly as we correct
these erroneous charges.

------
screwt
Apologies for nit-picking at punctuation, but occasionally it matters :-)

"Twilio is over-billing ..." = Twilio is charging more than they should.

"Twillio is over billing ..." = Twilio has given up on billing. It's _so_
over.

Just my personal bugbear - but sometimes it's worth getting this sort of thing
correct. Apologies if this is just a US/UK difference, also (I'm in the UK).

~~~
sjtgraham
Not a single person was confused by this, including me, a British dude sitting
in the middle of London.

~~~
krisoft
Now you got me confused. You are saying that no one was confused, not even
you. Or you say that it was confusing not only for a single person, but for
you too?

~~~
sjtgraham
I'm happy to take the karma hit I'll get for this, but IMO this is emblematic
of a large problem with HN, i.e. asinine bike-shedding. I address an unhelpful
nitpicking comment and get a nitpicking reply. It's turtles all the way down.

------
edanm
I see a lot negative comments here. Frankly, I understand people's
frustration, but I think Twilio are dealing with this in the right way.

I was one of the customers affected. Luckily for me, my credit card
automatically blocked the transaction after the 10th time I was auto-
recharged, meaning I had a cap on how much I paid. I can totally understand
the frustration of people discovering that they suddenly have 100's or even
1000's of dollars worth of charges to their credit card.

On the other hand, by the time I noticed that something strange was going on,
Twilio's status page already said they were looking into it. I contacted their
support right away, and they answered within 20 minutes, during what must be a
customer support crisis.

And now RobSpectre is on HN giving very helpful answers.

So, no, it's not their finest work (as RobSpectre has said), but at least
they're dealing with it quickly, and hopefully professionally as well. Let's
let this play out before we jump up and down on a great company that we all
love.

~~~
RobSpectre
Thank you very much for the support.

Lot of work to do to make up for this shortcoming.

------
k-mcgrady
This is bad. It'll badly affect people with low bank balances and high
overdraft charges - it'll be interesting to see how Twilio reimburses people
for charges banks levy on customers.

~~~
freehunter
Or if it's on a credit card and you hit your credit limit and they keep trying
to charge. How do you reimburse for a drop in credit rating?

~~~
nkozyra
This is unlikely to happen given most cards report monthly with a 30-day
delay. Assuming you're able to contest the charges there would be no effect on
your rating.

Debit/bank users are definitely the most susceptible.

~~~
lftl
Additionally as far as I know there is no credit rating penalty for repeated
attempted charges above your limit. That sort of information isn't even on any
report I've ever seen. The increased balance may impact your credit score, but
as you've noted it's unlikely this would make it onto your credit report
unless Twilio is rather slow to resolve the issue.

~~~
freehunter
However, running close to your credit limit _does_ impact your credit score,
at least according to everything I've seen on the subject.

------
btucker
I'm glad they (and most of us) are not in the business of making insulin pumps
or other software where such bugs could have potentially lethal consequences.

~~~
themckman
You mean you don't want to see a medical supply startup that "moves fast and
breaks stuff"? I'm surprised we haven't yet...

~~~
tbrake
Disrupt Health!

~~~
knodi
With death.

------
shakes
I've been using Twilio for over 3 years now and haven't experienced any
significant downtime or service issues. Their support has always been amazing.
Obviously this situation sucks but if there's any company I'd trust to make it
right it's Twilio.

From a developer point of view, we all have bugs. I remember the first time
one of my bugs impacted customers and it was awful. If there's any community
that should be understanding that things like this happen in the tech world,
it should be us.

~~~
RobSpectre
Thank you very much for that shakes. Grateful we have you in our community.

We fell short of your expectations today. More later today on how we'll get
this right.

------
SCDaniel
I think the fact of the matter is as a developer I want to get angry at them
and then remember this is my own worst nightmare - mistakes happen, the true
test will be in how Twilio reimburse people.

~~~
hobs
Yeah, I see a lot of (completely fair) angry comments in this thread. But
honestly the response has been pretty damn good for what happened, and we will
see how it plays out. Any one of us could have made this mistake and Twilio is
a great service, otherwise you wouldnt have seen many people here complaining
at all!

~~~
RobSpectre
Response is not good until everyone affected has their money returned to them
and we explain what happened.

Very much appreciate your support through a difficult night and morning. More
to come.

------
josh2600
I work in telecom, in a slightly different vein from Twilio's and I deeply
empathize with the pain of billing.

Even if Twilio's somehow able to ignore the pain of telco engine rating and
least-cost-routing, there are still so many places you can make an error and
cause something like this.

Kudos to Rob and team for owning the issue and I hope the Twilio folks get
this sorted out (I'm confident they will). It's worth mentioning that I've
never seen another billing issue of this size from Twilio so one in 3-4 years
isn't too awful. Compared to AT&T it's probably generous! (Facetious but with
a grain of truth).

Good luck guys and keep rocking. Here's hoping everything works out.

~~~
MichaelGG
I'd be really interested to know what ends up being the root cause. I know
properly transactional databases ain't so popular in some parts of telecom,
but it's pretty simple to insert a record of the charge transactionally. Then
the only real failure path is the lack of distributed transactions to the
credit card processor, but it's unlikely you're going to run into many
repeated commit failures.

~~~
josh2600
Our experience: We run BigCouch for Database and BrainTree for processing.
We're using DTH Software for our rating engine at the moment.

I don't believe any of these technologies have any failsafes built-in related
to credit card processing (or overcharging), and so anyone who uses these
systems has to perform their own sanity checks.

It's extremely rare for billing to screw up in Telecom, because, essentially,
that is the crux of all Telco business. Twilio is probably using a homegrown
or Non-telecom billing system, but I don't have any insight into their
operations. I would hazard a guess that it's homebrewed though as I can't
imagine a processing company introducing an error like this (stranger things
have happened).

So yeah, I think this will be a one time incident, and frankly, their response
was fantastic.

------
RobSpectre
Rob from Twilio again. Our CEO Jeffiel just posted an update on the situation
on our blog which can be found here:
[http://www.twilio.com/blog/2013/07/billing-incident-
update.h...](http://www.twilio.com/blog/2013/07/billing-incident-update.html)

Full text also here:

At 3:28am PDT/11:28am GMT, our monitoring systems reported an anomaly in our
billing systems, which resulted in erroneous credit card charges and in some
cases account suspensions. This incident affected 1.1% of customer accounts.
The on call team immediately began an incident response, using the Twilio
status dashboard at status.twilio.com to update customers at regular
intervals. By 6:24am PDT/2:24pm GMT all suspended accounts had had service
restored. The same problem briefly re-occurred at approximately 12:30pm
PDT/8:30pm GMT, affecting 0.3% of customer accounts, which were immediately
remedied.

At this time, the Twilio billing system is offline and account-balances are
not being updated in real-time until we fully resolve the issue. We are
actively processing credit card refunds, and you will see a transaction void
or refund on your credit card statement shortly (most banks process these
within 24-48 hours). While the billing issue is being resolved, all voice and
messaging services continue to operate normally.

In addition to refunding erroneous credit card transactions, Twilio will also
be crediting affected accounts an additional 10% of their last 30 days’ spend.
We recognize that it’s not about the money, but our responsiveness to the
situation that matters to you. If your account was affected, please consider
this credit an acknowledgement of the inconvenience we’ve caused you. If
affected customers incurred overdraft or over-limit fees due to this incident,
we will also make them whole. We are in the process of contacting all affected
customers via email. Additionally, we will be releasing a full postmortem on
the incident once all events and root causes are known, as well as detailing
the corrective steps we’ll be taking.

Our focus is on providing you the best quality service and experience, and we
recognize that today’s disruption came up short of what you expect from
Twilio. Please accept our apologies and know it is our mission to always be
earning your trust and business.

As many have you have pointed out on Twitter, this is not a fun day for our
team, and especially our engineers. We appreciate your support and your
patience.

Sincerely, Jeff Lawson CEO & Co-founder

~~~
pdknsk
I suppose this is because it was written in haste, but you're an hour off on
GMT times.

------
lysol
Haha, the one time in my life having an outdated credit card on file paid off.

~~~
uptown
Better double-check. Sometimes credit card companies allow charges to continue
to flow through old, expired, even cancelled / compromised credit cards. I
learned first-hand that a card can still process charges after AMEX allowed
about 7 months of Netflix charges to continue to be processed from a card
number I'd reported compromised.

~~~
jusben1369
What may be happening there is that Netflix is connected to services provided
by the card companies to stop interruptions to recurring billing for reasons
like this. Long story short if Netflix receives an error code from the payment
gateway that says something like "Card expired/Card changed" it'll make a call
out to this service with the old card data. That'll do a look up and return
the new data to Netflix who then updates their records and charges away
happily.

------
jerguismi
Goodbye, twilio...

~~~
RobSpectre
I'm very sorry to hear that, but understand your frustration.

As I indicated above, please shoot us a note to help@twilio.com so we can make
this right for you.

------
SCDaniel
Yeap had this ourselves. Auto recharge actually took $500 rather than $20!

~~~
RobSpectre
Absolutely unacceptable.

As I indicated above Daniel, if you haven't shot a note to help@twilio.com
indicating you're affected by the incident, please do so we can communicate
directly when this $480 mistake is corrected.

------
martin-adams
This is one of the reasons I like to use my credit card over a debit card.
Knowing that errors like this don't affect my mortgage payments makes me feel
safer.

~~~
claudius
The web portal of my bank has a link ‘cancel’ next to every direct withdrawal
which lets me cancel the charge immediately (at the cost of the people
charging me, of course).

------
drx
This has gotta suck. Many cards have steep overdraft fees.

------
imroot
My account was impacted by this as well, however, it looks like my credit card
company blocked the fifth recurring bill (and I woke up to a call from the
Fraud Team at my credit card), so no real damage has been done. Support
answered my email within 10 minutes, and promised to help me clean everything
up.

~~~
RobSpectre
Very sorry for the inconvenience here and appreciate you opening a ticket for
it.

More to come.

------
mathrawka
The worst part is that the calls aren't getting updated with prices right
away... so if you built a system that relies on getting the price info at the
end of the call, you probably aren't pretty happy now.

~~~
RobSpectre
We are working cautiously on restoring full billing service to all accounts.
We are trying to err on minimizing further billing impact to our customers as
we restore service and, as such, taking each step carefully.

This will also affect the Usage API for some users. If you would like an
immediate update when these are restored, use the OP link
([http://status.twilio.com/services/account-
portal](http://status.twilio.com/services/account-portal)) or send an email to
help@twilio.com.

We're moving as expeditiously as safety allows.

~~~
mathrawka
Thanks, I have been in touch with my account manager... just making a note
that developers should consider this type of failure to try and build
resilient systems using Twilio.

~~~
RobSpectre
Good to know.

Very sorry again.

------
andyhmltn
Oh dear, not nice at all if you're on a budget

~~~
RobSpectre
Definitely not nice for anybody.

We will make this right.

------
hndl
Hit -- but we only see an overcharge (3x). Service seems to be back up. Do you
have a support number our ops team can contact (they're paranoid).

~~~
RobSpectre
Very sorry for this mistake.

Can you have your ops team reach out via help@twilio.com and we'll get them
connected with a resource to keep them informed?

------
donaldmorgan
Another example of the amateur nature of Twilio. Good for $30 a month
developers but bad for businesses that depend on their phone system.

------
mkoble11
Mistakes like this really suck, but I have no doubt that Twilio will make it
right.

------
conformal
painful reminder that you need to test payment processing code, by hand or
however, after updating it.

i've run some ecommerce sites and this is basic stuff - new payment modules,
time to test card processing works properly.

------
TallboyOne
Should have used <= instead of <

Best of luck

------
robobrobro
Wow. Seriously? Learn to code.

~~~
h2s
Doesn't matter how good you think are. If you write code professionally for
long enough, you eventually experience an event like this first hand. It's an
absolutely horrible experience for a developer.

~~~
oconnore
Being a professional developer is hard. That's why it's generally a high
paying position, because you solve hard problems and are accountable for the
results.

So stop making excuses. An organization of this scale should be taking serious
precautions (against code errors, bad hardware, malicious attacks, network
problems, compiler bugs, cosmic ray induced bit errors, etc.) when building a
credit card processor.

If something truly extraordinary comes out to justify this problem, I may be
sympathetic. But Occams razor suggests this was avoidable (through good
process, not "I should have seen that off by one error").

~~~
cunac
I would think NASA has even more controls in place and still shit happens. All
problems are avoidable in hindsight but it is inevitable that you will have
problem at some point.

