

Billing Incident Update - agwa
http://www.twilio.com/blog/2013/07/billing-incident-update.html

======
aroman
Very well handled. Facts, gestures of good will, and straight-forwardness.

Call me crazy, but I tend to think issues like this ultimately serve, if
handled well, to _improve_ a company's public standing.

Though I suppose that depends on how one defines "well"...

~~~
stinky613
In 1982 some nutjob went around to supermarkets and pharmacies and put
potassium cyanide in Tylenol bottles. Johnson & Johnson responded by issuing a
recall for ALL Tylenol products nationwide. This incident is anecdotally
credited with having directly led to those paper or foil seals on over-the-
counter pill bottles.

[https://en.wikipedia.org/wiki/Chicago_Tylenol_murders](https://en.wikipedia.org/wiki/Chicago_Tylenol_murders)

~~~
BHSPitMonkey
Wow. That is utterly tragic and horrifying.

------
mikegirouard
It was surprising to read they will be reimbursing any overdraft fees caused
by the problem. Good move.

~~~
grimlck
Why are you surprised they are reimbursing overdraft fees? That is the bare
minimum that I would expect from any company which made this mistake

~~~
maaaats
Of course it is to be expected, but unfortunately one cannot expect it happen.
If you get it?

~~~
mikeash
In this case, I think one can. If they cause damages due to negligence (or
malice), they are liable for those damages. Only the dumbest of dumb companies
would refuse to reimburse overdraft fees in a case like this, because doing so
will open them up to lawsuits from every single affected person, lawsuits
which those people basically cannot lose.

------
ajtaylor
Simple, honest, open and straight to the point. Nice to see that they
recognise it's not just about the money, but the trust people had in Twilio.
I'm looking forward to reading the promised post-mortem, because I can't
recall reading one from them before.

~~~
RobSpectre
It has been some time since we have had an incident this disruptive for our
customers. Looking forward to delivering that investigation summary to you as
soon as we feel we got it right.

------
jmandzik
This is about as good as you can handle an incident like this, PR wise.

------
josh2600
Congrats to Twilio on a textbook example on how to handle a billing outage.
While I hope to never to through this I do admire a team that can execute like
this.

Cheers, from one telco nerd to your team of hackers.

~~~
RobSpectre
Always appreciate your support Josh. much work to do before this incident is
resolved fully.

------
ckdarby
So many updates on their status page! One of the first companies I felt like I
knew wtf was going on during the issue.

~~~
dustingetz
And you can see status from two years ago! Google won't even recognize that
gmail is down unless 5% of requests are failing (!!!! Try getting your work
done all day if 1/20 actions fails.)

~~~
RobSpectre
Very glad you found the stashboard useful during the incident. This is
actually one of my favorite open source projects of ours.

If you would like a similar service for your company, check it out here:
[http://www.stashboard.org](http://www.stashboard.org)

------
bdcravens
Interesting. I just came back from an event here at Lone Star Ruby Conference
they sponsored (movie - they used Twilio powered service to pick the movie we
watched at a local brew theater). Would have made for some interesting
questions had more known about this problem :-)

------
brador
Why not batch process billing every hour or 30 minutes? That way you can run
simple checks to look for anomalies.

~~~
RobSpectre
It's a good question. As our service is delivered through a utility pricing
model, most of our API requests are billing events - each call, text message
or recording ends up having a debit to the account balance associated with it.
As you can imagine then, our auto-recharge feature could be called any moment
to make sure the balance doesn't run out.

That said, we do have the monitoring you describe in place which is how we
were alerted to the incident and limited the damage to 1.4% of the account
base.

------
brianstorms
Twilio should really consider a lighter gray color for their text on the white
background. Right now the text is simply too legible still! Why, I can almost
read it, if I crane my neck and look at the screen at just the right angle.
What could they have been thinking? Hopefully they will take this feedback to
heart so that future urgent messages like these will be completely obscured.

~~~
maaaats
Kinda ironic that this comment shows up as light grey on a grey background. :)

~~~
RobSpectre
Appreciate the feedback. Will get it fixed.

------
happywolf
Nice response, bad text color. The gray on bright white background really
strains my eyes. Ouch.

~~~
neurostimulant
I use iReader extension whenever the website's color scheme strain my eyes too
much. It's essentially a clone of "Reader" functionality on safari.

[https://chrome.google.com/webstore/detail/ireader/ppelffpjgk...](https://chrome.google.com/webstore/detail/ireader/ppelffpjgkifjfgnbaaldcehkpajlmbc?hl=en)

~~~
RobSpectre
Good feedback. Thank you.

------
throwaway1979
That's how it is done. Classy.

------
contingencies
For anyone looking for multiple (redundant) SMS provisioning, I have had not
perfect but pretty good experience with
[http://fortytwotele.com/](http://fortytwotele.com/) over a good few years. I
am not connected to them at all, just throwing the option out there. As a
bonus, they're European, which means you are better positioned for compliance
with EU privacy laws for non-US clients.

