

Demystifying Site Outages - mcfunley
http://www.etsy.com/blog/news/2012/demystifying-site-outages/

======
DigitalSea
This is the most detailed server outage response I have ever seen. Looks like
Etsy understand the importance of accountability and I really appreciate the
way they wrote the post. A short version for non-technical and short attention
span minded people and a longer version for the nerds who like to know the
intricacies of big sites like Etsy.

~~~
gadders
And even the longer version was fairly understandable for non-technical folks,
I'd have thought. Very well written.

I always remember reading in a book once (Influence by Cialdini?) that the
main signifier for whether a doctor gets sued or not is not whether they make
mistakes, but how they handle them. 'fessing up honestly is the way to go.

------
patdennis
Good on them for taking the time to write such a detailed report. Shows
respect for their users.

------
tedunangst
_Without diving too far into technical details, suffice it to say that in
order to support the special characters that some languages require to be
complete, we needed to do an upgrade of the database server software._

Oh, hi, I see you're using MySQL. MySQL's 1024 character (oh sorry, not
characters, bytes, oopsie!) limit for indices was the bane of my existence.
You have a mountain of code and tables using varchar(200) and an index
spanning five such columns. Works with latin-1. Then somebody decides this new
fangled utf-8 is the new hotness. hahahaha, no database index for you! Now you
get to go back and decide if you can shrink said columns or get by with a one
column index (mysql assumes 3 bytes, so one 200 char column is already more
than half your allotment).

~~~
unwind
On a different but related note, I found that sentence to be pretty ...
strange. They go into lots of technical detail about servers, indices, users
and tables and things, but that wording sounds as if they're explaining the
world to a (small) child.

The idea that there are "normal" (=US ASCII, I presume?) and "special" (=the
rest) characters used in human languages is a bit 1980:s, or thereabouts, to
me.

~~~
gadders
I'm guessing the _primary_ audience is the site's users - i.e. people macrame-
ing plant holders and such like - and not other techies.

------
andrewcooke
that was interesting, but it felt like two different articles rolled into one
(a short non-technical and a long-but-shorter-than-that technical one).

anyway, so alter table modify on a primary key (int to bigint) requires that
the index be rebuilt? is that true for all databases? or was the problem
something else?

~~~
theatrus2
Since the allocation space is different, it's generally a rebuild. The only
exception would be a DB storing integers as a variable length int - unusual.

------
latchkey
In 2004, when etsy.com was created, we all were pretty aware of the importance
of specifying utf-8 when we setup systems.

How could a company, that now has 120+ engineers listed on the about page, get
to 80+ databases without fixing this issue?

