

SeatGeek Blog: What An Earthquake Does To Page Load Time - acslater00
http://seatgeek.com/blog/dev/what-an-earthquake-does-to-page-response-times

======
dorianj
Also interesting: a Sun engineer uses dtrace to show HDD latency spiking after
he yells at his disks

<http://www.youtube.com/watch?v=tDacjrSCeq4>

~~~
nkassis
probably the same thing happened here. Vibration will cause latency in reading
disk. If the hd platter are jumping around the needle will have a hard time
reading it. That probably explains the spike. Time to buy some SSDs ;p

------
brk
Disappointedly lacking on details.

Why did page load time go up during the quake? Was there a fiber fault that
took a few seconds to be routed around? Did the vibrations cause HDDs to
temporarily suspend?

This could have some interesting data behind it, but as it is the article
doesn't even have conclusive proof that the earthquake _did_ cause this
outage.

~~~
josegonzalez
As far as the reason, you'd have to ask who works on AWS. For obvious reasons,
we don't have direct access to the Amazon datacenter.

As far as conclusive proof is concerned, yes, we can't guarantee there wasn't
a gravitational singularity that affected response time, but it's very likely
that this was the case.

~~~
egiva
Yeah, latency is most likely due to the vibrations affecting the various rack
components, as commented here. Actually, earthquake-proofing datacenters is a
big business in places like the West-Coast USA and Japan:
[http://www.datacenterknowledge.com/archives/2007/07/17/earth...](http://www.datacenterknowledge.com/archives/2007/07/17/earthquake-
proofing-the-data-center/)

~~~
pistoriusp
I remember watching a video about how vibrations can have a negative impact on
hard drive latency (Video is at the bottom.):

<http://blogs.oracle.com/brendan/entry/unusual_disk_latency>

------
blantonl
Here at RadioReference.com had a MySQL Master server which is hosted on AWS
East in the N. Virginia data center inexplicably crash on us right after the
earthquake. The server uses a RAID-0 Stripe across 4 EBS instances and has
been running for over a year without a reboot.

And, we were featured on CNN live right after the quake as a source for
breaking news information.

We're scaled to handle a traffic floods because we get them occasionally when
something big happens public safety wise, but I'm really wondering whether or
not this crash was due to a huge influx of people or some hardware anomaly
during the quake (frozen disk, network problem etc)

A reboot of the server and an INNODB recovery fixed the issue, and all is fine
now.

------
tseabrooks
I was imagining it was the sudden violent shaking of the HDD. Thus the "lesson
learned" that "servers don't like earthquakes"

------
techiferous
"our Amazon AWS servers were all in Virginia, right near the epicenter."

Very little is near the epicenter. The epicenter was in one of the least
populated areas of the entire state. Here's a map:

[http://earthquake.usgs.gov/earthquakes/shakemap/global/shake...](http://earthquake.usgs.gov/earthquakes/shakemap/global/shake/082311a)

There are probably less than 50,000 people living inside the yellow circle and
there are no cities.

Amazon's data centers are in northern Virginia. This earthquake did not happen
in northern Virginia, it happened in central Virginia, between Richmond and
Charlottesville, about 60-90 miles away from northern Virginia.

~~~
JoeAltmaier
For some definitions of 'near'. In a country 3000 miles wide, 60 miles is a
fine definition of 'right near'

~~~
techiferous
I know, but it doesn't sound right if you live in Virginia. Just like saying
"Los Angeles is near San Francisco" probably only sounds right to east
coasters. And if you look at the shake map, it makes a big difference whether
you are within 20 miles of the epicenter or 100 miles.

------
jordanmessina
What does an earthquake do to ticket sales in the east coast? Now THAT would
be interesting.

------
bradleyland
I'm more interested in their real-time monitoring board setup.

~~~
kessler
From the post:

"For about six months, we’ve been using a combination of StatsD, Graphite, and
GeckoBoard to power a real-time dashboard of some of our system stats."

~~~
bradleyland
I saw that much, but I'd love to see a write-up on their implementation. We
primarily use Munin with a couple of custom plugins. It's fine for the
sysadmin side, but we were thinking of pushing some app data stats to a
customer facing interface. Tools like GeckoBoard look much better than Munin
graphs.

------
singlow
Would be useful to know if there was a corresponding traffic spike or if the
response time spiked on typical usage.

------
Shenglong
Looks like my server bandwidth graph during DDoS attacks.

------
DPS47
Pretty interesting article.

