

Google: 'At scale, everything breaks' - ca98am79
http://www.zdnet.co.uk/news/cloud/2011/06/22/google-at-scale-everything-breaks-40093061/

======
DanielRibeiro
It is always a pleasure to read from Urs Hölzle. I've been in touch with some
minor spec of his work[1], but I'm always impressed by it.

[1] <http://research.google.com/pubs/author79.html>

------
veb
All the best (and fun) scaling issues is best solved by _hacking_. Is there
really any proper decent methods out there that can actually take a company
from 5 users a day to 5 billion without any problems at all? I'm aware that
you could perhaps "foresee" it, and get a big data-center, and lots of
hardware solutions -- but is this really the solution that's called an 'out of
the box' solution?

Sounds like when Google gets a problem, they create something to fix that,
which is pretty damn cool in my opinion.

~~~
zaphar
As a google employee I can say it really is "pretty damn cool".

------
clarkbox
"...most applications don't use [Google File System (GFS)] today. In fact,
we're phasing out GFS in favour of the next-generation file system that is
very similar, but it's not GFS anymore."

what could this be? home grown? an open source project?

~~~
haberman
Colossus: see a slide about halfway through this presentation:
[http://static.googleusercontent.com/external_content/untrust...](http://static.googleusercontent.com/external_content/untrusted_dlcp/research.google.com/en/us/university/relations/facultysummit2010/storage_architecture_and_challenges.pdf)

------
delinquentme
"We use tapes, still, in this age because they're actually a very cost-
effective way as a last resort for Gmail. "

.. what kind of tape drives does Google use?

~~~
dholowiski
BFT... big freaking tapes. I wonder if it's really tape though. You can get
'tape' backup systems where the cartrige looks just like a tape, but it's
actually a special hard drive. I wouldn't be surprised if that's it.
Otherwise, I wonder how many miles (of tape) long my gmail inbox is?

~~~
zinkem
According to one source I found[1], 60 meters of DAT tape holds 1300mb of
data. That's 21.67mb per meter. My gmail inbox is 679mb, thats about 31 meters
of DAT tape.

[1]
[http://menehune.opt.wfu.edu/Kokua/SGI/007-2861-005/sgi_html/...](http://menehune.opt.wfu.edu/Kokua/SGI/007-2861-005/sgi_html/ch04.html#id5197689)

~~~
whatusername
LTO5 hit's about 1.8GB per metre. Or about 3.6GB compressed.[1]

There's a reason that people say tape is still cost effective -- and it's not
DAT.

[1] <http://en.wikipedia.org/wiki/Linear_Tape-Open>

------
mv1
I love this comment: "The reason why we put it in is not physical data loss,
but once in a blue moon you will have a bug that destroys all copies of the
online data and your only protection is to have something that is not
connected to the same software system." I think that is often overlooked when
designing HA storage systems.

------
herf
Am a little bit surprised Google hasn't incorporated SSD more.

~~~
arkitaip
Too expensive, too prone to failure
[http://www.codinghorror.com/blog/2011/05/the-hot-crazy-
solid...](http://www.codinghorror.com/blog/2011/05/the-hot-crazy-solid-state-
drive-scale.html)

~~~
wmf
SSDs are actually cheaper per IOPS.

Considering that everything we knew about hard disk reliability was shown to
be wrong (by Google), I wouldn't be surprised if the same holds for SSDs.

~~~
sliverstorm
It's hard to see SSD's being a clear win on the surface. It would be very
interesting though to see if the cost begins to change when you consider power
savings (much smaller than hoped, but they do exist), figure out how much
they'd save on HVAC in the datacenters, etc.

~~~
wmf
Having worked on power-aware hybrid storage for the last two years, the power
savings per GB is pretty much zero; there is significant W/IOPS savings but
that still doesn't pay for the capital cost. Performance is really the reason
to use SSDs.

~~~
sliverstorm
per GB is an excellent point, I had not considered that.

