

Facebook's photo storage rewrite - sr3d
http://www.niallkennedy.com/blog/2009/04/facebook-haystack.html

======
thwarted
Removed the reliance on expensive proprietary solutions, but how many non-
standard custom tools needed to be written (and thus maintained) to verify,
backup and access stuff? All that "extra" POSIX stuff could be avoided with a
DOS formatted file system, or one of the many filesystems designed for small
embedded storage systems (read and append only). Sites with the scalability
and size of Facebook or Flickr ("...Facebook's effort to design a more cost-
effective and high-performance storage system for their _unique_ needs") may
have a use for this, but the majority of smaller sites would most likely be
best served by a direct filesystem store which has known semantics, easy
access, and proven history.

I'd really like to see some performance numbers that shows which parts of
their old storage system started breaking down at which levels of scale.

~~~
lyime
If you go though the older presentation. It breaks down at the disk I/O level.
Not only that, it seems like one of the main reason was cost. CDNs can get
expensive, especially when are the biggest Photo service on the internet.

~~~
thwarted
I wasn't really including CDN in "storage system", even though it is listed in
those graphics.

------
wavesplash
I can understand disk I/O limits, but I'm not buying the metadata argument on
a read-mostly workload.

More detail about WAFL (netapp's filesystem) features and design in this
preso: <http://www.cs.tau.ac.il/~ohadrode/slides/WAFL.pdf>

Great for corporate documents. Overkill for photo storage.

~~~
Technophilis
But if it is overkill for photo storage why did facebook use that in the first
place ?

~~~
sokoloff
Because early in a high-growth company's life, speed and ease to market is
vastly more important than IOPS, $/TB, TB/KVA, or many other measures that
matter a lot when you're facebook's current size, and not much at all when
you're facebook's size 5 years ago.

Back in 2004, buying an appliance that plugs right in and works the way you
understand (presenting a Posix filesystem over NFS) means they can get on with
the Facebook-ness of Facebook. 20 hours of research and quoting, 2 hours to
rack, network and power it up and you're in business.

Once you're at 1/2 TB/day (I actually can't believe it's that low unless
they're counting the final storage requirement, not what the users are
initially sending them), the linear factor dominates the constant factor, and
they can now "afford" the luxury of doing a design that's better in $/TB
terms.

------
xelfer
I hope a new image compression algorithm is installed with it, their current
one is horrible.

~~~
jrockway
I don't use Facebook, but I think their goal is to minimize storage costs,
even if it means over-compressing images. If you want lossless copies of your
photos, use Flickr (or a similar service).

