Removed the reliance on expensive proprietary solutions, but how many non-standard custom tools needed to be written (and thus maintained) to verify, backup and access stuff? All that "extra" POSIX stuff could be avoided with a DOS formatted file system, or one of the many filesystems designed for small embedded storage systems (read and append only). Sites with the scalability and size of Facebook or Flickr ("...Facebook's effort to design a more cost-effective and high-performance storage system for their unique needs") may have a use for this, but the majority of smaller sites would most likely be best served by a direct filesystem store which has known semantics, easy access, and proven history.
I'd really like to see some performance numbers that shows which parts of their old storage system started breaking down at which levels of scale.
At the centre of this redesign is the need to save money.
Facebook doesn't need to run on the same hardware (NetApp) as banks and telcos. In this type of application, scaling up is done cheapest with custom software on commodity hardware.
If you go though the older presentation. It breaks down at the disk I/O level. Not only that, it seems like one of the main reason was cost. CDNs can get expensive, especially when are the biggest Photo service on the internet.
I'd really like to see some performance numbers that shows which parts of their old storage system started breaking down at which levels of scale.