Because early in a high-growth company's life, speed and ease to market is vastly more important than IOPS, $/TB, TB/KVA, or many other measures that matter a lot when you're facebook's current size, and not much at all when you're facebook's size 5 years ago.
Back in 2004, buying an appliance that plugs right in and works the way you understand (presenting a Posix filesystem over NFS) means they can get on with the Facebook-ness of Facebook. 20 hours of research and quoting, 2 hours to rack, network and power it up and you're in business.
Once you're at 1/2 TB/day (I actually can't believe it's that low unless they're counting the final storage requirement, not what the users are initially sending them), the linear factor dominates the constant factor, and they can now "afford" the luxury of doing a design that's better in $/TB terms.
Back in 2004, buying an appliance that plugs right in and works the way you understand (presenting a Posix filesystem over NFS) means they can get on with the Facebook-ness of Facebook. 20 hours of research and quoting, 2 hours to rack, network and power it up and you're in business.
Once you're at 1/2 TB/day (I actually can't believe it's that low unless they're counting the final storage requirement, not what the users are initially sending them), the linear factor dominates the constant factor, and they can now "afford" the luxury of doing a design that's better in $/TB terms.