This article is incoherent and incomprehensible. It seems to be a set of responses to things an unspecified person (possibly Stonebraker) may or may not have said at some time in the past (but mostly not in the Gigaom article it links to). But without the context of knowing what those things are, it doesn't make any sense.
oh well, maybe indeed I assumed more about general knowledge of new database technologies around.
voltdb (Stonebraker's solution) is in-memory store, so are few other of all this new wave of solutions (e.g. you don't want active mongo dataset to fall out of memory, as well as hit any contention on r/w lock).
I didn't want to attack statistics or numbers in any way, as to have proper picture it is not enough to know number of servers or shards, one has to know also workload that has to happen there, as well as access distributions, etc.
I could point out that average write transaction at FB is at around 5ms timing, so argument about transactional cost is not that important, as RPC times add up to that quite a bit, and multi-cluster dbms wouldn't reduce the RPC costs.
Pretty much everything he wrote about state of FB environment is very uneducated narrative.
Oh well, I guess general audience prefers something that has no basis rather to insights that are based on working today in that industry.
I think the author is missing the many orders of magnitude of the recent growth in computing power and corresponding drop in flash costs. Compilers and interpreters are often wasteful too, but the benefits were so great that we eventually embraced them and never looked back. As far as persistence goes, who is to say that flash memory in modern systems isn't just as permanent in the real world as their disk-based counterparts? There are many ways to lose data, and a corrupted database is just as corrupted on a disk as it is in flash.
I'd also add that it's bad form to ridicule the academics on whose shoulders your own career rests. Better to stick to the argument at hand rather than take things personally.
His "me" page explains that he's a database engineer at Facebook, and before that worked on performance-oriented systems and software at Wikipedia. It's part of his job to be aware of the drop in flash costs and employ it where he can while balancing performance with operating costs and expenses. No one is arguing that flash memory isn't as permanent as disks, but it's certainly not as cheap, especially on the scale of data that Facebook has.
Every year or so Stonebreaker feels compelled to come down from his ivory tower to troll industry. Remember a few years ago when he co-authored the paper "MapReduce: A major step backwards" (http://databasecolumn.vertica.com/database-innovation/mapred...)? Nevermind that Google probably processes petabytes of data using MapReduce every day that ends up getting served on practically every Google property. How can he argue with results? Can't we just ignore him?
I don't doubt the author's credentials or expertise. My criticisms of his article stand; precisely because he's stuck in the day-to-day technical minutiae of running a system as large as Facebook's, I think he's missing the forest for the trees. And his jabs at academia just come off as being thin-skinned.
Unfortunately, there are many facets of this discussion that stumble accross each other.
1) Stonebraker is an industry troll that likes to shamelessly advertise his current products while pointing 'problems' with the state of the art. And he usually emits clueless opinions about subjects he doesn't deal with.
2) In spite of Stonebraker, voltdb engineers are doing a serious job, and the product has its value;
3) It's virtually impossible (and high risky) to change the infrastructure of any company the size of Facebook/Google. As any software engineer research shows, it'll take years and lots of money to change a large code base and deploy other data management system. Of course, Facebook is using HBase now for new products, but the current MySQL based systems should account for millions of lines of code. It's worth to spend time developing new products or refactoring old code that is working?
4) Thousands of MySQL shards is a hell on earth, for sure. We should praise Facebook engineers for being able to manage it efficiently.
