I appreciate the effort and intention behind this article, but for all practical purposes, such numbers are really not helpful. From experience, if you really want performance from HBase, you need to spend significant amount of time coming up with the right way to structure your data in HBase. To name a few:
* choosing the right row key that optimizes bulk scans
* setting optimum client and server caching based on the size of each row
* pre-splitting regions, and setting custom region sizes
You will also run into various cluster-related issues. Things don't really scale linearly as you add more nodes. You need to also consider maintenance, upgrades, backups, replication and so on.
If you want to choose a NoSQL (or for that matter any database), spend some time thinking about whether it fits your data model and your own understanding of the technology. Performance is rarely gained by simply switching a few knobs.
It's frustrating, because a configuration like that makes the numbers completely irrelevant, unless you don't care whether your data gets persisted or not. I'd much rather see a more like-for-like comparison, and perhaps an unsafe configuration included for interest's sake.
You batch your changes in the client and then you commit them to the server. When that commit succeeds your data is persisted; until such time any client application must consider the data uncommitted. This is not unlike a client application would deal with transactions to a relational database.
So the benchmark is completely valid here.
(Well, only that here we also have deferred log flush, which means the changes will take a second or two to actually make it to the file system, but that still demands the same aggregate throughput from the HBase. That is where this test is not fair).
That was actually the bit that I was concerned about - I failed to read the parent comment properly and assumed it had the exact same concern I did. If you don't have to do a disk write because you're not guaranteeing durability, you have a completely unfair comparison.
cleanup calls flushCommits. So all requests are there. No get requests are served from the client's buffers so all of those must hit the region server. In addition the client side write buffer size is 12mb. So we're not talking about all requests being buffered at a time.
This may or may not be what is needed for your application so autoFlush is a per-client setting (HBase actually defaults it to on). Deferred log flushing is similar, though it's a per table setting.
The HBase default is correct for what YCSB is trying to measure ("live" updates, i.e., the update should be readable from the regionserver afterwards), but someone submitted a YCSB patch to override the default to make latency look better time some time ago, and most benchmarks including this one have inherited the mistake.
So I created a pull request for YCSB so that last little bit of the buffer is counted for HBase.
Also, rather than using M/M/1 or some reasonable analytic model, they deliberately trottled their request rates to hold throughput constant (thereby guaranteeing different loadings for different 'benchmarks')
Just reading the first graph, for example, and applying Little's Law, it's pretty evident that Cassandra was loaded more heavily than than the two MySQL systems, with HBASE and Riak trailing.
Looks like HBASE and Cassandra lead the pack to me, with different characterists for different purposes.
Advice to authors: buy a book by Neil Gunther.
Some people were skeptical of the tuning for HBase, and an HBase developer criticized their methodology, given the especially poor performance shown for HBase. The paper authors themselves indicated that they had considerable trouble setting up the HBase system. Which in a way validates the comments of some here that HBase is fickle and can be difficult to tune, though it still means that the results in this paper for HBase are not valid, when compared to the other systems.
In the interest of disclosure, it should be mentioned here that jbellis is a principle corporate backer of Cassandra.
"MyISAM caches index blocks but not data blocks."
InnoDB is faster at reads for most workloads on modern hardware. MyISAM has horrible scalability across multiple CPUs. MariaDB recently improved in MyISAM with the segmented key cache, but I haven't seen any results of head-to-head benchmarks since then.
Also, unless you're running all of the tests concurrently, we really need to see your IOPs records during the tests, since a single lagging EBS volume in a Raid-0 array will still negatively affect disk performance (either that or spring for dedicated iops), and thus skew your benchmarks.
Making sure you're not getting a spike CPU steal time would be great as well; same reason.
What if JVM instance crash under load? Data lose, but, see, it is not our fault - our code is OK.
And they are all just as easy to manage and backup as everything else. They all have their problems and issues.
Disclosure: I work for Basho, makers of Riak.
Plus, I think it would have scored very well here.