I guess will have to wait. The progress they made is obviously impressive but would really help if one could understand the overhead vs conventional RDBMS 5X might be OK
20X not so much.
The minimum latency will still be at least ~5ms (worse if nodes are very far apart) and especially bad for contended reads (because of the way they use clocks for serializable operations).
A traditional RDBMS does not have to worry about split brain decisions, but it can hardly do multi-master in the intuitive way.
I think you can still drive some insights. I clicked on the TPC-C results you shared and read their executive summaries.
The Oracle on SPARC cluster (at the top, 2010) performs 30.2M qualified tx/min vs the 16K tx/min in this blog post. The Oracle cluster also costs $30M, which is clearly higher than the Cockroach cluster's cost.
That said, the TPC-C benchmark is new to me. Happy to update this comment if I'm misreading the numbers.
A short note that the total cost of that SPARC cluster was $30 million. You're not misreading those numbers, but it requires a little context.
We're focusing today on our improvements over CockroachDB 1.1, using a small-ish cluster. We'll be showing some more scalability with larger clusters in the coming weeks. If you've found CockroachDB performance slow in the past, you will be pleasantly surprised with this release!
Sure thing. I was primarily answering the question above - in terms of how the numbers in the TPC-C benchmark fit in. I updated my comment to reflect the cost.
I think what's interesting with TPC-C is that you can sort the results based on performance or price/performance. On the price/performance metric, SPARC looks expensive. Dell has a $20K SQL Anywhere cluster that can do 113K tx/min.
I wonder if anyone tried to run these benchmarks on the cloud and how one would calculate total cost of ownership there now.
Yeah, but 1700 cores worth. That's still a lot of $300 boxes. Like qty 53 Sparc T3-2's for example. Which seem to be $1200 to $2k on eBay. And unsupported, end of life, etc.
I'd compare CockroachDB's number to some more recent result with a similar number of cores. (If you can find one)
My guess is that the benchmark setup would cost about 1m dollars to install (3 racks of commodity servers). The software is free. Naturally, Oracle aren't pushing this, when they charge 10s of millions for Oracle rack :)
TPC-C is much, much harder to do well on than that benchmark, even if you are using special hardware and even if you are using "tricks" like intentionally misreading the TPC-C invariants in order to avoid coordination. It's not directly comparable to what you posted; while TPC-C performs joins, reads, insertions, and updates, including ones that necessarily must cross partition boundaries some percentage of the time, and specifies invariants that must be maintained in order to qualify as a correct run, the benchmark you posted (flexAsynch) does only reads or only writes in the tested configuration--both options can be trivially executed without any coordination at all on as many nodes as you feel like. As such, it's more of a stress test of the overhead of the implementation than it is an indication of the actual performance you will get out of a normal workload on such a cluster.
And while I'd like to say that MySQL Cluster is nonetheless exhibiting very impressive performance, I can't really say that; they are using expensive networking and hardware that most people don't have available, microoptimizing both the client and server sides, and using a low-level API specifically designed for doing well in these sorts of benchmarks, but they still lag far behind the per-core performance of state of the art key / value stores in similar circumstances. For example, MICA can do 80 million key/value requests per second of a comparable size to the ones they listed on a single commodity server, with only regular NiCs and 10GiB ethernet (and in fact can saturate the Ethernet link). Granted, MySQL Cluster is a full-fledged database and MICA just does key/value, but I can pretty much guarantee you that on real requests MySQL Cluster's performance collapses, and in multi-master mode it's known to be inconsistent anyway.
If you really need hundreds of millions of k/v requests per second, you'll pay a lot less buying three servers and hiring all the researchers who wrote the MICA paper to manage your key / value store than you will buying MySQL Cluster :P Or, if you want a real database, you can play the same game with the many, many database systems that do much better than Oracle's cluster on TPC-C; the same person who wrote the MICA paper released one last year about Cicada, which can handle over 2 million serializable TPC-C transactions per second on a single 28-core server. Or you can try Calvin, which can do around 500,000 georeplicated TPC-C transactions per second on a commodity EC2 100-node cluster (2.5 compute units each back in 2014) and can operate on top of any existing key-value store. The database world has advanced a lot in the past ten years, and people who really need performance have no shortfall of options that aren't Oracle.
Great, thanks for the in-depth reply. Just saw it now. I agree that TPC-C is much harder than flexasynch, which is just a KV benchmark.
Here's a different benchmark. NDB got around 8m ops/sec on 8 datanodes when performing 1.2m HDFS ops/sec in https://www.usenix.org/conference/fast17/technical-sessions/... .
That is not a KV workload. It is, in fact, a very mixed workload with every transaction being a cross-partition transaction, lots of batched PK ops, lots of partition-pruned index scans. Transactions containing lots of ops. Full-table scans. Index scans. And get this, it's open-source. Research has, of course, advanced since NDB, but nothing open-source still comes close to it for some (sometimes complex) OLTP workloads - anything subscriber oriented. Not TPC-C workloads, they suck on NDB.
I'd never heard of MICA, will read up on it. Calvin, though, is extremely limited. They tried to build CalvinFS on top of Calvin, but it didn't work. Subtree ops didn't scale (at all), as the DB wasn't flexible enough. Features and maturity sometimes matter, and NDB has both.
Looking at the TPC-C page all the benchmarks seem quite old and only reflect commercial databases. Do you have any recent TPC-C benchmarks for OLTP databases such as Postgres, MySQL, and Cassandra so I can compare with CockroachDB?
Open source DBMS normally provide numbers using some open
source variant of TPC-C. One such implementation is DBT2.
TPC-C according to the standard is extremely expensive to
setup and is only interesting for databases with massive
disks. However most open source DBMS runs DBT2 with no wait
time, this means that also an in-memory DBMS can report
numbers on a TPC-C-like benchmark (e.g. DBT2).
MySQL Cluster (NDB) did such benchmarks a few years ago
where we ran 2.2M transactions per minute using DBT2.
This meant executing millions of SQL queries per second
towards the cluster and most of the CPU load is in the
set of MySQL servers.
Currently I am using the load phase of this benchmark to
test loading massive amounts of data into the new 7.6
version of MySQL Cluster using machines with 1 TB of memory.
