
CockroachDB is more scalable than Amazon Aurora for OLTP - manigandham
https://www.cockroachlabs.com/blog/performance-part-two/
======
carterehsmith
Eh, I find these comparisons of "X on local SSDs" vs "Y on networked storage"
completely meaningless.

Even worse:

"DISCLAIMER: this benchmark script was not validated and certified by the
Transaction Processing Council."

Of course. Try certifying it and see if it still stands.

Look, there is like a billion db vendors trying to push their ware every year,
every month. Many of them fake the benchmarks (like, not writing to the
permenant storage at commit). Which is why we have standards. So let's report
standard TPC-C numbers.

~~~
arjunnarayan
Author here. The point of certification is to ensure transparency. CockroachDB
is open-source. Our entire benchmarking - tools, scripts, are also open
source, and the results are from runs on public cloud providers. You can
replay our entire benchmarking runs. We think this is more transparent than
auditor certification.

Auditor certification is also time consuming and expensive. We'd have to do it
with an on-premises deployment, which means that the auditor certified version
wouldn't be easily reproducible the way our current benchmarks are. If we see
customer demand for auditor certified results, we'd certainly be open to the
idea.

------
thepumpkin1979
I'd love to try Cockroachdb for a reasonably sized service but it's active-
record adapter[0] doesn't work yet. I'm watching the repo to see what progress
is made towards supporting Rails 5.

[0][https://github.com/cockroachdb/activerecord-cockroachdb-
adap...](https://github.com/cockroachdb/activerecord-cockroachdb-
adapter/issues/24)

~~~
maxmcd
You might want to check out this PR:
[https://github.com/cockroachdb/activerecord-cockroachdb-
adap...](https://github.com/cockroachdb/activerecord-cockroachdb-
adapter/pull/20) looks like a lot of work has been put into compatibility but
it hasn't yet been merged

------
threeseed
This is one of the dumbest blog post I've seen in years.

How can you compare two databases in this way ? Each is running on different
clouds with different hardware. Aurora isn't even using Provisioned IOPS (or
is it we can't even tell). And in the Large Cluster scenario they have a
staggering 30x GAE instances. Where is the cost comparison here ?

Anyway this is just SEO/Content Marketing fluff. Shouldn't be on here.

~~~
thepumpkin1979
I could cost more but if CRDB is able to do it without vendor lock-in I don't
think it's just SEO/Content marketing fluff, the blog post invites you to try
it yourself anyway so I'd do that first and then call them bs.

~~~
scarface74
Aurora is "wire compatible" with either MySQL or Postgres depending on how you
create it. There is no "vendor lock-in".

~~~
thepumpkin1979
I can not install Aurora in my servers

------
beefsack
CockroachDB sounds amazing, and all of these blog posts are great to read, but
I'm yet to see someone talk about production experiences in comment threads
when the blog posts are put up.

Has anyone here actually run it in production yet?

~~~
tomsthumb
A previous company I worked at ran it in production. No problems with
cockroach per se, but the code they wrote on top of it (to manage multiple
writers) was a steaming pile. I’m not convinced that, even if the shim layer
was better, it would have been the optimal design decision, but cockroach
itself did pretty well while handling a relatively large amount of data all
things considered.

~~~
karmakaze
The thing I know to watch out for is the minimum latency of requests. CRDB
doesn't work well for workloads which make many DB queries per 'event' (e.g.
N+1 queries which can and should be avoided).

Why was it necessary to 'manage multiple writers' in that use case you
encountered?

~~~
tomsthumb
My somewhat embarassed apologies.

I have confused RocksDB with CockroachDB.

My understanding is that, at the time of development, Rocks itself did not
support multiple writers. This was in 2015, or maybe 2014. At this point it
looks like TransactionsDB might cover that, but I was not super familiar with
that particular project, nor am I super familiar with RocksDB in general. I do
know that they asserted Rocks could not handle multiple writers the way they
wanted, and they needed something which handled SSTs well. Rocks was chosen
and the shim layer was written on that assumption.

~~~
karmakaze
Thanks for clarifying. I'm also interested in RocksDB, in particular how it's
being integrated into the (poorly named) WebScaleSQL MySql variant.

------
wgjordan
So _30_ CockroachDB nodes running on Google Compute Engine instances compared
against previously-published reports of a single Aurora RDS instance only beat
the latter's reported throughput by ~10x.

Aurora already supports horizontally-scaling reads across multiple read-
replicas, and multi-master clusters for horizontally-scaling writes are
currently in preview.

~~~
arjunnarayan
Horizontally scaling TPC-C with sharding is nontrivial, as the TPC-C
specification requires cross-shard transactions. This is the intent of the
TPC-C benchmark: real world use cases seldom shard perfectly. CockroachDB
scales to 30 nodes in this benchmark while hiding all this complexity under
the hood.

As far as beating Aurora, CockroachDB's 3 node deployment has fewer vCPUs (48
vs 64) than an active-passive replicated Aurora RDS instance, and matches it
in performance. CockroachDB also scales horizontally, as we show, from 3 nodes
to 30 nodes, linearly increasing its throughput. I don't believe that 10
shards of Active-Passive pairs of Aurora would scale linearly to 10,000 TPC-C
warehouses, and as far as I'm aware, Amazon has not made that claim.

~~~
wgjordan
Aurora's horizontal scaling doesn't depend on sharding at all- read-scaling
uses read-replicas, and multi-master write-scaling uses a hierarchical
conflict-resolution strategy [1].

On 'beating Aurora', my point was that comparing a 30-node CockroachDB cluster
against a single-node Aurora benchmark and concluding that Cockroach is 'more
scalable' is disingenuous, considering Cockroach 2.0 just reached GA, and
Aurora multi-master has been available in preview since November and will
likely also reach GA in coming months.

Once Aurora multi-master reaches GA, I'll look forward to a more direct
comparison of an Aurora multi-master cluster against a similarly-sized
CockroachDB cluster.

[1]: [https://www.slideshare.net/AmazonWebServices/deep-dive-on-
th...](https://www.slideshare.net/AmazonWebServices/deep-dive-on-the-amazon-
aurora-mysqlcompatible-edition-dat301-reinvent-2017/36)

------
qaq
Problem is all the benchmarks so far are on dataset sizes that are meaningless
for solutions of this type. You could get almost 10X that number on a high end
commodity x86 box with say PostgreSQL. It would be interesting to see
something in 100TB+ size ideally closer to petabyte

~~~
arjunnarayan
TPC-C's intent is to make a tradeoff between two competing goals: benchmarking
total data stored and processing transactions on the stored data. It's one
point in the spectrum of possibilities, but one that has withstood scrutiny
for many decades. I agree that it would be nice to store 100s of terabytes in
a cluster, but one then also has to run transactions over that stored data.

But for perspective, TPC-C over a 100TB dataset would be 1 million warehouses,
and 12 million tpmC. Only one TPC-C benchmark EVER has hit this scale, Oracle,
in 2010, and had a hardware cost of 30 million dollars.

We'd like to get to that scale one day, and benchmark it, just as we have
today. But it's not where the industry is at right now. CockroachDB keeps up
with TPC-C 10,000: this means 800GB, replicated 3 ways across a 30 node
cluster. Aurora RDS doesn't keep up with the transaction rate at a mere 800GB
of data (the 10,000 warehouses number). They claim to store up to 64TB of
data, but what's the point of storing all that data if you can barely run any
transactions on it?

~~~
qaq
Industry for what type of databases? 100+ TB DBs are very common thing in the
RDBMS world. Doesn't have to be TPC-C just some benchmark with reasonable
dataset size. 2010 was 8 years ago in modern world an x86 box can have 8 CPUs
with close to 200 physical cores 6TB RAM and even consumer grade SSDs can
store 800 GB multiple times on a singe drive. Outside of very niche scenario
why would I use distributed database if I am not sure it can handle scale that
plain old RDBMs can handle? The interesting use case is dataset size that a
plain RDBMs would not be able to handle.

------
jacksmith21006
But more scalable than Spanner?

