
CockroachDB beta-20161013 - aphyr
https://jepsen.io/analyses/cockroachdb-beta-20161013
======
ysleepy
So I assume this means cockroachDB currently (probably) meets its promised
consistency levels. How does it compare to the usual default settings of
PostgreSQL for example? - I think you get SERIALIZABLE, so the behaviour
should be very reasonable.

I understand the txn/s numbers are in a pathological scenario, but will these
transactions block the whole db from moving faster or have unrelated
transactions still normal throughput?

I'm just glad we get linearizable single keys, I had to switch to HBase
because it was the only choice at the time. Cassandra did not even give me
read-your-own-writes. HBase also had issues while testing on the same host,
delete+write lead to deletes being reordered after the writes because they had
the same timestamp.

I'm a bit disappointed from HN in this post, everyone only talks about the
name. But 90% probably did not even read the article. Aaanyway, heres my
2cents: [https://cr.yp.to/cdb.html](https://cr.yp.to/cdb.html) is CDB for me
:P - so better choose another acronym!

~~~
jpgvm
I don't believe CockroachDB provides SERIALIZABLE consistency for the entire
keyspace, rather for subsets of the keyspace that reside in the same Raft
cluster.

In practice this means no cross key serializability without including a
serialization token in transactions.

I may be mistaken however but that is my understanding from reading
CockroachDB docs and Aphyrs post.

~~~
tschottdorf
Cockroach Labs employee here. We provide serializable consistency for the
whole keyspace. It's just that it's easier when you stay in one Raft ensemble
as less can go wrong.

------
BrentOzar
"For instance, on a cluster of five m3.large nodes, an even mixture of
processes performing single-row inserts and selects over a few hundred rows
pushed ~40 inserts and ~20 reads per second (steady-state)."

Wow, that's...not fast.

~~~
sametmax
Even if they manage to multiply this by 100 on the final release, it's still
way weaker than a regular sql db. I hope they have another selling point than
performance.

~~~
flyingramen
There is a better response to this up in this thread
([https://news.ycombinator.com/item?id=13661735](https://news.ycombinator.com/item?id=13661735)).

The test is testing the worst-case scenario of everything needing locking.
CockroachDB uses an optimistic "locking" approach which makes this bad. But if
you're use-case is strictly linearizable high-throughput reads good luck
finding anything that is not a single-machine database.

~~~
sametmax
That's actually the only good answer I received to my comment.

------
exhilaration
_Like Spanner, Cockroach’s correctness depends on the strength of its
clocks... Unlike Spanner, CockroachDB users are likely deploying on commodity
hardware or the cloud, without GPS and atomic clocks for reference._

I'm curious, do any of the cloud providers offer instances with high-precision
clocks? Maybe this is something one of the smaller cloud providers might want
to offer to differentiate themselves - instances with GSP clocks.

~~~
Already__Taken
Why can't these datacenters be using a high precision gps source for their NTP
source in the building?

~~~
tdrd
[Cockroach Labs employee]

In addition to being extremely precise, TrueTime is explicit about error in
its measurements.

In other words, a call to TrueTime returns a lower and upper bound for the
current time, which is important for systems like Spanner or CockroachDB when
operations must be linearized.

In effect, Spanner sleeps on each operation for a duration of
[latest_upper_bound - current_lower_bound] to ensure that operations are
strictly ordered.

Your suggestion might improve typical clock synchronization, but still leaves
defining the critical offset threshold to the operator, and importantly leaves
little choice for software like CockroachDB other than to crash when it
detects that the invariant is violated.

------
vonklaus
Wow. Probably fundraising and getting out front of googles new consumer
spanner service. I just started experimenting w/ cockroachdb and it
sounds/seems great. They def are campaigning hard, landing a Jepsen test and a
lot of posts/articles last couple.

I am pulling for them over G on principle. I infer they know people at Stripe
where the Rethink team & Aphyr are so hopefully they can learn from them and
build an awesome product. G has lost a lot of battles on UX, Docs, attitude
and just general product. They have won a lot of battles on sheer size &
resources. Be nice to have this stick around.

~~~
aphyr
Just as a headsup, I left Stripe in late 2015 to run Jepsen as an independent
consultancy.

~~~
vonklaus
Hey man, didn't realize you left Stripe but did sort of notice Jepsen getting
more serious. Jepsen has phenomenal reach:

Interesting and hilarious coverage of technical complexity which assumes just
enough knowledge to engage a keen novice while _seemingly_ engaging seasoned
experts in the field.

Thanks for that. Your entertainment firm is a bit out if my wheelhouse, but
it's such a crowded space and you end up getting really tied down. Maybe get
acquihired for an Iditarod team, but I reckon this is more scaleable.

Posts are great. Tganks

------
AsyncAwait
I'd be interested in any comparisons of CockroachDB vs TIDB[1], especially
when it comes to speed.

1 - [https://github.com/pingcap/tidb](https://github.com/pingcap/tidb)

~~~
placeybordeaux
Yeah I find it odd that most discussions of CDB don't reference TiDB, but in
this case speed is not really the focal point.

------
ohstopitu
CDB is a really cool db for the time I've been playing with it (just a few
months I think).

That said, I'd really appreciate if they'd launch a hosted DBaaS. By that I
mean, more in line with dynamodb or documentdb instead of say MySQL on AWS for
example.

I know it's on their timeline, but dbs generally get selected during the start
of the projects and unless absolutely necessary, companies don't want to
change dbs.

~~~
dianasaur323
CockroachDB employee here. DBaaS is definitely on our roadmap. There are a lot
of moving pieces to building out DBaaS, so our focus first is on honing our
core product. In the meantime, we are working on making it as easy as possible
to deploy CockroachDB across different cloud vendors and with orchestration
tools like Kubernetes.

------
bluejekyll
> Should the clock offset between two nodes grow too large, transactions will
> no longer be consistent and all bets are off.

A concern I have is that there could be datacorruption in this context, no? It
sounds like on a per cluster basis, i.e. Same Raft cluster, there is a strict
serializability guarantee. But without linearalizability system wide, you
could end up with data corruption on multi-keyspace writes.

~~~
aphyr
You're correct--large clock anomalies (or VM pauses, message delays, etc) can
cause serializability anomalies. I don't exactly know what will happen,
because our tests explicitly focused on staying within clock skew bounds.

To speculate: if clocks do go out of bounds, I don't... think... you can
violate single-key serializability, because that goes through a single Raft
cluster, but you can definitely see multi-key consistency anomalies. I think
you could also get stale single-key reads, because the read path IIRC relies
on a time-based lease. Might be other interesting issues if Cockroach is
shuffling data from one Raft ensemble to another--I think the range info
itself might have a time-based lease that could go stale as well. Perhaps a
Cockroach Labs engineer could comment here?

Note that the time window for these anomalies is limited to a few seconds-
nodes will kill themselves when they detect clock skew.

~~~
bdarnell
(Cockroach Labs CTO) The major issue with clock offsets is stale reads due to
the time-based read lease. Pretty much everything else is independent of the
clocks. Stale reads are still enough to get you in trouble (some of the jepsen
tests will show serializability violations if you disable the clock-offset
suicide and mess with the clocks enough), but it's tricky to actually get bad
data written into the database, as opposed to returning a stale-but-consistent
view of the data.

------
WhitneyLand
Could the name actually slow the project's momentum?

I only recently read some cool things about CDB and wondered if I could have
been subconsciously skipping articles about it for deep seeded reasons like
[http://bbc.com/future/story/20140918-the-reality-about-
roach...](http://bbc.com/future/story/20140918-the-reality-about-roaches)

Of course anything good may eventually rise to success on its merits, but in
case I'm not the only one with a subconscious aversion maybe the info below
will help put some people one less click away. How about SurvivalDB? :)

 _CDB is a distributed SQL database built on a transactional and strongly-
consistent key-value store. Scales horizontally; survives disk, machine, rack,
and datacenter failures with low latency disruption and no intervention;
strongly-consistent ACID transactions; SQL API for querying data. Inspired by
Google’s Spanner and F1. Completely open source._

[https://cockroachlabs.com/docs/frequently-asked-
questions.ht...](https://cockroachlabs.com/docs/frequently-asked-
questions.html)

~~~
finder83
I agree completely. I love reading about databases, but apparently have been
skipping reading about CockroachDB for that same reason subconsciously. I took
a deep dive the other day and was amazed that I've seen the name dozens of
times and have never bothered to ask what was special about it. It really is a
fascinating database.

~~~
zzzcpan
No, you have not been skipping it subconsciously or seeing it more often, than
others, you just feel like you did. People actually have an availability bias
[1] that clouds their judgement about unusual things.

[1]
[https://en.wikipedia.org/wiki/Availability_heuristic](https://en.wikipedia.org/wiki/Availability_heuristic)

~~~
finder83
Interesting, perhaps so. I did have a moment of realization that I had a
negative bias against the name though.

------
dijit
I wish everyone would stop bikeshedding the damn name.

From my perspective I'm glad something that's trying to do this is coming into
existence. a couple years ago I was given the arduous task of ensuring that we
never lost data. One of the requirements was that: At any time, any server can
(and will) fail.. if you acknowledge that you have data then you MUST never
lose it.

You cannot imagine how many database solutions that ruled out. Including
basically _all_ noSQL data stores.

We settled on postgresql, because, despite going into toast tables and having
to implement sharding on top- it was not only the most sane it could be
wrangled into basically doing the right thing in regards to fsyncing and
ensuring data wasn't in VFS.

So, Kudos to these guys, who, in a world of people who don't give a shit about
consistency (except eventual consistency) are actually trying to further it.

~~~
Perignon
It's not bikeshedding. Who the fuck wants to work for a company called
Cockroach Labs? Seriously, sure your tech friends and recruiters might
appreciate it on your resume but anyone outside of tech? Applying for an
apartment, mortgage, car, lol good luck. And also not too sure that anyone
outside the immediate tech/startup nerd scene would appreciate it other.

That aside why the hell do I want to thing of cockroaches day in and day out
when I am using their product? Seriously, is this someone's idea of a bad
joke?

The name has got to go.

~~~
dijit
Nobody is going to give a flying fuck what your company is named on a mortgage
application.

Honestly there are companies with _much_ worse names in active use like "Dick
& Daughters"[0] or "Ismail ANUS"[1] who even got an appointment to the queen.

Seriously, it's bikeshedding, and the point of the DB is that you don't think
about it all day.

I also don't think of women called Cassandra when I use the database of the
same name. Bikeshedding. Seriously.

[0]
[https://beta.companieshouse.gov.uk/company/08610385](https://beta.companieshouse.gov.uk/company/08610385)

[1]
[https://beta.companieshouse.gov.uk/officers/5-sFnEsibrDnUR2-...](https://beta.companieshouse.gov.uk/officers/5-sFnEsibrDnUR2-Y0_IgYEG2I0/appointments)

~~~
floatboth
Just one Dick, not "Dicks", and you posted the same link twice :D

~~~
dijit
oops, thanks for pointing that out.

------
irfansharif
I'm disappointed with the commentary here, I for one would think the
discussions around building a distributed system with the consistency
guarantees claimed here would be far more interesting than the name of said
system.

~~~
tshannon
The name may just be low hanging fruit to comment on if one has nothing else
to add to the conversation.

~~~
Gigablah
I think it's hilarious that people are up in arms about the name but
apparently something like MongoDB is fine.

------
will_pseudonym
CockroachDB might be the worst product name I've come across. I get it;
cockroaches are extremely hardy, but they're also much more strongly
associated with uncleanliness and disease.

EDIT: Also, cockroaches are BUGS! Bugs cause problems in computer systems.

~~~
artursapek
I think it's the most memorable database name I've seen. So at least it has
that going for it.

~~~
atmosx
I'm not a native English speaker and have troubles pronouncing "cockroach". In
my view the naming is rather unfortunate.

~~~
artursapek
Do you find it easier to pronounce "postgres"?

~~~
atmosx
Yes but I'm not sure the comparison stands. PSQL was created in 1996 and it
happens to be one of the 2-3 most common/famous databases ever made as far as
I can tell.

If you're postgres, it doesn't really _matter_ because you're already there.

------
sroussey
How does this compare to gundb.io?

~~~
010a
Gundb is garbage.

~~~
aphyr
To be a tad more concrete: GunDB can run in the browser as well as on servers;
CockroachDB is a traditional, client/server database. GunDB is aiming for
eventual consistency, but when I looked a year ago, it seemed like the
algorithm they chose wasn't actually convergent--nodes could diverge because
non-commutative updates could be applied in different orders on different
nodes. CockroachDB, by contrast, is a serializable, single-key linearizable
store. I don't think GunDB offers transparent sharding or SQL, both of which
are important aspects of CockroachDB.

Both depend on wall clocks.

~~~
marknadal
Aphyr, thanks! I appreciate seeing you here and replying. GUN Author here.

First, to answer @sroussey - Kyle is correct, GUN is an eventually consistent
graph database that runs browser/server with extremely high availability,
while CockroachDB is trying to be a strongly consistent, linearizable
key/value SQL store. To read about what we guarantee/don't, check out these
articles:

[https://github.com/amark/gun/wiki/CAP-
Theorem](https://github.com/amark/gun/wiki/CAP-Theorem)

[http://gun.js.org/distributed/matters.html](http://gun.js.org/distributed/matters.html)
(tech talk on how the algorithms work)

[https://github.com/amark/gun/wiki/Conflict-Resolution-
with-G...](https://github.com/amark/gun/wiki/Conflict-Resolution-with-Guns)

The last year we've focused on performance, and can now do 25M+ reads/sec (no
cache miss, disk I/O performance isn't particularly interesting to us). More
on that here: [https://github.com/amark/gun/wiki/100000-ops-sec-in-
IE6-on-2...](https://github.com/amark/gun/wiki/100000-ops-sec-in-IE6-on-2GB-
Atom-CPU) (we're not kidding about it being able to run in the browser)

With regards to commutative / non-commutative, @Aphyr, slight correction:
Different machines will converge to the same value, it is just that you'd need
to use a commutative CRDT on top of gun (we have one here:
[https://github.com/amark/gun/wiki/snippets-(v0.3.x)#counter](https://github.com/amark/gun/wiki/snippets-\(v0.3.x\)#counter)
) for them to converge to their combined value.

Example: GUN treats primitives as atomic, so if you try to
`gun.get('some').path('math').put(5 + 2)` it will converge to the atomic value
of 7. So if two machines do `gun.get('some').path('math').put(currentValue +
2)` at the same time, both machines will converge to the fixed atomic value,
not a commutative value - you need the CRDT for that.

I mention this in the "What could go wrong?" section of the talk (linked
above), and it is easy to add the necessary CRDT when you need it - actually,
this was inspired by you @aphyr, from our discussion a few years ago.

Note: @Aphyr, we built a distributed testing framework (
[https://github.com/gundb/panic-server](https://github.com/gundb/panic-server)
), and now that our "performance improvement" stage is done, in the near
future we're going to start building distributed correctness tests (we're
rolling out to a government client soon). We'll probably be in touch with you
in the next half year or so. :)

Great work, keep it up. You are always a huge inspiration. <3

Edit: With regards to sharding / SQL, we don't have SQL support yet but
somebody is currently building a prototype for it. Sharding, not built in, but
peers store the data they request, which acts as a natural shard, but nothing
fancy - for more info check out this article:
[https://github.com/amark/gun/wiki/sharding](https://github.com/amark/gun/wiki/sharding)
.

------
throwawasiudy
May I suggest renaming the thing already? 90% of the business world can't
touch it with a name like that.

How about Duradero ... Spanish for durable. The name doesn't seem to be taken
by any other database type software

~~~
dijit
I'm going to be the dissenting voice here and say: Use whatever name you like
as long as it isn't something like cuntbagDB.

CockroachDB accurately and immediately conveys meaning. It'll survive when
nuclear war happens just like a cockroach. And I might work in the games
industry but I would never have a hard time selling this. I would basically
force them to use it and explain the benefits.

If your company wont adopt technology because it uses a bug as it's name then
your company has much greater problems than technology adoption.

~~~
throwawasiudy
It's not my company or me, it's my clients. Here's a sort of long winded story
explaining what I mean that the corporate world can't use anything with the
name "cockroach". If you disagree after reading, I'm open for a friendly
debate.

Average project, some 60 year old VP signing off on 100k of custom software
with 8 pages of lawyer speak contracts is NOT going to allow "his" project to
run off of "cockroach database". VP's/CEO's like to take personal ownership of
projects they feel will go well, and ones done with us usually do. None of
these people want ANY chance of their project getting criticized internally,
so nobody will use CockroachDb. It's just not going to happen. Having
successful projects is an easy way for these people to gather promotions and
respect. Having any part of it do with "cockroach" could result in the whole
project being an embarrassment.

Imagine the database goes down one day. One of their low level staff calls
us(we have ongoing support contract). Problem gets rapidly escalated to
development. One of our lead developers replies to the email thread, that by
now has a third of the clients' company on it. "Yeah see the problem was the
cockroach database that wasn't supposed to go down needed to be restarted" .
Most of the staff think we're joking. Guarantee this becomes a big pun around
their office, every time anything breaks in the app they say "maybe the
cockroaches on the back end died again".

It's made worse by the fact that issues in software systems are commonly
called "bugs". Well, our system doesn't have general bugs anymore, it has
"cockroaches". Do you think this is an unlikely scenario? Because it's not.
Besides the word "cockroach" this is exactly how urgent issues tend to be
reported and fixed. Replace "cockroach" with "Microsoft" and I've probably
violated my NDA. After that incident, whoever championed the project, probably
our strongest ally in the company, resents us for making such a stupid mistake
that resulted in his project becoming a joke. Maybe we aren't as great of a
dev shop as he once thought? Maybe it's better to do open bidding next time so
whoever wins he can't be blamed directly for what happens.

You could say their company culture sucks, maybe it does? But that doesn't
change that I can't use it on any client projects because of the chance we'll
lose hundreds of thousands over a stupid name.

There's plenty of other names that convey meaning but don't make lawyers
nervous. Your attitude of "oh, well if you can't use it your company culture
sucks" is EXACTLY why nobody will rename it and why corporate american won't
touch it. Nobody cares about how cool it is to be edgy when millions of
dollars are on the line.

If you plan on forcing clients to use it you might as well hold a going away
party for any of your clients with more than 5M revenue.

~~~
dijit
If your VP isn't trusting his technical directors to make appropriate choices
when producing quality reliable software and infrastructure then I would say
that is a problem with company culture.

I wouldn't have a problem selling the technical merits of a highly consistent
database to a technical director. So everything else is moot to me.

I agree about the "bug" association, that is a negative stigma that I could
see attached to it- but there are plenty of projects who have stupid names
which pass by unnoticed.

take distro release names; Beefy Miracle.. Wheezy.. Natty Narwahl.

And these terms are used internally too. "Oh, that machine was running Squeeze
and not Jessie which is why we can't install the latest gdb" etc;

You're giving this much more credence than it will have in real life.

~~~
throwawasiudy
Fair enough, we can disagree cordially on this one. I agree that it shouldn't
matter but we disagree that it does.

The other names you mentioned have a more silly connotation then bad or gross
but I understand your point.

I will still serve as an example of somebody that will avoid cockroach db due
to name alone so there's at least one of us out there

~~~
dijit
Perhaps we can fork it, rename it and offer very expensive business class
support so your VP can feel good about using "IronTardigradeDB" :)

~~~
throwawasiudy
I had the same issue with another tool I used called "agent ransack". The
solution is simple, offer the same product with a different name to corporate
clients. They have a different download called "file locator pro" that's
literally exactly the same, without the bad name.

Guess which one we use in the office?

