
Tikv: Distributed key value store powered by Rust - adamnemecek
https://github.com/pingcap/tikv
======
xuejie
I know the authors of this project also read Hacker News, so it probably is
better if we wait till them to state their motives. But I suppose I could give
a brief first, since most of the confusing comments are due to language
barriers(sorry if you don't know enough Chinese, you might not have read
enough about this project).

This is actually affiliated to
[https://github.com/pingcap/tidb](https://github.com/pingcap/tidb), which is
another implementation of Google Spanner/F1, similar to CockroachDB. So think
TiKV as spanner, and TiDB as F1.

They actually started out using Go. And the reason they are changing to Rust
is not trying to be a Rust cool kid, but due to real constraints: Go's GC
introduces too much of latency time (and no, Go 1.6 is not enough despite
being better), they are also not satisfied with cgo's costs. That's why they
are changing to Rust. Note that according to their words, they still love Go
better, and TiDB is also happily using Go. But for TiKV's case, Rust is
perfect.

Disclaimer: I'm not working on TiKV or TiDB, I'm just a fan of their work(they
are also the founders behind Codis:
[https://github.com/CodisLabs/codis](https://github.com/CodisLabs/codis), tho
not working on that anymore). I'm happy to list references on where I got all
the information in case anyone are interested but they are all in Chinese,
which I think might not be that polite since not everyone here reads Chinese.

~~~
spullara
This looks like it could be a drop in replacement for FoundationDB but I don't
see enough documentation on the guarantees they give or limitations. Any more
information? Maybe even translated from Chinese?

~~~
xuejie
Unfortunately I don't see much material written in English at the moment.

To your technical question, I'm not that familiar with FoundationDB to
comment, but I can tell the author's intent is building an open source
solution for Spanner/F1. So I'd say TiKV/TiDB is really a drop-in replacement
for Spanner/F1, not FoundationDB.

So in that case, I think the guarantees they want to meet include at least
distributed transactions on a KV store(Spanner), and zero downtime SQL schema
changes(F1).

~~~
spullara
FoundationDB was a drop-in replacement for spanner as well. Can't wait to see
more docs.

------
arthursilva
I think everyone could benefit from extracting that raft implementation to a
crate. I'll see if I can get started on that.

~~~
viperscape
I think hoverbear has a raft crate: [https://github.com/Hoverbear/raft-
rs](https://github.com/Hoverbear/raft-rs)

~~~
bluejekyll
Yes, I was just wondering why they didn't use that? I reviewed that raft
implementation. Looks decent and uses mio for non-blocking io, which was quite
clean (I'm jealous).

~~~
justinsb
Most raft implementation don't implement dynamic reconfiguration or
snapshots/compaction. AFAICT hoverbear implements neither, and Tikv implements
at least snapshots.

~~~
bluejekyll
You're right. And thanks for the pointer on this implementing snapshots.

------
biokoda
Distributed KV store based on rocksdb. Isn't that just reinventing the wheel?
Is the fact that it is written in Rust that important?

~~~
dikaiosune
It looks like a parallel implementation of TiDB, which they had built in Go. I
would assume that avoiding garbage collection and reducing heap usage were
valid motivations for the project. But it doesn't really have enough
information on the repo to tell what their motivations are.

~~~
bpicolo
Not quite. TiDB can use it as a data layer, as an alternative to Rocks, Bolt,
etc. Different part of the system

~~~
dv_dt
So when the Readme says "Distributed KV Database", I'm imagining that it
refers maybe to being able to be used by TiDB which provides the distributed
logic, while TiKV concentrates on a compatible interface to TiDB and local
non-distributed storage?

~~~
bpicolo
Right. It should be roughly equivalent to InnoDB/MyISAM alternatives in Mysql
(I believe)?

------
skyde
I could not find a description of the algorithm used for distributed
transaction. I can only assume it's similar to CockroachDB but looking at the
code I see a lot of reference to "LOCK" in the protocol buffer definition
which mean they might be using something else.

Anybody know where I can find more documentation ?

~~~
hmottestad
Says they are using Raft: [https://raft.github.io/](https://raft.github.io/)

~~~
rystsov
Raft is a replication protocol which is responsible for the consistency of
multiple copies of the same data but it isn't applicable when the data is
sharded.

~~~
mtanski
You can use raft as to synchronizing other data structures. If this a KV store
then that's as easy as key level locks / ownership / placement.

------
llogiq
From the README:

> Thanks rust-clippy. We do love the great project.

This just made my day. :-)

------
alexnewman
I appreciate the sentiment, but this project is very difficult to understand
what's connected to what. That's not a good sign for a database

~~~
dikaiosune
The first PR was merged ~3 months ago, I'm not surprised that they don't have
more info in the README.

------
brokenmachine
Can someone ELI5 what a distributed key value store is useful for? Is it like
a DHT for torrents?

------
endymi0n
With "inspired by" meaning the very same thing as in the opening credits of a
Hollywood movie...

------
andrewvijay
I'm pretty much sure this software wouldn't get a lot of attention if it was
not built in rust. Darn I need to build something in rust and get some
attention!

~~~
dang
Sure, trends come in trends—in programming as in other things. But please
don't post comments that put down other people's work.

If you think other stories should get attention on HN, please submit some.

~~~
andrewvijay
It's a joke man. Why would I put down someone I don't even know!

~~~
dang
> _Why would I put down someone I don 't even know_

I share your question, yet it seems to be what most internet forums are for.

