
Tokyo Cabinet DB: Beyond Key-Value Store - igrigorik
http://www.igvita.com/2009/02/13/tokyo-cabinet-beyond-key-value-store/
======
amix
I have built a distributed key-value store on top of Tokyo Tyrant (which is a
network interface on top of Tokyo Cabinet). I have also tried to use
memcachedb (that's based on Berkley DB) and Tokyo Tyrant outperforms it -
especially with large amounts of items. memcachedb has also problems with log
(it grows furiously large and db checkpoints stalls the database for several
seconds).

One can also script Tyrant via Lua (which is pretty amazing) and there's
master-master replication. Here's how to implement incr via Lua:
<http://paste.pocoo.org/show/103848/> I have currently extended it with a
fixed list and Mikio (the dev of Tokyo *) has implemented an inverted index
for search.

Currently I have my key-value store in production (it has around 1 million
key-values) and the performance is pretty good.

I will release it open-source, it's only Python now, but the code is around
200 lines + a consistent hash library implementation ( which I implemented in
<50 lines of code <http://amix.dk/blog/viewEntry/19367> ) so it should be
possible to port it over.

~~~
babo
Is there something like CAS of memcached? It's fine to live without locking
but low-level atomic functionality is essential.

~~~
amix
There's global locking and key locking. I currently use key-locking when
updating.

You should be able to implement CAS in Lua I guess, but it isn't supported
from the start.

~~~
babo
Key-locking makes sense, I'll give it a try. Thanks for the info!

------
gcv
Sounds impressive and cool. I wonder about Table engine performance though.

My experiences with Berkeley DB (using the native C API) were positive, from a
performance point of view, largely _because_ BDB has no built-in query
support. When I wanted to look something up, and I had no index on the desired
value (BDB calls indices "secondary databases"), I always knew I was doing a
linear table scan, because I had to write the loop myself. No SQL meant that
inefficient queries could not hide in plain sight.

Berkeley DB trades off the flexibility of SQL queries for (1) excellent lookup
performance in simpler cases, and (2) the ability to mix relational and
object-based data storage, reducing the famous impedance mismatch of
relational data modeling. For some applications, this trade-off works
beautifully. I daresay that it holds true for most web apps which really don't
need a relational data store, and either outgrew pure in-memory storage or
just sensibly want to persist their state to disk. (Hint: if you use an ORM,
then you probably don't want a relational data store.)

For some other apps, the lack of SQL queries hurts. I once worked on a trading
system implemented on top of a proprietary engine which did not support
arbitrary queries. It made writing even trivial reports ("list all
transactions done two days ago by trader X using instrument Y") unnecessarily
painful. (Of course, on a typical RDB with a non-trivial schema and a large
dataset, these arbitrary queries would take 20 minutes to run, so catch-22.)

The Table engine for Tokyo Cabinet seems to do everything implicitly. Fast and
arbitrary queries. If it works on the kind of schema-less free-form data the
author refers to, and has working master-master replication, that sounds
almost too good to be true.

~~~
igrigorik
Some performance benchmarks: <http://gist.github.com/64391>

They're not conclusive, but give you an idea of relative performance.

------
amix
For Tokyo Tyrant there's a Python library:
<http://code.google.com/p/pytyrant/>

It uses the binary protocol and it's written in a very simple and very clean
way - and it should be production ready.

I would not recommend using the memcached protocol with Tyrant (it offers less
operations and it isn't binary).

~~~
chadr
I'm currently using Tokyo Tyrant with Rails and the Memcached client (in
production). It works great for my needs though I don't need to story binary
data. Overall, TT has been extremely fast and stable. The dual master
replication is a nice bonus.

------
nkurz
The article is mostly about the Ruby bindings, but Tokyo Cabinet is written in
C99 and has bindings for most of the usual suspects. English language
specifications are here: <http://tokyocabinet.sourceforge.net/spex-en.html>

~~~
bk
Just as an aside, using Ruby to access the db server via Net::HTTP should
limit the performance to at max a few dozen http requests per second per
(Ruby) process.

I think the more http is used to bridge web architectures, the more critical
this aspect will become for Ruby.

~~~
bjclark
It's safe to say there's some reasonable replacements already, and there are
some smart guys working on better ones.

[http://www.pauldix.net/2009/01/ruby-http-client-library-
perf...](http://www.pauldix.net/2009/01/ruby-http-client-library-
performance.html)

But, if you didn't know better, yes, Net:HTTP is going to negate the amazing
speed of Tokyo *.

------
esessoms
Haven't used it, but Erlang bindings:
[http://dukesoferl.blogspot.com/2008/06/tokyocabinet-and-
mnes...](http://dukesoferl.blogspot.com/2008/06/tokyocabinet-and-mnesia.html)

~~~
moonpolysoft
Those erlang bindings are very immature. I wouldn't use them in production.

~~~
esessoms
Good to know, thanks.

------
jrockway
So, this is a BDB clone without the years of testing of BDB? That's exactly
what I want for my critical data.

I am really confused as to why one would implement this in C. One pointer math
problem, and _boom_ , gigs of data are corrupted. If you are going to
reimplement BDB, you might as well do it in Haskell or OCaml and buy yourself
a little type safety. (Incidentally, this is on my TODO list, but BDB works
well enough for us at this point.)

~~~
staunch
The table DB type and Tokyo Tyrant for network access seems like a pretty big
deal to me. The full text search seems pretty damn awesome too. With regular
backups it doesn't seem that risky for certain things.

~~~
jrockway
This is really easy to add on top of BDB, though, that's why I ask.

(As an example, we implemented KiokuDB on top of BDB:
<http://www.iinteractive.com/kiokudb/>. We haven't added full-text search yet,
but we plan to do it soon.)

------
laktek
There is also a full-text search system for Tokyo Cabinet called Tokyo
Dystopia (<http://tokyocabinet.sourceforge.net/dystopiadoc/>)..Has anyone
written Ruby API for this?

Was wondering whether it be a superior option for Rails search ahead of Xapian
and Sphinx.

~~~
snowstorm
rufus-tokyo has the ruby interface: [http://github.com/cheapRoc/rufus-
tokyo/blob/7e524947825b6f7f...](http://github.com/cheapRoc/rufus-
tokyo/blob/7e524947825b6f7f4ad8801325f970a655c8439d/lib/rufus/tokyo/dys_words.rb)

------
leej
any PHP API??

