

Benchmarking BDB, CDB and Tokyo Cabinet on large datasets - anuraggoel
http://www.dmo.ca/blog/benchmarking-hash-databases-on-large-data/

======
vicaya
Sorry, but 500MB DB size is a tiny dataset these days (anything < 1GB is tiny,
< 4GB is small, < RAM on a single node (~8GB-64GB) is medium, < Disks on a
single node (~128GB to a few TB) is large, huge dataset requires multiple
nodes and typically above 128TBs.)

~~~
psadauskas
I agree, plus TC has a ton of parameters that can be tweaked, and the defaults
are pretty small. The one that has the most pronouced effect is the bucket
size, or the "width" of the hash table. The bigger, the less chance of
collisions, which means you have to follow a linked-list to find the exact
record. He used 11M keys, so a bnum in the range of 40M would be much quicker.

I benchmarked TC b+tree on a 1TB db with ~350M keys, and it worked great. I
would publish the numbers, but I'm embarrassed that they aren't very rigorous.

cdb docs say it has a limit of 4GB, which makes it pretty much worthless for
anything I would use it for.

~~~
rcoder
Actually, the core CDB data structure only limits keys and values to 4GB
_each_ , not in total:

<http://www.unixuser.org/~euske/doc/cdbinternals/index.html>

The hash algorithm used also only produces a 32-bit key, meaning you'll be
limited to 2^32 total records. Again, though, unless your data is of trivial
size, that gives you considerably more room to work with than a hard 4GB
limit.

 _Edit: doh! can't use double-asterisk for exponent on HN_

~~~
epi0Bauqu
Then why does the Perl implementation
(<http://search.cpan.org/~msergeant/CDB_File-0.96/CDB_File.pm>) warn of the
error:

CDB database too large -- You attempted to create a cdb file larger than 4
gigabytes.

?

~~~
asb
I believe rcoder is saying the 4GB database limit is a property of the
implementation rather than the cdb format.

------
asb
It would be nice if DJB would release CDB under the public domain like he has
now done with most of his other software. However, there's a great public
domain implementation by Michael Tokarev I'd recommend:
<http://www.corpit.ru/mjt/tinycdb.html>

~~~
zandorg
A caveat to TinyCDB in Windows:

Don't forget (In Windows) to specify O_BINARY eg:

    
    
      O_RDONLY |O_BINARY
    

Also the case in Berkeley DB 1.86.

Otherwise it opens a file in ASCII mode, when Windows writes \r\n when it
reads in \n... , and dies after just a few records.

~~~
asb
I have a Lua binding to tinycdb. I hadn't considered Windows compatibility and
this bug exists in my code. I've just committed a fix, thanks.

------
smanek
In my experience, the downside of BDB is simply the license. There is no way
for me to use it in my program (and distribute my program) without opening up
my source. I can't simply 'query' a BDB server - by using BDB at all I have to
link to it and also open source my own code as part of the Sleepycat License
(or so I've been told - IANAL).

~~~
zandorg
DB up to and including 1.86 are Berkeley license (basically 'do anything with
this but include this notice' but without open source requirements), but DB up
to and including 1.86 only allow < 2GB databases. :-(

I tried to upgrade it with 64-bit integers, but it's such convoluted C code I
can't make head nor tail of it.

Also try _buying_ BDB ($20,000). I emailed them asking for a discount, but
Oracle (who owns them now) never replied. :-(

~~~
jrockway
If by convoluted, you mean "half the source code is detailed comments", then
sure...

------
rcoder
I'm a bit surprised at the poor showing of Tokyo Cabinet in this test, given
the amount of buzz it's gathered in the last few months, but the CDB results
honestly don't surprise me in the least -- it's a special-purpose
datastructure wrapped in a thin library that just _smokes_ any other DB type
if your workload allows for a complete rebuild of the database on each update.

~~~
henryl
As someone mentioned up above, the author probably didn't tune the bucket
number appropriately.

------
antirez
It's a shame there is not Redis among the tested KV DBs. I guess the
networking layer is not an option in this specific use case.

