On: The Bw-Tree: A B-tree for New Hardware

[snaky]

>SQLite: We tuned SQLite's performance, by setting its locking mode to exclusive. We also enabled SQLite's write-ahead logging

>Databases are opened in asynchronous write mode. (LevelDB's sync option, TreeDB's OAUTOSYNC option, SQLite3's synchronous options are all turned off, MDB uses the MDB_NOSYNC option, and BerkeleyDB uses the DB_TXN_WRITE_NOSYNC option). I.e., every write is pushed to the operating system, but the benchmark does not wait for the write to reach the disk.

By the way there is SQLite with LMDB backend - https://gitorious.org/mdb/sqlightning

>Using tool/speedtest.tcl in the SQLite source tree, the time to insert 1000 records on my laptop SSD was 22.42 seconds using the original code, and only 1.06 seconds using MDB. Both tests were run 3 times, with results average

[easytiger → snaky]

Your point eludes me? For a start that is not tuning it. It is still using synchronous IO

[hyc_symas → easytiger]

Note that the LevelDB, Kyoto Cabinet, and SQLite3 configs are identical to what the Google LevelDB authors used in their original benchmark. http://leveldb.googlecode.com/svn/trunk/doc/benchmark.html

The only thing different about the Symas LMDB tests is that LMDB and BDB were added.

[snaky → easytiger]

Did you look at http://symas.com/mdb/microbench/db_bench_sqlite3.cc?

Search for 'write_sync'

[easytiger → snaky]

Ahh yea i missed, apologies

     std::string sync_stmt = (write_sync) ? "PRAGMA synchronous = FULL" :
                                           "PRAGMA  synchronous = OFF";

Still. how is it a valid compare if it is getting it to ssd/hdd and the others are writing to memory? Just open sqlite3 with :memory:

[hyc_symas → easytiger]

You're still not understanding. LMDB is a disk-based database. It is not only writing to memory.

[easytiger → hyc_symas]

MDB in those stats is writing to disk? I assume its some kind of lazy write though where the memory store is eventually persisted? in that case thats fairly impressive for a lot of use cases. Though a k/v store has a lot less to do than something like sqlite3.

I'll look into it (never heard of it before). Might fix fairly well with something I'm working on right now (currently implemented using sqlite3).

[hyc_symas → easytiger]

In fully synchronous mode, which is LMDB's default mode, everything is persisted upon commit. This is a memory-mapped database, not a main-memory/in-memory database. It is fully persistent to disk, it just so happens that it uses mmap() instead of read()/write() and a user-level page cache.

People are always confusing "memory-mapped" with "memory-only". LMDB is not a memory-only database.

http://www.openldap.org/lists/openldap-devel/201111/msg00064...

[easytiger → hyc_symas]

I guess (as i did) assumed it meant memory-mapped I/O as opposed to something like memory-mapped file I/O or port mapping.