

Symas Lightning Memory-Mapped Database (LMDB) - networked
http://symas.com/mdb/

======
justin66
Recently deleted from Wikipedia as being not notable by a former Oracle
employee.

[http://en.wikipedia.org/wiki/Wikipedia:Articles_for_deletion...](http://en.wikipedia.org/wiki/Wikipedia:Articles_for_deletion/Lightning_Memory-
Mapped_Database)

edit: that page refers to a deletion that happened a year ago but lmdb had a
page much more recently than that - even a month or two ago, I think - so
apparently deleting the lmdb page is a hobby for someone. I'm not familiar
enough with the tools to dig in and figure out what's going on there.

~~~
sitkack
[http://en.wikipedia.org/wiki/User:ThurnerRupert/Lightning_Me...](http://en.wikipedia.org/wiki/User:ThurnerRupert/Lightning_Memory-
Mapped_Database)

This is a problem with wikipedia in general. I think archive.org is/should be
archiving dumps.

------
mitchellh
As a data point, this is the underlying data storage mechanism we use for
Consul
([https://github.com/hashicorp/consul](https://github.com/hashicorp/consul)).

We also forked and improved some Go bindings to lmdb:
[https://github.com/armon/gomdb](https://github.com/armon/gomdb) (which is the
lib we use in Consul).

When building Consul, we were specifically looking for an in-process DB that
supports MVCC. The reason is because while Consul is doing a snapshot, we
wanted to be able to INSERT/UPDATE without affecting the integrity of the
snapshot. LMDB fit this role nicely and the performance has been fantastic for
our use case.

~~~
joshu
What is the difference between consul and serf? The top-level explanation is
very similar.

~~~
jdf
[http://www.consul.io/intro/vs/serf.html](http://www.consul.io/intro/vs/serf.html)

------
ddorian43
Hustle is a distributed, column oriented, relational OLAP Database that uses
lmdb.

[http://chango.github.io/hustle/](http://chango.github.io/hustle/)

------
pdq
Howard Chu, the author of LMDB, gave a great presentation on LMDB:
[http://parleys.com/play/517f58f9e4b0c6dcd95464ae/chapter0/ab...](http://parleys.com/play/517f58f9e4b0c6dcd95464ae/chapter0/about)

He can also rock the violin.

------
rdtsc
Benchmarks are very impressive:

[http://symas.com/mdb/microbench/](http://symas.com/mdb/microbench/)

Multi-threaded, vs memcache

[http://symas.com/mdb/memcache/](http://symas.com/mdb/memcache/)

------
lafar6502
Pls remember this database has no write ahead log and therefore is not too
fast for write-intensive applications. But i really like it and regret I
learned about it only recently.

PS and read the benchmark description carefully - some of the benchmarks are
performed in memory, without actual disk i/o.

~~~
hyc_symas
For _some_ write-intensive applications it is faster than everything else.
E.g. [http://symas.com/mdb/hyperdex/](http://symas.com/mdb/hyperdex/)

Depends a lot on your key and value sizes - the larger the values, the faster
LMDB is vs any other solution, due to the zero-copy behavior.

~~~
lafar6502
_wmd, hyc_symas I understand the principles of LMDB operation, the worst case
scenario is lots of random writes that hit random pages in the db file. In
such case the I/O will be heavy and random also, which leads to bad
performance of disk operations (not so bad with SSD ,though). But you're
right, if you avoid the worst case LMDB shines.

~~~
hyc_symas
I'm pretty sure the worst case is purely size-dependent.

The write pattern in the HyperDex benchmark is pure random but it still
obliterates HyperLevelDB (and don't even think of vanilla LevelDB there). Keep
in mind that LMDB uses free pages in sorted order, so even for a random write
workload, pages are allocated in ascending order, which generally translates
to unidirectional seeks on an HDD. This is why even for a purely random write
load, LMDB is still faster than all of the traditional update-in-place B-trees
out there - they really have to do _random_ seeks, LMDB doesn't. (And you pay
the highest in seek time when the drive head has to reverse direction.)

That's why LMDB write performance remains uniform under load, while all LSMs
suffer from GC/compaction pauses.

But ultimately, storage is moving to solid state, and seek time will be
irrelevant.

------
apendleton
I'd be curious to see how it compares to LSM, Sqlite4's new underlying KVS.

~~~
hyc_symas
The sqlite4 authors tested it. Never published the results, AFAIK. Probably
because sqlite4 LSM is a pig.
[http://www.sqlite.org/src4/info/51816384756d6c620e991bb4ef81...](http://www.sqlite.org/src4/info/51816384756d6c620e991bb4ef81a284d5ef56f3)

