

Bolt: A low-level key/value database for Go - sciurus
https://github.com/boltdb/bolt

======
gravitronic
Bolt was built at Shopify as a back-end to our Merchant Analytics reporting
stack.

If you're interested in working on projects like this, we are hiring - contact
me tom at shopify.com.

~~~
jbooth
Cool stuff but why didn't you just use an existing LMDB wrapper or write a new
one? What's this offer that an LMDB wrapper doesn't?

~~~
benbjohnson
We originally used szferi/gomdb which worked well but it didn't give us
performance visibility with "go pprof" and it was difficult to debug at times.
Bolt also takes the approach of simplifying the LMDB API significantly. There
are a lot of advanced, unsafe features that aren't available in Bolt (e.g.
WRITEMAP, NOMETASYNC). There's more detail in the README:

[https://github.com/boltdb/bolt#lmdb](https://github.com/boltdb/bolt#lmdb)

Bolt was originally just a side project but we found that it fit our needs for
our analytics database so we eventually swapped out LMDB for Bolt.

~~~
_wmd
This is a fairly minor nit, but a persistent one nonetheless: you keep
claiming to have removed unsafe features from LMDB, yet you left the most
dangerous one in there: NOSYNC provides no consistency guarantees whatsoever,
whereas e.g. with NOMETASYNC you merely lose durability. In particular
NOMETASYNC retained crash safety, which you don't get at all with NOSYNC.

The way this is worded, it sounds like an improvement, but in reality bolt is
no safer to use than LMDB was, it just lacks some configurable tradeoffs that
were present in the original engine, and were safer to use than those that
remain in bolt.

That said, I'm happy to see APPEND go ;)

~~~
benbjohnson
That's a fair point. Although I'd argue that WRITEMAP and APPEND are more
scary. :)

I try to point out in the docs that DB.NoSync should only be used for offline
bulk loading where corruption is typically less of an issue (since you can
simply rebuild).

[https://godoc.org/github.com/boltdb/bolt#DB](https://godoc.org/github.com/boltdb/bolt#DB)

The wording was intended to make it sound like an improvement for people
wanting smaller, more concise code bases. Bolt is 2KLOC vs LMDB's 8KLOC.
Smaller surface area makes it easier to test thoroughly. It's definitely a
tradeoff for people who want that really low level control though.

~~~
hyc_symas
LMDB author here - what have you got against APPEND? Fast bulk-loading is
pretty important when you're initializing a system.

Note that APPEND mode _won 't_ allow you to corrupt the DB, despite what the
doc says - you'll get a KEYEXIST error if you try to load out of order. We
just left the doc unchanged to emphasize the importance of using sorted input.

~~~
benbjohnson
Thanks, Howard. That's good to know. The docs always scared me away from using
it at all.

~~~
hyc_symas
Perhaps it's time for us to update the doc to reflect reality...

------
mrweasel
I've looked briefly at the Bolt documentation a few times and I can't say that
I like the idea of giving a function as a parameter to the Update and View
functions. It result in some weird looking code with functions within
functions, that can't really be refactored out or reused.

Pretty much Javascript callback spaghetti in Go. There might be a way around
it, but I haven't found any examples. Perhaps there could be a way where
transactions a manually managed.

~~~
okatsu
I'm still new to Go, but the functions don't have to be anonymous, right? What
would be wrong with defining all of your functions somewhere (for updates,
iterations, etc.) and passing them by name to Bolt's?

~~~
mrweasel
You could name your functions, but you won't really be able to parse in any
data. Unless you wrap then Update and View functions as well.

As others points out there's nothing wrong the Bolt code, it's fine and pretty
common, I just don't like it.

~~~
fishnchips
> Unless you wrap then Update and View functions as well.

This is how I'd do it.

------
NateDad
[http://npf.io/2014/07/intro-to-boltdb-painless-performant-
pe...](http://npf.io/2014/07/intro-to-boltdb-painless-performant-persistence/)

------
ginko
Namespace clashes in software are starting to become a real problem:

[http://developer.amd.com/tools-and-sdks/opencl-zone/bolt-
c-t...](http://developer.amd.com/tools-and-sdks/opencl-zone/bolt-c-template-
library/)

------
cbroglie
I'm trying to understand the use case for a persistent embedded DB for Go.
SQLite is embedded all over the place, often in clients.

But presuming Go is a server language, and you are running stateless web
nodes, why do you need persistence?

~~~
NateDad
It's awesome because it's just a go package - no external dependency... Your
application still ships as a single binary.

~~~
neilc
Note that using a sqlite wrapper like [https://github.com/mattn/go-
sqlite3](https://github.com/mattn/go-sqlite3), you also don't have any
external dependencies and your application still ships as a single binary.
However, there are other advantages of a Go-native storage library (e.g.,
doesn't require cgo, better integration with tools like pprof, etc.)

------
fishnchips
I'd love to have something like that built around some serialisation protocol
like protobufs. Something that would feel like GAE's Datastore API.

~~~
benbjohnson
I haven't used GAE's Datastore API but using protobufs on top of Bolt is
trivial. Bolt just uses []byte for keys & values. I use gogoprotobufs with
Bolt often.

