

NeDB, a pure Javascript database for Node.js and Node Webkit - louischatriot
https://github.com/louischatriot/nedb

======
sailfast
As someone that has a JS background and is looking to get started with node,
this is a welcome addition. I like the idea of being able to throw a require
statement in and get up and running. Syntax for inserts and queries look
straightforward too. Thanks!

~~~
louischatriot
Thanks! The syntax is the same as MongoDB's, so that developers used to
MongoDB (most Node.js devs are) can get started on NeDB right away.

------
bsg75
The trend continues where all things eventually get implemented in Javascript.
Waiting for JS.js

~~~
voidlogic
Indeed. In my mind much of the value of javascript is for rapid protoyping and
for allowing those who enjoy JS (I do not [1]), to build medium performance
tier applications.

What strikes me as odd is people wanting to write things that traditionally
are written in C/C++ and more recently also in Go/Rust/Java/D in JS. The
reason these arguably slower, in development speed, languages are used for
things like DBs is that they are write once re-use thousands of times, and
this use case warrants being slightly lower level and less rapid in
development and gaining performance and I would _argue_ maintainability.

[1] Note: In case you are wondering why _I_ dislike JS: 1. JS's extensive use
of spaghetti _cough_ I mean callbacks 2. Higher effort needed to use compute
resources for a single task in a parallel fashion compared to other langs 3. I
dislike dynamically typed languages (I belive they are harder to maintain) 4.
Performance:
[http://benchmarksgame.alioth.debian.org/u64/benchmark.php?te...](http://benchmarksgame.alioth.debian.org/u64/benchmark.php?test=all&lang=v8&lang2=gcc&data=u64)

~~~
louischatriot
The goal in this particular case is to have all your code in a single language
for easy distribution. If you write and distribute a Node.js webapp or Node
Webkit app (which I guess you don't since you don't like JS!), needing your
users to also install whichever DB you use is a pain for them (esp. for
desktop apps with Node Webkit, users may not be developers). That's the reason
I built this.

~~~
voidlogic
First off, louischatriot kudos to you for building this. Building embedded
DB's is fun and worthwhile. The fact I'm not a JS fan doesn't diminish that
value to me.

While I understand that packaging motivation; it seems to me to be less
relevant due to msi/rpm/deb packages. Those technologies make it pretty easy
to package an application built with heterogeneous components.

~~~
louischatriot
Thanks :)

It's true that it's easy to package such applications for unix systems, but
it's much harder on Windows. What makes Node Webkit great is the fact that
it's cross-platform with very little pain, and in that case having the DB in
the JS code makes packaging a breeze.

------
rdudekul
Great idea and project. Questions: a) Is the database loosely coupled to query
language implementation? b) In my biased opinion RethinkDB has a much better
query language. Is there a way to implement RethinkDB's query semantics and
how much work is involved? c) Underscore.js has a ton of collection
manipulation functions. How easy is it to use it from within NeDB? d) How
easy/difficult is it to run NeDB as a lightweight RESTful server, in its own
Node process?

~~~
louischatriot
Thanks! To answer your questions:

a) and b) The database is loosely coupled with the query language
implementation, so it could be possible to change the query semantics without
breaking it. That said I have other priorities for now (some adaptation for
Node Webkit, porting it to the browser).

c) NeDB queries return vanilla objects or array of objects , on which you can
use all the underscore functions you want.

d) Depends what you want to do. If you want to write a REPL I think it
shouldn't be too difficult, and it may be a good idea to do one. In the
meantime, since the API is the same as MongoDB's, you can learn NeDB using
MongoDB's REPL :)

~~~
rdudekul
Thank you. Can you elaborate your statement "you can learn NeDB using
MongoDB's REPL"? Specifically steps to use MongoDB REPL connected to NeDB.

~~~
louischatriot
I was referring to the fact that the syntax is the same so if you don't know
the NeDB syntax you can play with the MongoDB REPL. Appart from this there is
no connection between the two.

------
Cyranix
Perfect timing. I spun up a project fairly recently using node-webkit and
wanted this, exactly. Getting SQLite integrated wasn't particularly painful,
but a document storage model is just what I had hoped for. Thanks for the
contribution; looking forward to giving it a try!

~~~
louischatriot
Thanks! I am finishing a small helper that will enable you to specify the
location of the datafiles relative to the directory where Node webkit stores
data. You should find it helpful.

------
ivarvong
It might be interesting to persist snapshots to S3. It might be usable for
small projects on Heroku/etc with ephemeral storage. It could be patched here
(?) [1], but clearly there'd be a lot of issues with the long network call,
how often to do it, how durable it really is, etc.

1:
[https://github.com/louischatriot/nedb/blob/b1abd34384cefea16...](https://github.com/louischatriot/nedb/blob/b1abd34384cefea16e7cf86c7316f602d26d4e3d/lib/datastore.js#L368)

------
jdp
Reminds me of a beefed-up version of this similar project:
[https://github.com/chjj/node-tiny](https://github.com/chjj/node-tiny)

------
itsnotlupus
node-webkit seems to have a number of options for local storage:
[https://github.com/rogerwang/node-webkit/wiki/Save-
persisten...](https://github.com/rogerwang/node-webkit/wiki/Save-persistent-
data-in-app)

Amongst those are a native SQL engine (sqlite) and a native NoSQL engine
(levelDB, through the IndexedDB API)

It'd be useful to know what benefits NeDB brings over other existing storage
mechanisms.

~~~
louischatriot
I should stress this more in the README ... The key difference is that NeDB is
a "real" (meaning with an actually powerful query and update language)
database that is written in pure javascript. That's not the case for the other
engines that need you (your users actually) to install a third-party DB or
compile it. That makes prototyping, distribution and packaging much easier,
and that was the main motivation (appart from fun) for this project.

------
baudehlo
From what I can make out, this won't work under cluster, right? You won't get
new data written from one worker in another. It also has no locking so you can
easily corrupt the database if you access it from multiple programs at once
(or with cluster).

~~~
louischatriot
It's not supposed to work with cluster and using it with multiple programs
that update/remove/insert docs will cause unexpected behaviour, although no
data corruption since the datafile is append-only.

But if you need multiple programs to use your DB at once you need a more
powerful solution such as MongoDB. The goal of NeDB is to bring DB features to
desktop apps or low concurrency webapps, not replace Mongo and the like.

~~~
baudehlo
No I get that, I'm not saying it should support those things, just saying you
should probably document it clearly. People don't like to lose data.

And append-only doesn't make you immune to corruption. There's still a
possibility of two processes writing at the same time corrupting what gets
written to the file.

------
tantalor
What does this have to do with node-webkit? My understanding is that nodejs is
a subset of node-webkit, so by saying "this is for nodejs", you imply "this is
for node-webkit".

~~~
louischatriot
That's true, but I want to stress Node Webkit because NW apps are probably one
of the best usecases. Also, there is a bit of helper code specific to Node
Webkit, so that you don't need to worry about where the data is stored.

------
glenjamin
node.js used to see really slow GC cycles (>1s) at over ~5/600MB of heap
usage, and bomb out fatally at 1GB. Is this still the case?

If it is, you should probably put a larger warning about dataset size than the
line you currently have :)

~~~
louischatriot
I think it's better now, it handles as least 1.4GB in the heap. That said if
you have a database with more than, say, 300MB of data, I think you need a
more large-scale database.

As an aside, I'm considering enabling large datasets for NeDB as some users
asked, but that would be a lot of work so I'll wait for more requests.

------
gabhubert
Nice welcome addition to the node webkit landscape.

~~~
WarrickF
Agree. This just make writing simple apps that much easier. Great idea!

------
nollidge
How do you pronounce "NeDB"?

~~~
louischatriot
I may need to choose another name, you're not the first one to ask this :) I
pronounce it letter-by-letter as it is an acronym for "Node Embedded DataBase"

~~~
mistercow
My instinct was to pronounce it "knee dee bee", so I think there's definitely
an ambiguity there. On the other hand, that didn't stop people from naming
GIF, SQL, or nginx, so I don't see why it should stop you.

~~~
WayneDB
Indeed. "enn eee dee bee" (or "any dee bee") was mine.

------
WayneDB
Awesome! I've been looking for a project like this for a small app and have
been exploring things like EJDB and UnqLite, among others. I'll definitely be
checking this out. Thanks.

~~~
louischatriot
Great to hear :)

------
fourstar
How is this different from mongodb? Aside from being written in javascript and
having a non-existent community around it?

~~~
skeoh
It's different in that it's embedded in your application (like SQLite). It
does not require a separate process or server running Mongodb. There are
certainly appropriate use cases for this module, and since it has a Mongodb-
like interface it can be swapped out for Mongo at any time.

~~~
louischatriot
Yep. I also made a small script that automatically transfers all your data
from a NeDB to a MongoDB: [https://github.com/louischatriot/nedb-to-
mongodb](https://github.com/louischatriot/nedb-to-mongodb)

You can use that once you've outgrown NeDB.

