
UnQLite - An Embeddable NoSQL Database Engine  - ingve
http://www.unqlite.org
======
benjamincburns
As a hacker/tinkerer type programmer I can understand why it might be fun to
build something like this. All of the data structure fun with none of the
distributed difficulty! NoSQL! I can even build in that cool new language I've
been designing in my head! But like everyone else I'm straining to see a use
case. "Serverless" NoSQL? If you've got something small enough that you want
an embedded datastore, why use a NoSQL variant?

I understand that NoSQL literally means "no SQL," but typically it's a
buzzword reserved for unconventional datastores used under heavy distribution.

Maybe you'd want a local embedded "NoSQL" to act as a local datacache? But
then for that use case why create an impedance mismatch with the central
datastore by rolling your own?

Jx9? So wait, I need another language to interact with the thing that's
already embedded directly within my program?

Local offline data analysis? Now that makes sense. Does this have
intelligent/optimized paging mechanisms? I have a feeling not, since the
author seems to be couching this as an alternative to sqlite for people who
like NoSQL.

I'd sincerely love to hear more about what's motivating this project. Was it
just for yucks, or is there some problem that the author(s) needed to solve
that wasn't well-solved by another tool?

~~~
continuations
> I'm straining to see a use case. "Serverless" NoSQL? If you've got something
> small enough that you want an embedded datastore, why use a NoSQL variant?

Serverless NoSQL embedded datastore aren't really that unusual. BerkeleyDB,
Tokyo Cabinet, LevelDB are all commonly used serverless NoSQL.

~~~
benjamincburns
I think what's happening here is that NoSQL is just an anti-category.

When I think of NoSQL, I think of something like MongoDB or Cassandra which
are designed to handle tons of data under heavy load. When I think of
BerkeleyDB and friends, I think local persistence. It's a true statement that
these too are NoSQL stores, but, at least for me, they're not of the variety
that first comes to mind.

------
ominous_prime
The licensing is strange as well. The BSD license is listed as a feature, but
the embedded scripting language (Jx9) is SPL (their own GPL-alike/Sleepycat
license).

Since the Jx9 language is fully embedded (even stated as " All C source code
for UnQLite and Jx9 are combined into a single source file."), they seem to
contradicting their own licensing. The resulting library would also retain the
copyleft SPL license.

The base code may be BSD, but incorporating the whole database into your code
infects it with their SPL too.

~~~
symisc_devel
From UnQLite core developer: No, Jx9 (the standalone library) is under SPL,
while UnQLite is 2-clause BSD (including the Jx9 core), so there is no worry
about that since the two software are developed by the same company. In other
words, use UnQLite without restrictions

~~~
jessaustin
Care to respond to this complaint?

<https://news.ycombinator.com/item?id=5751928>

~~~
symisc_devel
Yep, sorry for the delay, We're quite busy.

I think this post will clarify the license issues
<http://unqlite.org/forum/note-on-the-licensing-situation>

------
willvarfar
This turned up recently on proggit too.

[http://www.reddit.com/r/programming/comments/1etfxi/sqlite_n...](http://www.reddit.com/r/programming/comments/1etfxi/sqlite_now_faster_with_memory_mapped_io/ca3lgyy)
was heavily downvoted for saying its an order of magnitude faster than SQLite

[http://www.reddit.com/r/programming/comments/1etkix/unqlite_...](http://www.reddit.com/r/programming/comments/1etkix/unqlite_an_embeddable_nosql_database_engine/)
raises concerns about the license, and there's a Global Lock in there too? The
scripting performance gives cause for pause too.

~~~
cpleppert
"It furthermore states that commercial licence must be acquired for closed
source applications."

I love the description of the scripting language as "Turing complete" based on
"JSON". That is the first time I have ever heard turing complete used to
market a programming language!

The scripting language is incredible, I have never seen a more verbose way to
program. It is a scripting language, yet seems to combine JSON, PHP and C
standard library functions with C type casting. Look at the samples:
<http://unqlite.org/jx9_samples.html>

I am speechless, the description of the project is the most vacuous collection
of buzzwords imaginable. "Built with a powerful disk storage engine which
support O(1) lookup." "Support Terabyte sized databases." Of course with a
global lock and after taking a look at the scripting language I have reason to
doubt the claims made

~~~
JulianMorrison
If you lived through the XML boom, you'll probably recall the "Turing
complete! based on XML!" languages (XSLT, Ant, and friends). This was before
people re-re-learned the essential lesson "languages based on markup are
painful as languages and unsafe as markup".

------
delinka
Isn't BerkeleyDB the quintessential "embedded NoSQL" database engine? I'm an
RDBMS+SQL kind of guy, so can someone enlighten me about the differences
between bdb and UnQLite?

~~~
boomzilla
Yeah, another alternative with friendlier license is leveldb

<https://code.google.com/p/leveldb/>

It's also well engineered (written by some of the best Google engineers) and
well supported.

~~~
willvarfar
In fact, the same engineer who previously wrote big table iirc

------
angersock
A very uncharitable reading of the features seems to suggest that this is some
strange bastard child of a hashmap, a scripting language interpreter, and a
JSON serializer.

I'm sure it's much more than that, but one wonders where this stacks up
compared to existing solutions.

~~~
tshadwell
LevelDB is similarly self-contained, and client/server API-less and is
extremely fast. Though it has a large advantage of speed, the databases are
not one files.

~~~
hosay123
LevelDB holds neither a read performance advantage in any scenerio over common
alternatives (LSM reads fundamentally require extra work by design), or a
write performance advantage for anything but very short benchmark-friendly
bursts. In the meantime its worst-case performance characteristics are orders
of magnitude worse than just about any alternative.

(Keep clicking "more" in my comment history for detail)

------
eliben
Is this a buzzword-laden way to describe a serializable hash table? [I'm
asking seriously]

~~~
angersock
You can also script it!

~~~
eliben
Why would I want to script a hash table from a programming language?

------
gizzlon
If you actually need something like this, there's Kyoto Cabinet, BerkleyDB and
others that are well known and well tested.

Why use something new with no apparent benefits? (and a lot of drawbacks)

<http://en.wikipedia.org/wiki/Berkeley_DB>

<http://en.wikipedia.org/wiki/Kyoto_Cabinet>

Edit: For extra points, read this: <http://www.aosabook.org/en/bdb.html>

~~~
josephg
BDB is really slow, and KyotoCabinet is GPL. For some better alternatives,
check out LevelDB: <https://code.google.com/p/leveldb/> and LMDB:
<http://symas.com/mdb/>

There are some decent benchmarks put out by the mdb guys here:
<http://symas.com/mdb/microbench/> .

------
ominous_prime
The site seems like a computer generated mashup of keywords and phrases. It's
a document store database similar to MongoDB, Redis, CouchDB etc _and_ and
key-value store similar to BerkeleyDB, LevelDB, etc.?

> UnQLite is 100% hand-coded, written in ANSI C, _Thread-safe_ , _Full
> reentrant_ ...

It also uses its own scripting language

> [jx9] uses a clean and familiar syntax similar to C, JavaScript and JSON

The copy on this site leaves me puzzled.

------
DoubleMalt
Not that I am envious for the karma, but I wonder why this was not merged into
my exactly identical submission 1 hour earlier
(<https://news.ycombinator.com/item?id=5749969>) -- afaict there was no
?repost or similar stunt ...

~~~
benjamincburns
Being that HN was designed to handle a metric shit ton of traffic on very
little hardware, I'd imagine that it has adopted a weak consistency model for
these types of things.

------
vinkelhake
What an unfortunate name. I spent some confused minutes thinking that the
author of SQLite had finally done a complete implementation of [1], which he
has been involved in.

[1] <http://unqlspec.org>

~~~
ccera
Yes, so did I, especially given that the author of SQLite said he was working
on a new database with exactly this name, UnQLite.

This UnQLite developer is a first-class tool.

------
mwcampbell
I imagine that Dr. Hipp won't be happy with the choice of project name when he
finds out, if he hasn't already. When I first saw the name "UnQLite", I
figured this was his project, since I knew that he was working on something
called UnQL.

------
Aykroyd
If I wasn't going to get the benefit of my data automatically being
distributed across servers, why would I use this? In many cases, NoSQL
solutions seem to be a compromise that you make, giving up ACID (yes I know
this has it) and other nice query features of databases and in return gaining
the ability to scale to much larger amounts of data and having redundancy
without needing to think about it.

In this case though, it's all embedded so I don't gain any of the benefits.

~~~
pyre
The point being that if you need a local data store, sometimes you would
rather have a document-based one where you don't have to create the code to
serialize your objects into (e.g.) a SQLite database, and to reconstitute them
back.

A good example would be something like an address book data store. Especially
if you want it to be dynamic so that the user can add/remove fields (e.g.
allowing the user to attach as many phone numbers as they like to the contact,
rather than just a static 5 numbers). If you have to implement this in SQLite,
then you have to develop the schema for it, and the (de)serialization code.
With a document-store, you can just do something like say "store these fields"
and you're done with it.

    
    
      | giving up ACID (yes I know this has it)
    

If you know that this has ACID, then why are you talking about giving up ACID?
The fact that many NoSQL implementations give up ACID doesn't have any bearing
on this discussion.

------
louischatriot
I am working on a similar project: a persistent database in Node for Node
projects, with no external dependencies. That means you can use it with a
simple require(), no external software needed.

This is useful for small projects that don't need the power of a behemoth like
MongoDB and want to be installable by a simple git-clone + npm install

I am very interested in feedback on it!
<https://github.com/louischatriot/nedb>

------
Millennium
Is this what Richard Hipp was working on? It doesn't sound like what I
remember hearing about back in 2011, but maybe things changed and I missed the
news?

~~~
mmcclellan
No. Dr Hipp and Damien Katz were working on UnQL
(<http://www.unqlspec.org/display/UnQL/Home>), which is unfortunately quite
similar in name to this new UnQLite project.

~~~
ccera
Actually, Hipp has said publicly that he intended to created a new embedded
database called UnQLite.

It seems this developer just blatantly ripped off the exact name Hipp was
planning to use. He also ripped off some of the core SQLite code (the VFS,
etc.), which is legal to do since SQLite is in the public domain, but still...

Not cool.

~~~
symisc_devel
OK, actually I've emailed D. Richard Hipp a few months ago asking for
permission to use the name UnQLite in a future open source project, here is a
copy of Hipp's reply:

It would be good if you can make it clear on your website, somehow, that yours
is an unaffiliated project. Otherwise, people might go complaining to me when
they find bugs in your code. (Don't laugh - that sort of thing happens a lot.)

Other than that, you are welcomed to use the name.

You might want to have a look at the LSM storage engine that Dan Kennedy is
working on for SQLite4. It is faster than the clunky and dated B-Tree used by
SQLite3. It is also faster than LevelDB. And it supports nested transactions,
with rollback. And concurrency. And it is more NAND-flash friendly. See
<http://www.sqlite.org/src4/timeline> for the latest code.

------
dorfsmay
It doesn't explain the advantages over all the other dbm alternatives:

<http://en.wikipedia.org/wiki/Dbm#Successors>

I'm yet to see a good replacement to Kyoto cabinet both in terms of ease of
use and performance. I feel it'd be better energy to pick up kyoto cabinet and
maintain it than re-write something from scratch.

~~~
jeltz
Isn't Kyoto Cabinet GPL licensed though which makes it less useful for
embedding?

~~~
dorfsmay
Never thought of this... It turns out it is dual licensed:

<http://fallabs.com/license/>

Although I'm not sure how difficult it would be to get a license, the author
now works for google and doesn't seem to answer to that email (when asking
technical question anyway - offering money might get a different reaction).

------
DonnyV
I would love an embeddable nosql database engine but this isn't it. I'm
looking for something like MongoDB but embeddable. RaptorDB comes close but it
doesn't have full support for Mono yet. <https://raptordb.codeplex.com/>

Can anyone recommend one?

~~~
jcromartie
Honestly, you could build a decent embedded document store on top of SQLite
pretty easily. It's hard to beat SQLite's long and solid track record for
embedded data persistence.

The default BLOB limit is around 1GB.

<http://www.sqlite.org/limits.html>

~~~
StavrosK
That's exactly what I did! <https://github.com/stochastic-
technologies/goatfish>

------
lucian1900
So dbm, but actually with a dbm-incompatible API? Why?

------
Sami_Lehtinen
Nice, I'm currently using SQLite for key value storage, because there aren't
better light options afaik. Python shelve module is totally useless with multi
gigabyte tables and millions of keys.

~~~
gizzlon
What's wrong with BerkleyDB, Tokyo Kabinet, Kyoto Kabinet, LevelDB .. ?

It's a real question because I haven't actually used them. SQLite is fine, but
if you don't need sql they probably fare better.

~~~
Sami_Lehtinen
Afaik, all of those require additional setup. SQLite3 is already included in
Python. I also tested year ago several database options and SQLite3 was also
the fastest one.

~~~
josephg
Maybe it depends on your work load, but thats not true if you just want a
regular K-V store. LevelDB and LMDB are usually way faster:
<http://symas.com/mdb/microbench/>

Of course, neither of these systems have indexing. If you want to use them
directly, you have to organise your data so that you can translate your
queries into range queries on an ordered set. The FoundationDB guys talk about
this here: [http://foundationdb.com/documentation/beta1/data-
modeling.ht...](http://foundationdb.com/documentation/beta1/data-
modeling.html#indexes)

------
brokenparser
This seems backwards to me. Sure, use the right tool for the job, but still.
SQL server -> noSQL / embedded SQL -> embedded noSQL. At every step there's
someone going "X is hard, let's not do X" and we end up with a serverless,
configuration-free, noSQL transactional database. The next logical steps are
presumably the removal of transactions (who needs them, anyway?) and volatile
storage (RAM is fast, let's use that).

I can't wait for the day malloc appears on a page with bullet points citing
all the latest buzzwords.

~~~
tracker1
Well, you can do MVCC over straight blocking transactions. As for using
volatile storage, see MemcacheD and Redis.

------
StavrosK
I wrote a very simple thing to sort-of emulate this on SQLite. It works rather
well for prototyping!

<https://github.com/stochastic-technologies/goatfish>

It also supports indexing on arbitrary fields and is only a few lines of code.
It did come in very handy.

------
hightowk
I think some folks are missing the point of being able to scale up when needed
by just changing the connection string. If you are trying to prove a big data
concept with small data, this might be a cheaper path. Now, if there were just
standards for NoSQL query interfaces, this might be 100% true...

~~~
kbenson
That's just it. Since it uses a specific scripting language, if you use it for
anything more than direct CRUD operations, you very well may need to do quite
a bit more than change the connection string. SQLite has less of a problem
with this because SQL is standardized (but of course all the RDBMS have their
variants).

It's unfortunate that it's called UnQLite. Richard Hipp, the creator of SQLite
is now involved with the UnQL specification[1], which looks unrelated to this.
In fact, there's an interview where Hipp states his plans to make an
UnQlite[2], which makes this, intentional or not, a namespace grab.

[1]: <http://www.unqlspec.org/>

[2]: <http://www.infoq.com/news/2011/08/UnQL>

------
iuguy
Does anyone know how this compares in terms of performance to leveldb, lmdb
and kyoto cabinet? I've been doing some potentially terabyte scale stuff and
while another option is welcome I don't feel like re-integrating yet another
local db to find out.

------
est
similar project

<https://github.com/Softmotions/ejdb>

------
dschiptsov
isn't leveldb is there for this?

------
RyanZAG
This would be a great replacement for the fairly awful sqlite on android -
accomplishes much the same goal, but should fit into Android a lot better.

Someone should write a android/java wrapper for it in the same way Google has
a built in wrapper for sqlite.

~~~
willvarfar
In what way is SQLite awful on Android? I'm just curious what your experience
has been.

~~~
j_s
Not really SQLite's fault, but Android automatically deletes corrupt SQLite
files

<http://code.google.com/p/android/issues/detail?id=10127>

and SQLite can corrupt databases (for sure back in 2010, with some discussion
on whether or not this is still an issue)

<http://code.google.com/p/android/issues/detail?id=8427>

Combined, this means the db file gets damaged (usually manually repairable) so
the Android OS just deletes it.

