
UnQLite – An Embeddable NoSQL Database Engine - rayascott
https://unqlite.org/
======
symisc_devel
Hey all. I worked on the UnQLite development team back into 2013 ~ 2014. It
was designed to be the primary storage for a home targeting router
manufactured by a French tech conglomerate.

The API is broken down into two independent layers plus one for various
utility stuff like cross platform mmaped files, RC4 random numbers generation
and so on. The first layer serve as a general purpose key/value store for the
host application blobs (i.e binary or text data). You can store whole disk
files inside a single database to generate a cross platform TAR archive for
example.

The second one is the document store layer similar in concept to what MongoDB
offer but without the networking layer since everything run in the same
process of the host application. Both layers are transactional and are able to
recover after some external failure thanks to the SQLite journaling mechanism
from which UnQLite is based on.

Since its release in 2013, there was four major bugs including two critical
that involves data loss under certain load. All known data corruption bugs
were fixed in the last release and no major bugs were discovered a year or so.

The library enjoy particular success among Python and C/C++ developers
especially Chinese that used to bombard the original developers with various
internal aspect of the library.

------
coleifer
I authored a python binding and have some experience that may be relevant.

The library seems poorly maintained in my opinion and might be abandoned.
There have been critical bugs leading to data loss. They have a similar
project vedis which is like an embedded redis that I think is completely
abandoned.

For the love of God just use SQLite if you want an embedded database for
structured data, even if it's just eav. If you just need key/value there are
battle-tested, well maintained options.

[https://github.com/symisc/unqlite/issues/41](https://github.com/symisc/unqlite/issues/41)

~~~
cwyers
You've posted the same two-year-old closed issue twice on this thread. You're
coming off as someone with an axe to grind.

~~~
sitkack
People often get passionate about data corruption.

------
interfixus
The licensing is not altogether obvious. There seems to be a "one time fee"
for downloading. The name sort of implies a connection with SQLite. No such
connection exists. There is a parent company, name of "Symisc Systems". Which
appears to be just one guy somewhere in France or Tunisia who likes to call
himself "we". The code is not too kindly treated by several who have actually
tried to use it.

I think I'll pass.

~~~
tyingq
[https://github.com/symisc/unqlite/blob/master/LICENSE](https://github.com/symisc/unqlite/blob/master/LICENSE)

It does use a fair amount of sqlite source.

------
dfabulich
I think the FAQ needs another question: when should I prefer UnQLite over
SQLite? When should I prefer SQLite over UnQLite?

~~~
sephware
That one seems straight forward: when you want SQL use SQLite, when you want
NoSQL use UnQLite

~~~
xellisx
From what I quickly gathered, it's just a Key-Value storage mechanism.

~~~
sephware
The title says NoSQL and the first paragraph on the page confirms that it's
more than just a KV store:

> _UnQLite is a in-process software library which implements a self-contained,
> serverless, zero-configuration, transactional NoSQL database engine. UnQLite
> is a document store database similar to MongoDB, Redis, CouchDB etc. as well
> a standard Key /Value store similar to BerkeleyDB, LevelDB, etc._

~~~
xellisx
Ah, I didn't read it correctly then.

------
teddyh
> _UnQLite is a standard key /value store similar to BerkeleyDB, LevelDB, etc.
> but, with a rich feature set including support for transactions (ACID),
> concurrent reader, etc._

This is slightly interesting, since the big reason I personally prefer
applications using SQLite over those using BerkeleyDB (which used to be _very_
common, btw) is that I’ve had _lots_ of problems in the past with corrupted
BerkeleyDB files. Like, almost all the time; if it used BerkeleyDB, it would
sooner or later get corrupted. If UnQLite can be a “non-corrupting BerkeleyDB”
for those who don’t actually need SQLite, it can serve an (admittedly niche)
purpose.

~~~
coleifer
Oh it corrupts...
[https://github.com/symisc/unqlite/issues/41](https://github.com/symisc/unqlite/issues/41)

------
UncleEntity
> This production-ready package is available for a one-time fee of $20.

Don't fault them for trying to make a bit of cash but going to have to pass
since I'm too poor to spend $20 to see if it's a viable replacement for a
little project I built on top of tinydb that's getting pretty slow since the
db has grown past what a pure python db can comfortably handle.

Or...I suppose they could provide a link from their download page to their
github repo for us somewhat lazy folks ->
[https://github.com/symisc/unqlite](https://github.com/symisc/unqlite)

------
evil-olive
Is there any documentation of the testing/verification process used in
UnQLite? One of the things that gives me confidence in SQLite is
[https://www.sqlite.org/testing.html](https://www.sqlite.org/testing.html) and
it'd be interesting to see what approach UnQLite takes to the same problem.

------
dfabulich
From the FAQ:

"7\. Is UnQLite thread-safe

Threads are evil.[1] Avoid them.

UnQLite is threadsafe and full re-entrant. But in order to be thread-safe,
UnQLite must be compiled with the UNQLITE_ENABLE_THREADS compile time
directive defined. If you are unsure if the UnQLite library you are linking
against is compiled to be threadsafe you can call the
unqlite_lib_is_threadsafe() interface to find out."

[1]
[http://www.eecs.berkeley.edu/Pubs/TechRpts/2006/EECS-2006-1....](http://www.eecs.berkeley.edu/Pubs/TechRpts/2006/EECS-2006-1.pdf)
"The Problem with Threads"

"Threads are evil" is not a useful remark to make in response to the question
"is this library thread-safe"

~~~
dominotw
sqllite has the same warning in their FAQ's :D

[https://www.sqlite.org/faq.html#q6](https://www.sqlite.org/faq.html#q6)

~~~
blattimwind
SQLite can actually do grouping and sorting (iirc) in parallel, though it's
disabled practically everywhere (Linux distros and other environments shipping
SQLite seem to pick compile flags kinda arbitrarily, e.g. json1 and fts4/5 are
fairly widespread on Linux, but not included in e.g. Python for Windows).

------
snissn
this is great - in my opinion there's a shortage of key value type data
storage engines and there needs to be more competition in this regard. There
simply aren't enough free/open source data engines to meet the demand of
today's software.

few quick thoughts - how does this compare to leveldb? How would it perform in
something like geth or parity as an ethereum client back end data engine in
compared to leveldb? do you have a python interface library?

~~~
manigandham
Really? There seem to be a ton of key/value engines on the market, especially
because it's such a simple model to implement.

~~~
dfox
There is plethora of embedded K/V stores, but most of them only solve the
simple to implement parts. That means either no concept of concurrency control
or global RW-lock, zero resiliency to power failures, no support for atomic
transactions...

If you want on-disk K/V store for critical data you are pretty much stuck with
BerkeleyDB.

~~~
hyc_symas
BerkeleyDB still has corruption issues, and is quite slow. The only k/v store
out there proven to be entirely corruption-free is LMDB.

[https://symas.com/lmdb/](https://symas.com/lmdb/)

------
dicroce
Unqlite doesn't support in order traversal of data because the default (and
only included storage engine) is based on hashing.

------
NonEUCitizen
Is there an "amalgamation file" like SQLite has for easy inclusion into other
projects?

~~~
rayascott
Yes. unqlite.h/c in the root at
[https://github.com/symisc/unqlite](https://github.com/symisc/unqlite)

------
koenigdavidmj
Sounds pretty similar in target usage to LMDB.

~~~
dominotw
thats log structured storage?

~~~
hyc_symas
No, LMDB uses a copy-on-write B+tree.

[https://symas.com/lmdb/technical/](https://symas.com/lmdb/technical/)

------
newscracker
The name though — what is it supposed to mean? “Un” as in “One” in French with
QL Lite or “Un” as in a prefix for an opposite in English with QL Lite (like
“No QL Lite”)?

~~~
JulianMorrison
Un SQL = no sql

And a pun on SQLite.

~~~
reelly
> a pun on SQLite.

And a bad one at that.

I normally don't chime in on HN, but this pun is a crime against humanity.

------
otabdeveloper2
> An Embeddable NoSQL Database Engine

So, std::unordered_map?

~~~
gmueckl
Looks more like an out of core unordered_map<string,string> with thread safety
and persistence to me, if you really want to make that analogy.

------
objektif
So what are the advantages of this over say python dicts? Genuinely asking
thanks.

~~~
Buge
The data in a python dict disappears when you turn your computer off. The data
in UnQLite doesn't disappear.

~~~
objektif
I could pickle the dict to a file right?

~~~
Buge
If you have 1GB of data, and want to change a small portion of it, that would
require you to read the whole thing, then modify it, then write the whole
thing back to disk. A database only requires a small amount of reading and
writing.

------
ankrgyl
This is a good example of someone twisting the term "serverless" to capture
attention:

> _Serverless Database Engine_

> _Most NoSQL database engines (i.e. MongoDB, Redis, CouchDB) are implemented
> as a separate server process. Programs that want to access the database
> communicate with the server using some kind of interprocess communication
> (typically TCP /IP) to send requests to the server and to receive back
> results. UnQLite like SQLite does not work this way. With UnQLite, the
> process that wants to access the database reads and writes directly from the
> database files on disk. There is no intermediary server process._

The fact that it's embedded doesn't actually have anything to do with what
popular "serverless" architectures are designed to solve, i.e. writing code
that does not need to run in a specific server process to be invoked on
demand. By their definition of serverless, pretty much any library is
"serverless"...

 _Edit: This is referring to "Classic Serverless"
([https://sqlite.org/serverless.html](https://sqlite.org/serverless.html)) not
"Neo-Serverless" architecture (my definition is more like "Neo-Serverless").
Thanks parhamn for pointing out the distinction._

~~~
parhamn
That doesn't seem fair. The word you quoted isn't even in the section you
provided. The use of "server" is interchangeable with daemon/isolated-process
here. That section seems like a decent explanation of daemon vs embedded for
the junior dev.

~~~
ankrgyl
Huh? It's literally in the "Serverless Database Engine" section. I can update
the quote to include the header.

~~~
parhamn
Even then, "twisting" is a stretch (especially the connoted maleficence). I
think most people would realize these are unrelated and there seems to be a
quite of precedence for talking about daemon-less as "serverless" [1].

[1] [https://sqlite.org/serverless.html](https://sqlite.org/serverless.html)

~~~
ankrgyl
Fair enough, the "classic serverless" vs "neo-serverless" distinction directly
addresses my point. I have added this clarification in my original point.

