
LokiJS – In-memory JavaScript Datastore with Persistence - joeminichino
http://lokijs.org
======
andrewaylett
While "Performance over everything" is obviously hyperbole, I hope they really
don't mean it and actually prioritise correctness: that it apparently does
actually store and retrieve data suggests that at some level, at least, they
do.

~~~
marknadal
(Full disclosure, I work at a database company) Their performance is pretty
impressive, but you can get over 30X their speed (30M ops/sec) by caching -
see [https://github.com/amark/gun/wiki/100000-ops-sec-in-
IE6-on-2...](https://github.com/amark/gun/wiki/100000-ops-sec-in-IE6-on-2GB-
Atom-CPU) .

And this can be done without compromising the storage or integrity of the
data. The Observable pattern lets you update caches while you perform writes
so the data stays correct, reflecting what is on disk. If caches don't exist
then you pull from disk (slow) and write to the caches.

~~~
joeminichino
this is actually something we are playing with :) we have a couple of drafts
in the pipe but you are welcome to bring some wisdom over to LokiJS - it's
free for all to use and for all to contribute, even if it's just on a
theoretical level. Cheers

~~~
marknadal
Nice, very excited to hear!

Honestly the wisdom I have has just been playing around with benchmark.js on
V8 and noticing the Chrome team doesn't do as they claim (although latest
Chrome Canary we saw major improvements), while FireFox is a different beast.
You've probably done about just as much of this as me.

What we found was that every function call in javascript significantly
decreased performance (even if it did nothing) at a quazi logarithmic rate. So
on a theoretical level if you're able to reuse function results that are
deterministic you can save a ton of overhead. However IDK how micro-optimizing
you want to get as it can sacrifice code legibility.

Would love to chat more if you have time, I'll see if I can find your email
and send you a message. Mine is (no spaces): a q u i v a @ google's email
provider.

------
egeozcan
> In-Browser NoSQL db with syncing and persisting

Is the syncing part as straightforward as Pouch[1]? I couldn't find anything
about syncing on its docs. If a db comes with an easy offline-first and sync-
when-possible model with granular permissions, it will be very interesting.
Current solutions have (IMHO unacceptable) fixed dependencies on the backend
or the architecture.

[1]: [https://pouchdb.com/](https://pouchdb.com/)

~~~
k__
I always wondered what the use-cases for PouchDB are.

Who wants to replicate their "whole" database locally?

~~~
egeozcan
You can filter what you are syncing or create a db per user and declare
interactions with foreign data in their own instance, which will be executed
according to permissions in the backend.

What I actually want is MeteorJS with neither MongoDB or the JavaScript
backend, or PouchDB without Couch, or Jaydata + ASP.NET WebAPI with more open
source compatibility...

~~~
k__
Have you tried Horizon?

~~~
egeozcan
Yes, but it's still WIP and you need to go all-in on JavaScript IIRC. I don't
dislike JavaScript but would like to have options.

------
thinkloop
V8 has some quirky memory mechanics. Object pointers, and similar, are subject
to a vague hard limit, loosely defined by `--max-old-space-size` and other
related flags, while the objects themselves can extend beyond it. For example,
if you had 1 object that contained 15gb text, that would likely be OK, however
if you had 1 billion objects, storing 1 character each, that would likely lead
to a crash - even if the process memory is below limits. Relatedly the shape
and nature of the data matters (nesting, types, etc.), limiting the ability to
use the full memory available in various ways. I imagine a lot of it has to do
with GC, but I don't think that's the full story.

Have you noticed such quirks building loki? Do you imagine being able to make
full use of, say, a 64gb ram server? Is this something being taken into
account and optimized in the project somehow? @joeminichino

------
formula1
What I find fascinating about a tool like this is how simple it is. An in
memory store is not very important to me and what I develop. And yet, Im very
happy this exists so that it sets a standard for other work in the future.

------
swalsh
Awesome, we're now 1 bored developer weekend away from Mumps.JS

~~~
ch4s3
Probably M.JS for clarity ;) /s

------
ldehaan
I used this on a internal project for a little while, it's easy to use, and
the promises integration is useful, however at high volumes it did very
poorly.

I was storing time series data.

I ended up moving to redis which has worked very well as long as I make sure
it has enough memory for the date range I am keeping.

I hope this project keeps at it and gets a handle on performance under heavy
load.

~~~
joeminichino
hey @ldehaan i'd love to hear more about the shortcomings of Loki, i never had
a real-life need for really high volumes so I would greatly appreciate some
insights in order to improve it.

------
finchisko
After 5min of introduction video I had to stop. Guy explains why he started
loki. Hi didn't want to use sqlite in cordova app, so he created loki. Did he
heard about localStorage or indexeddb? Probably not.

------
nilved
This website would actually be better if they removed all the CSS and images.

------
code_research
can I sync this with data in a postgresql db?

