
Ask HN: How to build something like redis purely in Node.js - kapv89
I want to build something like redis purely in nodejs for fun. The caveat is that I want the data structures to be &quot;distibutable&quot;. That is, I want this node-redis thingy to be horizontally scalable via adding more processes or servers. How can I go about this?<p>Thanks
======
tracker1
I would probably start by looking into hashing and sharding algorithms....
with memcached, this is done in the client, but you could just as well enable
a gossip-like protocol where the servers act as a relay client to other
servers... or do it at the client. In any case, if you look at some of the JS
clients for memcached, you can see how they are doing distribution...

Beyond that, for non-disk persistent keys, you can simple use an object in
memory, and have the keys be a property on that object. Avoid delete, set a
key to `undefined` on a delete command, you should probably expose the garbage
collector and force a collect on each delete operation, to keep GC cycles
small, and try to reduce memory (more on that below). you can do simple disk
persistence every N seconds via JSON.stringify (note, you may want to
partition objects in memory so that the persistence goes faster).

You may also want to look at some of the ZeroMQ bindings, as that could work
for your communication layering. Beyond that you may want to have a separate
shard key from your record key... this can keep multiple keys on the same
server, and reduce lookup times for related data, if you implement such a
thing.

It's worth noting that your whole process with die a fiery death if you hit
more than a couple GB of memory usage... node is a really bad environment for
this type of work under any significant load. Array searching, btrees, and the
like won't actually do any better in practice, it seems to be related to some
kind of reference handle allocation.

You can also use fork/cluster to run a localized cluster to minimize the per-
instance memory footprint.

Good luck, sounds like a fun project.

