
Redis Virtual Memory: the story and the code - antirez
http://antirez.com/post/redis-virtual-memory-story.html
======
va_coder
It's amazing what one super bright developer can do: DHH with Rails, Rich
Hickey with Clojure and Salvatore antirez Sanfilippo with Redis.

~~~
pierrefar
That's very true. I think two factors have helped in these kinds of cases:

1\. There wasn't a "boss" or someone dictating what they should do. Creativity
reigns and ideas simmer for a while to mature into good ideas.

2\. They have a mentality of "if I were to solve this problem from scratch,
what would I really need?" This question expands into "What is my wish list of
things that are hard to do now but me/everyone has asked for?"

~~~
kingkongreveng_
> solve this problem from scratch

I'm not detracting at all, but object databases with way more advanced feature
sets have been around for well over ten years
(<http://web.progress.com/en/objectstore/>). Making a stripped down version of
an object database is not exactly forging new ground. I know you weren't
necessarily implying otherwise.

~~~
antirez
Redis does not invent nothing new. It's 50 years ago technology mostly, at
work. VM, Data structures, and so forth. Even the language it is implemented
in, C, is old. But it's not an object database, and this is the point.

Databases are like languages. Does Ruby invented something new? No, but it's
my preferred high level language. It's a matter of "interpretation" of already
implemented concepts, not a matter of features set.

AFAIK Redis is one of the first DBs exposing basic computer science data
structures as abstract types. You have sets, linked lists, a sorted type that
is the equivalent of a balanced tree, and an dictionary (an hash table) on top
of this.

What matters is if this interpretation of already existing technologies fit
well with the needs of today. I'll leave the task of inventing new breaking
things to CS departments as I'm a programmer and not a scientist, with all the
limits and strengths that this implies.

As long as we, programmers, developers, hackers, computer entusiasts, don't
realize that our work is an interpretation work, like architects, we will
continue to be nerdy engineers that the society will continue to look at
without to recognize the real value.

Do you evaluate a building based on his number of rooms, the ability to resist
to forces, and so forth? I don't think so. Why you should use this meters for
software?

~~~
kingkongreveng_
Chill out man. I'm just pointing out this stuff has been around in commercial
products for a very long time.

> AFAIK Redis is one of the first DBs exposing basic computer science data
> structures as abstract types

With something like ObjectStore you've had persistent storage of arbitrary
templatized data structures for ages. You want a hash of hashes ultimately
pointing to a void* value? Easy.

~~~
antirez
Even with an SQL table you can build every data structure. Pointers to
pointers to pointers. But this is not the point(er) and I can't explain it any
better than in my previous message.

~~~
kingkongreveng_
You don't have to "build" the data structure. You are using your native
structures.

~~~
moe
As a recent redis convert let me put it differently:

Redis is a _tool_ in the unix-sense of the word. OOStore a _solution_ in the
enterprise sense.

Hackers tend to prefer tools, because they can combine them to build superior
solutions.

------
thomaspaine
The app we use for analytics currently works by logging events to the file
system and loading them into a mysql database every 5 minutes. I'm trying to
convince our sysadmin to let me switch the backend out to use redis, since the
only thing I'm using sql for is date filtering.

The main roadblock for him is redis' lack of support for automatic failover
(at least for the python client libraries I've seen), which I admit has some
validity to it. I don't think it's that big of an issue for our use case, but
try telling that to an opinionated sysadmin.

~~~
sedrik
If you have trouble with the Python client, please report a bug at:
<http://github.com/andymccurdy/redis-py/issues>

------
ehsanul
This is great, I was really hoping for Redis to take this direction, since
keeping everything in memory wasn't great for my current project.

In the same vein, I have an idea about forking memcached and turn it into
"memd". Instead of just caching for reads, it would allow writes to be done in
memory too, which are asynchronously synced to disk when an item expires, or
based on certain conditions (I'm thinking based on both time since last
updated, and when enough updates are made). Now obviously, this only works if
the data you store isn't all that important, which is probably the case for
most web applications.

~~~
leej
and that's called Redis

~~~
ehsanul
Well, the idea preceded Redis and applies more widely. The advantage of having
it the way I mentioned is that any database could be plugged in at the back,
just as memcached as no requirements for what method of persistence is used.
That makes it attractive for other reasons.

------
radu_floricica
I can't help but wish for a similar sql-solution. In-memory, with writes
journaled and close enough to mysql that porting wouldn't be a nightmare.
<sigh> one can only dream...

~~~
kingkongreveng_
There are absolutely super performant in memory sql databases. You just have
to pay for them.

There's also csql. <http://en.wikipedia.org/wiki/CSQL>

~~~
radu_floricica
Thanks!

------
kevingadd
I'm disappointed that he didn't try non-blocking IO. I suspect it would have
ended up simpler than the threading approach he went with.

Still a very cool piece of work, though.

~~~
tptacek
Nonblocking disk IO is a pain compared to nonblocking network, and redis
compiles with minimal deps.

------
w3matter
Well worth the read to learn the design decisions and the capabilities that
are opened up with Redis by keeping just the keys and least used data in
memory.

Antirez is my hero of the week :-)

