
Redis Loadable Modules System - djanowski
http://www.antirez.com/news/106
======
educar
The docs explain the module system
([https://github.com/antirez/redis/blob/unstable/src/modules/I...](https://github.com/antirez/redis/blob/unstable/src/modules/INTRO.md))
but it doesn't quiet explain what the system is for. I couldn't quite get that
from reading the blog post or the doc page. Specifically:

What kind of modules did you have in mind when designing this? Are any of the
below possible?

* ldap backend login system

* multi-tenant redis

* alternate db format/file system.

* implement a new redis data type. say built-in date support

* can i implement new commands and operators on said data types

An of course, thanks for this fantastic software. We use it everyday. Will
read in more detail about this over the weekend :-)

~~~
dvirsky
Actually the post explains the motivation well. The basic idea is to add
capabilities to redis at native speeds, without having to either convince
antirez that they're necessary or forking redis.

I work for Redis Labs, and over the course of the past few weeks we've
experimented with some modules using the API. Some of the things we did that
might answer some of your questions:

a PAM based authentication module, which IIRC can plug into LDAP potentially.

A bloom filter implementation on top of bitmaps.

An inverted index that doesn't use existing data types but rather creates
compressed indexes directly using redis strings (I wrote that one personally
BTW)

An API to store images on redis and retrieve them resized or cropped.

So right now the only way to create new data types is to piggyback strings as
raw memory, but the final version of the API will include actually registering
new data types.

You can see the modules we've released here:
[https://github.com/RedisLabsModules/](https://github.com/RedisLabsModules/)

~~~
vkjv
Maybe you can relay the information to the appropriate person, but on
[http://redismodules.com](http://redismodules.com), I can neither right click
"open in new tab" or ctrl+click to open in a new tab.

This breaks my normal browsing habit of opening new tabs as I see something
that interests me, but waiting until I finish skimming the list to read.

~~~
dvirsky
Thanks, will let the relevant person know.

------
astrodust
There's been a lot of unusual features I've been hoping Redis will support but
obviously given their edge-case nature it's unlikely that they'd ever be
implemented.

This is great to see. Instead of cluttering up the core engine with junk
people can mix in their own functionality.

------
aaw
This looks interesting. I've only glanced at the docs, but it looks like this
module system lets you define new commands but not necessarily new data
structures (since you'd have to implement RDB/AOF serialization for those).

Are there plans for that? It's probably possible to do most of that with this
module system but you'd have to serialize the data structure into a string and
"type X" wouldn't return your custom data structure type.

~~~
antirez
Definitely plans to register new data types with serialize/deserialize
callbacks! Thanks.

------
BadgerEmeritus
This is great and long awaited, please consider some kind of a namespacing
scheme for module-defined commands though, or things can easily become messy.
In the module example the "hello." prefix is specified for all defined
commands - it would probably be better to specify it once on module
initialization and have all commands defined by the module prefixed with it
automatically, the module registries could then manage the prefix
registration/allocation to module authors.

------
avar
Since Redis is BSD licensed I wouldn't be surprised if this is the start of
popular paid-for proprietary third party modules.

~~~
byroot
Even if redis was GPL licensed you still could do that. Linux is GPL and there
is proprietary driver modules.

~~~
ploxiln
There are proprietary modules for linux, and they are distributed and used,
but there isn't really a full consensus on whether they are legal. You can
certainly do it and get away with it though.

[http://yarchive.net/comp/linux/gpl_modules.html](http://yarchive.net/comp/linux/gpl_modules.html)

------
spullara
I once built something like this for Redis but never really thought he would
do it!

[https://github.com/spullara/redis/tree/plugins](https://github.com/spullara/redis/tree/plugins)

~~~
glenda
I've written something similar too. Adding plugins/modules was like a rite of
passage for anyone digging into the redis source. I'm definitely happy to see
modules getting official support now though.

~~~
spullara
I was blown away with how easy it was and with the code quality.

------
derefr
If these modules are going to be as fast as the native commands, is there any
reason to not lift the entire native command set out into a set of "base"
modules?

~~~
dvirsky
They're not going to be _as_ fast, but almost as fast. You could do that in
theory but IMHO there's not much point

------
shawn-butler
How people can code on airplanes amazes me. I have tried but the cramped
space, the constant interruptions.. I never get much done.

~~~
noir-york
It's not the constant interruptions (headphones do a good job of dealing with
that), its the lack of internet access that is the showstopper - I don't have
all the APIs + docs in my head.

~~~
swaroop
Use [https://kapeli.com/dash](https://kapeli.com/dash) on Mac or the open
source equivalents on other platforms.

~~~
avtar
[http://devdocs.io/offline](http://devdocs.io/offline) works well on Linux and
Windows.

------
akbar501
@antirez, is Redis 4.0 going to be part of the more rapid release schedule
(will we see it in 3, 6 months) or are major upgrades still annual releases
(we'll see 4.0 in 12 months)?

~~~
antirez
Definitely part of the short release cycle, we'll probably focus on getting
modules done and the features already pending into "unstable", and a few more
minor stuff, and release it.

------
manigandham
[https://github.com/RedisLabsModules/redex](https://github.com/RedisLabsModules/redex)

Shows some sample modules from Redis Labs called RedisEx, pretty useful stuff
(although most of these should really be in Redis by now but at least it's an
option to include as modules).

~~~
dvirsky
> most of these should really be in Redis by now

This exactly the point of the module system. You don't need to either convince
antirez to accept/implement a new feature, or fork redis if he doesn't agree.
It will make the dynamics of redis much more interesting IMO

------
maerF0x0
they also announced a couple of "hubs" for hosting/discovering these plugins,
plus a vetted one by redislabs

~~~
itamarhaber
Indeed - [http://modules.redis.io](http://modules.redis.io) will be the open
source place and the Redis Labs' Module Hub is at
[https://redismodules.com](https://redismodules.com)

------
evantahler
So who is adding node.js support?!

------
MasterScrat
Someone should really implement an interval module.

This custom implementation was pretty good: [http://blog.togo.io/how-
to/adding-interval-sets-to-redis/](http://blog.togo.io/how-to/adding-interval-
sets-to-redis/)

------
halayli
I bet a package manager will follow it. redis install moduleX and then we'll
have lib and target dependencies.

~~~
antirez
But everything will depend on the padding module.

~~~
halayli
and then the author will remove it one day. :)

------
lsiebert
Super excited to hear about this at Redis Conf, looking forward to
implementing a few ideas.

------
therein
Sounds exciting. Hopefully it won't be as under-appreciated as Apache Traffic
Server's plugin system.

------
awinter-py
databases (redis included) are awesome. It's always surprising to me that we
don't run web apps directly on the database; they're high-performance, durable
and parallel, and for some apps are already the perf bottleneck.

I think the only reason we don't is that deployment and testing isn't
straightforward. Fingers crossed antirez will get this right.

~~~
TheAceOfHearts
You might find CouchDB pretty interesting, then! [0]

[0] [https://couchdb.apache.org/](https://couchdb.apache.org/)

~~~
awinter-py
I know them for data sync. is there a stored proc deployment feature as well?

------
angelbob
I'm hoping there's an easy way to turn this off for the foreseeable future, to
avoid all the obvious security holes this is likely to introduce.

~~~
dvirsky
You don't have to use it, and you should not use modules you don't trust.

~~~
angelbob
In an ops-type capacity, I support developers. I keep up with news like this
_specifically_ to avoid winding up with modules somebody else trusts, but
didn't inspect.

~~~
dvirsky
What would be a must and should probably be a default, is not allowing to load
modules using the network API, only via config file.

------
MBCook
I tried to read the article but I couldn't. It's all one big <pre> block. The
word wrapping is wrong on input! Individual paragraphs are groups of
individual lines and extra long lines that the browser wraps for you. So you
end up with lines in the middle of paragraphs that only have one or two words.

Net result is it's very hard to read.

I tried Safari's reader mode, but that was even worse. Because all formatting
comes from happenstance in the <pre> tag reader mode loses _all_ formatting.
The entire article (and code!) becomes one long block of text.

I'm continually amazed at people's attempts to reinvent text layout in the
browser.

~~~
manigandham
Why is this downvoted? It's true that this blog has bad formatting considering
how simple it is, there's no reason to use _< pre>_ for the entire text.

Just because the author created Redis doesn't mean the site isn't badly
designed.

~~~
MBCook
I make these kind of comments every once in a while. They're usually
downvoted, I assume, for being off-topic or unnecessarily harsh ('We might
have cured cancer!' vs 'I can't read the article because of the video behind
the text.').

But I hope some people see them and work on the issue.

Sometimes, if enough other people agree, these kind of comments get voted up.
I guess this issue didn't annoy most people enough to get the up votes.

