
An update about Redis developments in 2019 - mykowebhn
http://antirez.com/news/126
======
stevekemp
@antirez: You continue to be an inspiration. Your words are thoughtful, your
thinking clear and well-documented, and despite rare lapses your English
skills are exceptional.

I use redis. I don't need ACLs, but I certainly don't disagree with your
assessment, nor begrudge the time, care and attention to detail which is
obvious in your design-work.

~~~
antirez
Thanks so much for the kind words! I'll try hard to find the balance in the
future before new, old and stable... Nobody wants a bloated Redis I bet! Since
I believe that many members of the Redis community have a similar aesthetic
mindset about software.

~~~
stevekemp
If you had the patience and available location you could publish an address
sometime in one of your blog-posts.

I bet you'd receive a hell of a lot of postcards of thanks from around the
world. Things like that feel more "real" than emails and words on forums.

~~~
leesalminen
I’d probably sent him some of our company swag. It’s not expensive stuff but I
love getting swag from my customers.

@antirez do you have dogs? I’ve got some branded leashes with your name on em!

~~~
antirez
No dogs just children :-D Thanks for your kind offer!

------
JesseObrien
>"I really want Redis to be a set of orthogonal data structures that the user
can put together, and not a set of _tools_ that are ready to use."

This is why I fell in love with Redis right from the beginning. Having a data
store that gets out of the way and just gives you the structures (a lot like
programming data structures) to do what you need feels _awesome_. Picking up
the commands and putting them to use feels as natural as reaching for any
vector/list/whatever structures I use anyway in my programs.

Thanks so much for everything you're doing antirez.

------
nicpottier
I've said it before and I'll say it again, thanks for Redis antirez. It is an
amazing piece of software in the utility it provides and I have used it in my
work for so long and just never ever had it fail on me. It does what is
written on the box every single time.

Thank you for the thoughtful design, for saying no to so many things and
keeping Redis pure and useful and for your awesome stewardship. You are an
inspiration in all those things!

------
Chris911
The ACL implementation:
[https://github.com/antirez/redis/blob/unstable/src/acl.c](https://github.com/antirez/redis/blob/unstable/src/acl.c)

~~~
Drdrdrq
Love the code and how it is explained! Beautiful.

Not a security expert, but in time_independent_strcmp(), first comment about
strlen()s: couldn't the attacker use his own accounts with known passwords to
determine the length of some other user's password? Also, given the name of
this function I would expect the comparison to be time independent, even if
attacker can change both strings' lengths... Or am I missing something?
Haven't touched C in a loooong time... :)

~~~
antirez
Hello, the attacker here can control only one string: we want the time taken
by the function to be independent from the POV of the string provided by the
user. That is, we don't want that the user-controlled string can affect the
time the function takes. The other string is the string set inside the
database. About information leaks about the length, that would be completely
acceptable: anyway the user is very likely to just it the user password length
by extracting a random number, there is no real protection there. The problem
of this kind of timing attacks is that it can leak the actual user string
_content_. Such function should hopefully prevent this problem.

~~~
cakoose
The comments in the code make it seem like absolutely no information about the
secret is leaked:

> /* Again the time of the following two copies is proportional to > * len(a)
> + len(b) so no info is leaked. */ > memcpy(bufa,a,alen); >
> memcpy(bufb,b,blen);

If the attacker controls one of the inputs, the execution time reveals
something about the length of the other input, right?

Or maybe you just meant that the length is leaked by the contents are not
leaked? (I agree that it's generally considered ok for "timing-safe equals"
functions to leak the length of the secret. But if you ARE allowed to leak the
length, you can simplify the code by just checking the length in the beginning
and exiting if they're not equal.)

And if you don't want to leak the length, it's easy: pre-SHA-512 the secret
and then only compare hashes instead of comparing the full strings.

------
sandGorgon
There seems to be a chronic need in the Redis community to avoid confrontation
(or even be bucketed) in the same space as Kafka.

Redis Streams is probably one of the most path-breaking things that Redis is
doing - and can completely change what it stands for. That's not a bad
thing..but it doesn't even get a title by itself but is rather called "data
structures".

Here's a question - do you anticipate competing with Kafka? There are a lot of
us that are cheering Redis on for this...but it seems there is internal
reluctance.

It's like the long standing "Redis is not memcached" statement - even
WordPress has a "choose Redis or memcached for object caching" option.

~~~
derefr
> It's like the long standing "Redis is not memcached" statement - even
> WordPress has a "choose Redis or memcached for object caching" option.

As was said in the post, _Redis_ isn't memcached, but "a bunch of Redis nodes
on the same machine together with a Redis Cluster proxy" _is_ kind of
equivalent to memcached.

Redis and memcached are on different _architectural levels_ —one Redis
instance represents one shard of a sharded data-store with an IO-concurrency
of r=1 w=1, while one memcached instance represents a complete data-store with
an IO-concurrency of r=N w=1. To build a caching "virtual appliance" for
deploying on some big hardware, a memcached appliance would just be a
memcached instance; while a Redis appliance would contain N Redis nodes + 1
Redis Cluster proxy.

It's a bit like comparing, say, UDP with TCP. There are features you can layer
on top of UDP to make it act equivalently to TCP (as e.g. QUIC does), but you
can also just use UDP by itself. UDP is not designed to be TCP or "compete"
with TCP; but UDP happens to be able to be used—when you add extra stuff on
top—for the same things TCP can be used for.

Redis, like UDP, is a flexible, low-level "infrastructure component" that can
be _customized into_ tools to solve any number of use-cases—but, like UDP,
Redis sets out to address only a _very limited_ set of use-cases when used as
a standalone "tool" without such customization. I personally think of Redis a
lot like an OS distribution—you could make any number of virtual appliances by
customizing e.g. Debian with your own extra packages and building the
resulting appliance-nodes into your desired architecture; and you can make any
number of data-store servers (solving any number of problems) by customizing
Redis with your own commands, and then connecting the resulting nodes into
your desired architecture.

Redis itself isn't going to compete directly with Kafka. Kafka, like
memcached, is on a higher architectural layer than Redis. But there's nothing
stopping some downstream developer from building something on that layer, e.g.
a "RedisMQ" server customized into a Kafka-killer.

~~~
sandGorgon
> _Redis itself isn 't going to compete directly with Kafka. But there's
> nothing stopping some downstream developer from building something_

So this is a philosophical rather than a technical thing. For a hobbyist open
source project, I would agree with you there - there's only so much resources
that someone can carve out of his day job.

For a company that raised a 60mil venture funding and specifically created a
license to prevent IP leakage, this is disappointing.

I want to pay Redis and I don't mind their licensing. But I'm then questioning
the philosophy if everything meaningful has to be done by unpaid open-source
developers/startups, who might be on tenuous footing to do anything meaningful
with it because of the licensing.

~~~
derefr
I think you're confusing Redis with Redis Labs.

Redis is built by antirez as a "hobbyist open source project." It is a small
core, and will stay that way. It is an infrastructure component. It is fully
open-source.

 _Redis Labs_ is the company that raised 60mil venture funding, and Redis
Labs' business model is exactly to create such "distributions" of Redis for
their customers. These distributions are not open-source; they are the thing
that has the weird license.

Redis Labs does not own or control Redis. antirez owns and controls Redis.

You could create a similar company of your own which is _also_ selling Redis
distributions (as a product, as a service, etc.) and make money off of doing
so. Since Redis itself—the thing antirez develops—is open-source, nothing
stops you from doing this, just like nothing stops Redis Labs from doing this.
No "unpaid open-source developers" are needed.

antirez just happens to _work for_ Redis Labs—but not in the sense that they
can really tell him what to do. They just want to pay him to ensure he keeps
making it, because their business depends on the continued health of the open-
source project at its core. (It's about the same situation as Guido van Rossum
working for Dropbox, or Yukihiro Matsumoto working for Heroku.)

If you want to talk about the well-being of some project, downstream of Redis,
to build a full-featured multipurpose DBMS "tool" with Redis at its core, that
is fully open-source... well, that project doesn't exist. Redis Labs is not(!)
that project; Redis Labs is a services company. But there's nothing stopping
such a project from existing, and there's nothing stopping a foundation
springing up around it to take care of it and its developers (like e.g. the
Linux Software Foundation, or the PostgreSQL EU+US foundations.)

But that software project, if it existed, would no more be "the Redis project"
than [https://openresty.org/en/](https://openresty.org/en/) is "the Nginx
project." It'd just be a customized distribution, by separate people.

------
didip
What is the current stance between Redis Streams vs Kafka? Serious question.

Because I prefer not to have both in production.

~~~
rainhacker
A key difference I observed was that if a Kafka consumer crashes, a rebalance
is triggered by kafka after which the remaining consumers seamlessly start
consuming the messages from the last committed offset of the failed consumer.

Whereas with Redis streams I had to write code in my application to
periodically poll and claim unacked messages pending for more then some
threshold time.

------
karmakaze
> I really believe that in the future of Redis “client side caching” will be a
> big thing. It’s the logical step in every scalable system.

As soon as this happens I will have no use for Redis. I already have better
scalable datastores. I often use Redis as a cache for those. If Redis isn't a
low-latency, consistent key-value/key-structure store anymore then in my mind
it isn't Redis anymore.

------
nerdbaggy
Interesting that coleifer wrote the comment. I really respect their open
source code. It’s no BS in their task cue program
[https://github.com/coleifer/huey](https://github.com/coleifer/huey)

------
rainhacker
> "I really believe that in the future of Redis “client side caching” will be
> a big thing. It’s the logical step in every scalable system."

How is this a logical step in the context of micro services ? I feel there is
a trade off. If I have to scale a service to N instances where each of them
cache the same type of data. Potentially, I'm duplicating that data N times.
Which means I need to reserve that much additional RAM per instance. I can
avoid this by storing the data centrally in a very fast cache aka Redis. On
the other hand even the fastest standalone cache will be much slower then
accessing data in process memory.

~~~
legohead
depends on what your application demands. I've worked on applications where
response times more than 3ms could cost the company thousands of dollars. And
I've worked on applications where 30ms and nobody would even notice...

~~~
antirez
I may add that, 99% of the very big Redis users out there do some form of
client side caching... at scale is basically something common for many years
now.

------
grezql
I have no valuable addition to the discussion, hope its okay but just want to
say thank you antirez for your impressive work. You're leading the way for us
others to follow

------
pdexter
Is there any more info on the redis cluster proxy?

~~~
antirez
Yet a lot to define, but it will be multi-threaded, written in C, in a
separated repository, but part of the Redis project conceptually. Please ask
any info you want (but not everything is defined yet...). In theory such
project is not needed because we at this point should have all the clients
with a great cluster implementation. Reality is a bit different :-) We need to
do something about that and allow developers to talk with the proxy like if it
was a single instance (but with the same multi-key operations limitations of
Redis Cluster itself, and the lack of SELECT to have multiple DBs), without
implementing any cluster protocol.

~~~
developer2
First of all, these two statements seem to be at odds:

> I/O threading is not going to happen in Redis AFAIK, because after much
> consideration I think it’s a lot of complexity without a good reason

vs.

> the way I want to scale Redis is by improving the support for multiple Redis
> instances to be executed in the same host, especially via Redis Cluster

Having to run multiple Redis instances, with a separate "Redis Cluster
[proxy]", sounds like "I didn't originally use threads, and while threads are
technically the correct solution (because it's the standard way to do things
and not all that complicated), I don't feel like rewriting Redis to use
threads the way it should have been done from the start; thus I've come up
with an alternative hack that is not ideal but makes my life easier". While I
have always argued that memcached is not _automatically_ better _solely_
because it is multi-threaded, the idea that an additional point of failure – a
technically unnecessary proxy/broker – is somehow a better solution than
threads seems like an attempt to save face for mistakes made during early
development.

As for the ACL:

> You get a library from the internet, and it looks to work well. Now why on
> the earth such library, that you don’t know line by line, should be able to
> call “FLUSHALL” and flush away your database instantly

What can I say other than "give me a break". Every large codebase has
hundreds/thousands of dependencies, and any one of them could slip in a
malicious block of code. My analysis: anyone who would be concerned about some
library calling FLUSHALL would never even install Redis - a C program that,
for all we know, is spamming emails or running a Bitcoin miner while it runs.
The fact is, all 3rd party software/dependencies come with risk; if I'm
trusting your software not to screw me while I sleep, I think I can trust a
basic library not to hijack my Redis server with a FLUSHALL command. Really,
this is not even remotely a valid argument. Whatsoever.

Then there's this:

> maybe you just hired a junior developer that is keeping calling “KEYS” on
> the Redis instance, while your company Redis policy is “No KEYS command”.

Ok, now you're just making shit up. Here's an idea: you don't need ACL to
restrict access to KEYS, as that command should have been deprecated and
entirely removed in favour of SCAN long ago. You mentioned "I think it’s a lot
of complexity without a good reason" regarding multi-threaded Redis; yet here
you are advocating an entire, complex ACL layer... for what purpose? For some
imaginary malicious scenario where my Redis library is going to call FLUSHALL,
or a "junior developer" is allowed to call "KEYS __* " – a command that should
not even be available in 2019?

I started this comment intending to provide unfiltered – but
polite/constructive – feedback. The more I reread your original post, the more
frustrated I got at the fact you seem to have lost all common sense. You're
trying to justify adding a massive amount of complexity, including a
clustering/proxy strategy instead of simple threading, and ACL to save us from
imaginary threats. I've previously been in the stage of development that I
believe you might be in now – you've run out of truly "good ideas" that would
actually benefit your user base, and are now attempting to justify new
features as necessary, when the truth is you're just looking to write new code
that you personally find to be "new and exciting", rather than iterating on
the existing, boring codebase.

I've spent the past 4-5 years advocating that memcached is effectively dead
because of Redis. The more you try to make Redis some kind of ACL-secured,
clustered database – rather than the lightweight cache store we learned to
love – the more likely another project will step in and replace Redis the same
way Redis replaced memcached.

------
fastbmk
TL;DR

* Clients will stay simple, RESP3 is backward compatible with RESP2

* ACLs are mostly an anti-fool protection.

* True multi-threading is impossibly hard to implement, but there will be some ad-hoc workarounds.

* Better persistence will be getting better.

* Existing data structures solve everything, so their number won't be extended.

* Read @antirez twitter to stay tuned.

------
blt
What is an ACL?

~~~
rjvs
Access Control List

[https://en.wikipedia.org/wiki/Access_control_list](https://en.wikipedia.org/wiki/Access_control_list)

------
munificent
This is completely tangential but it might help some people, especially those
where English isn't their first language:

 _> she/he (not sure)_

English is really strange around gender, and idiomatic styles have changed
over time. It has always been a valid style in English to use "they" to refer
to a single person of unknown gender [1]. This feels natural to English
speakers when the subject is unknown and could be one of many potential
people, as in:

 _When the first guest arrives at the party, give them a balloon._

Here, "them" only refers to a single person, but it's correct.

For the past hundred years or so, it was also common to use "he" to refer to
an unknown hypothetical person. You see a lot of textbooks that do this. In
theory, this wasn't supposed to come across as being exclusionary, but
obviously it is. It's an explicitly gendered pronoun.

The feminist movement rightly drew attention to this problem, and writers
experimented with a variety of approaches. Sometimes, at the beginning of a
book you'll see an explicit "apology" saying something like "we use 'he' but
don't mean it to only apply to males". Some simply switched to using "she" for
everything. Others switch between "he" and "she" throughout, randomly or use
the longer "he or she".

But, lately, it seems like the style is settling down towards simply using the
already established singular they for all of these cases. To a native speaker,
it feels a little weird at first, but you quickly get used to it. It's less
jarring than "he" or "she" for most readers now.

It has a lot going for it:

* It has more established history than any other form.

* It doesn't force the author to pick an arbitrary meaningless gender.

* It's shorter than the awkward "he or she" (which also still forces you to decide which gender to put _first_ ).

* It doesn't exclude people who prefer _neither_ "he" _nor_ "she" as their pronoun.

So, if you're trying to figure out what pronoun to use when you don't know a
specific one that is correct, default to "they".

[1]:
[https://en.wikipedia.org/wiki/Singular_they](https://en.wikipedia.org/wiki/Singular_they)

~~~
ummonk
Thought I just had: using just "he" is obviously exclusionary because it makes
males the default. Likewise, people don't just use "she" either. But given
that there are people whose preferred pronouns are "they/them", wouldn't that
make "they" equally exclusionary?

~~~
munificent
No, because singular they is a well-established idiom that conveys "unknown
gender", it subsumes people who actually prefer "they" as well as people who
prefer "he" and "she".

~~~
ummonk
"He" used to serve that purpose and subsumed people with all pronoun
preferences.

------
sjroot
Off-topic: this post, like other posts on the author's blog, is written in
Markdown. But rendered inside <pre> tags. Seems a little odd considering the
site also uses regular Serif fonts for the title / meta.

~~~
antirez
This is done explicitly, for the reader to have the same feeling as opening a
Markdown file with vim or Emacs.

~~~
jerrysievert
why not just deliver it as text/plain then? would save a little bit of
overhead.

~~~
antirez
The blog has the ability to embed images (ASCII graphs have their limits),
moreover I want to provide a readable font, spacing, ...

~~~
sjroot
I think you have to decide between providing a good web experience (readable
font/spacing/images) and making readers feel like they are in Vim/emacs.

That said, it was still a good read. Thank you for the time and effort you put
into Redis.

~~~
antirez
I agree, the current text is not very readable, but imagine it served raw as
txt... Btw soon or later I'll change it with something else indeed. Need to
find the time, also to convert the old posts.

------
chasers
This >> "An important thing to realize is that Redis has not a solid roadmap,
over the years I found that opportunistic development is a huge win over
having a roadmap. Something is demanded? I see the need? I’m in the mood to
code it? It’s the right moment because there are no other huge priorities?
There are a set of users that are helping the design process, giving hints,
ideas, testing stuff? It’s the right moment, let’s do it."

~~~
leesalminen
I’ve handled this “chaos” successfully for years. It starts to break down as
the company grows and matures. I used to be able to code something up and
explain it to 3 people so they could document it, support it, use it etc. Now
with ~30 it’s a lot more difficult and I upset folks when I deviate from the
roadmap.

------
jwr
I used to like Redis, because of its simplicity. But these days it is growing
so complex, with so many distributed algorithms being implemented, that I'd
much rather use FoundationDB, which at least has the benefit of being
extensively tested (in fact it was written test-first) and provides a full
distributed database with transactions.

I think there would be value in a "simple Redis": a well-maintained tool for a
certain class of problems which do not require a distributed database.

~~~
antirez
Redis is still very simple and most features are orthogonal and completely
isolated. Don't trust my words, check yourself, if you count the lines of code
with "cloc" in Redis unstable, it is composed of 64890 lines of code. In any
other system this amount of code is used, maybe, for the query parser. A
single developer can still read the Redis source code in a few days and
understand how it works and modify any part alone.

EDIT: I also want to stress on a fact, that at the same time in the newer
releases there was a simplification attempt. For instance the Lua scripting
side effects problem are going to be removed completely by simplifying how the
replication of Lua scripts is performed. Redis 5 is already like that, but the
dead code was not yet removed for safety. Redis 6 will completely remove many
parts of code. So there is not just the stress on adding, but also on
refactoring. ACLs themselves allowed to refactor authentication in separated
functions inside acl.c to lower the overall complexity.

~~~
OxidizedMetal
I'm surely not speaking for a minority when I say I couldn't disagree more
with the parent comment.

Redis is the most simple, rock-solid piece of software I have ever had the
pleasure of using. I find the comparison with FoundationDB jarring.

Don't let the negative comments like that sway your mindset; I think your
choices are spot on and the proof is in the pudding.

~~~
leevlad
To be honest, the top level comment in this chain reads a bit like an attempt
at a plug for FoundationDB than a legitimate criticism of Redis. You could
s/Redis/<literally any other DB>/g

