
Redis 6.0 GA - ingve
http://antirez.com/news/132
======
coleifer
Contrarian view here.

I really liked Redis for a long time. Simple, fast data-structures in-memory.
That's it. Along the way there have been some nice enhancements like Lua,
which solves a lot of the atomicity issues. But somewhere after 4.0 I feel
they have lost their way. First they saw all the attention Kafka/event-stuff
was receiving, so they baked-in this monstrous, complicated, stateful streams
feature. Now we have SSL (do people really expose Redis on the internet??),
ACLs, cluster stuff, and most relevant to me a new wire protocol.

To my thinking, Redis fit very well in the "lightweight linux thing" category.
It seems they aspire to be enterprise software, and this may be a good move
for Redis-the-Business, but it's not good for users like me who just want
simple in-memory data-structures and as little state as possible. Forcing a
new protocol that adds very little value (in my opinion) also seems like a
great way to alienate your users.

~~~
antirez
I understand the sentiment, but things are a bit different than they may look.
About SSL, there is no way out of this. I opposed to this feature for a long
time, but simply now because of changes in regulations, policies and so forth,
a lot of use cases are migrating to SSL _internally_ even if Redis is not
exposed. And frankly it is really a mess to handle SSL proxies now that
_everybody_ look like needs encryption. So what I did was the best I could do,
when checking for PRs to merge: 1) Opt in, not compiled by default, no SSL
libs requirements. 2) Connection abstraction, there is no SSL mentioned inside
the code. Everything is in a different file.

About the "Kafka" thing, actually streams were wanted by myself, very
strongly, and not suggested by Redis Labs. Let's start thinking at Redis ad a
data structure server and at streams without the consumer groups part (which
is totally optional). It was incredible we had no way to model a "log" in a
serious way. No time series easily, using hacks and with huge memory usage
because sorted sets are not the solution for this problem. But then why
consumer groups? Because for a long time people had this problem of wanting a
"persistent Pub/Sub": you can't lose messages just because clients disconnect
in most use cases. Btw this Kafka monster is a total of 2775 lines of code,
including comments. 1731 lines of code without comments. In other systems this
is the prelude in the copyright notice.

But ACLs, in order to manage to survive 10 years without ACLs we had to resort
to all kind of tricks: renaming commands to unguessable strings. Still with
the panic of some library calling FLUSHALL for error because the developer was
testing it in her/his laptop. Really ACLs have nothing to do with enterprise,
but some safety is needed. The ACL monster is 1297 lines of code, and is one
of the most user friendly security system you'll find in a database.

Actually all those features have a great impact on the users, huge impact on
day to day operations, and are designed in order to be as simple as possible.
And Redis Labs actually has only to lose from all this, because those were all
potential "premium" features, instead now they are in and every other Redis
provider will have it automatically as a standard. So... reality is a bit
different, and it's not a conspiracy to gain market shares or alike.

~~~
tedivm
My company has no choice- we have to use ssl internally for regulatory
purposes. Right now we're using an stunnel solution for having out clients
connect to redis- I am super excited that I'll be able to remove this
workaround in the future!

~~~
olavgg
There is a software named Hitch
[https://github.com/varnish/hitch](https://github.com/varnish/hitch) that is
super useful for enabling SSL to different services, like Redis.

~~~
stephenr
Putting the server behind TLS is a minor part of the process.

If you want any kind of HA, you'll have multiple instances of Redis, with
changes replicated from the writable node to the others.

That traffic needs to be encrypted too - and redis (pre 6.0) knows nothing
about TLS.

So now you need a tunnel to each other Redis node.

Oh but you also want Sentinel to make sure a failure means a new primary node
is elected... and sentinel doesn't speak TLS _either_ , and they need to both
speak to each other, and the redis nodes... so that's another set of TLS
tunnels you need to setup.

I setup redis on 3 nodes for a customer, if you tried to draw the stunnel
setup on paper, it'd look like you're illustrating a plate of spaghetti.

------
dudus
Thanks antirez for all the work you've done on redis. Personally I can say
that no one had such an impact on my work as you did. Reading through your
source code is educating and inspiring.

You are a hero.

~~~
antirez
Thanks, to me it means a lot that somebody out there perceives my work as
something of positive :-). Live long and prosper.

~~~
whalesalad
Dude... I do not think you know truly how many people love Redis. For every
contrarian armchair quarterback here on HN there are 1000 people who are out
using/coding/hacking w/ Redis and not caring about the noise. Myself included.
Thank you and all the contributors for the hard work on an absolutely killer
tool.

------
Dowwie
"Redis 6 is the biggest release of Redis _ever_ , so even if it is stable,
handle it with care, test it for your workload before putting it in
production. We never saw big issues so far, but make sure to be careful. "

Congratulations, team!

------
alufers
I have a bit of a tangent question for more experienced back-end developers:
where do you fit Redis (or other caching mechanisms) in the traditional MVC
model? I haven't had a use-case for Redis yet, but I'd like to know how should
I approach the architecture of an app using Redis.

~~~
vorpalhex
Redis can be a very powerful tool, it can also be a sign that something has
gone wrong.

Redis is a high performance key/value store with adjustable consistency. It
can be used as a caching layer, and it can also do a solid job of representing
a basic message queue. It typically fits in on the data layer to store
computed results for fast retrieval, but it can also behave like a read
replica (storing data from outside your domain).

That being said, when Redis becomes a critical part of your stack for
performance reasons it usually means something has gone wrong in your data
layer. I often times see teams use an antipattern of taking data from their
database, changing it's shape to what they want, and then storing that new
model in redis. If you're the only consumer of your data, then your data model
is incorrect and you're using Redis to paper over that fact.

~~~
takeda
Agree with you, that's bad, there are also places which use redis as a primary
data store. That's actually messed up.

~~~
A-Train
Is this that bad? At our company we are using redis for real time
recommendations. And this is our primary database for storing accumulated
statistics about items and users. It has been important to us to have a quick
way to share the data between the workers and we chose redis.

~~~
jakearmitage
You're fine. The simplistic interface that Redis provides to sets, and
therefore empowering pragmatic recommendation engines, is the perfect tool for
this job.

------
jrochkind1
> Threaded I/O

Woah, I missed that redis changed their mind about this. Clearly haven't been
paying attention.

~~~
antirez
It didn't, but we identified something that could be done without throwing
away our vision. Basically Redis remains completely single threaded, but once
is about to re-enter the event loop, when we have to write replies to all the
clients, we use threads just for a moment in that code path, and return single
threaded immediately after. It is not as performant as full threading, but we
believe the right thing to do is sharding and share-nothing. Yet this 2x
improvement for the single instance was almost for free so... btw it's
disabled by default.

~~~
ksec
>btw it's disabled by default

Why is that?

Thanks for the explanation on Threaded I/O, I was about to ask the difference
between Redis 6 and KeyDB.

~~~
jdsully
KeyDB multithreads the entire event loop from end to end. This provides better
performance [1], and combined with MVCC will put us in a place to support more
expensive commands like searches without blocking other clients.

1\. [https://docs.keydb.dev/blog/2020/04/15/blog-
post/](https://docs.keydb.dev/blog/2020/04/15/blog-post/)

------
kawsper
Congratulations to the Redis team, it's always nice to see improvements to the
cluster story and tooling.

------
maxpert
Threaded IO, Server assisted client caching, and Redis proxy in my opinion is
an important milestones. Threaded IO will fully utilize multiple cores; and
improvements are amazing. Server assisted caching will reduce client round-
trips. Redis proxy will remove the debate of which client library to use,
Jedis for example has not allowed reading from replicas or doing multiplexing
distributed commands; proxy is gonna solve all those gripes. I can't wait for
it to be stable and production ready.

------
elpatoisthebest
This is great timing. I was just spinning up a new service using redis this
morning. Thanks for the constant work you put into this project.

------
janpot
One caching pattern I find myself doing a lot on Redis is where multiple
clients try to access the same cached value, but if it doesn't exist only one
is allowed to revalidate it, all other clients wait until it's revalidated and
get notified with the new value. Currently for the clients it involves:

1\. subscribe for updates on the key you're trying to retrieve 2\. retrieve
cached value or set a lock 3\. if lock acquired, unsubscribe, then fetch value
4\. set the result in a key and publish event with the value

I wish there was a command that does: retrieve value of a key, if it doesn't
exist lock the key and notify me with an id, if it's locked subscribe for the
next update on the key. With a second command that acknowledges the lock with
a new value

Now I know this can be implemented and I've done so multiple times. It's just
that it's tricky to get right and consistent.

------
DethNinja
RESP3 looks very interesting, unfortunately I made the mistake of writing my
own RESP2 protocol and now I don’t have time to upgrade to RESP3. I guess I’ll
move to some third party C++ client library.

~~~
antirez
RESP3 is a strict superset of RESP2! You can just add to your implementation
and end with RESP3 finally :-) It means also that a RESP3 parser can parse any
RESP2 reply without issues, in case you use it against a RESP2 server.

~~~
DethNinja
Oh that is great to know, because there was a very specific reason why I
implemented my own RESP2 client that was not possible with other libraries.

~~~
skrebbel
out of curiosity, what is that reason?

~~~
DethNinja
I need to pipeline around 100000 commands every 30 seconds as a transaction.
So if any of the 100000 commands fail due to database failure, then I need to
crash the server and restart.

That is the easy part though, problem is these commands are coming from 12
different cores with possibly that cores will increase to 24 on future. One
core might issue 5000 commands, other core might issue 20000 commands, there
is no possibility to predict how many commands will come from a specific
thread. It is extremely important that this is done with highest performance
possible otherwise my system will have bunch of other issues with catching up
on other non-db related client-side commands.

So I devised a special Redis pipeline structure where each of the cores can
issue commands without any kind of mutex during the course of a tick (server
is tick based), mutexes for the threads are locked only at the end of the tick
to compile different pipeline commands coming from different threads into one
giant pipeline command along with their asynchronous callbacks.

So practically I'm utilizing a giant redis pipeline with 12 threads on Redis
client side where I'm locking only 12 mutexes at the end of the tick, this is
extremely high performance as far as letting server process other non-db
related client requests while all that data is getting written to server.

Truth to be told it is an over-engineered crap and I guess it might be
possible to implement something similar with other redis client libraries but
this is something I already build so I would like to keep it as it is.

~~~
rakoo
I know you've probably had a look at it and decided it wasn't worth it, but
could it be possible to generate a LUA script on the client side and EVAL it
once it's completely generated ?

You'd need to periodically SCRIPT FLUSH to not let old scripts linger

~~~
DethNinja
I think it can be done through LUA, or even just by using "MULTI" command but
I would still have to handle asynchronous replies on the client-side where it
would get hairy regardless.

In any case I'm fairly certain there are multiple ways to solve such problems,
normally I wouldn't even bother writing an in-house database client library
but RESP is so amazing and simple that it doesn't take much time build a
client library for Redis.

------
unrequited
Congratulations @antirez and the team! Can't wait to play with it.

------
munawwar
Having client cache a subset of data is very much needed for the most
frequently accessed data.. but it seems the various clients will yet have to
implement this? (it could have been the best feature in the list)

Previously I had to implement this myself ->
[https://github.com/Munawwar/redicache](https://github.com/Munawwar/redicache)

------
pachico
We use it also for queueing messages to be consumed by logstash. Not as
pub/sub but regular list and has never failed.

~~~
philjackson
Redis is a workhorse.

------
stephenr
Great work. I’ve been quite critical of Redis for its lack of TLS before.

This is good to see. Well done!

------
dirtydroog
Why does redis recommend to not use huge tables? I'd like to experiment with
using DPDK with it, but that requires huge tables

~~~
antirez
Because when Redis forks to persist, every write will copy a 2MB huge page,
and that's not good. All the memory will end to be copy-on-write-ed: latency
spikes, big memory usage, ...

------
Tim25659
What is ACL? Can anybody please explain

~~~
Tim25659
Looks like content is not available

~~~
antirez
Sorry, link fixed now.

------
uhoh-itsmaciek
>So what changed between RC1 and today, other than stability?

>...

That's quite the list of changes after a release candidate...

~~~
antirez
Yep I use a non confirming release cycle and even actively backport new
features form development branches to RCs or patchlevel releases if they are
safe and don't have interactions with other code. This is done to make things
available faster. Other times new features are almost like fixes because the
old alternative resulted in user pain.

------
Tim25659
Congrats antirez and redis labs!!

------
sandGorgon
Noob question - is redis streams completely able to replace the Kafka usecase?

~~~
itamarhaber
Nope - similar concept, different stories.

------
Thaxll
It's actually a big release with a lot of changes!

------
powerslacker
> 6\. RDB files are now faster to load. You can expect a 20/30% improvement,
> depending on the file actual composition (larger or smaller values). INFO is
> also faster now when there are many clients connected, this was a long time
> problem that now is finally gone.

Talk about humble bragging.

