
Forking Redis: SSL/TLS and Transactions - davidgl
http://www.dr-josiah.com/2018/04/yes-im-forking-redis-ssltls-transactions.html
======
antirez
I want to give just a few info about this:

1\. SSL is going to be merged in the next days, I'm in this moment here in SF
in the same room with the patch author, she made a wonderful work and the
patch is "opt in", simple, and good, so this is going to happen.

2\. Forking is a right that open source gives you. Have fun coding Josiah!

3\. I really will push for any fork not being called Redis. I care a lot about
the details, the stability, the quality and the ideas in my project, and I
really want that when people say "Redis", it's Redis, and not a Redis fork.

4\. I'll be a bit disappointed if the license of the fork is so that it will
be possible to merge my(1) BSD code but I will not be able to merge back
potential change. It's legal, it's possible to do for this approach, but it
IMHO ethically speaking sucks.

(1) my as mine and a number of other contribs.

~~~
simcop2387
As far as 3 goes, the Perl5 community tends to deal with that with software
licensing. I doubt it's possible to do that with Redis at this point, but the
Artistic 2.0 license (Perl5 comes under both GPL and Artistic 2.0, with the
choice between them) lets you make forks of the project and call it something
else without having to release those changes. I'd definitely agree about 4, at
least except where he talks about the cluster stuff, it sounds like there's
more murky waters there than I know about.

[https://opensource.org/licenses/Artistic-2.0](https://opensource.org/licenses/Artistic-2.0)

~~~
DrJosiah
Murky waters with cluster? No. Just being concerned about corporate liability
/ risk. I might have spent too long working under business folks, as some
companies won't consider using open-source software unless there is a vendor
they can call at 3AM and complain at. I'm saying that if someone needs
distributed transactions in cluster, and they want to run my software, I would
take that call and help them fix it.

------
piquadrat
Just a few days ago, AWS announced[1] that they will be contributing TLS to
Redis, and judging by one of his comments[2], antirez doesn't seem to be
completely against merging it.

According to a comment on Reddit[3], Josiah wasn't aware of this effort by
AWS.

[1]: [https://aws.amazon.com/cn/blogs/opensource/open-sourcing-
enc...](https://aws.amazon.com/cn/blogs/opensource/open-sourcing-encryption-
in-transit-redis/)

[2]:
[https://github.com/antirez/redis/pull/4855#issuecomment-3831...](https://github.com/antirez/redis/pull/4855#issuecomment-383171007)

[3]:
[https://www.reddit.com/r/redis/comments/8f373k/a_new_redis_f...](https://www.reddit.com/r/redis/comments/8f373k/a_new_redis_fork_with_ssl_transactions/dy19br1/)

~~~
fapjacks
Nice! At first blush, the TLS PR looks pretty good. It's not just shimmed in
there like it could be (for example if I'd have written it haha). I've always
appreciated Redis for having generally excellent code quality over the years.
Looks like a great contribution to the project!

~~~
DrJosiah
It is better than the code I actually wrote, I've already switched to it as a
base, and have been merging my changes in to my local branch today. :)

I look forward to everyone having SSL :)

------
decebalus1
But... why? SSL will eventually make it into the official Redis release.

Anyway, I don't think anyone would use a fork of Redis just because of SSL. I
would take the perf hit and go with socket tunneling just so that I don't have
to deal with the maintenance burden of depending on a fork.

~~~
originalsimba
> But... why? SSL will eventually make it into the official Redis release.

"Eventually" it's 2018. It's too late.

I'm especially cynical and jaded about the insanity of running any plaintext
protocols in 2018, why wasn't SSL/TLS implemented by now? It should have been
at the top of the priority list.

People who think unencrypted traffic is okay should have their competency
challenged. Sure, in very limited specific deployment models you can maybe get
away with it without compromising security, but actually even in the most
restrictive networks it is still an open door for exploitation.

It's the same kind of stupidity as Docker running as root by default... this
stuff is amateur hour and these apps are mission critical enterprise tools.
That's a bad recipe. In the 90s it was forgiveable, today it is not.

Kudos to the guy. Maybe it'll light a fire under the official project. Or
maybe the fork will pick up steam. It'll be interesting to see what happens.

~~~
tptacek
This is reductive. I don't know what the faction of major Redis deployments
(by transaction volume or data size or however you want to measure) that are
housed in AWS is, but I'm guessing it's north of 50%. In an AWS Redis
environment, running Redis over TLS is not in the top 10 systems security
concerns relative to database security. The access you need to attack Redis
for lack of TLS is almost certainly gameover by itself with or without TLS.

I'm not saying Redis TLS is a bad thing (honestly: I'd prioritize "not having
Redis in a design" over "making sure Redis runs over TLS"), but I think in a
lot of very mainstream designs, it's mostly a cosmetic security control.

~~~
aeorgnoieang
> I'd prioritize "not having Redis in a design"

Why?

~~~
tptacek
Not in general, just over optimizing the transport security of Redis.

~~~
DrJosiah
In one example test I ran, the SSL/TLS unwrapping daemon used 5x the CPU of
Redis itself. And generally, the unwrapping daemons always used more CPU (for
the reasons explained in the article). With integrated SSL, you're looking at
substantially reduced CPU overhead, just from reduced interrupt handling and
memcpy operations.

For the examples I was running, it's like going from 2 hours of total CPU time
to 50 minutes, versus 40 minutes for no SSL/TLS. Or going from 1 hour to run
the test to 45 minutes, versus 35 minutes with no SSL/TLS.

------
gtrubetskoy
I forked Redis years ago providing threads and SQL support, though I can't
remember if transactions were supported now. This was done mostly for fun and
as an experiment, learned a lot about locking and database internals in the
process:

Main concept: [http://thredis.org/](http://thredis.org/)

Implementation details: [https://github.com/grisha/thredis/blob/master/README-
THREDIS](https://github.com/grisha/thredis/blob/master/README-THREDIS)

~~~
coleifer
I read your code a couple years ago after finding your repo through a search
for Redis and SQLite - what a fun project! Learned a good deal from your
changes, so thanks for open-sourcing your experiment.

------
doug1001
i love redis and have been using in production for over five years. But i'm
interested to read the story of this fork because the practicioner-focused
redis book this guy wrote is one of the best software-related texts i've read
in the past 7 - 8 years. ("Statistical Rethinking" by Richard McElreath, and
"The Art of SQL" by Stephane Faroult are two others that had the same impact)

~~~
DrJosiah
Thank you very much, that is the best review I've ever received on my book.

As for the story; the clues are there in public if you dig a bit, but aren't
all that interesting to anyone.

------
calebio
Did I miss something here around the rationale for forking vs. attempting to
contribute this to the official project?

Edit: found some more details over on reddit[0]:

 _I didn 't see the existing TLS PR, and I'm not finding it now. Do you have a
link?

As for why fork and not PR, Salvatore already closed the Transactions PR and
said he didn't want Redis to go in that direction. And when searching about
SSL/TLS in Redis itself, I found this:
[https://redis.io/topics/encryption](https://redis.io/topics/encryption) ,
read the implementation of spiped (it uses fixed 1k block sizes), then
realized that SSL/TLS is the right answer in this situation.

Could transactions be a module? I was about halfway through the cluster
transaction bits as a module when I hit a collection of "oh wait, I can't even
call this entire class of things unless I create new module wrappers for both
directions" problems. Then I just added a new .c file, new .h, did the right
includes, a make clean && make, and my life was 10x better.

Also, this just includes redis-benchmark, redis with SSL/TLS, etc., is still a
couple weeks out. I need to get redis-cli > and redis-sentinel speaking
SSL/TLS._

[0] -
[https://www.reddit.com/r/redis/comments/8f373k/a_new_redis_f...](https://www.reddit.com/r/redis/comments/8f373k/a_new_redis_fork_with_ssl_transactions/)

~~~
DrJosiah
Other changes that I had been working on weren't going to be merged upstream
(already rejected), and though I wasn't going to make any attempts to prevent
SSL/TLS from being merge-able (I was actually starting from 4.0.9 for ease of
current users, and had moved between 4.0.9 and unstable a couple different
times), my code was derived from MIT code, and I can't really re-license
someone else's code.

This fork announcement was a, "Hey, you can probably use this stuff, but it
might not be usable for everyone now. In the future, it might be a materially
better solution for some of your problems (transactions). I'd appreciate some
sort of vote of confidence if you all think it's cool, or even an 'eff off' if
I've somehow pissed in your coffee."

With the SSL work in PR #4855, some of the announcement has no real use now,
and I've redirected my own efforts in its wake.

~~~
calebio
I appreciate the followup and reply!

------
t0mbstone
Very cool! Any reason why this wouldn't make its way into the official
releases of redis?

~~~
tedmiston
Presumably they'll have to sort out what kind of performance hit it has and
how to address that longterm, but I think you're correct.

It's probably more complicated than that given that the Redis security page
begins with:

> Redis is designed to be accessed by trusted clients inside trusted
> environments.

[https://redis.io/topics/security](https://redis.io/topics/security)

I think the current recommended approach for this use case is to wrap it with
stunnel.

~~~
DrJosiah
Please re-read. It's spiped. Fixed 1k blocks, but otherwise reasonable.

But #4855 is going to be merged, and my tests show that #4855 is actually
faster than my changes, so ... :)

I like not having to be responsible for writing SSL/TLS code, even if I just
spent 9 weeks doing it. :P

------
mythz
Not expecting this to see much traction given there's already an open PR to
add SSL to Redis (from an AWS employee):

[https://github.com/antirez/redis/pull/4855](https://github.com/antirez/redis/pull/4855)

~~~
DrJosiah
I'm about 99% sure that this will get about < 10 downloads. Heck, I've already
switched to PR #4855 code, and am applying my other changes forward on top of
it today. I've also got a followup post where I talk about the next steps I'm
taking.

Good code is good code.

------
enz
From the article:

> And as a basic need, relying on third party tools for SSL/TLS termination or
> a transparent VPN solution is a great first step from running without
> encryption, but it can leave speed on the table. And part of the reason why
> we use Redis is for speed, right?

> With 3rd party SSL/TLS termination, that can only get worse. How much worse?

I'm curious to see what the results could be with HAProxy as a SSL termination
frontend. (via a Unix socket or a TCP connection)

~~~
DrJosiah
As I was going through setting up HAProxy, I ran into a roadblock that stopped
me from testing. I will look into it for the next round.

------
salmo
Meh. A short-lived fork addressing issues being addressed in mainstream redis
in a way that is probably more consistent with the codebase.

I think it's an interesting piece of work, no doubt. But I'd never adopt
something like this in Real Life. Given the gorgeous state of the codebase, I
trust the current stewardship of the project to make the right calls. This
sounds more personal than technical.

~~~
DrJosiah
This specific fork, yes (has to be renamed). Another fork, perhaps not. Per
other threads, I've got more work going in for transactions and other
features, some of which Salvatore wasn't even interested in hearing about, let
alone merging.

Feel free to keep using Salvatore's Redis, it'll be around as long as he is,
I'm sure.

And feel free to take a look (or not) at my fork in a few weeks. Maybe I'll
disappoint you again. :)

------
ryanmccullagh
I watched about 10 seconds of the video. I stopped after that because the guy
was still talking about himself, not the software.

~~~
fapjacks
You mean Dr. Josiah Carlson, PhD?

~~~
needz
You ask that as if it means something

~~~
scandox
Poe's Law I think...and for the avoidance of doubt I think he was being funny.

------
iampims
Sounds like it adds native SSL termination for redis-server and benchmarks
show its much faster than external ssl termination.

------
gingerlime
Anyone knows a redis cli client that supports rediss/TLS? Tunneling the server
is one thing, but I do miss the convenience of redis-cli and tunneling the
client is a bit of a nuisance.

------
make3
why not just submit patches to redis, it feels like something a lot of people
would benefit from.

~~~
DrJosiah
Because up until I posted, this was the official party line as far as I had
been made aware:
[https://redis.io/topics/encryption](https://redis.io/topics/encryption)

And this was the start to several other changes I am in the process of adding,
which Salvatore had historically not been interested in. The patches I'd been
working on for SSL/TLS were small enough to be easily cherry-picked, at any
point in the future, if Salvatore had wanted them.

It was easier to release some SSL/TLS bits to start, than to do everything
else I need to do before adding SSL/TLS.

------
aphextron
Why does Redis need SSL/TLS? It's meant to be used as a backend service across
trusted networks. And if you need to use it over untrusted networks, you're
best off using an actual server like NGINX or HAProxy to mediate it. Not to
mention the killer feature of Redis is not neccessarily throughput, but it's
best-in-class latency, which could be severely affected by SSL termination
(and if you are doing SSL termination on the same machine as your Redis
instance, you're doing it wrong.). This seems like added bloat to a really
great lightweight tool.

~~~
derefr
Most Redis DBaaS services “expose a Redis instance to the Internet”, because
they’re not running in the same cloud as you are (even if it’s the same
underlying datacenter.)

As well, if my understanding is correct, the OP’s goal is to create globally-
distributed Redis clusters, where lower-level solutions like VPN network
peering have relatively bad performance compared to plain userland TLS.

~~~
aphextron
>Most Redis DBaaS services “expose a Redis instance to the Internet”, because
they’re not running in the same cloud as you are (even if it’s the same
underlying datacenter.)

Sure, but they are almost certainly not just directly exposing Redis on port
6379. How would they handle auth, rate limiting, etc.?

~~~
zeroxfe
You do auth with the AUTH command -- you wouldn't want to do AUTH without
SSL/TLS.

You don't need anything more sophisticated than that for the use-case where
you're just making Redis on one cloud available to trusted clients on another.

