
Client side caching in Redis 6 - mfrw
http://antirez.com/news/130
======
Deimorz
This might be a bit of a strange question, but it's interesting to think about
and I know antirez will probably read it here, so I want to bring it up:

Could there be some potential privacy or data-leakage concerns with this,
since the server is now storing and acting on info about what data has been
accessed, by which clients, and even when (to a limited degree)?

For example, say that some kind of sensitive data is stored in redis (e.g.
user private messages). By watching for relevant invalidation messages (or if
there's any way for clients to read the data in the Invalidation Table
directly), it could be possible to learn some info about how that data was
accessed.

Without knowing more about the implementation, I don't know if this is even a
concern at all, and I haven't come up with a specific method of abusing it
even if it is. I think it's probably something to keep in mind though.

~~~
antirez
Everything could be a risk, but in the specific case of a software device
holding data, I believe that the main problem is the exposure of the data
itself, so in the specific case of Redis the fact that the database stores
informations about which key was retrieved by which client should not be an
issue AFAIK. A few random arguments:

1\. The database anyway sees such connections and there is a continue stream
of activity about clients, so there is also an instantaneous state that
discloses who does what, at any time.

2\. Companies trace everything in order to have data to do debugging when
there are problems, so this information is stored at a much better level of
details in other places as well.

3\. There is no obvious way this could be exploited.

4\. In general it is supposed that the clients can trust the server.

5\. If you already violated Redis in order to read such info (not accessible
via APIs), you can collect a lot more.

So based on initial thoughts I don't think this is going to be ever a security
problem.

~~~
jonny_eh
Also, the client needs to opt-in.

------
naphaso
Redis continue to wonder me, it's yet another feature I thought to implement,
already implemented by redis itself. And I had same reasoning about hash
slots.

~~~
core-questions
I always appreciate antirez' very candid writing style. Comes across as such
an earnest and reasonable person. Very low ego, very excited about the
technology, but with an eye for how people will be able to adopt it. Wish we
saw more tech leaders like that.

------
bironran
Interesting. I built something somewhat similar about 2 years ago (not big on
blogging so it’s not published - probably end up hurting my career :( )

The gist is using pubsub to send invalidation messages consisting of item ID
and time stamp, but also store the latest invalidation message into a set that
is polled on every longer time period (10 seconds in my case). Listening to
pubsub and polling are done per server that hosts the in memory cache for all
items in the cache.

One interesting complication was how to make sure I don’t end up with the
wrong (stale) cached item due to race condition between the read and the
message passing. Imagine 2 thread, thread A reading and populating the cache,
thread B listening to invalidation events. If thread A get held up (context
switching, slow network, etc) it can actually populate into the cache after
thread B has received and processed the invalidation message. To avoid that I
ended up holding the invalidation messages and their receive time in the
receiving server for a short duration (20 seconds - twice poll time). I
required and in memory cache population to tell me the time from before the
read command was sent to the cache. At the time of cache population, if I saw
in had any invalidation message marked after the time the read command started
I discarded that item instead of caching it. In highly concurrent, highly
distributed application this turned to be a must-have.

Also, mainly to placate my fears, I had “panic mode”. If I didn’t see any
message (I sent periodic maintenance messages) or if I couldn’t execute
multiple polling attempts I dumped the whole cache and wouldn’t let any new
item be ca he’d until communication was restored.

~~~
bironran
Forgot to mention - writes also invalidated the cache locally so maintain
“read your own writes” for multi step algorithms. That was crucial for us and
I suspect for a lot of other use cases.

------
blaisio
Redis is already super useful, but this would be an absolutely killer feature.
Server assisted client side invalidation! Something people have always wanted
but has never been built.

Implement it quick so I can use it on AWS in a few years!

~~~
kevan
Any reason you can't achieve this today with DynamoDB+DAX?

~~~
stephen
Isn't DAX only a network-based in-memory cache, and not a in-process in-memory
cache? Or is it both?

It's definitely at least network-based, b/c you have to spin up instances of
it in your VPC.

However, they also require custom clients, which is a little odd afaiu (b/c in
theory DAX is API-compatible with DynamoDB, so why would regular dynamodb sdk
clients not work?) so maybe the custom clients also add some in-process-based
caching?

------
baq
I read ‘time’, I think ‘jepsen’... as the saying goes, there are only two hard
problems in software engineering: naming things and cache invalidation.

Disclaimer: happy user of a redis cluster :)

~~~
antirez
If you re-read you'll find that I suggest an incremental epoch to avoid any
issue :-D however starting with the idea of time to later hint about a better
solution is a lot more understandable IMHO.

------
estebarb
It reminds me of OVSDB + IDL (C client):
[https://tools.ietf.org/html/rfc7047](https://tools.ietf.org/html/rfc7047) .
The C client keeps a partial replica of the database, and the database
notifies all the clients with the changes.

It makes super fast to work with data (it is already in process memory), and
the centralized server handles consistency and transaction serialization.

------
redis_mlc
You can see the diagram I did of Ben Malec's multi-level cache system here:

[https://images.app.goo.gl/RgQcYt3MTU3AA3CF9](https://images.app.goo.gl/RgQcYt3MTU3AA3CF9)

------
iampims
I feel like this would be neatly implemented as a Redis Proxy, keeping the
same semantics but removing that concern for simpler clients.

~~~
Deimorz
The simple clients just wouldn't enable client tracking, it's off by default.

It also wouldn't work very well as a proxy because then if any clients don't
go through the proxy but change some of the data, there's no way for all the
proxy-using clients to know.

~~~
zzzcpan
You can make all the changes propagate to all the relevant servers and
clients.

------
ricardobeat
What's the effect of this system on consistency guarantees? Seems like clients
could end up in a state where they serve stale data due to (undetectably) not
receiving the push updates.

It also looks like client / protocol implementations are about to get a lot
more complex, even if the feature is being used by the majority of users?

~~~
antirez
This is discussed in the blog post itself. If you want to ask a more specific
question I'll be happy to reply to my best knowledge.

~~~
ricardobeat
My bad for not being clear enough. I was thinking of, for example, a
networking issue where both server and client are still “connected” but
delivery never happens. Can be solved with tuning timeouts / client-side TTL
as mentioned elsewhere, but I was wondering specifically if this would change
Redis’ known guarantees for data consistency, what happens during partition
etc. More of a theoretical question.

On the second point, the post talks about having the feature disabled by
default; I was curious regarding the protocol itself - it is currently quite
simple, and by adding extra cache-control metadata it sounds like correct
client implementation might become a lot more complex, but it’s hard to tell
without having seen what the new protocol looks like.

~~~
antirez
Yep your concern about consistency is definitely correct, but the good thing
is that the invalidation connection can be pinged continuously by the client,
as often the client wants (for instance 10 times every second). As soon as the
connection is timing out, the client may decide to no longer serve the cached
data. Of course every time the invalidation connection is closed at all, the
client should flush the cached data completely or apply a very short TTL to
the set of cached data.

About your second point: today I implemented even the feature (in the unstable
branch) to support RESP2, that is, the current Redis protocol. But even using
RESP3 the complexity is very low. On the data connection side you have just to
use the CLIENT command to activate it. In the data connection, what you could
receive, if you associate a callback in your client, is push data that
notifies you the invalidation connection is no longer active. Push data is
normally discarded by the client implementation of RESP3 if no handler is
associated. On the invalidation connection, in RESP3 mode, you get again push
data with a callback, or if you are in RESP2 mode, you just subscribe to the
special channel __redis__:invalidate, and you get the messages about the
caching slots that need to be invalidated.

------
js2
> But later also client ID 444 will ask about keys in the slot 2, so the table
> will be like:

I think you mean slot 5 in this sentence.

~~~
antirez
Thanks, fixing

------
lessclue
How will client libs that do connection pooling handle this? If a client opens
a pool of 50 connections, will they be identified as 50 different clients, and
receive 50 different pushes for invalidation? I guess the client libs will
have to build a layer to balance caching on top of the pool.

~~~
zmj
Just curious, why would you pool redis connections?

~~~
lessclue
High throughput. Pooling is the default behaviour for many Redis clients too,
isn't it? eg:
[https://github.com/gomodule/redigo](https://github.com/gomodule/redigo)

------
PhilippGille
Is this similar to Hazelcast's "near cache"?

------
nullwasamistake
Shameless plug for in-language storage. I've found that using a fast
language/framework with a local cache gives you most of the advantages of a
shared cache like Redis without all the complexity. This feature feels like a
tacit admission of that.

In "fast" languages (C# Java Go Rust Cxx) allocation tends to be fairly
efficient so you're not wasting tons of space keeping objects in RAM.

These fast languages can max out the network interface most of the time using
the right frameworks. In them, memory storage is also over 100x faster than
remote like Redis.

Now if you've got something in PHP or Rails that can only manage 100
requests/second remote caches make tons of sense. The network overhead is
nothing compared to how slow the language is. This advantage erodes when
you're closer to native speed and local memory access is 1000 times faster
than retrieving cached data over network.

TLDR: Redis and alike make sense when your language or framework is slow and
the bottleneck is CPU use. They don't give you much when networking and memory
bandwidth are your bottleneck because the language runs near native speed.

~~~
sk0g
Yeah but this doesn't work with something like microservices, or even
Kubernetes based deployments, right?

It might take you 3, 40, or a 100 tries for the same request till you hit a
node that has your data cached. It could work with extremely high req/seconds,
though in an unpredictable way.

~~~
heavenlyblue
To be fair, what does it take you to write a simple caching microservice based
on protobuf or similar?

~~~
sk0g
A lot more knowledge in the relevant space that I have. I could spend a week
or two learning necessary information, analysing our use cases and then
implementing it finally (which I would never get the time to, from a business
perspective).

Or I could use an off the shelf product that is probably better than whatever
I could come up with, given, even, unrealistic amounts of time to perfect it.
I could come up with something more suited to our use case only, but the
moment the use case changes, I'd wonder why I didn't just go with Redis.

I haven't actually looked into a DIY method though. I just know I want
something like Redis for our user table, because it's hit a lot, and the
traffic tends to be repetitive.

~~~
heavenlyblue
>> A lot more knowledge in the relevant space that I have.

Relevant to what? You've already got a microservice mesh, which means you've
also got a microservice architecture that's supposed to handle all of the
RPCs.

Adding a layer of Redis on top is probably going to introduce more entities
into the system and make it less homogenous.

------
coleifer
Whether or not you like this feature or think it's a good idea, the brunt of
this will be born by library writers. Yeah you can use a dedicated thread to
listen to these push messages, but really this is tacitly blessing the
asyncio's and whatever else.

As you can probably tell, I'm not at all a fan of the way Redis is moving. It
hit a sweet spot as a "better memcached" but apparently aspires to be a real
database, so now this nice simple tool is all crazy. The new protocol just
kinda sucks for synchronous use.

I'd go with a hand-written in-memory caching scheme before I ever considered a
Redis sandwich with this feature. Seems like a complexity nightmare and very
difficult to reason about. Yeah maybe I'm being crusty, but I'm just thinking
about maintaining and debugging something like this and having cold sweats.
The linked redisconf talk clearly bears this out. Sounds like they had a hell
of a time. But then that also provides some measure of job security?

~~~
ec109685
Your hand tuned client cache won’t have active invalidation like this new
proposal does, so you are stuck waiting for ttl to expire and ttl windows
where data is inconsistent between nodes.

