Redis doesn't guarantee that expiry happens exactly at expiry time ([1] gives up to a 1 millisecond delay), so if your system assumes that all related keys are removed at exactly the same time it is still broken (although the race window is now tighter).
Maybe you could store the related keys in a hash or something?
I'm also guilty of trying to use features in redis in ways they weren't really designed for - it's such a fantastic bit of technology that a lot of the time you can still get away with it but it's not redis's fault when it doesn't work!
Right. Relying on atomic and absolutely reliable expiration is the problem anyway. If it's really that important then it needs to run in a transaction that manually sweeps the keys.
yep - part of our fix was ordering the expires such that the invariants are maintained even if the expires arn't exactly at the same time (we are assuming expires are at least ordered)
yep - as i mentioned in a comment below - we arn't concerned about when the key is actually deleted - only that its not returned when looked up if its TTL is hit
I’m not aware of the docs making any guarantees that expires are ordered either, just that at some point after the expiry time the keys will be expired.
sorry i should have explained. by ordered i meant after expiring a key its TTL will be set, which will never be a time before a previous expire. when it looks up the key, if its TTL has been reached it will not be returned (which we checked in the redis source). we weren't concerned about then the keys are actually removed
I'm pretty sure they aren't ordered. Redis active key expiry is a random process: https://redis.io/commands/expire#how-redis-expires-keys. TL;DR: Redis will randomly sample the keyspace and delete any expired keys it finds in the sample. If a large enough percentage of the sample was expired, it will repeat the process. Since the sample is random, it cannot be ordered.
There is also a "passive" key expiry where any read to a key will check the expiry time and delete the key if the current time is larger than the expiry time.
>The integrity of a set of related keys requires that either all keys exist, or none exist.
Using a Key-Value DB to store relational data eh - that never leads to problems.
Their solution isn't any better because the API does not guarantee a particular order or atomicity of key expiration - so great that it works for you now[1], but next version may change the behavior.
[1] Does it work though? Are you really really sure?
>The integrity of a set of related keys requires that either all keys exist, or none exist
Sorry it wasent that clear that part of our fix was such that it no longer matters if all keys exist or none - we reordered the expires such that the invariants still hold even if all the keys dont still exist
My understanding of expiry is - its not guaranteed when keys are expired if they are not accessed - but if a looked up keys TTL is hit it will not be returned - which is all we cared about
It should be noted that an EXPIREAT triggers a deletion if the timestamp being set is in the past and could lead to an increase in memory and disk pressure during script execution. Worth checking if the behavior there differs in any meaningful way since y'all made the change. EXPIRE usually just marks a key and relies on either subsequent operations to the same key or garbage collection to actually excise the entry and do the subsequent bookkeeping, I believe.
It's been a while since I had to deal with this kind of problem though, so please tell me if things are different now.
It seems the article's implication is that they are relying on redis TTL and expiry to guarantee all keys exists or no keys exist, at all times, for a given key set.
I never got the impression that Redis gave guarantees on when expired keys were deleted, just that they would be eventually.
If the issue is that you want a consistent all or nothing, why not just use a single key to make that determination?
we reordered our keys such that they can expire at different times - also redis checks a keys TTL before returning - and wont return if its expired. i think it will 'eventually' delete if its not accessed, but we only cared about keys being returned, not when its actually deleted
Thanks for clarifying. I wasn't aware of the fact that redis removes expired keys even if they aren't deleted. I've always thought of expiry as more of a resource management annotation. Interesting way to use it.
- If the keys are strings, just use a single hashmap and you're done.
- If the keys have different types, then you need a Redis module that implements something like `MPEXPIREAT time key1 key2 ...` and then you need to either be able to handle the case where you successfully retrieve key1 but not key2 (because they expired half-way through) or you use a lua script to execute the multi-key access atomically. That said, without the lua script the whole point of having the keys expire at the same time makes less sense, IMO.
I don't want to sound snarky, but since you mentioned that Redis does not document if/that "multiple key expiry is guaranteed to occur at the same time if keys have the same EXPIREAT setting", I am tempted to ask - why did you not check the implementation you're using? That's the unique selling point of using FOSS - you don't have rely on documentation or observational guesswork - you can wade in and really see for yourself! Redis' source code is especially readable in my opinion (thanks again, antirez, for creating this little load-bearing gem of a data structure server), so I would very much encourage you to try, even if C is not your particular forte :)
Because documentation is what is needs to be: A simplified and necessarily incomplete description of program behavior. It's a trade-off, since a complete and exhaustive description of all intricacies of potential program behavior would be at least as complex as the definition (= its source code) of said program.
If you happen to have a question that ends up on the wrong side of that trade-off between documentation's completeness and accessibility, you will have to descend into the depths that lie beneath. I believe the redis documentation actually strikes a rather OK balance in that regard.
That's a great suggestion, and OP should do this, but even if OP verifies behaviour today through code inspection, it's possible a future release could change it. The problem is that Redis doesn't want to make that guarantee so you can't rely on it.
Would Redis Keyspace Notifications [1] help in this case? Like a script that runs when a specific key expires, such that all other related keys can be deleted too.
I ran into a similar issue and used a hashset of all the related keys with ttls on the individual keys. the hashset would expire making the keys unreachable, and the TTls basically garbages collects the individual keys. anything that required to transact with the set would do so atomically by being wrapped in a lua script.
Maybe you could store the related keys in a hash or something?
I'm also guilty of trying to use features in redis in ways they weren't really designed for - it's such a fantastic bit of technology that a lot of the time you can still get away with it but it's not redis's fault when it doesn't work!
1: https://redis.io/commands/expire