
Distributed Locks Are Dead; Long Live Distributed Locks - cangencer
https://hazelcast.com/blog/long-live-distributed-locks/
======
mjb
This is good work, and the lock semantics and fencing token (epoch) make a lot
of sense. I can't help but think that implementing
java.util.concurrent.locks.Lock will turn out to be a liability. The problem
here is that the code looks like a Java lock, but the semantics are entirely
different with regards to failure. Specifically:

> While we’re on this subject, the same logic applies even to the primary
> FencedLock.lock() call: at the very next line of code in your program, you
> may no longer be holding the lock.

That's not, in most programmer's experience, how locks work. This behavior is
necessary (at some level) to deal with partial failures and stalls of clients,
but means that if you use this like Lock your code will be very wrong.

> Note the key message here: all external services must participate in the
> fencing-token protocol, with guaranteed linearizability, for the whole setup
> to uphold its invariants.

So this isn't really like a Java lock at all, and instead is a nice convenient
way to build part of an epoch/view change implementation. That's useful, but
in my mind the API they chose will reduce the likelihood that non-experts will
use this correctly.

------
roro159
I think it's worth to add a few previous discussions about distributed locks.

Redlock is a distributed lock using Redis:
[https://redis.io/topics/distlock](https://redis.io/topics/distlock)

Martin Kleppmann criticized Redlock and mentioned the fencing solution:
[http://martin.kleppmann.com/2016/02/08/how-to-do-
distributed...](http://martin.kleppmann.com/2016/02/08/how-to-do-distributed-
locking.html)

Antirez disagrees with the analysis and the HN post has a good discussion:
[https://news.ycombinator.com/item?id=11065933](https://news.ycombinator.com/item?id=11065933)

------
sriram_malhar
More accurately, these are leases, not locks in the traditional sense. The
lease expires when the corresponding client session ends, which is detected by
the absence of a heartbeat from the client.

~~~
hcnews
Locks are always leases in a distributed system, at least in my experience
(and for a sane system).

------
cangencer
There was also a follow up blog post again by Basri regarding how it's tested
using Jepsen for those interested: [https://hazelcast.com/blog/testing-the-cp-
subsystem-with-jep...](https://hazelcast.com/blog/testing-the-cp-subsystem-
with-jepsen/)

------
heavenlyblue
What happens when some of the quorum members are not available?

The fact that split brain is not allowed implies that liveness is given up for
it.

More importantly, what can I possibly do in the scenario where I would like to
obtain several locks at the same time?

Distributed lock frameworks usually imply there’s some sort of transaction
reversal mechanism implied by the architecture.

~~~
mey
Each lock has an associated name. If you have locks for separate purposes, you
have separate names. If you need to grab multiple locks before processing a
single task, I would suggest looking for a design that doesn't need it. If
there is no other way, then acquring each like in a defined order would still
work.

~~~
hinkley
This is one area where code reuse bites us and I think this is why you see so
many systems starting to resemble functional code principles.

There are many code bases where you delegate an action to a module that goes
out and modifies state (or doesn't) based on its own rules. Composing these
things becomes challenging.

It leads to situations where verbs that sound like they should be side effect
free are unknowingly causing an avalanche of updates behind the scenes.

And when two calls can touch the same objects in a different order (eg, when
you have many-to-one relationships), then you are up a creek without a paddle.

------
PaulHoule
I love Hazelcast. It's the best thing since the Coupling Facility on zSystem
mainframes.

~~~
PaulHoule
I don't understand why people think comparing something to the 360 is ever an
insult.

