
Reader/Reader blocking in reader/writer locks - dmit
https://blog.nelhage.com/post/rwlock-contention/
======
drewg123
I encountered this issue a year or two ago on the Netflix CDN nodes when
running with 100K TCP connections or more. We have metrics collectors that
need to report how many TCP connections are open, and what state they are in
(ESTABLISHED, TIME_WAIT, etc). Until recently, the only way to do this was to
ask the FreeBSD kernel to copy out the entire TCP hash table to userspace,
where an application can examine it & count connections. The problem with that
is that the thread wanting a copy of the table had to take an rlock on the
table while it was being copied. As the author points out, this blocks any
writers, which in turn blocks more readers. So we'd see latency spikes when
this script was running.

The problem has been fixed twice: by moving the top-level locking for that
table to epoch (FreeBSD RCU equivalent), and also by using per-cpu counters to
track the number of TCP connections in every state. So now the user-space
collector just needs to fetch a small array of tcp states from the kernel.

------
twmb
This is a decent reminder blog post about something that is obvious on the
surface, but subtle when you forget to think about it and make a mistake.

I've found that thinking of a lock as something you don't want to hold goes a
long way. The point of locks is to release them.

Jeff Preshing has a good series on locks and concurrency. This[0] post
(potentially with an extra dozen I read in the same sitting) is the one that
really changed how I thought about locks: before I read it, I was very into
lock freedom at every opportunity.

0: [https://preshing.com/20111118/locks-arent-slow-lock-
contenti...](https://preshing.com/20111118/locks-arent-slow-lock-contention-
is/)

------
tntn
> So reader/writer locks tend to rely on the additional concurrency, and on
> readers not blocking other readers.

> software has a constant tendency to get slower, absent deliberate and
> careful efforts to hold the line, as developers add features and complexity.
> And if slower processes don’t immediately impact performance (because they
> only hold read locks), they’re more likely to go unnoticed. Thus, it’s not
> unlikely that over time read lock durations will creep upwards, mostly
> without effect, until one happens to coincide with an attempt to grab a
> write lock.

For me, the takeaway here is to maintain discipline in software regarding
locking behavior. It's almost always a bad idea to hold a lock (rw or not) for
an extended period of time.

Notably, this is how the Linux kernel developers responded to the case linked
in the discussion of mmap_sem - the operation quadratic in number of threads
was thrown out to avoid holding the rwsem for extended periods of time.

Also, re "and the lock is held by readers for a significant length of time": I
would imagine that Rusty Russell would consider a few microseconds a
"significant length of time" and wonder if you were insane if your software
held a lock for 60 seconds.

~~~
dmit
In a perfect world "maintaining discipline" is indeed the correct solution,
but my experience tells me that humans cannot be relied upon to do that. Some
individual humans can, but not the collective whole.

The discipline has to come from elsewhere. Either from the management
(reducing the number of people you have to rely on from N to log(N)), or from
tooling (reducing the number to log(1)).

As always, education is the silver bullet. Instill proper values, enable the
skills to follow through, and in a mere 40-60 years the industry will be in a
better place. As long as there is no incentive to mass-produce junior software
developers on the cheap, we're golden.

------
kazinator
> _What to do about it?_

Design with RCU, if you can.

> _Time out writer acquisitions._

> _If potentially starving writers is acceptable, adding a timeout to write-
> side acquisitions will bound how long readers can be forced to pause.
> GoCardless’ library for online Rails migrations implements this option for
> Postgres, and also discusses the challenges and implications a bit._

This is nearly the same as simply not having writer priority. If "potentially
starving writers is acceptable", that implies we can consider reader-priority
locks, which do exactly that: potentially starve writers.

Under writer priority, we hold back new readers so that the old readers can
leave and let the waiting writer in. If we balk when the new readers are
blocked for too long, and bump the writer, then it's simply not writer
priority any more.

------
ggambetta
Please make the second "reader" in the title lowercase, otherwise it's very
confusing to parse (looks like three things separated by slashes, instead of a
single sentence with two words that have slashes in them).

------
dahfizz
For the database example, could you change your isolation level for the
reading queries?

