
“POSIX advisory locks are broken by design” - okket
https://www.sqlite.org/src/artifact/c230a7a24?ln=994-1081
======
tonysdg
I'm pretty sure this is a well-known problem in general on Linux -- file
locking is almost never a good solution. It sucks, since there are definitely
times where it would be useful.

In any event, this is probably a low priority for glibc/kernel devs -- the
fact that syncing doesn't actually behave as expected is much more important
([https://lwn.net/Articles/752063/](https://lwn.net/Articles/752063/)).

~~~
mjw1007
They added a solution for this three or four years ago, by adding "Open File
Description Locks" which don't have the misbehaviour that that SQLite comment
is complaining about.

See
[https://www.gnu.org/software/libc/manual/html_mono/libc.html...](https://www.gnu.org/software/libc/manual/html_mono/libc.html#Open-
File-Description-Locks)

~~~
SQLite
Yes. And there is code in the SQLite source tree (currently on a branch) that
supports OFD locks. The problem we have is that SQLite is so widely deployed
and is in so many systems and on so many platforms, that we have to continue
to support Posix Advisory Locks (PAL) for the foreseeable future, for
compatibility. It will be great if someday we can remove the PAL code. But for
now, it has to stay in the tree.

~~~
101km
Ah, darn, I'm a few hours late so you will probably miss it but I wanted to
ask for a long time:

Sounds like a lot of stuff like this has accumulated. Your dedication to
backwards compatibility (and testing) is always very impressive, but don't you
ever get the urge to do a parallel "SQLite Next" effort as it were? Kind of
like python had the 2/3 years.

~~~
timClicks
There was an attempt to re-architect sqlite. Try searching for sqlite4.

~~~
okket
Here is a discussion about how the sqlite4 experiment ended, from 8 months ago

[https://news.ycombinator.com/item?id=15648280](https://news.ycombinator.com/item?id=15648280)
(157 comments)

And the relevant commit (timestamp 2017-10-27):

[https://sqlite.org/src4/artifact/56683d66cbd41c2e](https://sqlite.org/src4/artifact/56683d66cbd41c2e)

> All development work on SQLite4 has ended. The experiment has concluded.

> Lessons learned from SQLite4 have been folded into SQLite3 which continues
> to be actively maintained and developed. This repository exists as an
> historical record. There are no plans at this time to resume development of
> SQLite4.

------
CaliforniaKarl
An unrelated comment: Looking through the rest of the file, I love how many
comments there are in general! I am able to get a sense of what each function
does, and what the individual code blocks within the function are doing.

This is something that I do not see as often as I want, and I'm really glad to
see it here!

------
mmt
My (admittedly quick) reading of this is that the issues stem, primarily, if
not exclusively, from using multiple file descriptors for the same file,
necessitated by using/supporting threading for concurrency.

If that's the case, then perhaps it's not so much that PAL was broken by
design but that the original design merely didn't anticipate threading. I'm
not sure how the _advisory_ locking scheme compares in terms of timeline with
threading support, but Unix file locking in general easily predates even SMP.

~~~
Hello71
I don't think that's correct. According to Wikipedia, SMP dates to
approximately the early 1960s, whereas according to the FreeBSD man pages,
flock and fcntl were both introduced in 4.2BSD, released in 1983, two decades
later. It's at most fair to say that Unix file locking came around the same
time as Unix SMP.

~~~
throwaway76543
"Unix SMP" is not synonymous with a system threading interface -- processes
give parallelism as well and predate the introduction of threading interfaces.
The pthread interface came about in the 1990s.

The unix interface is broken with respect to threads in many places besides
this. Signals. Forking. Problems abound because the concept was bolted on
decades later.

~~~
kazinator
signals, forking, and simple things like chdir.

A huge area that is broken w.r.t. threading is the whole internationalization
stack: setlocale and all.

What if I want to write a server such that each request is executed in a
potentially different locale (because it services globally distributed users?)

~~~
caf
_What if I want to write a server such that each request is executed in a
potentially different locale?_

You can use uselocale() to install a thread-specific locale.

[http://pubs.opengroup.org/onlinepubs/9699919799/functions/us...](http://pubs.opengroup.org/onlinepubs/9699919799/functions/uselocale.html)

~~~
haimez
Then your thread better only execute one request at a time and carry through
every operation involved in that request. Works fine in PHP, not fine in most
performant and even most non performant web contexts.

~~~
kazinator
But that's fine. For example, you pull the thread from a pool and bind it to
the request. As part of this, the thread pulls the locale object from the
request and attaches to it. It services that request, and then returns to the
pool. Next time it is pulled from the pool it gets a request in a different
locale.

Other disciplines are possible. E.g. some operation involving multiple
locales, all in one thread, can just do the uselocale calls as needed.

~~~
int_19h
That's not fine, because processing a single request often involves waiting on
some I/O, and the thread can service (a part of) another request in the
meantime. Which is what modern event loop driven servers do - but this doesn't
play well with thread locals, or abstractions implemented on top of them,
because now your single physical thread effectively runs multiple
cooperatively scheduled logical threads/requests.

------
emmelaich
If you want locks from the same user you'd use other mechanisms that file
locks. File locks are for processes that don't have a lighter weight way to
cooperate. (e.g. semaphores.)

Also, there's a reason that databases have processes dedicated to writing
(e.g. Postgres writer process, Oracle dbwr)

Fun fact - IBM's MVS didn't have file locks.

~~~
pvg
_there 's a reason that databases have processes dedicated to writing_

That's going to be a little tricky in an embedded db.

~~~
emmelaich
Sure, but you could still make sure writes are done in a mutex?

~~~
int_19h
Don't forget that it still needs to synchronize across process boundaries, too
- two processes, both running SQLite, and both accessing the same database
file, is a supported scenario.

------
tinus_hn
As far as I know these alternative thread libraries are ancient history. I can
understand you want to defend against sabotage like people using hard links to
open a database under two names but is that really a dealbreaker?

------
Annatar
When he gets a file descriptor, he could also get an inode. Since on hard
links inode will be the same, the prior lock state, if any, can be obtained
and action taken accordingly.

