
Libdill: Structured concurrency in C - rumcajz
http://libdill.org/structured-concurrency.html
======
Matthias247
Many of the ideas inside the library are also explained in the Structured
Concurrency blog post on the authors homepage:
[http://250bpm.com/blog:71](http://250bpm.com/blog:71)

I really sympathize with a lot of them, e.g. scoping the lifetime of all child
processes (goroutines, ...) to the lifetime of their parent and having good
cancellation/timeout/close mechanisms for everything. I'm trying to apply
these patterns in Go and it can solve a lot of scenarios in a more obvious and
less spaghetti fashion.

Libdill seems to bringt a view more primitives to the table which for example
make cancellation and waiting for child processes to finish easier. But the
basic mechanism can be also applied in Go (e.g. via context.Context and a
WaitGroup/Channel for waiting on the childprocess to finish) and therefore
most likely also to libmill.

~~~
the_duke
Go does not allow external cancelling of goroutines though, which I have
wrestled with a few times.

The workaround is, of course, to have a "done" bool channel which is checked
by the routine, but that's not always applicable or convenient.

~~~
jerf
First, since people generally assume replies are disagreement, yes, goroutines
can not be externally cancelled and I don't expect that to ever change.

However, you can often mostly simulate what this library does in Go fairly
effectively, since the goroutines end up stereotypically used. For instance,
if your goroutine's purpose in life is to read from a socket, you can close
the socket from another goroutine and that will result in all reads
immediately failing. If you close the resources the goroutine expects, you get
similar results.

You do have to structure your code for this, though; if the goroutine that
you're trying to close both opens a socket and listens for it, then another
goroutine trying to close that socket may fail if it tries before the first
one even has the socket open. One solution I've sometimes used is to lock
around a struct with a bool and a socket reference; the goroutine opens the
socket, takes the lock, closes if the bool is true and updates the socket
reference if it isn't to point to the socket, then drops the lock. (One lock
is generally not a great penalty next to the costs of acquiring the socket.)
If an external goroutine tries to close the resource, it takes the lock,
closes the socket reference if available, and sets the "closed" bool if it
isn't. It sounds complicated in English but isn't that much code.

It's a cute trick in C to turn all blocking calls into ECANCELLED results (no
sarcasm, it is a cute trick), but generally goroutines should be small enough
that they have an obvious and small set of "resources" that can be closed,
where one such resource is definitely the dominant case. (And Go is all about
considering the dominant case and not inserting tons of language features for
the longer-tail cases.)

A "done" channel is a good solution if your goroutine already has a "select"
statement in it. I've found that correct use of the select statement actually
requires more such "structural" channels than is sometimes advertised; in
particular, whenever you write code where you want the receiver to close a
channel for some reason, the correct solution under current Go constraints is
actually to add a channel that goes back to the sender, which the sender must
be modified to check, and close that. I find rather more of my channels end up
being "struct{}" than I expected at first.

~~~
rumcajz
To me, the trick with closing the resource sounds like a hack.

Imagine you wanted to generalise it: We have a goroutine that may use some
resources: sockets, files, keyboard, screen etc. To shut it down you cut it
off from all those resources and hope that it will die.

It sounds wrong somehow.

~~~
jerf
I'd agree it was a hack if it were ill-defined, but I don't think it is. I
don't think Go introduced that trick. In many cases you have to write your
code to be able to deal with the resource being unavailable at any moment
anyhow, especially networks, where Go is on its home turf. (As I've said more
than once, the farther you are from writing something that looks like a
network server, the less happy you will be with Go.)

------
simonlindholm
In the same vein, but with less idiomatic C, note also
[https://github.com/hnsl/librcd](https://github.com/hnsl/librcd). I had the
opportunity to work with it two years ago at Jumpstarter, and it was quite
wonderful.

I've rehosted the long and opinionated introductory blog post about it at
[https://github.com/hnsl/librcd/wiki/Announcing-
Librcd:-Concu...](https://github.com/hnsl/librcd/wiki/Announcing-
Librcd:-Concurrency-and-Segmented-Stacks-in-C). It's well worth a read, I'd
say -- there's a lot of interesting ideas in librcd.

------
the_duke
How is this different from
[https://github.com/sustrik/libmill](https://github.com/sustrik/libmill)?

It's from the same author.

~~~
the_duke
Just noticed that there is an entry in the FAQ.

Pasting here for others:

[http://libdill.org/documentation.html](http://libdill.org/documentation.html)

It's C-idiomatic. Whereas libmill takes Go's concurrency API a implements them
in almost 1:1 manner in C, libdill tries to provide the same functionality via
more C-like and POSIX-y API. For example, choose is a function rather than a
language construct, Go-style panic is replaced by returning an error code and
so on.

Coroutines and processes can be canceled. This creates a foundation for
"structured concurrency". chdone causes blocked recv on the channel to return
EPIPE error rather than a value.

chdone will signal senders to the channel as well as receivers. This allows
for scenarios like multiple senders and single receiver communicating via
single channel. The receiver can let the senders know that it's terminating
via chdone.

libmill's fdwait was replaced by fdin and fdout. The idea is that if we want
data to flow via the connection in both directions in parallel we should use
two coroutines rather than single one. There's no networking library in
libdill. This is meant to be a separate project.

~~~
norswap
Is it meant to be a successor then? Or are they complementary? It's misleading
that the FAQ only seems to mention advantages.

~~~
the_duke
For reference:

libmill was a project to copy Go's concurrency model to C 1:1 without
introducing any innovations or experiments. The project is finished now. It
will be maintained but won't change in the future.

libdill is a follow-up project that diverges from the Go model and experiments
with structured concurrency. It is not stable yet and it may change a bit in
the future.

------
adrianratnapala
Looks like a very nice API. Now I'm waiting for an excuse to use it!

The only minor blemish I see is the need to call `fdclean()` before closing a
file-descriptor. It introduces a slight non-orthogonality between libdill and
straight posix.

Another approach might have been to wrap the fd up as a channel -- perhaps not
one that transfers actual bytes, but which does signal availability.

The user would still need to to call chdone to clean up before calling fclose.
But that would be a natural usage of the interface rather than a special-
purpose wrinkle.

~~~
rumcajz
fdclean() is not strictly needed and it haven't even existed in earlier
versions of the library.

But it turns out that when handling 10,000 concurrent connections, the version
with fdclean() is 1700x faster...

~~~
adrianratnapala
I am not really objecting to the need to clean up the fd-bookeeping data, just
the interface.

And it's not much of an objection. I guess the intention behind libdill is
that applications will do their I/O using some abstraction layer anyway -- and
layer can be libdill aware enough to call fdclean().

If that is the design philosophy, then does that means there would be problems
if someone used some big external libray that did lots of its own I/O inside a
libdill coroutine?

~~~
rumcajz
Here's the use case that drives the design of fdclean: Imagine a third party
library makes a callback to your code and hands you a file descriptor. You can
use it temporarily, but you have no idea what will happen to it after you
return from the callback function. Will it live on? Will it be closed? The
only way to deal with that is to purge any cached info about the fd before
returning and pretend that you never had anything to do with that file
descriptor.

    
    
        void callback(int fd) {
            fdin(fd);
            recv(fd, buf, len, 0);
            fdclean(fd);
        }

------
aristidb
If you can use C++, give Intel TBB a try. It has a lot of structured
primitives but still makes it easy to just parallelize a for loop.

~~~
Narishma
This is about concurrency, not parallelism.

~~~
aminorex
It is equally "about" both, if you look at what the API actually provides.

------
zzzcpan
> What that means is that one coroutine has to explicitly yield control of the
> CPU to allow a different coroutine to run. In the typical case this is done
> transparently to the user:

And this is exactly what's wrong with coroutines. It's not "transparent" to
the user, but actually implicit to the user and unpredictable. With that user
has no control over switch between coroutines and no idea when it's happening,
which forces him to think about synchronization and generally have a very bad
time if the essential complexity of the task he needs to implement is not
something as trivial, as talking to a database.

In contrast, having a keyword, like await, to let the user keep control of the
switching doesn't have that problem and user doesn't have to think about
synchronization. But such things cannot be called coroutines.

So, be wary when someone is trying to push coroutines onto you. It's never a
good idea to have them.

------
partycoder
I am more familiar with the "futures and promises" approach.
[https://en.wikipedia.org/wiki/Futures_and_promises](https://en.wikipedia.org/wiki/Futures_and_promises)

~~~
adrianratnapala
It seems libdill supports channels -- blocking FIFOs.

In Golang (which seems to have indirectly inspired libdill), you would
implement such logic by returning a channel (future) over which is sent just
one value. That result itself might be a channel and so on.

No doubt the syntax in C would not be as convenient as in some other language
-- but lightweight channels that work as first-class objects are a lot like
futures.

------
kgabis
Looks very interesting. The only problem I see with it right now is that
functions aren't prefixed with dill (or some other unique prefix).

------
norswap
One thing that I would love to see is the ability to schedule coroutines
accross threads. Currently, you have to choose to spawn a new thread for your
coroutine or keep it on the same thread. Although I believe adding a thread a
pool and an assorted dispatch capability shouldn't be too hard.

------
siscia
It looks like a very nice and well thought beginning.

Now we should wrap this into safer abstraction and maybe we should explore if
it is worth to create a supervision mechanism and a communication model.

Would love to explore this further, but my time is lacking, feel free to drop
me an email for any kind of discussion :)

------
Keyframe
Is this portable among compilers/OS'/architectures?

~~~
rumcajz
gcc/clang, linux/osx/bsds, x86_64, ARM...

~~~
jackhack
I'm far out of my area of expertise here, but can anyone offer thoughts about
the potential of wrapping these C routines and accessing them from python
(Cython) to give python this level of concurrency?

~~~
rumcajz
Don't know about Python but someone did a similar thing with Swift:
[https://github.com/VeniceX/Venice](https://github.com/VeniceX/Venice)

