
The Common Lisp Condition System – Upcoming Book - phoe-krk
https://www.apress.com/us/book/9781484261330
======
phoe-krk
Author here. The book has just gone to production and will be available at the
end of the year in dead-tree and electronic formats.

The book happened because I got annoyed at the fact that there is no good
reference that shows how the condition system works, what are the foundations
required for it to work, what are the basic building blocks that it is
composed of, and what are its uses, both in the domain of handling exceptional
situations and, more importantly and less obviously, outside that domain.

The book also contains a description of a full implementation of a condition
system that can (and hopefully will!) be used outside Common Lisp. There's
hope that some languages will pick up these parts of Common Lisp that fully
decouple the acts and means of signaling conditions from the acts and means of
handling them, therefore allowing for means of dynamically controlled control
flow that are currently very hard to achieve in other languages.

The accompanying code is available at [https://github.com/phoe/tclcs-
code](https://github.com/phoe/tclcs-code) and
[https://github.com/phoe/portable-condition-
system](https://github.com/phoe/portable-condition-system) \- it will be
copied over to the Apress repository soon.

Linked Reddit thread:
[https://www.reddit.com/r/lisp/comments/hrjzs8/](https://www.reddit.com/r/lisp/comments/hrjzs8/)

AMA, I guess.

~~~
eismcc
While this may seem absurdly naive, what is the CL condition system? Based on
what I see (read) It seems like you need to know what it is before read the
book.

~~~
phoe-krk
Let me answer by posting an introduction to the condition system by Kent M.
Pitman. It is the first subchapter of the book.

\------

 _There have been many attempts to declare the Lisp family of languages dead,
and yet it continues on in many forms. There are many explanations for this,
but an obvious one is that it still contains ideas and features that aren 't
fully appreciated outside the Lisp community, and so it continues as both a
refuge and an idea factory.

Gradually, other languages see the light and these important features migrate
to other languages. For example, the Lisp community used to be unusual for
standing steadfastly by automatic memory management and garbage collection
when many said it couldn't be trusted to be efficient or responsive. In the
modern world, however, many languages now presume that automatic memory
management is normal and natural, as if this had never been a controversy. So
times change.

But proper condition handling is something which other languages still have
not figured out that they need. Java's try/catch and Python's try/except have
indeed shown that these language appreciate the importance of representing
exceptional situations as objects. However, in adopting these concepts, they
have left out restarts --- a key piece of the puzzle.

When you raise an exception in Python, or throw one in Java, you are still
just performing an immediate and blind transfer of control to the innermost
available handler. This leaves out the rich experience that Common Lisp offers
to perform actual reasoning about where to return to.

The Common Lisp condition system disconnects the ability to return to a
particular place in the program from the necessity to do so, and adds the
ability to "look before you leap." In other languages, if you create a
possible place to return to, that is what will get used. There is no ability
to say "If a certain kind of error happens, this might be a good place to
return to, but I don't have a strong opinion ahead of time on whether or not
it is definitely the right place."

The Common Lisp condition system separates out three different activities:
describing a problem, describing a possible solution, and selecting the right
solution for the right problem. In other languages, describing a possible
solution is the same as selecting that solution, so the set of things you can
describe is necessarily less expansive.

This matters, because in other languages such as Python or Java, by the time
your program first notices a problem, it already will have "recovered" from
it. The "except" or "catch" part of your "try" statement will have received
control. There will have been no intervening time. To invoke the error
handling process IS to transfer control. By the time any further application
code is running, a stack unwind already will have happened. The dynamic
context of the problem will be gone, and with it, any potential intervening
options to resume operation at other points on the stack between the raising
of the condition and the handling of an error. Any such opportunities to
resume operation will have lost their chance to exist.

"Well, too bad", these languages would say. "If they wanted a chance, they
could have handled the error." But the thing is, a lot of the business of
signaling and handling conditions is about the fact that you only have partial
knowledge. The more uncertain information you are forced to supply, the more
your system will make bad decisions. For best results, you want to be able to
defer decisions until all information is available. Simple-minded exception
systems are great if you know exactly how you want to handle things ahead of
time. But if you don't know, then what are you to do? Common Lisp provides
much better mechanisms for navigating this uncertain space than other
languages do.

So in Common Lisp you can say "I got an argument of the wrong type. Moreover,
I know what I would do with an argument of the right type, I just don't happen
to have one or know how to make one." Or you can say "Not only do I know what
to do if I'm given an argument of the right type (even at runtime), but I even
know how to store such a value so they won't hit this error over and over
again." In other languages, if the program doesn't know this correctly-typed
value, even if you (the user) do know it at runtime, you're simply stuck.

In Common Lisp, you can specify the restart mechanism separately from the
mechanism of choosing among possible restarts. Having this ability means that
an outer part of the program can make the choice, or the choice can fall
through to a human user to make. Of course, the human user might get tired of
answering, but in such a case, they can wrap the program with advice that will
save them from the need to answer. This is a much more flexible division of
responsibility than other languages offer._

~~~
pierrebai
Am I allowed to think the described use case less than compelling?

It's especially not compelling that the caller passed the wrong type, the
called function would like a different type and somehow a different piece of
code would know both end of the situation and fix it instead of the caller or
callee.

It all seems like an artificial and convoluted use case.

Software design tend to know if they'd like to abort (throw) or report
(callback) statically. Conditions may be general, but they seem to me to
mostly allow unnecessary open-ended complex design.

If the supposed gain is that a single system can do it all, I again don't feel
it is a convincing argument. In fact, I tend to prefer a one-goal system,
where different use case are easily differentiable because they are different.
IOW, that throwing an exception, calling a callback, sending a signal or
converting a value should look different is a plus.

This is something I've noticed: one starts with a rigid design, then adds
abstractions. But one reach a point over over-abstracting where the design
becomes uncomprehensible because it is so generic that it becomes meaningless.

~~~
phoe-krk
Sure. The type-error example shown here is easily fixable by languages which
utilize static typing and therefore make invalid code uncompilable, but it has
the advantage of being easily understood by almost all programmers.

A more contrived example would be a situation in which some piece of data
(e.g. a worker's monthly timesheet report) is passed between modules of a
programming system, but the receiving module, upon performing validation,
discovers e.g. that the employee was working during a holiday.

Handling that situation is hard, since there are multiple ways of handling it,
each of them valid in its own specific context. If the employee is in another
country where that day is not a holiday, we should proceed without any other
actions. If the employee has an agreement with their manager that they are
having crunch time, then the system should proceed after applying overtime
payment. If the employee is on a flexible time schedule, we should proceed and
log this somewhere else; if the employee has no justification for that
overtime, we should abort and signal an error; more examples follow.

In other words, when we signal the condition, we do not have full information
about what should happen to it. We have the question "What should we do with
this timesheet?" and the answer to that question is "It depends."

It is possible to model this situation by a condition type named e.g.
EMPLOYEE-WORKING-DURING-HOLIDAY, and instances of that condition type being
signaled inside the programming module. We create a dynamic environment where
the proper handler routines for EMPLOYEE-WORKING-DURING-HOLIDAY are
established, and we call the module's validator function inside that
environment.

This process fully decouples the act of signaling a condition from the act of
choosing _whether_ to handle that situation and also from the act of choosing
_how_ to handle that situation.

One can (and should) document the condition type in the design specifications,
and also describe functions that are allowed to signal it.

~~~
pierrebai
I'm afraid to sound like a negative type, but to me this example is simply a
callback?

The described EMPLOYEE-WORKING-DURING-HOLIDAY is akin to a system has a slot
to register a callback (or signal which can be connected to receiver, to use a
Qt-like design) which can handle the situation or decide raise an exception.

Again, I understand that it might be attractive to have a single unified
system to handle the different possible situations instead of separate
systems. I find it hard to imagine a case where the decision to be an
exception-like, callback-like or restart-like is chosen dynamically at run-
time by any or all participants. And like I said, I' d be afraid that if such
a case come up, it would make understanding the design harder, not simpler.

For example, as much as Qt signal/slot mechanism is powerful and flexible,
I've found that when used fully, it makes understanding the code hard because
it becomes impossible to know what will happen when a signal is raised because
the handling is so well uncoupled.

~~~
db48x
They are callbacks, but when they return they don't return to you. Instead
they return back up the stack to whoever called the function that signalled
the condition.

~~~
phoe-krk
> Instead they return back up the stack to whoever called the function that
> signalled the condition.

Not necessarily. They may return to _any_ point on the stack that has been
"announced" as a valid return point. One can compute a list of these return
points, choose which one to utilize, and under which conditions.

That's what constitutes the power of restarts (and non-local exits) in Common
Lisp.

------
drcode
For those unaware, Common Lisp exception handling is heads and shoulders above
almost any other programming language: Imagine having your program crash in
the middle of a function, being able to inspect the call stack to find the
bug, fixing it, then completing the function call from the middle of the
function, at the exact point where the exception happened, with the fix in
place.

~~~
hardeeparty
This seems really valuable for development, but is there a way to use this in
production systems? Say a user hits a bug in a CL web app. Could I write an
exception handler that (1) immediately returns a 500 error page and (2)
persists an image somewhere so that I could fire it up later and have the
debugger ready at the offending point?

~~~
phoe-krk
I think that it's possible. Immediately returning a HTTP 500 is easily doable,
and you should be able to fork the Lisp process, continue serving requests in
the parent, and save stack information (such as function parameters, etc..)
and dump the Lisp core for further analysis in the child. (Do note, however,
that this process breaks all FDs, which includes open files and network
sockets.)

~~~
zeveb
Depending on how the HTTP library works, one might be able to return an error
to the user, but preserve all the other call state in an open thread, and
later on join it from an interactive debugger … perhaps even a Lisp debugger
running in a Web browser!

~~~
phoe-krk
That's doable. Common Lisp has a programmable debugger hook which allows for
implementing one's own debugger - also, possibly, as a web application. And
for situations where the standard debugger hook is not enough (e.g. BREAK or
INVOKE-DEBUGGER), one can use [https://github.com/phoe/trivial-custom-
debugger](https://github.com/phoe/trivial-custom-debugger) to completely
override the system debugger with a custom one.

------
gumby
The condition system is often hard for non-lisp people to understand as the
best they’ve had in the past is a core dump or a c++-style exception system.
The main problem for me with those is that by the time you see the exception
the state has already been destroyed. This is the problem with core dumps as
well: all your files and network connections have been closed. The only
alternative is to use a debugger which is a high overhead situation.

The roots of condition system evolved from two related hacker environments:
the ITS system at the MIT AI lab and the lisp machines (both CADR and its
descendants and the PARC D-machines) as well as influence for the Smalltalk
folks.

In ITS what is called the “shell” these days _was_ the debugger. If your
program signaled an error you were immediately in the debugger and could
debug, just exit, or Force a core dump. This was the routine way to interact,
even for non-programmers. The lispm did the same. Debugging was a lightweight
operation.

So it was natural in the 80s to develop a system that formalized this mode of
operation. Because an exceptional situation by definition requires greater
knowledge than is available _in situ_. When you catch an error in java or
can++ it’s too late to look at the state of the object that caused the error
without standing on your head and possibly redesigning the try block say to
allocate something on the heap that can be passed to the error (which means no
more RAII etc etc).

In addition, try/catch typically puts the hand lin too close to the site of
the problem for you to be useful; if not, by the time you get the exception
there’s not much you can do but terminate the calculation politely.

~~~
phoe-krk
> The main problem for me with those is that by the time you see the exception
> the state has already been destroyed. This is the problem with core dumps as
> well: all your files and network connections have been closed. The only
> alternative is to use a debugger which is a high overhead situation.

This seems like two core issues with understanding the condition system.
First, the currently popular exception systems always destroy the stack by
unwinding it, allowing the programmer to debug only dead remains instead of a
living program; second, debuggers are bolted on top of programs instead of
being their internal parts, which gives many people an impression that using
them as an integral part of the whole programming lifecycle is inconvenient
and unnatural.

The condition system in Common Lisp solves both of these issues, which IMO,
paradoxically, contributes to it not being well understood outside Lisp
circles.

------
mjbishop
This book looks great!

I tried to implement this in Ruby with a gem about 6 years ago and got really
close. If anyone wants to know more about this, but is more fluent in Ruby,
you might find this a good jumping-off point. There’s a good explanation in
the README.

    
    
      https://github.com/michaeljbishop/mulligan
    

Why did it not work in the end? Because even though I was able to implement an
error-recovery mechanism that wouldn’t pop the stack, it still triggered all
the `ensure` statements that were up the stack. It was still a valuable
exercise. I'd welcome this addition in any exception-based language.

~~~
phoe-krk
Thanks for that link! I'll include it in the book.

------
steveklabnik
Fun trivia fact: Rust had conditions at one point. They weren't really used by
anyone and so ended up getting dropped.

~~~
phoe-krk
The book mentions that fact. The commit that removed them (along with the
rationale) is available at [https://github.com/rust-
lang/rust/commit/454882dcb7fdb03867d...](https://github.com/rust-
lang/rust/commit/454882dcb7fdb03867d695a88335e2d2c8f7561a)

~~~
gumby
Those reasons seem more rust-specific than being a general reason to avoid
restartable conditions — as the commit message acknowledges at the end.

~~~
steveklabnik
They're also "2014 Rust" specific; Rust changed a bunch between then and 1.0.
Which doesn't mean they still wouldn't apply, just it gets harder for anything
earlier than may 2015.

------
dan-robertson
A problem I don’t have a good answer to is how to extend conditions (or even
exceptions) to async programming models.

Two problems are:

1\. some code may fail multiple times (with non-async code, once an exception
is caught the program can’t throw another one from the same dynamic extent of
the try block; similarly once a restart is invoked and control is transferred,
a condition won’t be signalled from the same dynamic extent where the restart
was bound (for the purpose of this clause, we consider a retry restart to mint
a new dynamic extent to bind the restart in again after retrying))

2\. Some computation may happen after you don’t care about its result because
a parallel dependency has failed. E.g. suppose you have some rate limiter
where you pass an async function, get a promise for it’s result, and the rate
limited will queue your function to be called soon but with the condition that
it will only allow say 5 calls to be in flight. Now you have some resource-
downloading function which puts many separate download requests into the
ratelimiter in parallel, and combines the results in such a way that the whole
download fails if any request fails. But now if an early request fails, your
function will continue trying to download all the other resources even though
it doesn’t care about the result now that it has failed. You could imagine
having a way to cancel promises and maybe you would be clever and write really
solid programs and never have issues but to me this seems to be inviting very
difficult to debug problems (just like expecting any thread to be killed at
any random point (or at least a bunch of different points) would make correct
multithreaded programming harder.

Perhaps a more succinct description of problem 2 is that unwinding the “stack”
isn’t well-defined with an async programming model because “stack” doesn’t
really mean the call stack from the scheduler you whatever bit of async
computation you happen to be doing, but rather the combined dynamic scope from
everything that started the computation and is currently awaiting its
completion.

A related and, I think, very useful thing to have is a way to represent
computations which may signal conditions or be restarted. I think CL doesn’t
have a great story for this (threading isn’t really a part of the language
though). An example use case would be some program which has to process some
stream of events. If it gets stuck on one or some, you may want to continue
processing the rest of the events while deciding what restart to invoke for
the problematic ones. Perhaps an effect system could help with this.

~~~
phoe-krk
I'll be frank: my book does not touch this topic in the slightest, and this is
an open question. Common Lisp does not have any asynchronous features built in
and, as I've said in some earlier comment, the condition system is fully
synchronous.

This topic is still waiting to be explored, either with an extension of Common
Lisp or a completely different language that permits asynchronous programming
with an adaptation of the condition system to this programming paradigm.

------
daly
I tried to buy (pre-order) the book. I created an account on the Apress site
as described in their help section.

There is no link to buy the book. Sigh.

~~~
phoe-krk
It is an upcoming book, as the title describes. It will be available later in
the year.

~~~
MycroftJones
Amazon generally has a "pre-order" option. If there was a pre-order option, I
also would order your book. By the time your book is in print, I will have
forgotten about it. Fascinating topic; nice to see good coverage. I always
felt that callbacks and exception handling left a little something to be
desired. It looks like you've addressed that.

When your book is in print and available for sale I hope you post again here
as a reminder.

~~~
phoe-krk
The pre-order option is there! [https://www.amazon.com/Common-Lisp-Condition-
System-Mechanis...](https://www.amazon.com/Common-Lisp-Condition-System-
Mechanisms/dp/148426133X)

Thank you for the kind words; I hope that the book suits you well once it's
out.

