
Lisp Macros, Delayed Evaluation and the Evolution of Smalltalk - mpweiher
https://blog.metaobject.com/2019/03/lisp-macros-delayed-evaluation-and.html
======
lispm
That's a common misconception that Lisp macros are mostly used to 'delay'
evaluation.

What Smalltalk calls 'blocks' are just (anonymous) functions in Lisp. Books
like SICP explain in detail how to use that for delayed evaluation in
Lisp/Scheme:

[https://mitpress.mit.edu/sites/default/files/sicp/full-
text/...](https://mitpress.mit.edu/sites/default/files/sicp/full-
text/book/book-Z-H-24.html#%_sec_3.5)

~~~
reitzensteinm
What are lisp macros for if not to delay evaluation? I assume you're making a
distinction between purpose and mechanism here?

I've always understood the semantics to be:

fn: (a b c) => (call a (eval b) (eval c))

mac: (a b c) => (eval (call a b c))

~~~
lispm
That might be more like what was called an FEXPR mechanism in some early Lisps
or some niche Lisps, or even in languages like R.

[https://en.wikipedia.org/wiki/Fexpr](https://en.wikipedia.org/wiki/Fexpr)

When an FEXPR is called, it gets unevaluated args and can then at runtime
decide what to do.

Macros OTOH are a different mechanism, where expressions get rewritten at
macro expansion time, which can be for example at compile time. The macro then
gets called with arbitrary expressions it encloses (those don't need to be
valid code by themselves) and computes new source.

Thus macros are code generators from expressions. In a compiled
implementation, macro expansion is done with compilation interleaved: each
form gets expanded, even repeatedly until it no longer expands into a macro,
and then the resulting non-macro form gets compiled.

Thus in a way a macro does not delay execution, it does the opposite: it
actually shifts computation to compile time -> the computation of code from
expressions and the computation of arbitrary side effects in the compile-time
environment.

In an interpreter version of Lisp, the macro gets also expanded at runtime -
but in its own macroexpansion during evaluation. There eval will call the
macroexpander repeatedly until it gets a non-macro form.

Now, what can you do with arbitrary complex code generators at compile time?
[https://stackoverflow.com/a/2563308/69545](https://stackoverflow.com/a/2563308/69545)

Actually Paul Graham wrote a classical Lisp book explaining a bunch of things
around macros. Available here for download:

[http://www.paulgraham.com/onlisp.html](http://www.paulgraham.com/onlisp.html)

The classic Lisp article which motivated the move from FEXPRs to macros:

[https://www.nhplace.com/kent/Papers/Special-
Forms.html](https://www.nhplace.com/kent/Papers/Special-Forms.html)

~~~
reitzensteinm
I don't think what I wrote is a FEXPR, which doesn't evaluate the result of
the function call. I'm having a hard time parsing what you wrote or any of the
sources you linked in such a way that says the semantics of macros differs
from what I wrote (certainly the implementation gets a lot more complex and
there are subtleties).

~~~
lispm
That's why I wrote 'more like'. Given that you haven't defined any semantics
of your operators, it's more like a guess.

Not sure if this helps you. But let's define a macro A:

    
    
        CL-USER 32 > (defmacro a (b c)
                       (print (list :macro-expansion b c))
                       (list 'print (list 'quote (list :runtime :b b :c c))))
        A
    

This macro does two things: it prints something at macro expansion time and
then generates some code it returns as a value.

Now we can use this macro in some code:

    
    
        CL-USER 33 > (defun test ()
                       (a 21 42))
        TEST
    

If we now compile the function, the macro gets executed and prints something:

    
    
        CL-USER 34 > (compile 'test)
    
        (:MACRO-EXPANSION 21 42) 
        TEST
        NIL
        NIL
    

We can also call the macroexpander independent of the compiler. The we a) get
the side effect of the print statement and we see the result:

    
    
        CL-USER 35 > (macroexpand '(a 20 30))
    
        (:MACRO-EXPANSION 20 30) 
        (PRINT (QUOTE (:RUNTIME :B 20 :C 30)))
        T
    

At runtime we call the test function:

    
    
      CL-USER 36 > (test)
    
      (:RUNTIME :B 21 :C 42)  ; <- the printed output 
      (:RUNTIME :B 21 :C 42)  ; print also returns its arg
    

Thus all the macro expansions have been done at compile time and we have
generated some code there. No macro expansion at runtime.

Thus it has to do with code generation and code execution at macro expansion -
nothing about 'delaying' something.

The example isn't useful, but imagine a macro INFIX

    
    
       (infix a * b + c)
    

which rewrites the expression to the Lisp expression:

    
    
       (+ (* a b) c)
    

There is nothing about 'delaying' -> it's just rewriting the form. Ideally at
compile time. There are many other examples which do something different.

~~~
hibbelig
A normal function call evaluates the arguments and then calls the function.

Because macro expansion does something before evaluating the arguments, you
can say that evaluating the arguments has been delayed.

I feel this is just looking at the same thing from two different directions.

Of course, macro expansion does _more_ than just delaying the evaluation of
the arguments, and if people say that macros delay evaluating the arguments,
you might think that's all they do.

~~~
lispm
> Because macro expansion does something before evaluating the arguments

It does something independent of evaluation. When the code gets compiled, the
macroexpander already has transformed the code. The code might never be
evaluated in this Lisp. It might be written to disk and later be loaded into
another Lisp.

If something does not get evaluated, gets evaluated later, gets always
evaluated or never -> that depends on the generated code.

Thus 'delaying' something is the wrong idea and it limits the imagination of
what macros are used for. Think of 'general code transformation', often
independent of execution in a compilation phase.

------
j-pb
Lambdas are about delayed evaluation. Macros are about disabling evaluation.
Lisp code is just an abstract syntax tree notation format (s-exps) that comes
with default evaluation semantics. Macros allow you to disable those
evaluation semantics to reuse the AST for a different programming language.

So for example to add pattern matching facilities to the language, you come up
with a syntax and its representation in s-exp AST and then write a macro that
describes the unification operation using the default semantics.

Same for logic programming or any other paradigm not initially supportet.

Languages like clojure also bootstrap quite a bit of the language from a
simpler to implement dialect. (look at all the functions with a * in clj
source they're the base language)

~~~
lispm
lambdas are anonymous functions. Macros are code transformers. Lisp code is
not an AST.

~~~
j-pb
Different name for the same thing, whats your point? Lambdas/AF are used to
delay evaluation but keep the default semantics.

Macros are more than simple code transformers, that wording somehow implies
that they somehow retain the semantics of the data passed to them, which mighy
be the case but is not required at all.

S-Expressions are just a serialisation format for the m-expression AST.

~~~
lispm
Lambdas are just functions. Delaying functionality is just one use of
functions.

> that wording somehow implies that they somehow retain the semantics of the
> data passed to them

Since they can do arbitrary transformations, retaining semantics is not in
focus. Since the input may not have any semantics defined, the semantics is
actually provided via the macro implementation.

> S-Expressions are just a serialisation format for the m-expression AST.

S-expressions know nothing about 'syntax'. Thus they can't be an 'abstract
syntax tree'. (3 4 +) is a valid s-expression, but carries no information
about any syntax (what is the + ? in an s-expression it's just a symbol) and
is also an invalid Lisp expression.

An abstract syntax tree would be the result of parsing a program according to
some grammar and it would represent the syntactic categories. The parsing
stage would already eliminate invalid programs of that programming language.

[https://upload.wikimedia.org/wikipedia/commons/thumb/c/c7/Ab...](https://upload.wikimedia.org/wikipedia/commons/thumb/c/c7/Abstract_syntax_tree_for_Euclidean_algorithm.svg/800px-
Abstract_syntax_tree_for_Euclidean_algorithm.svg.png)

In Lisp it is important that the s-expression is NOT a syntax tree. Otherwise
it would be difficult to write macros which violate Lisp syntax.

~~~
agumonkey
I know you're rigorous and mostly right, but you forget to admit that for most
practical uses s-exps encode trees and are used as ad-hoc AST's. People just
make implicit grammars based on spec like predicate patterns.

~~~
j-pb
He's not even right. He's pseudo rigorous to support is CL zealot trolling.

He made it far enough into the wikipedia article to find a graphic that pseudo
supports his claim, but not far enough to actually read the definition of an
Abstract Syntax Tree (the thing we talk about) vs Concrete Syntax Tree (the
thing he talks about).

> This distinguishes abstract syntax trees from concrete syntax trees,
> traditionally designated parse trees, which are typically built by a parser
> during the source code translation and compiling process.

~~~
lispm
You might want to reread the Wikipedia article and tone down a bit:

Check out the abstract syntax tree on the right:

    
    
      https://en.wikipedia.org/wiki/Abstract_syntax_tree
    

It has a node which is a BRANCH and which has three relations CONDITION, IF-
BODY, ELSE-BODY.

In an s-expression this is just

    
    
      (if (> a b)
          (setq a (- a b))
          (setq b (- b a))
     

Thus there is no representation that IF is a branching expression, there is no
representation that A and B are variables. There is no representation that >
is a compare op. And so on.

The s-expressions are just nested lists of tokens without any idea what the
tokens refers to or what language construct it stands for. All we know is what
the tokens are and a hierarchy. A is a symbol, but what kind we don't know: it
could be a data object, a variable name, a function name, a goto tag, a name
of a class, a name of a type, ...). In the abstract syntax tree the > is
identified as a compare op, IF is identified as a branch, A is identified as a
variable identifier, ...

The Lisp reader also does not create that information. It just creates a data
structure, which could be anything, any kind of data.

------
User23
Algol 60 is wildly underappreciated, and pass by name[1] is a great example.

[1]
[http://www.cs.sfu.ca/~cameron/Teaching/383/PassByName.html](http://www.cs.sfu.ca/~cameron/Teaching/383/PassByName.html)

Edit: An elementary error is to assume that call by name is equivalent to pass
by reference. It's not.

~~~
mpweiher
"Here is a language so far ahead of its time, that it was not only an
improvement on its predecessors, but also on nearly all its successors" \-
Hoare

[http://www.computernostalgia.net/articles/algol60.htm](http://www.computernostalgia.net/articles/algol60.htm)

------
supernintendo
Macros work much better in homoiconic languages like Lisp than they do in
other languages. Most of the code I write these days is in Elixir and I avoid
macros unless absolutely necessary. The benefits generally don't outweigh the
(long-term) costs.

~~~
lispm
One of the differences between macros in Lisp and some other languages which
provide macros is that the expressions themselves don't need to be valid code
in some programming language -> they don't get parsed by a language parser
upfront.

Thus I can write a postfix macro and then write code like this:

    
    
      (postfix 2 3 + 3 *)
    

even though Lisp requires code to be nested prefix expressions.

Some other languages won't allow this, because the expression 2 3 + 3 * is not
legal in their language. Thus it only may allow macro transformations from
legal expressions to other legal expressions...

~~~
mpweiher
> don't need to be valid code

Yes. One of the examples from the talk was a comment macro. Very cool (and the
talk was about fun/cool stuff, not about practicalities).

The question is whether you want that sort of power in day-to-day programming.
My guess is no. That's also what the PARC/LRG folks found out with
Smalltalk-72. It's also something I hear from some very seasoned LISP hackers.
It's also the sense I am getting from these very powerful DSL/LOP workbenches.

From TFA:

 _The reason the question is relevant is, of course, that although it is fun
to play around with powerful mechanisms, we should always use the least
powerful mechanism that will accomplish our goal, as it will be easier to
program with, easier to understand, easier to analyse and build tools for, and
easier to maintain._

It's also why I use the C pre-processor _very reluctantly_. Though I do use
it. From time to time. And then try to get rid of that use if I can[1]

And no need to explain how much better LISP macros are :-) In a sense, like
Smalltalk, LISP may be just too powerful, in the words of Alan Kay "Lisps
frequently 'eat their children'" so that there's always an answer (use a
macro) that will cut off an interesting question.

[1] [https://blog.metaobject.com/2018/11/refactoring-towards-
lang...](https://blog.metaobject.com/2018/11/refactoring-towards-
language.html)

~~~
m00natic
"we should always use the least powerful mechanism that will accomplish our
goal"

I like this when implementing something for non proficient users. But when it
comes to providing tools for (supposedly) advanced users, like programmers...
There's late-"socialism" joke in Bulgaria: "thrift is mother of misery". A
designer doesn't know ahead of time what problems "creative" users will face
long term. Providing a set of simplest mechanisms for today's challenges would
possibly constrain them in the future - combination of multiple mechanisms in
ways not foreseen may add large incidental complexity (like OO design
patterns). Which could be avoided if less by count but more powerful
mechanisms were used in first place. Macros have main role in keeping Common
Lisp relevant to the latest paradigm hypes despite the standard being set in
stone. Opposite to this, for example, C++ must keep introducing piles of new
least-powerful mechanisms to keep pace.

~~~
mpweiher
>> ... _use_ the least powerful mechanism ...

> ... _providing_ tools ...

Use ≠ provide. :-)

See:

 _The Rule of Least Power_ , Tim Berners-Lee

[https://www.w3.org/2001/tag/doc/leastPower.html](https://www.w3.org/2001/tag/doc/leastPower.html)

See also:

 _Rule of least expressiveness_

When programming a component, the right computation model for the component is
the least expressive model that results in a natural program.

From _Concepts, Techniques, and Models of Computer Programming_ , Peter van
Roy,
[https://www.info.ucl.ac.be/~pvr/book.html](https://www.info.ucl.ac.be/~pvr/book.html)

~~~
m00natic
I was more after

> The question is whether you want that sort of power in day-to-day
> programming

It's good to use the least powerful mechanism, no doubt. But it seems you are
trying to sneak the usual "macros are too powerful for everyday use" so better
be left out of a language altogether? I think when the storm comes - you'd
better be equipped. Having varied ways to tackle problems (and macros are sort
of linguistic abstraction orthogonal to lambda calculus/Turing machine derived
toolboxes) allows for less complex solutions.

~~~
mpweiher
> sneak the usual "macros are too powerful for everyday use"

Not trying to "sneak" anything, I openly say that language design, which is
what macro usage is, is not something you should have to engage in everyday.
In fact, I would turn it around: if you have to (repeatedly) resort to
language design in your everyday programming, your programming language is
(woefully) inadequate. Most are.

Which is why the reasons for hitting that boundary interest me: where do I
have to resort to metaprogramming, why, and what can I do about it? What non-
metaprongramming facilities are missing here so that I don't have to resort to
metaprogramming? And if I don't want to just add those facilities to the base,
which I don't, what mechanisms can I add to the language so that users of the
language can use plain, non-meta mechanisms to provide those facilities
themselves?

This is a bit tricky, but I am making good progress using a software
architectural approach[1], with frequent surprises as to how much simpler
things can be.

> left out of a language altogether?

Quite the contrary. I think "escape hatches" (metaprogramming) are fundamental
and your everyday language(s) should be built 100% on top of those mechanisms.
Heck, I named my company "metaobject"[2] 20 years ago, after _The Art of
Metaobject Protocol_.

[1] [http://objective.st/](http://objective.st/)

[2] [http://www.metaobject.com/](http://www.metaobject.com/)

------
syastrov
Scala has implemented support for async-await syntax using its experimental
macros: [https://docs.scala-lang.org/sips/async.html](https://docs.scala-
lang.org/sips/async.html)

Just an example of how powerful macros can be.

They also have an example in their docs about implementing printf using a
macro so that formatting parameters are typechecked.

------
panzerklein
There is whole set of "with-x" macros that aren't about delayed evaluation.

~~~
User23
What's the with-x macro that isn't providing dynamic extent, that is, delaying
execution of some cleanup logic?

~~~
bluefox
Consider for example CL:WITH-SLOTS, or CL-WHO:WITH-HTML-OUTPUT... such macros
establish context, and are not really about delaying evaluation. On Lisp has a
section about uses of macros, which is not exhaustive, but shows there's more
to them than "delayed evaluation".

------
stevelosh
Lisp user here. I'll chime in with another example of using macros for more
than just delaying evaluation.

I wrote a library called Chancery[1] for procedurally generating strings (and
other data). It's inspired by Tracery[2] but takes advantage of macros to make
it easier to read and feel more like part of the language. I use it to write
stupid Twitter bots like
[https://twitter.com/git_commands](https://twitter.com/git_commands) and
[https://twitter.com/rpg_shopkeeper](https://twitter.com/rpg_shopkeeper) for
fun.

As an example let's say we want to generate a message about the loot we
receive from a monster in a fantasy, D&D-style story. Maybe we'll start with
some random weapons:

    
    
        (chancery:define-string weapon-type
          "sword"
          "spear"
          "lance"
          "flail"
          "mace")
    
        (weapon-type) ; => "mace"
        (weapon-type) ; => "sword"
    

This expands like so:

    
    
        (macroexpand-1
          '(chancery:define-string weapon-type
            "sword"
            "spear"
            "lance"
            "flail"
            "mace"))
        ; =>
        (DEFUN WEAPON-TYPE ()
          (CASE (CHANCERY::CHANCERY-RANDOM 5)
            (0 "sword")
            (1 "spear")
            (2 "lance")
            (3 "flail")
            (4 "mace")))
    

This simple case actually _could_ be done just by delaying evaluation, as long
as we get every body clause as a separate thunk. Now let's define a rule for
generating the material of a weapon:

    
    
        (chancery:define-string (weapon-material :distribution :weighted)
          (100 "iron")
          (40 "steel")
          (5 "silver")
          (4 "gold")
          (1 "adamantine"))
    

This will generate the materials according to a weighted distribution, and
macroexpands to:

    
    
        (DEFUN WEAPON-MATERIAL ()
          (CASE (CHANCERY::WEIGHTLIST-RANDOM #<CHANCERY::WEIGHTLIST ((100 0) (40 1) ...)>)
            (0 "iron")
            (1 "steel")
            (2 "silver")
            (3 "gold")
            (4 "adamantine")))
    

This case needs more than just delayed evaluation. If you receive `(100
"iron")` as an opaque thunk, where all you can do is evaluate it, there's no
way to pull out the weight and body components.

If we add a few more rules, we can see more cases where we need to go beyond
delayed evaluation:

    
    
        (defun currency-amount ()
          (+ 10 (random 100)))
    
        (chancery:define-string (currency-type :distribution :zipf)
          "copper"
          "silver"
          "gold"
          "platinum")
    
        (chancery:define-string loot
          #((weapon-material weapon-type) chancery:a)
          (currency-amount currency-type "coins"))
    
        (chancery:define-string discovery
          ("You open the chest and find" loot :. ".")
          ("You find" loot "in the monster's hidden stash.")
          ("You find nothing but dust and cobwebs."))
    
        (discovery) ; => "You find nothing but dust and cobwebs."
        (discovery) ; => "You find an iron sword in the monster's hidden stash."
        (discovery) ; => "You find 61 copper coins in the monster's hidden stash."
        (discovery) ; => "You open the chest and find a steel sword."
    

Macroexpanding the last one:

    
    
        (DEFUN DISCOVERY ()
          (CASE (CHANCERY::CHANCERY-RANDOM 3)
            (0 (CHANCERY::JOIN-STRING "You open the chest and find"
                                      (PRINC-TO-STRING #\ )
                                      (LOOT)
                                      "."))
            (1 (CHANCERY::JOIN-STRING "You find"
                                      (PRINC-TO-STRING #\ )
                                      (LOOT)
                                      (PRINC-TO-STRING #\ )
                                      "in the monster's hidden stash."))
            (2 (CHANCERY::JOIN-STRING "You find nothing but dust and cobwebs."))))
    

Here we can see the macro walking the lists and doing different things to each
element: strings are included raw, symbols are turned into function calls, and
the special keyword :. suppresses the usual joining space character inserted
between everything. There's also some special handling of vectors, in the LOOT
example, which I won't go into. This is more than just delayed evaluation —
we're inspecting the actual structure of the code received by the macro at
macroexpansion time. If all we had were an opaque thunk that we could evaluate
later, we couldn't do this.

Delayed evaluation is enough for certain kinds of abstraction, like writing
basic control structures, but isn't as powerful as full macros. Macros let you
transform arbitrary code into other arbitrary code using the full power of the
language.

[1]: [https://sjl.bitbucket.io/chancery/](https://sjl.bitbucket.io/chancery/)
[2]: [http://tracery.io/](http://tracery.io/)

