

Of Lisp Macros and Washing Machines - MrJagil
http://www.loper-os.org/?p=401

======
ekidd
I like macros and have used them heavily for most of my career. But in
practice, macros have some limitations:

1\. Macros allow you to easily improve your syntax, but offer less guidance
for improving your semantics. To use a mathematical analogy, it's nice to have
a better notation, but what you really want are better definitions and
theorems. Most macros are thin syntactic wrappers; few offer profound semantic
insights. (Although there are some lovely examples of the latter in PG's
excellent _On Lisp_ : <http://www.paulgraham.com/onlisp.html> .)

2\. Macros almost never call _macroexpand_ recursively, except inside of code
bodies. This means that macros like _defclass_ usually cannot be extended in
an incremental or modular fashion. Instead, you need to declare a _def-your-
class_ macro that can't usefully compose with any other _def-her-class_ macro
that you might encounter.

3\. Weaker mechanisms than macros can accomplish nearly as much, with fewer
issues similar to (2). For example, you can re-implement many Lisp macros
using Ruby's block syntax, metaobject protocol, and ability to call class
methods from inside a class body. And you can usually compose the resulting
DSLs freely, even when they're analogous to _defclass_ extensions—witness all
the ActiveRecord add-ons.

So I like macros, but I no longer think that they're the ultimate high-level
abstraction. They're mostly a nice way to incrementally improve your notation,
except in the hands of programming language designers who are prepared to pull
out all the stops.

Vladimir Sedach: _The Haskell gang is primarily interested in advancing
applied type theory._

If we're talking about high-level abstractions, this is probably not the best
way to describe Haskell. (In fact, Haskell's type system is notoriously _ad
hoc_ , compared to languages like ML.)

Originally, Haskell started out a project to explore lazy evaluation and
programming without side effects. But in recent years, the Haskell community
has been investigating ways to build better domain-specific languages using
ideas from mathematics: combinators, monads, co-monads, arrows, derivatives of
types, and so on.

So far, many of these Haskell ideas have a steep learning curve, but the
results are nonetheless impressive. If I were trying to design a "hundred-year
language", I would definitely pay close attention to the Haskell
community—they just bubble with fascinating ideas. And mathematical ideas tend
to endure.

~~~
sedachv
> Macros allow you to easily improve your syntax, but offer less guidance for
> improving your semantics.

This IMO is the biggest problem with macro usage today and needs to be
overcome. I've started a campaign against the "macros are syntactic sugar, and
I have a sweet tooth!" camp (<http://carcaddar.blogspot.com/2010/08/input-
needed.html>). I recommend reading this excellent parody blog post (written at
the height of PG-inspired Lisp mania) for background: [http://classic-
web.archive.org/web/20070706135848/brucio.blo...](http://classic-
web.archive.org/web/20070706135848/brucio.blogspot.com/2005/03/concision-is-
equivalent-to.html)

> Macros almost never call macroexpand recursively, except inside of code
> bodies.

This is by design. The expansion of a macro needs to be opaque - otherwise
where is the abstraction? When you macroexpand, you're opening the black box.
You can only rely on macroexpanding macros that you have some control over.

> This means that macros like defclass usually cannot be extended in an
> incremental or modular fashion.

This is because it's the same problem as extending an arbitrary grammar. You
can't compose two grammars and expect the result to be non-ambiguous.

> Instead, you need to declare a def-your-class macro that can't usefully
> compose with any other def-her-class macro that you might encounter.

This is completely, totally wrong. Things like _def-your-class_ is the reason
I started my education campaign. You should never write a _def-your-class_
macro. Macros are the wrong way to solve this problem, which is why CLOS and
the CLOS metaobject protocol include all the facilities you could want for
extending _defclass_ in a composable way (Art of the Metaobject Protocol is a
comprehensive, but confusing, reference).

~~~
ekidd
_You can only rely on macroexpanding macros that you have some control over._

I think we're looking at this from exactly opposite directions. I'm not
talking about code-walking pre-existng macros, but rather providing hooks for
other people to extend yours:

    
    
      ;; Completely hypothetical package management-system.
      (define-module foo
        (require 'bar)
        (file "foo.ss")
        (file "foo-tests.ss" :in-mode 'test))
    

There's a lot of macros like this in the Lisp world, and they define non-
extensible DSLs. For example, you can't define a macro _tested-file_ to
eliminate the duplication above:

    
    
      (define-module foo
        (require 'bar)
        ;; Doesn't work:
        (tested-file "foo"))
    

This could be supported if _define-module_ internally called _macroexpand-1_
on unrecognized child forms until it saw either _require_ or _file_. Then you
could define a _tested-file_ macro to extend _define-module_.

Note that the actual advisability of this approach varies greatly between Lisp
and Scheme dialects.

Contrast the following ActiveRecord example:

    
    
      class Person < ActiveRecord::Base
        belongs_to :organization
        
        state_machine :initial => :starting do
          state :starting
          state :online
          state :stopping
          state :offline
        end
      end
    

Here, we have a third-party state machine library extending ActiveRecord. This
is just one of several places where many Lisp macros tend to be broken or
inflexible.

Of course, macros _are_ a very useful tool. But they have limitations, and
other languages also provide interesting mechanisms for "writing programs that
write programs", with different tradeoffs.

 _You should never write a def-your-class macro._

Well, even if the CLOS metaobject protocol _does_ solve this problem, it
doesn't fix _defsystem_ , or any of the other thousand non-extensible
_defblah_ forms in the Lisp community.

~~~
swannodette
The amount of OO runtime metaprogramming crap out there far outweighs the
amount of macro metaprogramming crap from simple fact that hardly anyone knows
anything about macro metaprogramming. I've seen only a few non-extensible
_defblah_ and a _metric ton_ of non-extensible OO bullshit in major projects.

I agree that you have some valid points but your overall argument holds little
water. In good Lisps you have access to both runtime and compile time
abstractions. Used wisely they will always trump a system that can only lean
on runtime abstractions.

I'll also note that in my experience runtime metaprogramming is horribly
painful. And many, many, many others have pointed out this fact. There are
numerous cases where structural transformation is far simpler to reason about.

~~~
ekidd
Oh, I absolutely agree that OO languages do gross things, too. If somebody
gets me started on AbstractRequestProcessorFactoryFactory, or inappropriate
use of instance_eval, I can keep going for hours. :-)

Overall, macros are a very useful tool. But there's a school of thought that
sees them as the ultimate, universal abstraction. I'm no longer convinced by
that argument.

However, I do agree with your remark, elsewhere in this discussion, that _The
Reasoned Schemer_ is an extraordinary fine example of what macros can be used
for. It's one of my all-time favorite programming books.

------
swannodette
If you want to understand the beauty of macros don't read silly trollish
blogs. I've said it a million times and I'll say it a million more - look at
very good Lisp code that wisely unites elegant runtime code with just enough
macro sugar to have something beautiful and idiomatic. So far I can think of
no better text than The Reasoned Schemer,
[http://mitpress.mit.edu/catalog/item/default.asp?ttype=2&...](http://mitpress.mit.edu/catalog/item/default.asp?ttype=2&tid=10663).
You get the soul of Prolog in _200 lines_ of Scheme R5RS. The entire system
fits on 2 pages.

This book is written by three true masters of macros and functional
programming Oleg Kiselyov, William Byrd, and Daniel P. Friedman.

~~~
silentbicycle
Thank you for suggesting good code to read, so many comments like this just
leave people hanging. :)

I know _great_ C code to suggest, but draw a blank when it comes to Lisp
macros done right.

------
primodemus
"Wadler created ML to help automate proofs". Minor nitpick: actually, ML was
designed by Robin Milner:
<http://en.wikipedia.org/wiki/ML_(programming_language)>

------
kenjackson
What do CRUD apps have to do with Lisp macros? The points are disconnected and
points are introduced with no evidence or logic to back them.

The only thing I can gather is this guy is about to become a billionaire,
because he has found a way to rid the world of spreadsheets and CRUD apps...
apparently with steel mills.

~~~
troutwine
I try to think the best of people. With no code to Loper OS that I can find
and a writing style that perpetuates lisp elitism using peculiar analogies to
the Industrial Revolution without any evidence that this elitism is warrented
--I use lisp, I like lisp but it's one tool in a bag--I find it hard to
believe that Mr. Datskovskiy is serious. Surely this is brilliant satire?

~~~
asciilifeform
_> without any evidence that this elitism is warranted_

What kind of evidence would you accept?

And if you won't accept it from R. Gabriel, P. Graham, P. Greenspun, and other
"household" names, why should the rest of us bother sweating to assemble a
watertight case for your wastebasket's eyes?

~~~
troutwine
> What kind of evidence would you accept?

Code. Every author takes on himself the onerous to justify his claims. I am
familiar with the works of the authors you mention; they present a good
argument that lisp is a fine language for fine works. It is an entirely
different argument to assert that, merely through the use of lisp, you will
revolutionize the world.

That, as I read it, is the claim made by the original author. It is that claim
which has gone unsubstantiated, for years. The rest of you--incidentally, no
need to use exclusionary language: we are all friend here--need do nothing.
Mr. Datskovskiy has some code to deliver. A bootloader in two years is no
accomplishment, though Mr. Datskovskiy has excelled in punditry in this
period.

~~~
asciilifeform
Original author speaking.

If you actually read the blog (unbearable ordeal, I understand) you would know
that Lisp (in available incarnations on available hardware) is not enough:

<http://www.loper-os.org/?p=284>

Let's see how much of an OS _and hardware architecture_ you can design in
several years - _without using a single line of code written by another
person._ Starting with the NAND gate - because that is where you will have to
start if you truly intend to avoid recapitulating the mistakes of the past
three decades.

There was no sequel to the bootloader - I have been occupied with reverse-
engineering the bitstream format of a major brand of FPGA (no naming names) in
order to dispense with the proprietary x86 toolchain (and eventually with the
abomination called Verilog.)

Publishing incremental progress in bite-sized chunks is vastly over-rated. It
wastes everyone's time and contributes to the proliferation of hideously
ragtag systems reminiscent of "Junkyard Wars" (aka much of the software
industry.)

Punditry, on the other hand, is a relaxing and inexpensive hobby. The hosting
bill for the past year of the blog was around $30.

Most of my time is spent working on unrelated efforts which I am not at
liberty to make public. Currently, they leave precious-little time and energy
for my long-term (think lifetime) project.

Feel free, of course, to imagine that I am an idler sitting in an armchair
drinking the days away, if it makes you happy.

~~~
troutwine
Clearly you are a man of strong opinion. You are also needlessly rude and
insufferably self-important. I apologize that you took my critique so
personally, without, of course, apologizing for the critique itself. May you
succeed in your ambition.

------
agentultra
I told a friend how I was going to implement my next project in Lisp. He
immediately thought I was nuts to do so -- that it would never get acquired or
it would be too hard to hire people to work on when it grows. All of the
tried-and-true objections to using Lisp for a "real" project.

Then I read an article like this and realize I've been explaining it all
wrong.

This is a great read and a real eye-opener.

~~~
sedachv
Give your programmers a copy of _Practical Common Lisp_, it's not difficult to
learn.

------
joev
Coral cached version:

<http://www.loper-os.org.nyud.net/?p=401>

------
wnoise
Eh, in lazily-evaluated languages such Haskell, there is much less _need_ for
macros.

~~~
raganwald
True, but can't we say that about every non-trivial PL feature? For example,
we could say that:

 _In OOP languages such as Smalltalk, there is much less need for macros_.

The author's point is that programming languages are great when you're cutting
with the grain, when you're working in the domain the author optimized the
language around. But when you leave the domain, macros are an important tool.

Don't get me wrong, I think that lazy evaluation is an incredibly valuable
tool for separating what from how. But I wonder how it obviates the need for
macros any more so than a well-defined OO system or any other non-trivial
language feature.

~~~
wnoise
sedachv actually got my point. The standard _need_ for macros rather than
functions is to handle staged computations, distinguishing between names and
objects. Lazy evaluation really does strike right at the heart of this, unlike
something like "object system".

~~~
kenjackson
Closures give you this (staged computations). Macros really give syntactic
abstraction. That's it. That's nice, but you get the rest via other
mechanisms.

~~~
sedachv
Closures can't do partial evaluation, macros can.

~~~
kenjackson
I wasn't trying to imply you can do everything with closures. But rather you
could do lazy evaluation with it (I guess staged computations could include
various forms of pre-rutime evaluation).

