
Lisp: it's not about macros, it's about READ (2012) - lisper
http://jlongster.com/Lisp--It-s-Not-About-Macros,-It-s-About-Read
======
bsder
Okay, I'll put something heretical out there:

Lisp just isn't that much better than our current languages. Sorry.

Sure, when Lisp came out it was "advanced". Garbage collection, AST parsing
and manipulation, macros, packages, a standard library, modules, homoiconic
data structures, etc.

However, the good features of Lisp are now _normal features_ of any modern
language. And some of those features have gone into the dustbin for good
reasons. And some of them (static vs dynamic typing) are personal taste.

I remember using Lisp in 1984. It was completely eye-opening relative to the
BASIC, assembly, and C I was using up to that point. However, it didn't run
well on small machines that we plebians were using back then. So, we plebians
moved back to assembly, C and possibly Pascal.

Lisp _STILL_ has the same problem on small processors. The embedded world is
desperate need of a good language for rapid development, and yet Rust seems to
be the only new contender? No offense to Rust, but is that the best we can do?

Where is a good dynamic language that is always online and runs in <8K of
flash/RAM?

~~~
aidenn0
> However, the good features of Lisp are now normal features of any modern
> language. And some of those features have gone into the dustbin for good
> reasons. And some of them (static vs dynamic typing) are personal taste.

I definitely agree that with each passing year, languages that are not
recognizable as being related to lisp have more features that mad lisp special
in the past.

Features I regularly use that are good, but aren't "normal features of any
modern language" (some are implementation details, but common to most CL
implementations)

1 CLOS

2 Homoiconicity

3 A good interactive debugger with incremental compilation

4 Generalized references

5 An equivalent to reader macros

In addition, while some form of AST parsing and manipulation may exist in many
modern languages, it is typically much less accessible than in LISP. I think
this is largely because of the lack of homoiconicity, but also partly a
culture issue.

There are probably other features that I don't use that others make good use
of as well.

As a last comment, it baffles me that #3 is something that is extraordinarily
rare to find outside of the lisp, smalltalk and forth families, but is
regularly mentioned as a huge productivity boost by programmers that use it.

There's actually no reason I can think of that non-pure statically typed
languages couldn't have such tooling either, with the restriction that global
variables (including functions) never change type.

~~~
mbrock
Some more stuff:

1\. The condition system including restarts to recover from errors.

2\. A uniform syntax that can be fluently manipulated with paredit-style
modes.

3\. VM image saving/loading.

4\. Runtime access to the compiler and very complete reflection.

5\. A package system (ASDF) that lets you dynamically recompile and reload an
entire system with all its dependencies just by calling a function.

6\. First-class namespaces.

7\. Well-defined multiple dispatch.

8\. Metaobject protocol.

9\. Language specification with conceptual clarity.

10\. High-quality optimizing compilation despite extreme dynamicity.

I think the list could go on for a while...

~~~
aidenn0
I think VM image saving/loading is still debatable as to whether or not it's
good, assuming that it's the primary way of generating a binary image, as it
is in most lisps.

A lot more work has to go into guaranteeing reproducible builds, and it tends
to be a fairly non-compact representation of your executable. If CL were a
little bit less customizable at load-time, that would allow for much faster
FASL loading, which in turn would make the reliance on saved-images less
important.

Obviously there are uses for that feature, and having a feature doesn't by
itself make a language worse, but when that feature is a square-peg, but still
the best fit for a particular round-hole, it causes problems.

~~~
mbrock
I found it pretty easy to generate images in a reproducible way: just load
your packages, run whatever setup you want, then save the image. Compactness
hasn't been a problem for me but I suppose it could be. Admittedly I don't
have much real experience with deploying Lisp applications in any serious way;
I'm curious about what problems people run into.

~~~
aidenn0
Have you read any of the XCVB papers? They dig deeper into the issues with
generating reproducible images, in particular with parallel builds.

------
beders
It's all about the read. But a different one. The human ability to read code.

You can write very dense code in Lisp (and the author makes the point several
times on how little lines of code you need for this vs. that). However, who's
going to read that and understand it?

Especially problematic with code that looks like a regular s-expression but
has a completely different semantic thanks to macros.

Lisp is great if you want to write code. Reading it...you mileage may vary

~~~
junke
> a completely different semantic thanks to macros.

Most of the time, you use functions. Sometimes, you need to define "WITH-X"
macros or "DO-Y" iterators, or a custom "DEFINE-Z" to shorten declarations (I
tend to use macrolet for that).

    
    
        (defun rgb (r g b) (format nil "#~@{~2,'0x~}" r g b))
    
        (defparameter *background* (rgb 10 10 30))
        (defparameter *foreground* (rgb 90 90 255))
    
        (with-html-output-to-string (s)
          (:html
           (:head
            (:title "Hello")
            (:style :media "screen"
                    :type "text/css"
                    (str (cl-css:css
                          `((:body :background-color ,*background*
                                   :color ,*foreground*)))))
            (:script :type "text/javascript"
                     (str
                      (ps:ps  
                        (setf (ps:@ window onload)
                              (lambda () (alert "Message")))))))
    
           (:body
            (:h1 "Heading"))))
    
    

The above is an example of what I consider a heavy usage of macros, and yet
somehow I find it readable.

(the output is here:
[http://pastebin.com/raw/S5cb5ufC](http://pastebin.com/raw/S5cb5ufC))

~~~
SeanLuke
For me the #1 rule for macros is to never, ever use one if you can use a
function instead. With this in mind, surely you could just write

    
    
        (html
          (head
            (title "Hello")
          (style :media "screen" :type "text/css" ........))))
    

What is the value of a macro in this context?

~~~
junke
The macro is documented here: [http://weitz.de/cl-
who/#syntax](http://weitz.de/cl-who/#syntax)

In particular, each list beginning with a keyword is transformed into an
(X)HTML tag. It is an easy way to handle all current and future HTML elements
without defining a function for each element and their attributes. You might
be thinking: "we could generate invalid HTML", and yes, it is true. Other
libraries exist to generate HTML, but this one is quite popular.

The other interesting thing is that by processing the tree at compile time,
you can make some optimizations. For example, the following form:

    
    
        (:html (:body (:h1 "Hi")))
    

... is expanded as:

    
    
        (WRITE-STRING "<html><body><h1>Hi</h1></body></html>" *STANDARD-OUTPUT*)
    

... because it does not depend on external data. However, if instead of a
constant string, you place (str (something)), you have instead:

    
    
        (WRITE-STRING "<html><body><h1>" *STANDARD-OUTPUT*)
          (LET ((CL-WHO::*INDENT* NIL))
            NIL
            (STR (SOMETHING)))
        (WRITE-STRING "</h1></body></html>" *STANDARD-OUTPUT*)))

------
DalekBaldwin
> So what are macros? All they are is read packaged up nicely into formal
> system.

Macroexpansion happens during a different phase altogether, and would work
exactly the same even if programmers had no access to the reader at all. The
reader transforms an unstructured character stream into structured data, while
macroexpansion transforms already-structured data into different structured
data. (Read-macros are another story, but it's clear they're not what the
author had in mind.)

Too often these kinds of posts only add to the confusion they're attempting to
clear up.

------
auganov
tldr it's not about macros it's about homoiconicity;

~~~
imglorp
Surprised op didn't refer to it by name. We can handle big words.

------
dschiptsov
Nope. Lisp is about extending itself with new special forms and embedding DSLs
into itself ( SETFs, LOOPs, DEFSTRUCTs, even whole CLOS, to name a few).

With macros one could introduce new special forms, which differs from mere
high order procedures by having its own evaluation rule implemented as a
macro. In particular some of its arguments could be left unevaluated before
application, which gives one laziness and other nice things (for an ordinary
procedure, according to the general evaluation rule, all the arguments will be
evaluated in order before application of the procedure).

Lisp isn't about hipsters. It is about fundamentals of programming.

Here is a simple illustration:

[http://karma-engineering.com/lab/wiki/Bootstrapping8](http://karma-
engineering.com/lab/wiki/Bootstrapping8)

------
pramodbiligiri
Previous discussion:
[https://news.ycombinator.com/item?id=3607248](https://news.ycombinator.com/item?id=3607248)

------
capitalsigma
How is this any different from `eval`, except that lisp's syntax is
particularly well suited to being `eval`'d by virtue of being so easy to
parse?

~~~
chowes
I think the main benefit comes from being able to build tools to transform,
refactor, etc. your code in very little effort.

~~~
agumonkey
Yes, it raises the abstraction level to allow 'meta' programming as the normal
level. Brown university had a chapter on their old PAIP book saying that they
voluntarily skipped parsing because it's of no interest, so they used a scheme
as a basis for classes.

------
rpazyaquian
This article helps explain the how, but doesn't really cover the _why_. Coming
from a functional programming perspective (Ruby, Elixir, Clojure [yes I
know]), I don't see where macros become useful or anything more than a
curiosity outside of a couple situations:

1\. Implementing a DSL, which is its own beast and is more often a bad idea
than not; or

2\. Actually, I can't think of anything else.

Granted, I might be spoiled by modern conveniences already implemented in
Clojure, such as cond. It's just hard to see what macros can give me that I
can't accomplish with functions. If code is data, then aren't functions, which
operate on data, already what we're looking for?

~~~
nickpsecurity
"Implementing a DSL, which is its own beast and is more often a bad idea than
not"

Those using languages supporting that disagree. Let's assume the DSL is syntax
embedded in a language like LISP or Haskell so you don't get stuck in the DSL.
Essentially, there's not much difference between a DSL and library in terms of
what's required to use it. You still learn what you type to achieve a certain
effect. You still can use rest of language. Main difference is both (a)
matches the problem domain more closely and (b) limits you to expressing just
what you need to solve the problem.

These are why the BASIC-oriented 4GL's were quite successful. LISP, too, given
it's also easy to process. REBOL & recently Red take it further. Kay et al are
working wonders where a whole system is expressed in readable code that's a
tiny fraction of code in a non-DSL system.

The combo of easy-to-modify language and DSL toolkit (esp Racket or Red) also
leads to methodologies like this one from sklogic where it's easy to do
components with a series of DSL's that are thrown together & many pieces
autogenerated:

"The method is very simple:

* describe the problem in plain English (maybe with some diagrams)

* iterate it a few times until you have a syntax you think is unambiguous enough

* strip this syntax from all the sugar you just introduced in order to define an AST

* find DSLs in your toolbox that are potentially close to the one you're building and cherry-pick the necessary language components from them

* write a sequence of very simple transforms that would lower your source AST into a combination of the parts of the ASTs of the target DSLs of your choice

* Done. An efficient DSL compiler is ready, with a language designed as closely to your current view of the problem domain as possible.

* If you later find that your DSL is inadequate and your understanding of the domain was insufficient, than just start over again, this entire process is so cheap and simple that it does not really matter."

It also helps with verification. You can define DSL's for specific types of OS
components or app software that make correct-by-construction easier. CertiKOS
is doing that:

[http://flint.cs.yale.edu/certikos/publications/ctos.pdf](http://flint.cs.yale.edu/certikos/publications/ctos.pdf)

[http://flint.cs.yale.edu/certikos/certikos.html](http://flint.cs.yale.edu/certikos/certikos.html)

------
thelazydogsback
It always struck me as odd that many programming languages have standard
printers that will output any data structure expressible in the language, but
they don't have accompanying readers - this forces one to roll one's own or
use a completely separate serialization technique. This is useful even when
not in a homoiconic language for day-to-day tasks.

------
golergka
Fantastic article, but I still don't quite understand what kind of real-world
tasks on typical projects become easier with AST manipulation. After all, how
often do you write a debugger?

~~~
sedachv
> Fantastic article, but I still don't quite understand what kind of real-
> world tasks on typical projects become easier with AST manipulation. After
> all, how often do you write a debugger?

A good example comes from Python: decorators had to be added as a special
extension in 2003
([https://www.python.org/dev/peps/pep-0318/](https://www.python.org/dev/peps/pep-0318/))
and the `with` statement in 2005
([https://www.python.org/dev/peps/pep-0343/](https://www.python.org/dev/peps/pep-0343/)).
Both are very useful for web programming, but do not need PEP committees and
years of work when you have macros (examples:
[https://github.com/vsedach/cliki2/blob/master/src/accounts.l...](https://github.com/vsedach/cliki2/blob/master/src/accounts.lisp#L213)
[https://github.com/vsedach/cliki2/blob/master/src/wiki.lisp#...](https://github.com/vsedach/cliki2/blob/master/src/wiki.lisp#L155))

------
dogfishbar
Ha! LISP macros and read both work because of a very simple bug in the
original definition of LISP. Don't even bother with it or Scheme or Racket ---
they're all utterly broken and needlessly confusing!

~~~
ZenoArrow
Which bug are you referring to?

~~~
dogfishbar
The definition of quote is broken. I personally explained it to John McCarthy.
He agreed.

~~~
sesquipedalian
Care to elaborate?

~~~
dogfishbar
OK, you asked! LISP was originally developed as a language for writing
recursive functions of symbolic expressions (S-expressions). It was roughly
based on lambda calculus, the language developed by Alonzo Church. But roughly
is the key word. S-expressions are given by the context-free grammar:

S ::= A | [] | (S . S)

One can write data structures this way and lists by using the abbreviation:

(S1 . (S2 . ( ... ( SK . ()) ...))) == (S1 S2 ... SK)

The terms that manipulated these S-expressions were called M-expressions. They
were first-order terms. McCarthy's key idea was to use a conditional in
conjunction with a label form to define recursive functions (in the service of
various AI applications). The M-expressions were defined roughly as follows:

M ::= S | x | if[M; M; M] | f[M; ...; M]

f ::= lambda[[x1; ...; xn]; M] | label[g; M]

(I'm not 100% confident that I remember the exact details on this syntax but
the idea is correct!)

McCarthy wanted to show that his new language was Turing-complete. So he
wanted to exhibit a universal function APPLY (derivable from f above) such
that for any function f and arguments M1; ...; Mk such that f[M1; ...; Mk]
evaluates to S-expression S, well, given a representation of f[M1;...;Mk],
lets call it, hat(f[M1;...;Mk]), well

APPLY[hat(f); hat(M1);...;hat(Mk)] would evaluate to hat(S). This is pretty
much a standard formulation of the recursion-theoretic argument. In order to
close the sale, McCarthy had to exhibit such an APPLY and also the hat(.)
function. Sadly for all of us LISP lovers (!) he botched the definition of
hat(.)! It left people utterly confused for 30+ years. Such a shame.

Fixed a couple of typos. Sorry! Fixed one other typo! It's been a while...

~~~
akkartik
I still don't understand. You said the definition of _quote_ is wrong, but
didn't mention it at all in your elaboration.

~~~
dogfishbar
Fair enough, I felt I was droning on but it's true that I didn't show the key
mistake. Here it is.

If you want to represent an arbitrary M-expression as an M-expression, it's
most natural to use S-expressions for the representation language, these are
the -values- in M-expression LISP. (In lambda calculus we have more choices,
normal-forms or weak-head normal-forms). McCarthy defined hat(.), naturally
enough, by induction on the structure of M-expressions. For each M-expression,
we need an S-expression representation. (Note that we use uppercase symbols
for the symbolic constants and lowercase symbols for identifiers.) Here goes:

hat(S) == (QUOTE S)

hat(x) == X

hat(if[M1; M2; M3]) == (IF hat(M1) hat(M2) hat(M3))

etc..

But HOLD ON! The S-expressions have inductive structure(!). The definition of
hat(S) should have been:

hat(A) == (SYM A)

hat(()) == (NIL)

hat((S1 . S2)) == (PAIR hat(S1) hat(S2))

There are sensible mathematical properties that this latter representation has
that the former doesn't. It's a bit of a long story. But the bottom line is
that QUOTE, was defined erroneously. (And John McCarthy burst out laughing
when I explained it to him.)

RM

apologies, more typos.

~~~
abecedarius
I've written a Lisp compiler that handles quotations this way, so that QUOTE
is not part of the target language. (Not that this was my own idea.) I can
sort of see why you'd call McCarthy's s-expression Lisp a mistake, in that it
adds something (QUOTE) not in the original M-expression Lisp, which you can do
without. It still seems to me like a strange thing to emphasize, but OK.

BTW, the M-expression syntax for IF was "test -> consequent; alternative"
which got encoded as a COND expression. (Doesn't matter, I'm just pointing it
out as long as I'm commenting.)

~~~
dogfishbar
Thank you for reminding me, McCarthy also invented COND which eventually led
to the great modern pattern matching forms.

An ironic side-story of this that may or may not be of interest:

Because QUOTE was mis-defined, McCarthy had to hack his definition of
APPLY/EVAL to get it to work. One consequence of this hacking was that the
S-expression LISP "defined" by his version of APPLY/EVAL was a higher-order
language while the M-expression LISP that he was attempting to model was
strictly first-order. So in his S-expression LISP he could write the MAP
function (called "mapcar" back in the day) but the syntax of M-expressions
leaves no way to express MAP.

I find it so ironic that it took this little representation error to lead to
LISP having the essential property of lambda calculus. (Guy Steele fixed most
of the trouble with the grammar and introduced proper lexical scoping in
Scheme but he didn't catch the quote bug.) It's also fair to say that
M-expression LISP wouldn't have changed the world as S-expression LISP did.

I don't know if Paul Graham reads HN but Paul once wrote a book on macros in
LISP. As far as I know, he doesn't know this story about QUOTE. It doesn't
seem to have slowed him down.

------
wallforbidmen
I can say an image with sbcl (lisp implementation) with a load of functions
and libraries. I can't save an image with python, ruby and javascript.

~~~
naveen99
You can checkpoint at a lower level
[https://en.m.wikipedia.org/wiki/Application_checkpointing](https://en.m.wikipedia.org/wiki/Application_checkpointing)
or using a virtual machine.

But in practice it's not that useful to checkpoint a running program because
of external state. You can't checkpoint sockets when they are connected to
some other machine.

------
mdpopescu
An article that mentions abstracting things and then immediately uses CAR and
CDR...

