
Ask HN: Are macros still an enormous edge in the power of a language? - revenant
I use Lisp (Clojure) a lot and like it, but it seems like macros, except in specialized cases, aren't quite the game changer they used to be. They're much tougher to reason about than functions, can't be passed around as parameters, and can lead to "magic" DSLs that only their owners understand. Obviously they are sometimes very useful, and the built-in macros (e.g. cond, if-let) are essential, but I don't see them as such a game-changer in modern programming. I feel like over 90% of the edge Lisp has over Blub comes from its functional support, with less than 10% coming from macros. This makes the edge small enough that, in many circumstances, I can see languages like ML and Haskell winning; <i>good</i> static typing (e.g. ML, not Java) becomes a benefit on large projects.<p>I could be convinced that I am wrong and just don't "get" macros yet, but it seems like much of what's shown off in <i>On Lisp</i> et al is available nowadays in non-macro languages. I'd like to see modern uses for macros that aren't easy to implement in macroless languages.
======
sedachv
I used to like arguing over the Internet about this subject. There are many
good technical and management/organizational arguments you can make for and
against macros. What I've come to realize is they're all pretty much
irrelevant.

The entire point of programming is automation. The question that immediately
comes to mind after you learn this fact is - why not program a computer to
program itself? Macros are a simple mechanism for generating code, in other
words, automating programming. Unless your system includes a better mechanism
for automating programming (so far, I have not seen any such mechanisms),
_not_ having macros means that you basically don't understand _why_ you are
writing code.

This is why it is not surprising that most software sucks - a lot of
programmers only have a very shallow understanding of why they are
programming. Even many hackers just hack because it's fun. So is masturbation.

This is also the reason why functional programming languages ignore macros.
The people behind them are not interested in programming automation. Wadler
created ML to help automate proofs. The Haskell gang is primarily interested
in advancing applied type theory.

Which brings me to my last point: as you probably know, the reputation of the
functional programming people as intelligent is not baseless. You don't need
macros if you know what you are doing (your domain), and your system is
already targeted at your domain. Adding macros to ML will have no impact on
its usefulness for building theorem provers. You can't make APL or Matlab
better languages for working with arrays by adding macros. But as soon as you
need to express new domain concepts in a language that does not natively
support them, macros become essential to maintaining good, concise code. This
IMO is the largest missing piece in most projects based around domain-driven
design.

~~~
mwh
Excellent analysis!

------
pg
Yes. It's part of good Lisp style not to use a macro when a function would
suffice, and yet the source of HN is full of macro calls. If you want to see
examples of modern uses of macros, that would be a good place to start.

<http://ycombinator.com/arc/arc3.tar>

If you want a single, self-contained example, here's one:

    
    
        (mac zap (op place . args)
          (with (gop    (uniq)
                 gargs  (map [uniq] args)
                 mix    (afn seqs
                          (if (some no seqs)
                              nil
                              (+ (map car seqs)
                                 (apply self (map cdr seqs))))))
            (let (binds val setter) (setforms place)
              `(atwiths ,(+ binds (list gop op) (mix gargs args))
                 (,setter (,gop ,val ,@gargs))))))
    

How do you implement that in a language without macros?

~~~
rbanffy
Care to explain it for the Lisp-impaired?

~~~
pg
It lets you apply an arbitrary function to a variable or a position anywhere
within some structure:

    
    
        arc> (= flag t)
        t
        arc> (zap no flag)
        nil
        arc> flag
        nil
        arc> (= foo '(1 2 3 4 5))
        (1 2 3 4 5)
        arc> (zap (fn (x) (* x 10)) 
                  (foo 0))
        10
        arc> foo
        (10 2 3 4 5)
    

BTW, in more idiomatic Arc you'd say

    
    
        (zap [* _ 10] foo.0)
    

Here's a line in HN that uses it:

    
    
        (zap [firstn votewindow* _] (uvar user votes))
    

This expands into roughly the equivalent of:

    
    
        (= ((profs* user) 'votes)
           (firstn votewindow* ((profs* user) 'votes)))
    

Not quite, though, because you only want to evaluate the code that gets you to
the location once, and the operation needs to be atomic. The actual
macroexpansion is much more complicated.

~~~
viraptor
If that's all it does, then I don't see a problem with implementing it in any
language without macros. Let's take simple C. (I didn't test it, or even try
to compile - just want to show the idea)

    
    
        void zap(void *object, void* (*selector)(void *), void (*action)(void*)) {
            action(selector(object));
        }
    

Given your explanation: "It lets you apply an arbitrary function[1] to (a
variable or a position anywhere within some structure)[2]."

Where object is the structure you're operating on, `selector` is a function
taking an object and returning a pointer to the position [2]. The `action` is
a pointer to an arbitrary function [1].

Your first example would look like this:

    
    
        int foo[5] = {1, 2, 3, 4, 5};
        void *select_first(void *x) {return x;}
        void act_times_10(void *x) {int *i = (int *)x; (*i)*=10;}
        
        zap(foo, select_first, act_times_10);
    

Or am I missing something you didn't include in the description?

~~~
gruseom
I think you're missing the fact that in the C version you have to write a
specialized selector function each time (to find the thing you want to zap),
whereas the macro works generically on any place. ("Place" here is an
abstraction meaning "settable thing" that is closely related to Lisp macros
and is analogous to the left-hand-side of assignment statements in most
languages, but far more manipulable by the programmer.)

~~~
viraptor
PG asked "How do you implement that in a language without macros?". I
responded with a simplest translation of the example, but a more general
solution exists too. Nothing stops me from using GObject instead of void*,
which can provide named "places" and as much generic functionality as needed.
`selector` and `action` can even operate on property tables, which would be a
closer translation from lisp.

I was simply surprised that PG asked about something like that - I still think
it's pretty simple and the code in C is as close to the macro as possible
(with whatever object abstraction you need).

However I have to admit that I thought about other languages now and it seems
that the described zap() is not possible in python... and that's interesting /
worrying:

    
    
        action(object[place])
    

cannot assign a new value to `object[place]`, while

    
    
        object[place]=action(object[place])
    

has to do the lookup twice.

------
magice
The issue of macro is that it is, by itself, not a power. It's like how knife
by itself is not food. However, knife can make you food. That's the main
point.

Imagine this Christmas, Perl 6 is out with some killing features. Now, your
language (Python? Haskell? ML? Whatever) does not have these features. What
would you do?

You have 2 choices, basically: a. forget about that features, write a long
blog in which you prove how your language can do just the same thing, albeit
"a little" less elegant b. Switch to perl. This is not always possible, btw,
since you may sacrifice some OTHER features that your language has.

How about a LISPer? Well, simple, let's implement these features. Keep in
mind: All languages are Turing equivalent, meaning they can do roughly the
same thing. There are two differences: 1\. some do some task more elegantly
(aka easier to maintain) than others. 2\. Some have better support (eg.
corporates, community, etc.) than others

Thus, you CAN always write the exact same features in LISP. Now, the
definition of the macros WILL be messy. However, as a good practice, you spend
2 days debug that holy mess, and jam it into a file you banish somewhere with
a dot precede the directory. Next time, you can use the features, pretending
that you are using Perl 6!

This is my strategy. My Scheme code always has for-loop (purely functional,
too), foreach-like, OO-like features. Of course, Scheme (remember, it's not
even CL) does not provide these, but I can either implement them or find the
implement somewhere. I have once even try to implement Icon-like mechanism
with Macro + Continuation (okay, the Scheme implementation that time was a bit
fuzzy, so I failed. But, with pure Scheme, it SHOULD be possible). That's the
main power: you can copy others' ideas extremely fast! Without throwing away
your old code, too. Hurray to Macro!

------
apu
Are macros still at the "top" of the features pyramid?

In <http://www.paulgraham.com/avg.html> , PG argues that Lisp is more powerful
than other languages because it has macros. However, in the 50 years since the
invention/discovery of Lisp, is there now some language that has a new feature
X that lisps don't have (and can't easily add)? Or are macros still the
"pinnacle" of language features?

~~~
pg
_Lisp is more powerful than other languages because it has macros_

It's a bit more complicated than that. The key idea is that programs are
composed of lists. But that idea is tied to several others:

<http://www.paulgraham.com/diff.html>

~~~
apu
Fair enough, although I was asking a different question (not very clearly, I
suppose). Is there a language X and a feature (or set of features) Y for which
the following statement holds?

"Language X is more powerful than Lisp because it has feature(s) Y."

------
zandorg
In 2006 over 2 months, I wrote an OCR (optical char recognition - turning
images back into text) system in Lisp using macros.

First it finds all 'features' by traversing a 1-pixel wide representation of
the text character's image. Then you describe which features you want to look
for, in a macro language.

So (in layman's terms) , a 'c' is a top curve curving up, a lower curve
curving down, and a stem at the left-hand side close to the left-hand of those
curves. So 'c' would be about 3 rules, in this macro language.

It was fun but monstrously sluggish and I never got the 1-pixel wide algorithm
(called skeletonisation) that I used, to work 100%.

One thing it did have going for it was you could identify really small font
chars and even 10-pixel-high truetype fonts, as long as you normalise so each
char is 60 pixels high.

I ported the system to C++ also in 2006 and had to write a simple Scheme-ish
interpreter to turn the Lisp ruleset into C code.

------
gruseom
Macros expand into other code, so in the literal, obvious sense, there isn't
anything you can do with them that you can't do without them. The question is
whether you can do it as briefly, as clearly, or as composably.

Here's another example. I work on a spreadsheet app. A spreadsheet has many
operations that apply to either a row or a column (e.g. selecting a row/col or
resizing it). The fact that they occur in transposed dimensions make them
difficult to abstract over with functions. You probably end up with a bunch of
boilerplate that says things like "if this is a column, change the first co-
ordinate, otherwise change the second", or most likely would just write the
code twice, once for the horizontal case and once for the vertical. In Lisp,
we can write such a function for one dimension and then use macros to generate
a transposed version of the same function that does exactly the same thing in
the other dimension.

~~~
seunpy
That's the first use of a macro I can wrap my head around, but it can also be
done elegantly without macros - each function that acts on a row or column of
data will have a "list of values" argument that could either be a list of
column values from a row or a list or row values from a column.

~~~
gruseom
Not quite. I wasn't clear enough. These operations are purely UI
manipulations. They don't operate on lists of values or even need to know what
they are. What they care about are things like: should I set a _width_
property or a _height_ property? To which you might reply: fine, just pass the
dimension around and have a _set-size_ function that knows whether to set
width or height depending on the dimension, and so on. I say: yeah, you can do
it that way, but it takes more code and is harder to read, because now you
have to write it in a dimension-agnostic way, which is not how humans (or I at
least) like to think about these things. It's easier to write or read the
concrete version for one dimension and then say "the other dimension is just
like this only transposed", which is exactly what the macro does for us.

Do you need macros to make such a program work? No. Can you use them to make
the program shorter, clearer, and more generalizable? Yes.

------
Hexstream
I couldn't live without _iterate_ , a very intuitive, lispy, simple, mostly
"backwards compatible" replacement for Common Lisp's clunky, unlispy, ugly
_loop_ iteration facility.

<http://common-lisp.net/project/iterate/doc/index.html>

[http://www.lispworks.com/documentation/HyperSpec/Body/06_a.h...](http://www.lispworks.com/documentation/HyperSpec/Body/06_a.htm)

In a language without macros, I'd be stuck with _loop_ and curse every time I
need to write some non-trivial iteration code. But thanks to macros, I can use
this wonderful iteration facility as a library, and have great integration
with CL, just like if it was part of the language.

~~~
Hexstream
There's also this macro I wrote that I couldn't live without, I call it
_fmask_.

Problem: I always hated that _(loop for element in list collect (my-fun
element))_ translates trivially to _(mapcar #'my-fun list)_ but _(loop for
element in list collect (my-fun element my-constant))_ doesn't. You'd either
have to stick with the loop version or use something ugly and inefficient like
_(mapcar #'my-fun list (make-list (length list) :initial-element my-
constant))_ or write out the lambda: _(mapcar (lambda (element) (my-fun
element my-constant)) list)_ (my-constant could actually be an arbitrary
form).

None of these solutions appealed to me. So I wrote fmask. With it, the example
is simply rewritten as _(mapcar (fmask #'my-fun ? (? my-constant)) list)_

Another example:

    
    
      (mapcar (fmask #'list ? (? (1+ (length ?))))
      		 '(3 8 2) '(("some" "items") "A string" (nil "test")))

=>((3 3) (8 9) (2 3))

Here's the implementation:

    
    
      (defmacro fmask (function unknown (&rest args))
        (check-type unknown symbol)
        (macrolet ((autotraverse-let (bindings &body body)
    	         (let ((gensyms (iter (repeat (length bindings))
      				    (collect (gensym "TRAVERSED")))))
      		 `(let (,@(iter (for (nil value) in bindings)
      				(for gensym in gensyms)
      				(collect `(,gensym ,value))))
      		    (symbol-macrolet (,@(iter (for (var nil) in bindings)
      					      (for gensym in gensyms)
      					      (collect `(,var (prog1 (car ,gensym)
      								(setf ,gensym (cdr ,gensym)))))))
      		      ,@body)))))
          (labels ((unknownp (arg)
      	       (etypecase arg
      		 (atom (eq arg unknown))
      		 (cons (some #'unknownp arg))))
      	     (literalp (arg)
      	       (or (typep arg '(or array keyword pathname number))
      		   (and (consp arg)
      			(or (and (eq (first arg) 'function)
      				 (cdr arg)
      				 (not (cddr arg)))
      			    (eq (first arg) 'quote))))))
            (let ((unknowns (iter (repeat (count-if #'unknownp args))
      			    (collect (gensym "UNKNOWN"))))
      	    (knowns (iter (for arg in args)
      			  (if (not (or (unknownp arg) (literalp arg)))
      			      (collect (list (gensym "KNOWN") arg)))))
      	    (known-function (and (consp function)
      				 (eq (first function) 'function)
      				 (symbolp (second function))
      				 (second function))))
      	(autotraverse-let ((an-unknown unknowns)
      			   (a-known (mapcar #'first knowns)))
      			  (let* ((unknown-function (gensym "UNKNOWN-FUNCTION"))
      				 (call-function (if known-function
      						    (list known-function)
      						    `(funcall ,unknown-function)))
      				 (main (let ((inner `(lambda (,@unknowns)
      						       (,@call-function
      							,@(iter (for arg in args)
      							     (collect (if (unknownp arg)
      									  (subst an-unknown
      										 unknown
      										 arg)
      									  (if (literalp arg)
      									      arg
      									      a-known))))))))
      					 (if knowns
      					     `(let (,@knowns)
      						,inner)
      					     inner))))
      			    (if known-function
      				main
      				`(let ((,unknown-function ,function))
      				   ,main))))))))

~~~
mahmud
Please use Lisppaste to show gorgeous code :-)

<http://paste.lisp.org/new>

~~~
Hexstream
Good idea!

fmask's implementation with syntax coloring:

<http://paste.lisp.org/display/81488>

I also have a more extended example usage in an annotation at the bottom.

------
ggchappell
You might be interested in what John Wiegley had to say on the subject (in
particular, Haskell has no macros; is this bad?) a couple of months ago:

[http://www.newartisans.com/2009/03/hello-haskell-goodbye-
lis...](http://www.newartisans.com/2009/03/hello-haskell-goodbye-lisp.html)

The comments below the article also have some interesting and relevant things
to say.

~~~
lispm
For somebody with 15 years of Lisp experience he sure managed to mention the
most trivial use of macros.

~~~
mahmud
Public language changes are usually just trolling for the love of a community;
it's like those public "converters" who go from one mainstream, often
Abrahamic, religion to another.

The rest of us just rewrite the critical stuff in whatever that's the most
efficient, and the non-critical stuff in whatever that's easiest.

Oh screw this! Guys, I just converted from Common Lisp to CSS! The _cascade_
is like OOP inheritance, without the annoying CLOS multiple-inheritance and
over-configurable multiple dispatch. It has _keyword_ methods, :hover,
:active, etc, I can even implement list processing with getElementById! Take
that, you Lisp weenies!

------
smoofra
Haskell can do almost everything people generally use macros for without them.
But there's always some things only macros can do, so for that you have
template haskell.

~~~
prodigal_erik
As I understand it, if Haskell didn't already have syntax for

    
    
      let vars
      in expr
    

you wouldn't be able to use Template Haskell to create it from scratch. Any
domain-specific languages you make have to mimic existing Haskell syntax.

~~~
mahmud
Most people think macros are for "compacting" code, or factoring out common
parts.

Macros are for introducing new control primitives, syntax and evaluation
models into the language.

~~~
gruseom
It's both, no? You can always write macroexpanded code by hand, only with much
repetition, so in that sense macros are always compacting.

Except, I suppose, for perverse macros which expand into shorter code :)

~~~
mahmud
It's for both, but utility functions also serve the same purpose (which is why
you see people claiming they could do whatever Lisp macros do with
function/methods/classes/monads/templates/makefiles, etc.) :-P

------
justin_vanw
Personally, I think compiler macros are far more interesting than 'regular'
macros. I would love to see compiler macros even in mundane languages such as
python.

Example, in pseudo code:

<code> defun regex_find(some_string, regex) return
regex.compile().find(some_string)

def compilermacro regex_find(some_string, regex) if
IS_STRING_LITERAL(some_string) and IS_STRING_LITERAL(regex) SUBSTITUTE
exec(regex.compile().find(some_string)) elif IS_STRING_LITERAL(regex)
SUBSTITUTE exec(regex.compile()).find(some_string) else DO_NOTHING return

</code>

regex_find(MYSTRING, '\w(.+)\b')

A language without compiler macros would compile the regex at runtime, even in
those cases that it is a string literal, and could be compiled at compile
time. And in the case that both the string and the regex are string literals,
the result itself could be computed at compile time. Another alternative, used
by lots of languages/libraries, is to maintain a cache of recently compiled
regular expressions. Of course, this doesn't help when you have a loop that
uses 15 regular expressions but it only caches the last 14, for example.

This is one reason why CL-PPCRE is faster than the C PCRE in benchmarks, it
makes great use of compiler macros.

This is a very concrete real world thing, that isn't endlessly debatable,
unlike regular macros, and it could be included in languages without any
special macro syntax.

One example of a time I _really_ wanted compiler macros in Python was when I
was doing some parsing of binary files, using the struct module. The struct
module lets you do stuff like struct.decode(data, 'UUUIUH'), where the UUUIUH
is some code for Unsigned long, unsigned long, 32bit int, whatever. However,
it parses the little code each time you want to use it, which ends up taking
way longer than the actual decoding itself. In python 2.x, whatever I was
using, they didn't have any way to make any sort of compiled decoder, you had
to just use it as above, which is silly, since the code is almost always going
to be a string literal and 99% of the work could be done once when the code is
compiled.

~~~
swolchok
> Another alternative, used by lots of languages/libraries, is to maintain a
> cache of recently compiled regular expressions. Of course, this doesn't help
> when you have a loop that uses 15 regular expressions but it only caches the
> last 14, for example.

Nitpick: it depends on the replacement policy whether it helps. If the cache
is MRU, for example, it would work fine.

~~~
justin_vanw
It doesn't matter if each one is used once, say in a loop.

If you have 14 slots in your cache, and you loop over 15 regular expressions,
no cache policy (unless it also tracks regexes it has already lost from the
cache, which effectively gives it more than 14 slots) will work, except maybe
some kind of cache that just remembers the first 14 regexes it sees, and only
forgets one randomly every 10 more regexes it sees. Such a cache policy would
be really really stupid in most cases not specifically designed to thwomp,
say, an LRU cache or a regular deque or something.

------
russell
I used macros a lot in C to initialize structs and arrays with constant
information, like keywords for a parser. It would still be useful for
languages, like Java, which do not have literals for initializing objects,
maps, and arrays. Another use would be DSL's on top of a language like Python.
Sure you can do all of this within the languages, but macros would make it
more expressive and concise. I really don't buy the argument that we need to
be protected from idiots. They will piss in the soup regardless.

~~~
mahmud
C "macros" != Lisp macros.

~~~
karlgluck
Why with the "sarcasm"?

~~~
mahmud
Because C macros have no sense of context; they're replaced in the raw
character stream while lexer is reading in a file. C macros are mostly stupid
string-replacements of the s/foo/bar/g type.

Lisp macros operate on the abstract syntax of the code forms, _after_ it's
parsed. They're tree-transforms. They could also be nested, so one macro would
operate on the result of another. All of them could also operate on the values
read by _reader_ macros, which are Lisp's C-like preprocessor macros: A lisp
macro that frobs vectors could, for example, processes code that looks like
any of the following: foo[], vector foo, (vector foo), (array 1 ..), etc. if
the appropriate reader syntax is defined for them. That's right, ONE macro to
do all that.

~~~
karlgluck
Ah, that makes sense. Thanks for the explanation.

------
prodigal_erik
Sure, working with code in someone else's domain-specific language is hard.
But not as hard as working with a lower-level translation from a domain-
specific language which only exists in their head. Even if Clojure were only
used for one project in the world, I'm still better off learning Clojure than
trying to maintain the JVM bytecode it happens to compile to.

------
stonemetal
Note the reason discussion on programming languages.

[http://gmarceau.qc.ca/blog/2009/05/speed-size-and-
dependabil...](http://gmarceau.qc.ca/blog/2009/05/speed-size-and-
dependability-of.html)

Lisp and scheme do pretty poorly on the power scale. Even going back to the
original data neither finish in the top 10 and Lisp gets beaten by java.

