
Rebol vs. Lisp Macros - vanderZwan
http://blog.hostilefork.com/rebol-vs-lisp-macros/
======
ejbs2
So this was a very interesting post. I would like to note that we can actually
do something like definitional scoping in CL too :), but we need to write it
ourselves!

We would do this using CLOS and its MOP, more precisely by using the
FUNCALLABLE metaclass and having a class with an environment slot, source slot
and effective function slot which both would be altered by some convention
(USE, COMPOSE and FUNCTION it seems?).

Then you'd SET-FUNCALLABLE-INSTANCE-FUNCTION to a lambda that invalidates the
efficient function when source or environment has changed and composes a new
function with (compile `(let((,@environment(( ,@source)).

Voila, you now have a type of function that carries with it an environment and
its source code.

Now I don't know if this fully satisfies what REDBOL is doing to be honest,
because the post talks about "deep walking" and that sounds like code walking,
which is a scary place to go to in CL.

EDIT: Yeah, it seems like USE would do a full code walk(?) Code walking really
sucks, because you need to recognise the special forms of CL and deal with
them appropriately. This doesn't apply if the list of vars used is always
passed to USE (just let LET take over binding the lexical environment)

~~~
junke
I have tried to use SBCL's code walker on a pet project and so far I have no
problem with it; in fact it is surprisingly easy to use. By the way, the
following post is a good introduction to it:
[http://christophe.rhodes.io/notes/blog/posts/2014/naive_vs_p...](http://christophe.rhodes.io/notes/blog/posts/2014/naive_vs_proper_code-
walking/)

------
lispm
See FEXPRs in Lisp.

[https://en.wikipedia.org/wiki/Fexpr](https://en.wikipedia.org/wiki/Fexpr)

and

[http://www.nhplace.com/kent/Papers/Special-
Forms.html](http://www.nhplace.com/kent/Papers/Special-Forms.html)

Passing unevaluated code into Lisp functions was widely used in Lisp. Long
ago.

It is no longer, since macros are typically simpler to understand and easier
to integrate into a compilation model.

In Symbolics Genera, which has weak support for FEXPRs:

    
    
        (defun print-reverse zl:fexpr (form)
          (print (reverse (first form))))
    

Calling (print-reverse (+ 1 2)) does print (2 1 +)

~~~
eggy
@lispm see the reference I made to FEXPRs in reply below
([http://axisofeval.blogspot.co.id/2011/07/meaning-of-
forms.ht...](http://axisofeval.blogspot.co.id/2011/07/meaning-of-forms.html))

You seem to be a lot more knowledgeable than I on Lisp. It seems there's a
divide on FEXPRs and macros when I look into further references. What's your
take on it aside from the Pitman paper you cited? Is is possible something
useful was thrown away from CL, or Pitman's argument holds? PicoLisp seems to
implement FEXPRs a bit from my limited knowledge. I am currently in a death
spiral of LFE, Chez Scheme, PicoLisp, Extempore (xtlang), and Shen, not to
mention my necessary forays into elisp, since I use spacemacs! I need to
choose one, only one. Thanks.

------
ACow_Adonis
Perhaps I am missing something/unable to interpret the article properly.

I remember there was a day when it clicked in my head that the way the lexical
scope/compiler in common lisp works means that the variable names don't stick
around at run-time, but are compiled away. So of course, using the lexical
scope of (let) forms in lisp means that yes, it is quite impossible to have a
naive variable symbol passed into a let form and looked up at run time as
though it were still around.

But does that mean its not possible? I mean...doesn't Common Lisp at least
have both lexical and dynamic scope? What if there was some kind of primitive
function or form available that told lisp that you actually want to keep the
names around and use that name to do run time environment/variable lookup
instead. Then you could pass in such a quoted form and evaluate it and it
would work.

Hang on a second...

Lets see if we can re-write that form from the article that "doesn't work" in
lisp...in an easy and convenient fashion.

Here's his function definition:

(defun greater10 (value code) (if (> value 10) (eval code)))

And now lets set up a dynamic binding/evaluation using the less common progv
form:

(progv '(msg) '("Hello") (greater10 20 '(print msg)))

And the result:

"Hello"

~~~
sedachv
I am not familiar with Rebol but from what I gathered from the "So...What
Can't Be Done?" section of the article is that `use` is more like a code
walker that does lexical binding before the given expression is expanded and
evaluated (hence the need for `compose`).

If I understand things correctly this is a lot like the `defmacro!` and sub-
lexical scoping that Doug Hoyte writes about in _Let Over Lambda_. There the
intention of code walking and rewriting is to prevent unwanted variable
capture (walking over the code and replacing certain names with unique ones
that have their own unique locations is basically implementing lexical
scoping).

So where `progv` might run into trouble is:

    
    
        (progv '(msg) '("Hello") (greater10 20 '(foobar msg)))
    

Where foobar is something like:

    
    
        (defmacro foobar (x)
          `(let ((msg "abc"))
             ,x))
    

And a Common Lisp `use` might be something like:

    
    
        (defmacro use (var binding expression)
          (let ((var1 (gensym)))
            `(let ((,var1 ,binding))
               ,(subst var1 var expression))))

~~~
draegtun
You should find this of interest because this is how USE is written in Rebol:

    
    
      >> source use
      
      use: make function! [[
          "Defines words local to a block."
          vars [block! word!] "Local word(s) to the block"
          body [block!] "Block to evaluate"
      ][
          apply make closure! reduce [to block! vars copy/deep body] []
      ]]

------
eggy
I learned quite a bit reading this. I am just looking at RED, but I program in
Lisp/Scheme. FEXPRS come to mind [1]. PicoLisp sort of has them, and I think
Newlisp does too, but I don't use Newlisp. I think this addresses some of the
Lisp comparisons in a different light.

    
    
      [1]  http://axisofeval.blogspot.co.id/2011/07/meaning-of-forms.html

------
guicho271828
Q. definitional scoping in CL? A1. Use a quasiquote and comma instead of a
quote, which embeds the value inline. Not the direct answer though.

    
    
      (let ((msg "Hello"))
        (eval `(print ,msg)))
    

A2. Declare the variable as a SPECIAL variable and give it a dynamic scope
(rather than lexical). Lexical environment/scope and dynamic environment/scope
(resp.) are independent.

    
    
      (let ((msg "Hello"))
        (declare (special msg))
        (eval '(print msg)))
      ;; -> "Hello"
    

So, when people "points out" any language-level flaw in common lisp most
probably s/he simply doesn't read the ANSI spec.

------
pklausler
Non-strict (lazy) evaluation should be mentioned as another way to approach
the problem.

~~~
junke
Which problem?

~~~
__s
Give me lazy evaluation & I can define if as a function which takes 2
expressions & evaluates only 1, whereas in an eager language I'd have to pass
a hand rolled if function callbacks

~~~
pklausler
For example, Haskell's ifThenElse is just a normal function of three
arguments.

~~~
draegtun
"IfThenElse" in Rebol (called _either_ ) is also just a function with 3 args:

    
    
        either 1 = 1 [print "true!"] [print "false!"]
    

Here's an example of writing this in Rebol directly:

    
    
      if-then-else: function [
          cond [logic!]
          true-block [block!]
          false-block [block!]
        ][
          switch cond [
              #[true]  [do true-block]
              #[false] [do false-block]
          ]
      ]
    

This works because blocks aren't evaluated until explicitly called:

    
    
      >> if-then-else 1 = 1 [print "true!"] [print "false!"]
      true!
    
      >> if-then-else 1 = 2 [print "true!"] [print "false!"]
      false!

------
kazinator
> _because ordinary Lisp functions are limited in how they can work with
> "context-dependent code fragments" passed as arguments. Rebol attacks this
> problem by making bindings of individual symbols "travel along" with their
> code fragments. The result is that many cases requiring macros in Lisp can
> be written as ordinary Rebol functions._

Bzzt, no. "Context dependent code fragments" are "lambdas" in mainstream Lisp
dialects, which carry the environment (i.e. bindings of symbols) via lexical
closure.

Now, indeed, one of the uses of macros is to provide some (usually mild)
sprinkling of syntactic sugar to conceal lambda syntax.

> _But packing code into a [functional /lambda] "black box" creates barriers
> to transformations that are a big part of the appeal of using a homoiconic
> language._

The history of Lisp has recorded considerable experience with the FEXPR
approach to language extension. Lispers had worked out decades ago that this
largely sucks, and so interest in FEXPRs waned. They do not appear as a
feature in Scheme or Common Lisp, and it's not for lack of knowledge or
experience.

Essentially, Lisp performed a "Rebol-ectomy" on its semantics by the end of
the 1960's or thereabouts.

Anyway, to understand macros fully, you have to step out of the role of
language user and put on a "compleat hacker" hat. Suppose you have complete
control at your disposal: you can hack any aspect of the language, as deeply
in the implementation as you want. You can add new features that cannot be
made out of anything which is already there. In spite of that control, you
still want macros. Because macros let you separate the concern of designing
the low-level abstractions that are the most convenient for a given feature,
and a user interface to them which is nice for the user. (Without macros, you
face the annoyance of having to extend the repertoire of special operators,
even in situations when it is obvious that these new ones should just expand
to some other syntax in terms of functions, plus the old ones).

If I'm extending a language deeply at the implementation level, I'm doing
something that can't be done with macros or functions anything else; yet
macros help in a specific way that other things don't. I still want them in
spite of having total control. If I add non-strict evaluation into the
language core, I want macros. If I add continuations, I want macros. If I add
functional reactive programming, I still want macros, etc. Macros mean I don't
have to go into the compiler, or code walker or elsewhere and add new cases to
handle special operators. I don't have to go into the grammar of some parser
generator to add the syntax, where the parser action is just "construct a node
for this new syntax, exactly like some equivalent longer expression would
generate".

(If you say you already have all the syntax you would ever need and everything
else is just done with that syntax, then you are not wearing the "compleat
hacker" hat.)

------
draegtun
Only got one comment but this article was also posted a week ago -
[https://news.ycombinator.com/item?id=11541170](https://news.ycombinator.com/item?id=11541170)

~~~
vanderZwan
Ah, must have missed it. Maybe if we let this one up it might attract more
discussion a second time? I posted it hoping that some fanatic Lispers had
some interesting thoughts to add.

~~~
aidenn0
Semi-fanatic lisper here. First of all, search the other comments for a
discussion of f-exprs.

I was not familiar with Rebol before this article, but this author shows a
very good understanding of metaprogramming and homoiconicity, if they picked
up this much about lisp macros without having used lisp previously.

Something that throws a lot of people new to common lisp is that macros are so
powerful that they seem like magic, when they are probably the simplest
metaprogramming tool around. In fact a big part of why I find CL to be such a
high local maxima is the very high power/complexity ratio.

Complicated metaprogramming techniques tend to have either high compile-time
overhead, high run-time overhead, or both. Sometimes efficient implementations
can be discovered with the application of a large number of PhDs.

Lastly, I want to point out that bolting on a full-code walker to Common Lisp
is not possible to do in a fully portable way (and by "fully portable" I mean
"will run on any future conformant CL implementation"). This is due to the
fact that conforming implementations are allowed to expand macros (and even
special forms) from the standard into code that contains non-standard symbols.

Obviously you can do code walkers that are portable across N existing
implementations with O(N) amount of programming effort; see cl-cont[1] for an
example of this.

On the other hand, it would be very doable (if not quite trivial) to implement
something in CLOS that works like a code-carrying closure that allows
recompilation with declarative scope.

I do sometimes miss the fact that EVAL doesn't take an environment. The upside
is that judging by questions I've seen in #lisp, if it did have such a
construct, there would be a lot of fairly spaghetti-like EVAL code when a two-
line macro would do the trick, as EVAL is the more obvious tool for code-as-
data when macros are simpler and easier to understand.

1: [http://quickdocs.org/cl-cont/api](http://quickdocs.org/cl-cont/api)

------
xaduha
If possible, do things without macros. Problem is - it's pretty much always
possible to do things without macros.

~~~
sklogic
No. If possible, do things _with_ macros. And it is always possible to do
things with macros.

Why? Because macros are the best way to implement simple, nice, clean,
maintainable eDSLs. And any problem is best solved with its most natural
language, where the very problem description is already a working solution.

Unfortunately, most people do not understand what macros are, and how they
should be used to build multi-stage, simple DSL compilers. They're using
macros to implement obscure syntax instead, all that awful LOOP macros,
anaphoric ifs, etc., which was exactly what resulted in the bad reputation of
the compile-time metaprogramming.

~~~
wschroed
I have only ever heard the opposing argument, but I am interested in
understanding to its fullest extent how to implement good DSLs with macros. I
understand CL and defmacro; I am just looking for good patterns of use. Can
you point me to any resources, documentation, or examples of this?

~~~
adrusi
When you design a eDSL out of functions you get a lot of expressive power for
free because you can lean on the host language's composition facilities.
However this also means you have to shoehorn your semantics into an existing
set of constraints (this is addressed by the "If possible" qualifier in "If
possible, do things without macros") and you have to adopt all the complexity
that the host language enables.

One big advantage of a DSL is that it limits complexity so that it's easier to
predict what code will do, even if you haven't read it yet. With a macro-based
DSL, you can know exactly what functionalty expressed in the DSL can and
cannot do, which makes it easier to reason about without studying the code for
hours.

