Hacker News new | past | comments | ask | show | jobs | submit login
What Template Haskell gets wrong and Racket gets right (ezyang.com)
172 points by adamnemecek on July 18, 2016 | hide | past | favorite | 66 comments

FYI, Racket's syntax macros is from its Scheme roots. For further information on what Scheme provides, by standard revision:

R5RS: [0]

R6RS: [1]

R7RS: [2] (Page 22)

And many Schemes provide non-standard extensions; for instance, Chicken supports non-hygienic macros[3].

FWIW, I prefer to think of the two stages as "syntax expansion" and "evaluation", and not "compile time" and "run time". Compilation and running are loaded terms that don't appropriately capture the totality of situations where this behaviour applies, in Scheme.

0: http://schemers.org/Documents/Standards/R5RS/HTML/r5rs-Z-H-8...

1: http://www.r6rs.org/final/html/r6rs/r6rs-Z-H-14.html#node_se...

2: http://trac.sacrideo.us/wg/raw-attachment/wiki/WikiStart/r7r...

3: http://wiki.call-cc.org/explicit-renaming-macros

I strongly dislike Racket's macro system, from my admittedly limited usage of it.

Does Racket have a way to address the need to, say, create a dispatching table during syntax expansion and read from that table at runtime? I don't know Racket well, but I've been able to do similar things easily in Guile, Common Lisp, and Clojure without an issue. Googling this problem leads me to believe that it can't be done, which is a pretty severe handicap to have if you're a Lisp.

Yes, you can certainly do that in Racket. For example, the `case` macro does exactly that when there are many possibilities. Unless you mean something different that I mean by "create and then read from".

Ah. It seems I have misunderstood some of the Racket documentation. Thanks for the comment -- helped me out:)

Also adding that what I was doing in Guile, Common Lisp, and Clojure was sloppy. After searching for the case macro, saying 'aha!', and applying what I learned into creating an equivalent Racket macro, I realize that it was a portable solution to Guile, and the same idea applies for non-hygienic macro systems.

If you send a minimized example to the mailing list, I'm sure someone will translate it from Guile to Racket in a few hours. https://groups.google.com/forum/#!forum/racket-users

(If the example is not too long, perhaps you can paste it here.)

FWIW, the phase system that is the main topic of the article was created for Racket [1], though some Scheme macro systems have since adopted this Racket innovation.

[1]: https://www.cs.utah.edu/plt/publications/macromod.pdf

Racket also has the concept of a documentation phase. Matthew Flatt wished it into existence when creating Scribble.

I think it (like Guile) just supports the reader syntax, rather than the semantics (which is heavily based on Racket's macro system).

Note that R7RS does not really have any sort of sophisticated phase system for macros.

What exactly is the difference between Racket and Scheme, how are they related?

Racket is derived from Scheme. Sometime after R6RS I believe they decided to just call their own not-necessarily compatible language Racket, but Racket can also run RxRS scheme code.


Racket was formerly PLT Scheme, but it became such a superset of R6RS that it is now considered its own language. However, I believe it has options for Scheme compatibility.

The big differences in my experience are defaulting to immutability and embracing the idea of being a big language...so big that Racket is many languages, not just one.

This is tangentially related but it seems like you might know - I heard something about a big vs small release of R7RS? Do you know anything about that?

Scheme has traditionally had a minimalist philosophy about the language itself. Basically, the language defines only very low-level syntax and operations -- think arithmetic, variable and function definition, basic control flow, etc. It was then up to libraries to add additional functionality.

R7RS small maintains this same mentality. R7RS big is basically R7RS small with a large standard library.

And as political background, this is basically the "we'll offer both options" resolution to the R6RS acrimony. R6RS tried to take the language in a more "batteries included" direction, which led to a lot of resistance from some quarters, and was part of why Racket ended up disclaiming itself as being a Scheme and going in its own direction (the Racket team was quite involved in R6RS, and I believe stung by the way it was essentially rejected by a big part of the Scheme community). R7RS-small is sort of a spiritual successor to R5RS, and R7RS-large is trying to carry on what R6RS attempted, but in a way that can be factored out of R7RS-small and used to complement it.

The problem people have been trying to work through is that many people like a small, clean, minimalist core Scheme, but many other people (well, some are the same people, myself included) don't think the current standard is enough to produce useful interoperability. Especially implementors like being able to write small self-hosted experimental or embedded Schemes without a large standard. But many people writing Scheme code are not all that happy about the poor interoperability story that ends up resulting, because there is not quite enough standardized to write libraries that will work on all the major schemes, which is why there's no Scheme equivalent of CL's Quicklisp repository of cross-implementation libraries (the SFRI system is the only real attempt to make this sort-of happen, and is not really good enough).

Thanks so much for your very detailed explanation! With interop, do you mean FFI?

For R7RS-large, are there plans for a module standard?

Maybe interop isn't quite the right term. I meant being able to write portable Scheme code, something that can run on, say, both Guile and Bigloo. The current standard leaves so much undefined that most practical Scheme code is nonportable.

Well said.

And yet, R7RS-large _still_ will not have a standardized FFI.

You actually CAN'T have a standard FFI, without having a blessed implementation, as so much of FFI is implementation specific. Besides, it would force FFIs in scheme to do The Right Thing to handle continuations (many don't). And call/cc is expensive enough as it is.

There are many languages without blessed implementations that define ways to interoperate with other languages. For instance: C++[0], Haskell[1], Java[2], Common Lisp[3], and others[4]. The FFI needn't describe how Scheme structures should behave, it need only describe how to send and receive data that a foreign library can interpret; IE, C data structures.

0: http://en.cppreference.com/w/cpp/language/language_linkage

1: https://www.haskell.org/onlinereport/haskell2010/haskellch8....

2: https://docs.oracle.com/javase/8/docs/technotes/guides/jni/

3: https://common-lisp.net/project/cffi/

4: https://en.wikipedia.org/wiki/Foreign_function_interface#Exa...

Alright. I didn't know. But I will say that Scheme is a particularly hard language to build an FFI for. In order to do The Right Thing, and have continuations work across language boundries, you have to integrate continuations into a system that is gleefully unaware of their existance. Depending on the design decisions made, making the FFI do The Right Thing is either impossible, or painfully expensive. So many implementations don't. The only thing RnRS could do is define as implementation inspecific an API as possible, and define any use with continuations undefined behaviour. And that would only be moderately better than not having them at all.

Why should continuations need to safely work across language boundaries? Plenty of Schemes support FFIs with the enormous warning that call/cc exists and you should be aware of that.

This solution is hugely superior to having none at all; as it means that bindings to foreign libraries may trivially be ported across many Schemes. Choosing the right Scheme will be less about what libraries are available, and more about which provides the best execution environment for your needs.

In this case, doing the right thing happens to be doing nothing more than warning the user, as is often the case. Attempting to design a system that safely captures foreign state with continuations is almost certainly an over-engineered solution.

Actually, you're right. God, what was I thinking. The only potential problem is that implementations that DO implement callcc across ffi will wind up with code that won't run on implementations that DON'T. That would be bad, but not the end of the world. The real problem is that it's unlikely such a thing would be standardized.

R7RS would have to been an extraction, because big came first. That is probably the way to do things

  1. kitchen sink
  2. extract core
  3. redefine kitchen sink as `small` + extras

R7RS-small was ratified back in July 2013, while R7RS-big still isn't ratified.

That is great news. I remember when they got announced, -big was first with -small to follow.

Maybe this is heretical among Haskell lovers, but my personal, rather undeveloped, feeling is that Haskell syntax is too complicated for macros. Sure, Haskell code feels natural to read, but in fact they had a lot of "superficial complexity"[1], i.e. the syntax supports many ways of expressing the same thing. This makes any sorts of introspection at compile-time troublesome. Compile-time code generation is less of a problem but still it makes the experience less pleasant.

[1]: Hudak P., Hughes J., Peyton Jones, S., Wadler P. (2007). A History of Haskell: Being Lazy With Class.

Many of the answers in the SO thread[0] linked from TFA agree with you: there are other means of abstraction in Haskell that "fit" with the language better, and they (and I) think those abstractions mostly obviate the need for Template Haskell.

[0]: https://stackoverflow.com/questions/10857030/whats-so-bad-ab...

Not at all! I like Haskell because it nails the two things that are most important to me: a good type system and the ability to restrict side effects. Constructive criticism of other parts of the language can only be a good thing.

That's not heretical, most Haskellers agree that macros are a misfeature because (a) they are only useful in homoiconic languages (b) they are only useful in strict languages (c) they interfere with type safety and code clarity.

Why would they only be useful in strict languages? You don't need a macro to write `unless` in Haskell, but you do to (say) automatically generate typeclass membership given a datatype, or autogenerate lenses, and that use is orthogonal to evaluation strategy.

I guess parent is just saying that macros allow programmer control of evaluation order, which you can't get otherwise in a strict language, but you can with laziness.

The point could have been worded better I think.

It's a misunderstanding that has unfortunately caught on among many Haskell programmers. People who say "macros are only useful in strict languages" mistakenly think that macros can only be used to delay evaluation. Of course, there are many more uses of macros, as you point out. Macros can define new binding forms, e.g., in Racket, pattern matching is a library, while it in Haskell it must be part of the core language. Macros are also useful for abstracting over non-first-class language forms like top-level definitions, import/export statements, and even type definitions.

Agreed. We're always trying to move Template Haskell code into new and more powerful type system features.

I admittedly have limited experience with macros, but I'd like to know more about why (c) is the case. It seems like, if you've got type inference at compile time or even before compile time, then it should be feasible to type check the macros.

It is a bit difficult to have static type checking, type inference, and dynamic code generation all working together seamlessly. If you have a macro expansion whose types are entirely inferred, then naturally they should check properly, even if they are inferred improperly.

The larger the macro, the more cryptic the errors are as well, since you could conceivably be generating entire functions whose output and input types are inferred based on very distant macro input and context, and such a type error is going to be very confusing to your users. C++ has this problem: simple errors can dump a dozen screens of template garbage on you.

Can't you solve problem (a) with quasiquoting?

I mentioned it in the comments of Edward Yang's article but I'll mention it here again, Template Haskell is based on C++ templates, here's the original paper http://research.microsoft.com/en-us/um/people/simonpj/papers.... Haskell was written to mimic C++ more than lisp, at least when it comes to meta programming. And if you read the paper having the compiler check the template code before the expansion is there by design decision, that's why it's compile time only.

> Haskell was written to mimic C++ more than lisp, at least when it comes to meta programming.

Depends what you mean by "metaprogramming". Most people would consider type class "abuse" metaprogramming, but it's definitely not something inspired by C++.

The wikipedia definition is good: "Metaprogramming is the writing of computer programs with the ability to treat programs as their data. It means that a program could be designed to read, generate, analyse or transform other programs, and even modify itself while running."

In my own words, code that can read and write code (whether that's at compile time or run time).

Right, so then Haskell type classes qualify, and my original point stands. Certainly the type class language is limited, but common extensions make it Turing complete [1].

[1] https://mail.haskell.org/pipermail/haskell/2006-August/01835...

If you're interesting in learning more about syntax-case in general, I've been learning more and FameInducedApathy on reddit sent me the following resources that I have found very useful:

Macro By Example: ftp://www.cs.indiana.edu/pub/techreports/TR206.pdf

Dybvig's classic paper: https://www.cs.indiana.edu/%7Edyb/pubs/tr356.pdf

Great resources. I'd like to add "Syntactic Abstraction: The syntax-case Expander" (http://www.cs.indiana.edu/~dyb/pubs/bc-syntax-case.pdf), which gives a high-level overview of how syntax-case may be implemented.

I'd also like to point out that, contrary to popular belief (for some bewildering reason), syntax-case is capable of expressing non-hygienic macros via the `datum->syntax` function, which takes an identifier and a symbol, and returns a new identifier with the same name as the symbol and the same scope as the provided identifier. The R6RS even gives an example of writing a completely unhygienic, Common Lisp-like macro transformer via syntax-case (section 12.6 in the R6RS Libraries document [0]).

syntax-case has a bad rap, but it's really not a difficult system to learn, and it's incredibly powerful. In particular, I think its "hygiene by default, with per-identifier hygiene-breaking when and where you want it" is the best possible policy for a macro system (as far as I'm aware, the only other macro system with the same policy, though imo with a much more intuitive interface, is Chicken's ir-macro-transformer for implicit-renaming macros [1]). The combination of syntactic pattern-matching and procedural expansion is extremely convenient, too: you never need to have a massive `let` form that binds the car, cadr, caddr, cadddr, ... of the input form because the patterns let you destructure and switch (as in case analysis, thus syntax-case) on the input form much as in ML-style pattern matching. Of course, you could argue that the pattern matching ought to be disjoint from the notion of a macro transformer, and I'm inclined to agree, but my understanding is that implementing it the way it is makes it easier to keep track of which variables are "pattern variables" which helps to provide hygiene (someone please correct me if I'm wrong).

Whenever a new language comes out with an unhygienic macro system (as in, unhygienic by default), I immediately headdesk. Hygiene is a very desirable property of a macro system, and there's really no reason not to provide it. A simple hygienic macro system (like syntactic closures, for example) is only slightly more complicated to implement than an unhygienic one, and you're doing your target audience a huge favor. No, gensym is not enough.

I'm also bewildered by arguments that define-macro (Common Lisp-style macros) is easier than syntax-rules. The builtin pattern-matching offered by syntax-rules makes writing macros so much easier. Yes, template macros are less powerful than procedural macros, but in my experience they cover upwards of 90% of macros you might want to write, and they do away with tons of boilerplate code. For the other 10%, most Scheme systems do offer procedural macros in addition. I was a little disappointed that R7RS left syntax-case out because it was nice to finally have a standard facility for defining low-level and procedural macros. Perhaps there's hope yet for R7RS-large.

[0]: http://www.r6rs.org/final/html/r6rs-lib/r6rs-lib-Z-H-13.html...

[1]: http://wiki.call-cc.org/man/4/Macros#ir-macro-transformer

A great tutorial on macros in Racket is Fear of Macros by Greg Hendershott.


In Lisp the macro for a symbol, is called a symbol macro. Symbol macros are defined with DEFINE-SYMBOL-MACRO and SYMBOL-MACROLET.

Functions defined in a file where a macro is defined and used are also not seen by the file compiler. The use of (EVAL-WHEN (:COMPILE-TOPLEVEL) ...) instructs the file compiler to execute the enclosed forms at compile-time. Thus one can tell the compiler to define a function in the compile time environment or to load some code at compile time via LOAD or REQUIRE.

Interesting. I haven't written much Lisp but I was under the impression DEFUN had some black magic that obviated the need for that.

Common Lisp has a number of various namespaces that symbols can be defined in. Interfaces like DEFUN, DEFMACRO, and DEFINE-SYMBOL-MACRO all are explicit within their domain, and have explicit rules as to what is available when with respect to read/compile/evaluate times.

Common Lisp is a very explicit language, and that explicitness allows macros to easily coexist with other code, because nothing is hidden by syntax or "black magic". (By default. Because you can morph the language to do nearly anything you'd want to.)

I don't know macros in Racket, only Common Lisp, but I am curious what are some good uses of non-reader macros that cannot be well expressed in normal Haskell?

Anywhere you see a "deriving" clause is a place where macros would have probably have sufficed instead of requiring support from the compiler.

I think "would have sufficed" is a strange choice of words in this case. I do consider macros a lot stronger than "deriving", but I am not sure they are entirely appropriate in a language which is so much rooted in type theory as Haskell is.

I think of "deriving" in Haskell as a form of automated theorem proving, and I think it would be beneficial to have better theorem prover integrated in the language (I think it is in languages like Coq).

The reason why I think theorem prover would be more appropriate than macros is because with macros you, as a programmer, have to do the proof explicitly, and you're responsible for it being type correct. Whereas with a real theorem prover behind the scenes, this work is not needed.

In general, I think it's still quite unclear how to well integrate macro system of Lisp power with type checked language.

So risking "moving the goalpost", I am not really satisfied with your answer. Are there some uses cases for macros that do not involve theorem proving (i.e. having compiler build some function automatically based on type signature alone)?

The deriving (and related generics) functionality is quite a bit cleaner and more appropriate to the problem than macros.

Except when you want to derive something that isn't supported by the compiler, and isn't creating a newtype.

So you haven't used Haskell? I can use `Generic` or TH to derive anything for anything I want. In the case of the latter, it doesn't even have to be a typeclass instance.

Should I mention now that I think syntax-case is an awkward hacky mess, that syntactic closures solved the problem better, as did er and ir transformers from chicken? (Although ir-macro-transformer is O(n), because it crawls your macro so that sucks)

Should I also mention that I feel that this is endemic of Racket itself, which tries to be the be-all-end-all by taking everything from everyone, but winds up being an awkward mess, as it tries to integrate DBC with FP and a (fairly weak, iirc) OO system, not to mention custodians, and a ton of other stuff that should have been spun off into modules and libraries, but is cluttering up core and makes me long for chicken, and want to hurl whenever I look at it?

Don't get me wrong, a big stdlib is always handy, but for god's sake, pick a SMALL set of core concepts and stick with them.

Now if you'll excuse me, I'll go get some kevlar. Knowing you lot, I may need it after this.

Note that the contract system and the OO system are built entirely with libraries, rather than being in the core. Also, syntax-case is implemented with a library on top of a much simpler core. We Racketeers spend a lot of time working on how to simplify and shrink the core -- Matthew Flatt's recent work on hygiene via sets of scopes is an example of exactly that simplification.

Custodians are in the core, but it's not possible to move them to a library -- they necessarily have to integrate with things like the garbage collector. You couldn't have the things they provide, like the ability to limit part of your program to a specified amount of memory, without that.

What I meant was that the contracts and OO are integrated strongly into racket's stdlib: instead of providing a basic paradigm, and then providing alternatives, like most schemes do, Racket's default paradigm is, "all of them." That creates an overcomplicated mess.

Right, we think that contracts and OO programming are both important things that programmers want and need, and so they deserve high quality support from the language. The traditional Scheme approach of not making choices and not integrating things hasn't served anyone well, and isn't the approach that other languages like Python or Ruby take.

But Racket isn't making choices either. Python and Ruby say that procedural programming is the choice for small projects, and OO is what you should be using for larger projects. You can do other things, but that's the default. Racket's approach is to toss 50-odd paradigms at you, and say: "screw it, you figure it out." At least with the traditional scheme there was a default: a sort of procedural paradigm, with some functionalism mixed in. With Racket, there isn't even that. Give us contracts and OO, and all the other paradigms, but give us a reasonable default paradigm as well.

Racket provides contracts for all paradigms you might want to use -- DBC is not a separate paradigm.

The reasonable default is writing programs with structures and functions. Most people use OO primarily for the GUI (where OO works quite nicely). This is the same paradigm that you'd find in ML or Rust or Go in many cases.


The docs could have fooled me. You try to learn anything about Racket, and pretty soon, you're 10 levels deep into nonsense about contracts and other junk that you shouldn't need to see just to find the semantics of cons, or define. Which is to say, the docs are bad. But that's excusable.

What I do NOT find exusable, and what started this conversation, is that syntax-case is overcomplicated compared to, say, er and ir macros, or syntactic closures, all of which were already available. So use a more complex system, when it's been shown a simpler one will do?

For posterity, here are the docs for `cons`: http://docs.racket-lang.org/reference/pairs.html?q=cons#%28d...

I don't see 10 levels of nonsense there. But clearly you're not happy about larger aspects of the system that it's probably not worth debating here. But let's just say that (a) syntax-case is a pattern matching library, not a macro system, and (b) Racket's macro system has demonstrated its usefulness by building those libraries that you claimed were built in, such as OO systems and contracts.

Racket's macro system has proved it is as capable as any other macro system. No more, no less. I merely argue that it is overly complicated.

in response to b, I knew full well how those libraries were built. When I said built in, I meant part of core. And before you claim they aren't in core, I mean what the rest of the scheme world means when they say core: stdlib.

Racket is a programming language programming language to program and prove programming languages. See eopl. (edit: yes it is a pun and a brain twister. i love racket (and scheme too))

Applications are open for YC Winter 2021

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact