
Stop Writing JavaScript Compilers, Make Macros Instead - jlongster
http://jlongster.com/Stop-Writing-JavaScript-Compilers--Make-Macros-Instead
======
sync
I'd be afraid of inheriting any codebase that made extensive use of these JS
macros.

You really have no idea what this is doing (the canonical example from
sweetjs.org):

    
    
      class Person {
        constructor(name) {
          this.name = name;
        }
    
        say(msg) {
          console.log(this.name + " says: " + msg);
        }
      }
    

It's not javascript -- it's your own made up "language" of macros.

I do believe with a steady hand this could lead to some great things, but the
example on sweetjs.org seems rather heavy handed. Having different `class`
macro definitions between codebases that use sweet.js will devolve into
madness.

~~~
klibertp
I have only one thing to say here, to you and to all people who are opposed to
macros and new languages in general: go learn some more (let's say 5)
languages and then come back[1]. Don't bother before that.

Why? Because learning languages gets easier every time you do it. And it opens
your eyes to languages' internal structure, the way languages are composed,
which makes using given language effectively easier.

So now you know 5 or 8 languages and you discovered a few basic principles
behind most of them and you understand how semantic constructs compose and you
know about various syntactic representation of any given semantic statement
(and the other way around, similar syntactic constructs having different
meanings). And you come upon a macro, like the one in the example.

You don't know what it's doing - well, you just check the docs and then you
know. It's as easy as learning what a function does, really. So what was
learning those other languages for? It was just to reduce your fear of
introducing new syntax for things. You won't think "oh, how dangerous this
could be" but rather you just start using it, because you do it (change
syntactical constructs you use) all the time when you switch between
languages.

At this stage you can learn some Lisp. And write some real macros. And maybe
try Nimrod, which has wonderful macros too. And maybe Rust, which I didn't try
but I read good things about. After this you'll understand what macros are,
how they are constructed and how they compose - and write many of your own -
and then you look at the class example above and you instinctively[2] know
that there has to be a syntax transformer which matches class keyword, a
keyword and a block, and then there has to be another transformer, perhaps
recursive (like in Scheme) or just iterating over the body (like in defmacro)
and that all they do is to add some keywords here and there and a statement or
two at the top of the block. In short - you know exactly what to look for and
even docs are unneeded, you just take a quick look at a macro source and go on
using it happily. Alternatively you can look at generated code and work out
macro implementation from it easily, too.

I'm a happy user of LiveScript, and CoffeeScript before that. The only reason
for not switching to JS with sweet.js is the fact that I would need to write a
whole host of macros by myself, while in CS and especially LS they are already
written. And of course modifying their grammar is not _that_ hard either. Had
I somehow been unable to use LiveScript, I'd use sweet.js for sure. What I'd
implement? Probably some kind of "let" statement for sane scope management, a
loop macro for iterating over custom collection classes (loop in CL, for/* in
Racket, also in others), function currying (partial application) and maybe
something like "with" context managers from Python would be among the first.
There are many things which are better abstracted with syntax than with (for
example) functions.

Would the language be JavaScript? Well, hard question, it would be a superset
with underlying semantics intact, but it sure wouldn't "feel" like JS. But
would that matter? Not at all - syntactic extensions would save me a ton of
time and I'm used to many different syntaxes anyway. And I expect anyone who'd
like to work with me to be able to pick up any language in reasonable time -
the ability to pick up a few additional syntactic constructs on top of already
known language is essentially the same thing, so I think such person would
have no problems with it either.

Anyway, macros are good; using them is good; the only dangerous thing is
having many people implement essentially the same macro over and over again,
but I think this could be solved with a sane module system for macros which I
read is already planned. There should be more languages rather than less; more
exploration of different syntaxes and semantics for things, not less - and
macros are one way of making this happen. And if you have problems with
learning what a "class name body" does, then I can only refer you to the first
paragraph.

[1] After I wrote this post I realized I could sound a bit rude. It was not my
intention and I'm sorry if I offended anyone; while written generally, it's
actually only a description of a road I personally traveled, so obviously
YMMV.

[2] I know completely nothing about sweet.js in particular, so I'm completely
just guessing.

~~~
nostrademons
I have written non-trivial programs in C, C++, Java, Javascript, ActionScript,
PHP, Python, Haskell, Scheme, Go, and Sawzall, and also played around with
prototypes or toy-quality programs in Common Lisp, Ocaml, Dylan, Erlang, Ruby,
and Perl. I've written compilers, templating languages, desktop GUIs, web
GUIs, mobile GUIs, parsers, machine-learning based data extractors, and
distributed systems. I also lead a team, and I'm trying my hardest to bring a
large, heavily used product into the modern era when it comes to web
technologies. I was a huge fan of macros in college and the few years
immediately afterwards. I've now done almost a complete 180 on them, and would
encourage anyone I work with not to use them.

The reason is because I've come to understand that social factors trump
technical factors nearly every time. The hardest part of writing a software
system is almost never solving engineering challenges or writing code, it's
communicating the solution to everyone who will have to touch it. Over a
software system's lifetime, you will have to read code much more often than
you will have to write code. Macros make writing code really easy and
communicating everything about what it does, how it does it, and what
invariants need to hold for it really hard. That's usually a poor trade-off
for anything non-trivial.

~~~
sheepmullet
I've found that working with a DSL structured around the problem domain
improves communication and readability. Macros are a tool to allow you to
easily implement these DSLs and have them fit nicely into the host language.

~~~
nostrademons
I actually like DSLs, but I think the right level of abstraction to develop
them is by exposing the tokenizer, parser, AST, typesystem, semantic analyzer,
and execution engine of the language as libraries in the language itself.
Things like 'ast' and 'parser' in Python, package 'go' in Go, Clang for C++,
etc. I've tried to fill this gap for HTML5 with Gumbo - it was initially
designed to support a semantically-aware templating language in Google.

The reason I think DSLs are good but macros are bad is because DSLs make you
think long and hard about the costs of implementing a new language. You have
to pay for it up-front by writing a compiler and hooking it into your build
system, even if you have help from the host language's libraries, and then
you'll end up with something that's clearly a separate language and must be
documented and learned separately. That cost makes it clear that creating a
DSL is not something to do lightly, and you need a problem domain where there
really are much higher-level abstractions that can't be captured by an
existing programming language.

The problem with macros is that you can create a new language construct with a
half dozen or so lines of code, use it in a few hundred places in the
codebase, and now your maintainers have a few hundred problems.

~~~
pixelcort
But at that point what is the difference between a macro and a function, from
the maintainers' perspectives?

Is it just that, when a maintainer sees a function call it's easier to guess
what it does, then when they see the use of a macro?

------
orthecreedence
Glad someone finally wrote a post about this. Coming from the lisp world,
that's the thing I constantly see missing from languages wrapping javascript.
They generally change the syntax a bit, add a few features to minimize
boilerplate in some areas, and call it a day. Why would I want to learn a new
syntax to program in a language I already know well? Unless you give me the
tools to write code that writes code, I'm not going to bother learning/using a
new javascript syntax.

~~~
nawitus
At least TypeScript brings to the table something very valuable: optional type
checking.

~~~
aaronem
And sweet.js macros can be used to implement the type system of your choice in
otherwise stock Javascript. (For that matter, you could un-overload the +
operator so that it only works with numbers, and invent a . operator to do
string concatenation.)

I can see a potential argument that doing type checking via macros costs more
than precompilation in efficiency terms, but on the other hand, that's not
necessarily so. How does TypeScript do type checking for non-constant values?
That is, if I define function foo(bar: string) in a TypeScript library, then
compile it, load the library in a Javascript interpreter, and call
foo(Integer.parseInt("42")), what happens? I'm guessing that either foo()
misbehaves, or that foo() throws an exception.

In the former case, there is no benefit to TypeScript because your type
annotations are evanescent and misleading -- you can't rely on them, so you
either have to write fences around input values just like you would without
them, or risk your code misbehaving because it's written to rely on unreliable
type annotations.

In the latter case, TypeScript is adding those fences automatically, so that a
function which expects an integer argument can check whether it actually got
one and fail if it didn't. That's basically what a macro system implementing
type checking would do, too, and it seems to me that in both cases
optimization would be merely an implementation detail.

~~~
nawitus
It's not practical to implement a type system using macros. The problem is the
software ecosystem. First of all, I don't think sweet.js macros can implement
a full type system like TypeScript has which can load type definitions from
other files, do type inference and support interface definitions.

Secondly, it's difficult to get a community around a single type system.
Thirdly, it's difficult to get IDEs to support these sweet.js macro type
systems.

The fact is, a sweet.js macro type system which is widely in use and supports
everything TypeScript has and is supported by many IDEs doesn't support, and
is unlikely in my opinion, if not technically impossible. In theory it might
work.

If you compile TypeScript and then call it using JavaScript with the wrong
type the code will misbehave. It's a downside, true, but a dynamic type
checking system written in JavaScript would be too costly. Sweet.js macro type
system wouldn't bring any benefit in that regard either.

You can rely on TypeScript type checking to make sure your code is correct.
Similarly, if you develop in JavaScript, you can use unit tests instead of
type checking to make sure your code is correct. Unit tests nor type checking
can't guarantee that _other code_ is correct, even if that other code happens
to use your own code.

The benefits of TypeScript's type checking include the elimination of a class
of unit tests, automatic refactoring, full intelli-sense support, better
readability and automatic documentation (which doesn't remove the need of
manual documentation).

~~~
samth
You should have a look at Typed Racket [1] which is a type system for an
existing language (Racket) built entirely with macros [2] that satisfies all
of the criteria you want -- works with the IDE, safe interop with untyped
code, etc.

[1] [http://docs.racket-lang.org/ts-guide/](http://docs.racket-lang.org/ts-
guide/) [2]
[http://www.ccs.neu.edu/racket/pubs/pldi11-thacff.pdf](http://www.ccs.neu.edu/racket/pubs/pldi11-thacff.pdf)

~~~
nawitus
Wow, that's impressive. I didn't realise macros can be that powerful in
practise.

~~~
innguest
Maybe this will help see the full scope of the power of macros: macros receive
their arguments as an AST (i.e. a list) and are free to transform it as they
please; they also have full access to the host language and are free to
implement a type system checker that walks code and checks that all the types
match. There are no restrictions here - the type checker would be regular
scheme code that would be wrapped into a macro simply to make it run at
compile time. So there's nothing less powerful about macros at all. It's just
code that runs at a different stage (compile-time instead of run-time). After
macros, in terms of power, you have reader macros, that receive their
arguments as raw text and are free to turn it into any AST.

------
sheetjs
Since sweet.js code can't be directly run in the browser, you need some
"compiling" or "interpreting" step. It's not obvious why this is any different
from a compile-to-js language that had support for macros.

For example, someone wrote a macro system for coffeescript:
[https://github.com/mrluc/macros.coffee](https://github.com/mrluc/macros.coffee)

Related discussions: [https://github.com/jashkenas/coffee-
script/pull/3171](https://github.com/jashkenas/coffee-script/pull/3171)
[https://github.com/gkz/LiveScript/issues/328](https://github.com/gkz/LiveScript/issues/328)

~~~
jlongster
This isn't intended for people interested in other languages; if you like
CoffeeScript, certainly use it. I really like ClojureScript myself. But
there's a lot of reasons you already need a build step for JS anyway
(especially for client-side), and this gives you a lot of power without having
to completely switch languages.

It's worth noting that sweet.js is a robust macro system grounded in academic
research (the core of it from the Honu paper). That CoffeeScript system is a
bit of a hack and its flaws would show it the whole community started using it
(I don't have time to go into this right now, but things like hygiene, etc are
really difficult).

~~~
mr_luc
Hi, I made macros.coffee, and I'd like to say that this guy is right. It's
non-hygenic (on purpose) and it's a hack.

And _purely_ a labor of love! Slides: [http://mrluc.github.io/hammy/12-7-rum-
macros/deck.html#1](http://mrluc.github.io/hammy/12-7-rum-macros/deck.html#1)
Code:
[http://mrluc.github.io/macros.coffee/docs/macros.html](http://mrluc.github.io/macros.coffee/docs/macros.html)

I wanted to see how easily I could add (non-hygenic) macros to CoffeeScript,
and if I could do it as an npm-installable module instead of a fork of the
language.

It was also an experiment in making macros very clear to myself. Non-hygenic
(CL-style) macros are very simple conceptually.

But the basic hack that made it possible to get true CL-style macros also
makes the whole thing incredibly dependent on the structure of the
CoffeeScript AST. Which makes it _fragile_.

I'm excited by BlackCoffee, actually. This is the first I've heard of it.
Thanks a lot for giving me a nod in the github PR, and of course let's mention
Oran Looney as well, who wrote the best deepcopy ever for Javascript (I
wrapped this up as owl-deepcopy and made an npm module for it).

------
cromwellian
JavaScript compilers aren't only for syntax sugar. Closure Compiler, Dart2JS,
and GWT are _optimizing_ compilers. Most large Google applications for example
would be hideously large without an optimizing compiler.

~~~
jlongster
Definitely. It was a terse statement for simplicity. I was hoping that my
sentence "it's not good for extending javascript" clarified that.

------
chc
This is basically just saying "Choose Sweet.js in combination with some
unknown array of macro libraries as your only language that compiles to
JavaScript."

But because the article doesn't really frame it this way, it doesn't really
explain to me why I would choose Sweet.js over ClojureScript or Scala.js or
Black Coffee or whatever.

------
aryastark
I barely trust Guy Steele with macros. You know who I don't trust with macros?
Random JS coder dudes that haven't learned what true hardship is, when
debugging a macro that is 5 levels deep and subtly changes the language
because, hey, I have a macro! And having a macro means using a macro.

Language design is hard. Scheme has gotten it wrong. Over. And over. And over.
They didn't get hygiene right for more than a decade. They _still_ don't have
it right. But people think they can easily invent new syntax that somehow
doesn't do unexpected things. Your clever macro isn't so clever when some poor
son of a bitch has been beating his head on a bug all day long just to find
out your macro-that-looks-like-a-function isn't evaluating all its arguments
because your macro-that-looks-like-a-function is really a function-that-is-a-
macro. Don't get me started on missing IF branches and undefined behavior...

------
kirtijthorat
I would also recommend the WISP: A homoiconic JavaScript dialect with Clojure
syntax, s-expressions and macros. More details at:
[https://github.com/Gozala/wisp](https://github.com/Gozala/wisp)

------
skrebbel
I strongly agree with everything the author writes, but I believe that
sweet.js might not go far enough. For example, I am very fond of TypeScript.
I'm also very fond of React's JSX. Yet, I cannot mix the two, and neither
JavaScript extension could be expressed as Sweet.js macros.

------
charlieflowers
>>>> "What if JavaScript actually adopted macros natively so that we never
even had to run the build step?"

Honestly, that's pretty damn exciting. In my book, if JavaScript did that, it
would very nearly complete a "worst to almost-first" transformation as a
language. Starting with its early versions and all their ugly "bad parts", and
advancing all the way to one of the most exciting, expressive languages to
work with.

------
Fishrock123
I find the title kind of ironic, given that sweet.js is still a compiler. :P

However, it is true it would be better to have one standard-ish compiler that
can just execute whichever macros you throw into it.

That being said, you still run into the issue that `sync` brought up.

------
rcarmo
Best way I know of using macros in JS is
[https://github.com/Gozala/wisp](https://github.com/Gozala/wisp), which
generates pretty readable code. If you're going to pre-process, you might as
well write in a nicer language...

------
ahunt09
So, after puzzling around with the macro for swap a bit, it seems like macros
are like functions, except without the scoping rules, which suggests to me
that we now have two kinds of functions, to keep track of in the code, each of
which look identical, but behave differently. swap(a,b) looks like a function,
but can interact with all the variables in the same scope (or above) as when
it's called (is this the same for global functions?). Maybe I'm missing
something big, but that seems dangerous when the macro is declared in another
file, and I have to go figure out how it behaves.

~~~
MBlume
Macros are not functions, macros are a mechanism for adding features to the
language itself.

~~~
ahunt09
Maybe I wasn't clear. I am not saying that macros are functions. I'm saying
that without seeing the implementation, both look identical, and thus could be
conflated, but do not treat scope identically.

------
chromanoid
How do you deal with naming collisions of macros from different developers?
Are you considering to introduce name spaces? To me macros as well as other
surprise code modification stuff seem unfeasible in teams with more than a few
developers. Don't you think you risk maintainability with a huge macro
apparatus?

~~~
disnet
We're working on integrating macros and modules to address this. Once it's
implemented you'll be able to do something like `import { m } from
"macros.js"`.

------
lightblade
sweetjs was a good idea, but I've found it hard to use, or rather..it's not
the using part that's hard, but writing your own macro.

sweetjs has this hygienic thing that renames identifiers. To prevent it from
renaming, there are many hoops that you have to jump through to prevent it.

sweetjs introduces of a lot of new concepts. These concepts are not as easy to
understand as the ones in C. The macros are certainly more powerful. It's just
that assuming it's C-like macro will give you the wrong kind of expectation
with sweetjs.

~~~
aaronem
Of course! These aren't C "macros"; they're Lisp macros, which is a wholly
different, and far more valuable and reliable, proposition.

Granted, I'm not sure how much I like the idea of a macro system which finds
macro instances by string-matching the source it's given. But in a non-
homoiconic language, I'm not sure anything else is feasible, and I can easily
see how something like this could improve Angular.js's dependency injection,
for example.

~~~
taejo
Unhygienic macros aren't a C-ism: hygiene was an innovation (I think
originally in Scheme). Ordinary Common Lisp macros are unhygienic.

~~~
aaronem
So are sweet.js macros, or else they wouldn't need to have reimplemented
GENSYM. But I get the impression (parent (parent)) had sweet.js macros mixed
up with C-style string-replacement "macros", which was what prompted my
comment.

~~~
mnemonik
Sweet.js is hygienic but you can choose to be unhygienic if you want.

It is the best of both worlds.

[http://sweetjs.org/#getting_dirty__breaking_hygiene](http://sweetjs.org/#getting_dirty__breaking_hygiene)

------
badman_ting
I'm not super into the idea of code running on code because I likes it as
simple as I can gets it. But, I know this notion is popular with lispers so I
get it.

~~~
kaoD
Don't be fooled by a legion of old-time lispers. Macros are a last resort in
real-life Lisp (aka Clojure).

Hard to write (way harder than a function), hard to debug, hard to reason
about, they're not functions (so fit worse in the rest of the language)... but
powerful when you _really_ need them.

"When you have a hammer...", and a desire to stand out from other languages
made the lisp-macro myth flourish.

~~~
6cxs2hd6
Well actually, in a modern lisp (like Racket), you use macros in a variety of
ways.

In addition to "deep" things, often you use them as practical, simple
alternatives to using stuff like "snippets" or heavy IDEs. You can "DRY"
annoying patterns, without resorting to external tooling.

Although macros used badly can be mysterious, so can any excessive pre-
processing and build magic.

Macros provide "an API for the compiler", and since the compiler is involved,
it can be smarter than external tools.

~~~
kaoD
Racket? I said real life! (just kidding :P)

As I said, when you _really_ need them they're useful (e.g. in typed Clojure).
Most (if not all) DRY patterns can be fixed using only functions.

The problem is, the macro mantra has been parroted for so long now it's part
of the Lisp culture and its external image ( _" Lisp is homoiconic! You can
modify code! Macros! MACROS!"_), when it's actually one of the ugly (but
powerful) parts of Lisp you should avoid most of the time. This confuses
beginners and people interested in Lisp.

I see macros fitting mainly in DSLs (which is probably a code smell most of
the time) and extending the language, as typed Clojure does. What other real
use cases do you see for macros?

~~~
6cxs2hd6
The real use cases for macros boil down to 3 main areas:

1\. Changing order of evaluation.

2\. Creating new binding forms.

3\. Implementing a data sub-language a.k.a. DSL

Although arguably #3 doesn't _require_ macros, arguably it requires macros to
do _elegantly_ (Rubyists might argue not) and _reliably_ (instead of monkey-
patching, using hygiene and a macro-aware module system that can handle
towers/layers of such systems).

~~~
chc
Well, 1 and 2 are the only cases where they are absolutely required, but I've
seen other use cases. For example, the ClojureScript templating library Dommy
uses macros to optimize selector code at compile time, which gives some
impressive speedups (IIRC Prismatic found it to be twice the speed of jQuery).

------
yuchi
Did you had a look at
[gorillascript]([http://ckknight.github.io/gorillascript/](http://ckknight.github.io/gorillascript/))?
Its
[prelude]([https://github.com/ckknight/gorillascript/blob/master/src/js...](https://github.com/ckknight/gorillascript/blob/master/src/jsprelude.gs))
is written in gorilla macros :)

------
nessus42
I enjoyed the article. I just have a factual correction: Lisp had real macros
as early as the 70s, not the 80s. (Maybe even earlier, for all I know.)

~~~
daniel-cussen
Lisp macros were introduced in 1963 by Tim Hart at MIT.

~~~
pmcjones
Here's the original technical report:

ftp://publications.ai.mit.edu/ai-publications/pdf/AIM-057.pdf

------
norswap
If you need the same for Java:
[https://github.com/norswap/caxap](https://github.com/norswap/caxap)

------
pork
Very interesting! I've often thought that Javascript could actually be a
pretty language with some syntactic sugar. It'd be even more interesting if
you could run it client-side without preprocessing an sjs file into js first,
sort of like how you can include LESS stylesheets and load a JS module to
compile them in the browser.

------
voidr
You won't have good IDE support.

You won't be able to directly debug your code.

Also it can be really fun to reason about nested macros.

------
vpj
You can also use C macros with Javascript [http://vpj.svbtle.com/coffeescript-
and-macros-clean-and-fast](http://vpj.svbtle.com/coffeescript-and-macros-
clean-and-fast)

------
14113
So wait, what does this have that's better than C macros apart from hygiene?
I'm afraid I really don't follow.

------
smrtinsert
There are many reasons why people are writing *->js compilers, lack of macros
isn't the least of the problems for js. Please keep the compilers coming, no
productive coder really wants to write js.

~~~
bryanlarsen
The author's point is that the best way to implement Javascript sugar like
CoffeeScript is by writing sweet.js macros rather than by writing compilers.

------
Dewie
I think things like syntactic sugar is perfectly fine, _as long as I can
desugar it in a straightforward way_. I want to be able to programmatically
desugar some piece of code, not have to Google it each time I am curious.

I think that if it is easy to investiage things like syntactic sugar, and not
have it be buried in something like a compiler or lang spec, then DSL/language
implementers (and anyone else, if the language permits it) could get away with
implementing things that objectively make the language more complex to deal
with, because to decipher it is only a query away, anyway.

~~~
charlieflowers
Right. IDE features would be needed such as Emacs features which show the
expansion of the macro you're on.

------
al2o3cr
Sup dawg, I heard you wanted to stop writing JS compilers and start using
macros so here's a JS compiler that implements macros.

/head asplodes

------
johmacindu
Don't tell me what to do. If I want to write a JS compiler, I can write a JS
compiler, and it's nobody's business to tell me not to.

~~~
pork
Don't post garbage on Hacker News.

