
Stop Writing JavaScript Compilers. Make Macros Instead (2014) - tosh
https://jlongster.com/Stop-Writing-JavaScript-Compilers--Make-Macros-Instead
======
overgard
Its not exactly equivalent, but having spent the last month debugging a
nightmare of C++ template meta programming mixed with a lot of C macros... I
think things like this can be nice if they’re applied tastefully but I
wouldn’t trust 99% of programmers to be judicious with these things. As
someone else said, it’s catnip to a certain type of coder. It also creates a
really nasty implicit dependency — you can’t pull out individual components
because they’re dependent on the DSL, and who knows what dependencies the DSL
pulls in? I would actually argue that macros might be why Lisp didn’t become
mainstream; it creates a lot of fragmentation. Most people don’t even want to
learn the syntax of their build system, asking them to learn some weird
language extensions they can’t take to their next job seems like a rough sell

Edit: one more thing I forgot to add: this is a nightmare for tooling. I know
there’s the contingent that wonders why you’d ever want anything more than vim
or emacs, but for the rest of us, this just means refactoring/code
navigation/syntax highlighting/autocomplete/static code analysis are totally
broken.

~~~
rauhl
‘Most programmers can’t be trusted with macros’ is a sentiment I read a lot,
but at one point in time people might have said the same about functions, and
arguably they could say the same about pointers or recursion today.

I think that being able to think symbolically about one’s code is a vital
skill for a programmer (and, more generally, that the capability of thinking
abstractly is a vital skill for a learnéd human being). Thus, if someone lacks
that skill, perhaps _he should probably not be a programmer_? That may sound
revolutionary, or elitist, but I think most of us would agree that someone who
can’t effectively reason about a depth-first tree traversal will have trouble
as a professional software developer; mayn’t that also be the same of someone
who can’t effectively reason about computational structure?

Your comment re. macros & fragmentation makes me wonder if you’ve ever used
Lisp’s macros: I think that they actually mitigate _against_ fragmentation, as
e.g. CLOS started out as macros atop Lisp and ended up being standardised as
part of it. Lisp’s packages help manage namespace collisions, and thus prevent
fragmentation by making differences manageable.

And Lisp tooling typically handles macros very well indeed.

~~~
AkshatM
I think the point being made is about incurring unwanted technical debt,
rather than about the skills of programmers. A macro enthusiast can leave
behind a codebase that requires substantial expertise or siloed knowledge to
work with.

It's not that you can't trust people to make good decisions or invest in
understanding what they're doing. It's that in real life people take
shortcuts. Time constraints, deadline pressures, a brief skimming of a
StackOverflow answer with no followup or further exploration - any of these
can lead to code that does the job, is ignored until only one person is left
who understands it, and needs to be parsed out and refactored. I think you can
only trust programmers with macros (or functions, or pointers, or object-
oriented programming) if you have good practices in place to handle technical
debt (clean naming, clean separation of concerns, adequate testing, etc.).

As a side note: the other big part about macros is YAGNI. Macros are fancy,
but I've yet to come across a use case for it that can't be replaced with a
simple function call. I'm also curious about how one would test a macro.

~~~
pjmlp
> I'm also curious about how one would test a macro.

The same way you test any other kind of function call?

------
andy_ppp
After using Elixir I love that 90.1% (according to github) of the language is
written in Elixir and the excellent macros system is a major reason for this.

I actually think baking in the ability to add language constructs with macros
is exceptionally useful and allows concepts to be expressed very clearly. A
great example of this is Ecto which allows you to generate SQL directly from
within the language.

    
    
        from p in Friends.Person, where: p.last_name == "Smith"
    

Will produce a select query at compile time, no ORM in sight taking up
resources and getting in-between you and the database.

~~~
sdegutis
Clojure had this same property, and it seems like a benefit at first: because
new "language features" aren't tied to releases of the language, anyone can
create them and instantly share them with the community for everyone to
immediately start using.

A great example is destructuring, which someone wrote a macro for, around
version 1.2 or so, that got bundled with the language shortly after. You can't
do that without changing the language, or being able to extend it with macros.

But in practice, having community-written macros means you'll get several
versions of the same one, and there often won't be any clear winner since each
will have strengths and weaknesses, and at least some authors will be
unwilling to merge or add features or change their version, so that you end up
with a bunch of almost-perfect macros and have to settle for their annoying
bugs _that you know you could fix in 5 minutes_ if you just made a private
fork, but then you end up in the xkcd-standards paradox.

Babel showed that you can successfully achieve the same thing macros do, but
outside of the language, by writing a compiler and just targets the language
you want to execute. No need for macros to get destructuring, just enable the
"destructuring" plugin.

Unfortunately, Babel misses the mark on this dream, because _literally every
Babel "plugin" is actually implemented in Babel's core library_ and all the
plugins do is just enable a flag or two in the core parser/compiler. It
doesn't expose any parser or compiler for you, and you can't actually add new
tokens without forking Babel.

~~~
TeMPOraL
Changing a macro you own > forking a macro library > forking a transpiler >
forking the language.

I admit I never worked with Babel on the plugin-writing side, but that sounds
much more complex than writing macros.

With Clojure macros, and Lisp macros in general, one of the important part is
that they run within the same system as the rest of your code. That is, macro
logic can be split out to _regular functions_ , that will be executed at
compile-time. Macros can reuse functions you wrote to be used at runtime. This
property makes a language and your codebase much more coherent wrt. macro use.

------
tzs
In this example:

    
    
      macro swap {
        rule { ($x, $y) } => {
          var tmp = $x;
          $x = $y;
          $y = tmp;
        }
      }
    
      var foo = 5;
      var tmp = 6;
      swap(foo, tmp);
    

the article says it expands to this:

    
    
      var foo = 5;
      var tmp$1 = 6;
      var tmp$2 = foo;
      foo = tmp$1;
      tmp$1 = tmp$2;
    

Is that really true? The statement "var tmp = 6" appears in the source code
_before_ the place the macro is invoked. Can a sweet macro reach back and
modify statements that occurred earlier in the file?

~~~
jlongster
Macros cannot reach back like that, no. The renaming of the variable is due to
sweet's hygiene system that operates on the program as a whole - if two
variables with the same name exist in the same scope it will rename them to
ensure uniqueness.

~~~
ianbicking
But then shouldn't the rewriting be like this?

    
    
        var foo = 5;
        var tmp = 6;
        var tmp$1 = foo;
        foo = tmp;
        tmp = tmp$1;

~~~
jlongster
It could rewrite it like that, yeah, but sweet renaming all collided
variables, likely to make it clear that they collided. It doesn't try to keep
the distinction between what a macro introduced and what was already in scope
(not worth the complexity).

This was back in 2014 though and sweet has changed a ton since then, so the
output is probably different

------
nialv7
> so they (C macros) are pointless except for a few trivial things

Oh, believe me, they are NOT.

~~~
jedimastert
Right? This seems a little obnoxious. I could totally see doing all of the
things he talked about with C macros.

But whatever.

~~~
ChrisSD
Since C macros are glorified copy/paste, you can do anything with C macros...
which was the point, I think.

~~~
KingMob
??? Not sure what you're getting at. You can't actually use the language
itself from inside a C macro, like you could a Lisp.

------
antihero
Please no, I'd much prefer the logic of taking nice things like types and new
features and having them implemented by a central team that knows what they're
doing as opposed to being implemented on a per-team/project basis by people
have no idea what they they are doing.

~~~
jjnoakes
Do you let the people who have "no idea what they are doing" write functions
too? Put code into production? You pay them right? You train them?

Why not teach them about macros and have code reviews on things they submit,
just like you would normally?

------
pmarreck
This was written in 2014. What's the js macro scene look like today? Has Sweet
grown in usage/utility?

Also, does this macro system have a concept of quoting/unquoting?

------
lhorie
Worth noting that sweet.js mostly fell by the sidelines after Babel came
along.

With babel-macros, one can write macros that look exactly like function calls.
Or, more accurately, one can make any valid syntactical construct expand as a
macro. Webpack and friends also do this to an extent with loaders.

In the current Javascript landscape, I don't generally see macros being used
for syntactical sugar as much as they are used for code generation: things
like "take this SVG file and generate a React component that renders it" or
"generate the boilerplate to inject this stylesheet that I added via a ES6
import declaration".

------
ljm
I remember when Sweet.js was first announced, I was excited by it so I started
playing around.

I still like it a lot as an idea, but maybe not in the way it was originally
intended. In the last four years people have built parser libraries for JS (of
varying kinds), and of course you have what became Babel. And Webpack. You
might as well see them as way of plugging in features into the language you're
writing and taking macros to an extreme, because it's not pure JS any more.

Yet most of that could be represented as macro definitions if you were to put
the work into it. So I actually like the idea of Sweet.js as a really simple
way to experiment with JS syntax without investing in parsing a language from
scratch, or trying to plug it into Babel.

In the context of JS itself though, it's not a solution I would promote as
sustainable. As soon as you see the idea of macros you start to want to think
of everything as a macro ("because it produces more optimal code!" or
whatever) and then you get a codebase that cannot possibly be maintained,
because changing one line of code in your macro isn't always the same as
changing the same line inside a function.

------
lolive
I have never used macros. Is it really that much productive and readable?
Versus proper compiler features (async, =>, type system) + a proper code
design.

~~~
lucozade
> Is it really that much productive and readable?

It can be. Used judiciously, they can reduce boilerplate and help highlight
the important parts of the codebase.

The problem is that they're catnip to a certain type of developer. If they're
allowed to proliferate in an uncontrolled way, you can easily end up with
unmaintainable code.

Worst case, you'll end up with someone implementing a half-arsed type system
at which point you may as well delete the repository (ok, slight
exaggeration).

I can't really comment on whether or not JS would be a better language with a
good macro system. My guess is probably not but I have no evidence for that.

~~~
shawn
Emacs is a very important ecosystem that effectively uses macros. The danger
of macro overcomplication is real, and emacs' conventions are the primary way
they avoid introducing problems.

For example, macros and functions are nearly the same thing: one runs at
compile time, the other at runtime. But they both have documentation, and you
can jump to a macro's definition the same way you would for a function. You
can also compose them together and generate new ones on the fly, etc.

Basically, a lot of the problems with macros in other languages is that macros
are considered this special type of not-really-a-function. It's either
templating sugarcoat, or restricted turing-noncomplete layer. But if you treat
macros as literally functions that just happen to run at compile time, the
problems are reduced. And if you treat them with discipline and document what
they do, and show examples, and add test cases, then you never have to worry
-- just like normal software.

One helpful rule of thumb: If you copy some functionality more than twice,
it's probably time to make it a function. If you _define a set of functions_
repeatedly -- like test cases -- or find yourself writing the same pattern
over and over, or if you can generate most of your structure automatically
based on e.g. "I know this is a React component", it might be time to write a
macro.

If this concept sounds intriguing to anyone, I highly encourage you to check
out Lumen: [https://github.com/sctb/lumen](https://github.com/sctb/lumen)

It's a Lisp that's literally javascript:

Compiled code:
[https://github.com/sctb/lumen/blob/master/bin/reader.js](https://github.com/sctb/lumen/blob/master/bin/reader.js)

Original code:
[https://github.com/sctb/lumen/blob/master/reader.l](https://github.com/sctb/lumen/blob/master/reader.l)

So if you understand Javascript, then there's not much to learn. And you can
use it in your own projects right away.

Most of Lumen's brevity is thanks to macros.

(One hack for quickly understanding Lumen: read the tests. Every language
feature is meticulously demo'd:
[https://github.com/sctb/lumen/blob/master/test.l](https://github.com/sctb/lumen/blob/master/test.l))

~~~
lolive
As a OO programmer, I would still argue that: if I define a set of functions
repeatedly, I will try to find an elegant object design to abstract that. One
good point, then, is that I can debug that object design at any line of code.
Whereas (I feel that) macros are undebuggable by design.

~~~
shawn
Suppose you were forced to use a language with no objects, and no closures.
Since you're an effective dev, you'd find ways to cope. But we invented object
systems to do better work.

Closures were another step up the ladder of abstraction, and now it's hard to
imagine not being able to use them.

Macros are no different. It seems worth being skeptical of the idea that it's
better for a programmer to have less power. And Emacs Lisp shows macros can
work at scale without causing friction.

To your point on debugging, it all depends on the language. Elisp has an
excellent debugging facility:
[https://www.gnu.org/software/emacs/manual/html_node/elisp/Ed...](https://www.gnu.org/software/emacs/manual/html_node/elisp/Edebug-
and-Macros.html#Edebug-and-Macros)

The key is to internalize that macros are no different from functions. If you
know how to call a function, you know how to use macros. The only difference
is that they return _code_ rather than _values_.

~~~
lucozade
This was kind of my original point though. If you do treat them as compile
time functions i.e. for a bit of light code gen then they can be super useful.

But that's often not the case. In fact, OO is a nice example. It's probably
less true now that OO seems to be a bit out of fashion, but the number of
crappy, poorly documented OO systems I've seen in Lisps over the years is
staggering.

It's not really a criticism of macros so much as an observation. It's also
possible that the type of developer who is attracted to adding a bijou OO
system into their application is the type of developer that is attracted to
Lisps. It may well be the case that adding a macro system to JS wouldn't have
the same attraction so wouldn't have the same downside.

------
benjaminjackman
Something I have been experimenting with what I call "persistent snippets",
over frustration with boilerplate. Basically when inserting a snippet into an
editor it leaves behind a magic comment, that lists the parameters of the
snippet, and wraps the snippet code in a //#region to enable folding in
editors and languages that support that.

e.g.:

    
    
        import {insertSnippet} from "polish-and-release-this-someday"
        import {mstBoilerplate} from "./some-local-snippet-store"
        
        insertSnippet(mstBoilerplate, {name : "XCardModel"})
        //#region mstBoilerplate 20180911
        export interface XCardModelInstance extends Instance<typeof XCardModelImpl> {}
        export interface XCardModelCreation extends TypeHelp.Id<typeof XCardModelImpl["CreationType"]> {}
        export interface XCardModelSnapshot extends TypeHelp.Id<typeof XCardModelImpl["SnapshotType"]> {}
        export interface XCardModel extends TypeHelp.Id<typeof XCardModelImpl> {
          Type: XCardModelInstance
          CreationType: XCardModelCreation
          SnapshotType: XCardModelSnapshot
        }
        export const XCardModel: XCardModel = XCardModelImpl
        //#endregion mstBoilerplate 20180911
    

This means a simple editor extension and very simple command-line tools can be
written to expand / refresh the "insertSnippet" lines as needed.
mstBoilerplate, basically is a template literal string in typescript /
javascript land but you can use whatever makes sense for your language. So
this is useful for language that don't want to / can't support macros
natively. You end up getting a lot of the boilerplate reduction power, and
everything just works.

I feel for a lot of cases this works better than macros, the expanded code is
checked into version control so there is nothing hidden going on, debugging
works, if the code ends up needing to be specialized a bit, it's pretty easy
to erase the insertSnippet line / comment lines and start modifying.

~~~
hinkley
Interesting. I had the opposite idea a while back, inspired by a coworker who
struggled greatly with indirection.

In my scenario I’d like to extend the code folding mechanism to show inherited
methods and macro expansions inline, folded by default. Then you could drill
into a five function and/or macro code flow, without ever leaving the current
editor window, if the code has good cohesion.

And if it doesn’t, then this is more incentive to fix your code.

------
romanovcode
> when run through the sweet.js compiler.

So.. which one is it?

~~~
carry_bit
The title says writing compilers, not using compilers.

It's almost the case of "one compiler to rule them all", though IMO pervasive
changes (like adding a type system) are better done at the compiler-level
rather than the macro-level.

------
platz
Macros typically have limited scope compared to a compiler, and tend to
operate locally.

Also, macros typically do not carry state between macro instantiations.

As such, with a macro it would be very hard to eliminate null from the
language.

Therefore, we need compilers.

------
loosetypes
So CL macros often follow the pattern of several forms of processing before
returning a quasiquoted expression using the results of those earlier forms.
That is, some regular code lisp code, followed up by, I guess I'll call it, a
DSL.

Is that really all that different from where much of mainstream javascript-
land has ended up?

It might take a moment to familiarize oneself with macros or a DSL, but does
that not almost precisely define react component definitions?

Component definitions execute some javascript then spit back JSX and these JSX
components can be used in a DSL that doesn't follow javascript syntax?

~~~
shawn
_It might take a moment to familiarize oneself with macros or a DSL, but does
that not almost precisely define react component definitions?_

Surprisingly, the answer seems to be no.

Experienced react devs rely heavily on metacomponents: components that take
other components as input, and return a combined component as output. It's
like oldschool template metaprogramming in C++, but you end up merely wanting
to stab yourself in the eye with a fork rather than hurl yourself out the
nearest office window.

A proper macro system would make this pattern unnecessary, because you would
rely on macros to generate the specialized components. It would spit out react
code that you'd otherwise have to write -- or that you'd have to write a
metacomponent for.

Note that a metacomponent is _not_ the same as a macro. It's similar, but it
still operates at _runtime_.

Here's a rule of thumb. Can your macro system embed the contents of your
/etc/passwd file as a string literal into your codebase? If the answer is no,
then you're missing out on significant power.

~~~
coding123
Hopefully etc/password is a bad example? Security issues aside, I feel like
reading files that are configuration in nature are best done "at runtime" not
compile time. Can you imagine the code that loads a configuration file
directly into the code and the ops team unable to reconfigure things? That
would be the epitome of "well it worked in MY build".

~~~
TheDong
Note that `/etc/passwd` contains no passwords or other security sensitive
stuff. You might be thinking of `/etc/shadow`.

That is an example of something you might not actually want to do, though
embedding a file from the code repository (e.g. as [https://doc.rust-
lang.org/std/macro.include_str.html](https://doc.rust-
lang.org/std/macro.include_str.html) does in rust) is a reasonable and common
thing to want to do at build time.

------
dang
Discussed at the time:
[https://news.ycombinator.com/item?id=7025261](https://news.ycombinator.com/item?id=7025261)

------
amelius
How do you define in what order macros are applied?

~~~
KingMob
Dunno about sweet.js, but in Lisps, it's usually source order. E.g., read the
next token, is it at the head of the list and names a macro, if so, call the
macro function, and pass in the code as the param, take the output and replace
the macro form with the expanded version, resume reading...

------
fithisux
But there is Biwascheme.

------
anfilt
Honestly, can we just stop writing java-script. Even better remove java-script
from webpages. Oh wait that boat sailed a while ago.

That aside I don't see what's the problem is with cross compiling to an other
language. Other than if your writing a compiler why not compile to machine
code. Honestly, I think code generators are better than macros.

~~~
vertline3
I enjoy writing JS, so no, I won't stop.

~~~
raxxorrax
Seriously. The hate for JS is strong, but I don't really see why.

Use typescript if you want, but I don't really see the issue. There are funny
things related to implicit type conversion and some rules produce funny and
unintuitive outcomes. These conversion can be very handy on the other sides.

"Math.min() > Math.max()" and they are damn right that this is true!

~~~
spdionis
> Seriously. The hate for PHP is strong, but I don't really see why.

> Just use PHP 7! There are funny things related to implicit type conversion
> and some rules produce funny and unintuitive outcomes. These conversion can
> be very handy on the other sides.

> "PHP_INT_MIN < PHP_INT_MAX" and they are damn right that this is true!

Feeling the perception difference?

~~~
Klathmon
I always felt the same about PHP.

It gets a lot of shit because some of the oldest parts have some weird
function names or parameter order, but outside of that it's a fantastic
language, especially if you stick with php ~5.5 and newer. It's fast, it has a
nice package system, and the pepole behind the language are making it better,
faster, and more feature complete every day, with standards bodies which
includes the biggest players in the space.

IMO languages like Go could learn a thing or 2 from PHP about how to stop
worrying about the "best possible choice" and just start giving the language
the tools that developers can use to solve real problems.

~~~
akatechis
The difference is that JS has to be backwards compatible in order to not break
50% of the internet. With PHP, it is theoretically possible to make a major
version upgrade that fixes most if not all the issues without worrying about
breaking backwards, since upgrading is opt-in.

~~~
Klathmon
Upgrading was opt-in with python as well, and look at where breaking changes
got them... A decade in and there's still a massive rift in the python
community over 2 vs 3.

PHP is backwards compatible almost to a fault, but that doesn't mean you can't
use new libraries or frameworks that hide those warts away from you, and
that's a MUCH better choice in my opinion than making a significant percentage
of the internet incompatible with the latest PHP so that your function
arguments for a handful of old function calls can all look the same...

~~~
anchpop
"massive rift" is a bit of an exaggeration. Most Python libraries have Python
3 versions and it's not at all uncommon to see libraries that don't have
support for Python 2 (see Django)

~~~
Klathmon
I don't think it is. Sure, it's not like they are 2 different languages, but
it's something that all python developers will have to keep in mind when using
libraries. Not to mention the headache of having both installed on many
systems (what does `python` give me?), and there are still many things that
don't have python3 versions still (node-gyp is the one that still bites me
every day).

It's not the end of the world, and it is getting better every day, but having
to spend the better part of a decade dealing with multiple "main" versions of
a language is not something that I want any other languages I use repeating.

And if that means needing to lookup if the function is urlencode or
url_encode, well i'm happy to pay the price.

