
Scala Macros: “Oh God Why?” - thebootstrapper
http://blog.empathybox.com/post/19126121307/scala-macros-oh-god-why
======
odersky
The blog post and tweets seem to say the Scala community only cares about
fancy language stuff and not about the developer experience. Nothing could be
further from the truth. Take Typesafe for example. We have three full time
engineers on staff to improve the Scala IDE for Eclipse. We made a lot of
progress and are continuing to do so. Nobody pays us for any of that; we do it
because we know that IDE experience is crucial for Scala developers. Typesafe
is just one company and it cannot pour unlimited resources into all aspects of
Scala development, so we need to rely on the community for that. And the
community does step up to the task. One of the criticism was on documentation.
I wonder whether people have recently looked at <http://docs.scala-lang.org/>
? In my mind, that's community-driven development at its best!

Why macros? I was a long-time sceptic. I now tend to think about them
differently because I believe we hit on a brilliantly simple scheme that can
express a lot of different use-cases. In particular we'll be able to do the
analogue of Microsoft's LINQ with macros. And we can express optimizations
such as turning foreach applications into while loops. And we can remove a lot
of compiler plugins. And finally it looks like we can remove down the road
several special cases in the language and the compiler. So macros might pay
for their own complexity handily. But I should also say that at present this
is a SIP, a Scala Improvement _Proposal_. It's not yet accepted. And part of
the current proposal is also that macros will be enabled only under a special
compiler flag.

~~~
stuhood
We definitely appreciate the work that you and Typesafe are doing: thanks! I
apologize for my brusque reaction in that tweet.

Regarding macros: it's quite clear that they simplify the job of the compiler
writer: you can provide more features, more efficiently to the end user. But
every feature has a cost, and increasing the end user's surface area onto the
language by providing 1) macros, 2) new features enabled by macros, is very
much a double edged sword. Being enabled by a compiler flag definitely helps
to minimize end-user exposure to the former.

Macros in lisps can be used as a replacement for many of the features that
Scala provides, including call-by-name parameters, (certain types of)
implicits, and many uses of generics. If we could deprecate some of those
features in Scala by replacing them with macros, then this would seem like
more of a net win to me. But that doesn't seem likely: instead we will
probably have all N, and Scala programmers will need to decide which of the
growing list of ways to accomplish cat-skinning is best.

~~~
modersky
I am very much aware of the double-edge sword nature of macros, and therefore
follow and aid their progress with much trepidation. The single thing that
sold be was that macros by themselves already could replace a compiler phase,
which did code reification (i.e. generate ASTs at run-time; something that's
needed for LINQ like technologies). So in that sense, macros, if they arrive,
would already have paid for themselves. Concerning possible misuse, I believe
the best thing we can do is give an option to developers. So we plan to enable
macro definitions only if a special import is present. Something like:

import language.macros

Of course that will not prevent misuse, but it will make it much easier to
enforce policies, if someone desires that.

------
hp
The bug in this post is thinking that if you have someone willing to
contribute feature X to an open source project, they were also available to
contribute feature Y.

Open source patches are all itch-scratchings. For Scala macros, it's a
researcher at EPFL, I believe. I don't think you could write a thesis about
fixing bugs in the IDE. So this contributor was not available to submit IDE
bugfixes; they _were_ available to submit macros.

It's the same for other contributors. Someone building an app on Scala may be
available to submit a fix for some scalability issue they are hitting in
production; they will not be available to work on macros, or on IDE bugfixes.

Open source projects get the fixes they get. You can't act like the priorities
are set by some central product manager.

So I think it's just not correct to argue that "the priorities are wrong, why
are they doing this instead of that, etc." - everyone is doing what matters to
them. Some people _are_ working on IDE bugfixes, other people are working on
macros. People can do what they want.

~~~
cageface
Sure, but the communities behind all the FP languages seem to share this same
weakness. For obvious reasons, they attract people much more interested in
hacking on funky compiler techniques than on mundane things like I/O libraries
and documentation.

In contrast, I think Python and Ruby both owe a lot of their success to the
people willing to put a lot of energy and polish into this essential but less
glamorous work.

~~~
tikhonj
I don't know, this has not been my experience with Haskell at all. Not only do
plenty of people work on all sorts of libraries (there are some really great
web frameworks, for example), but they also work on all sorts of "less
interesting" stuff. For example, there are some people working to improve
Hackage, which is the site that contains Haskell's libraries and their
documentation, and Haskell has really good support in Emacs (which is much
better than good IDE support!).

~~~
cageface
I haven't spent much time with Haskell lately but the last time I did Cabal
was still way less polished and useful than Ruby's gem system or even Python's
pip/distutils stuff. Robust, easy to use package management is step one in
getting any new language off the ground, IMO.

~~~
gtani
Yup. This week's cabal anguish:

[http://www.reddit.com/r/haskell/comments/qxopq/announcing_th...](http://www.reddit.com/r/haskell/comments/qxopq/announcing_the_yesod_platform/)

[http://www.reddit.com/r/haskell/comments/qwj5j/the_cabal_of_...](http://www.reddit.com/r/haskell/comments/qwj5j/the_cabal_of_my_dreams/)

------
stephen
Where I work (i.e. write code for money, to satisfy Ted Nyman's requirement
that my opinion matter), our reaction was exactly the opposite--we're very
much looking forward to having macros in Scala.

If this was a more esoteric language feature, I'd be inclined to agree, but
macros (if done well), I think will be a huge boon to the language.

I do all sorts of hackery in Java now (external code generation, heavily using
annotation processing tools (APTs) which are a very, very limited form for
language-based code generation) to do semi-metaprogramming stuff and can't
wait to do the real thing.

(And do it well, e.g. not to make things "even more cryptic" just for the fun
of it, but to solve real problems.)

------
chalst
Macros make language design experimentation easier. This issue is not
orthogonal to tackling the other issues the OP cares about.

There are good reasons to doubt the overall value of macros in production code
- I won't argue this either way - but OCaml has had success by separating the
macro expansion mechanism (camlp4) from the main language. People who want
macros can have them, people who want a production language without macros can
have that too.

------
lihaoyi
"I understand macros to provide two things (1) forced code in-lining and (2) a
kind of non-strict semantics or lazy evaluation of arguments"

False. There isn't much else to say, but if this is all he thinks macros are
about, he doesn't know anything about them, their uses, or their potentials.

Macros would basically take a large amount of nasty extra-lingual stuff (code-
generation, byte-code generation, compiler plugins) and codify it into a
systematic, easier to use and easier to understand framework.

F#-style Type Providers? Compile-time typechecked regex literals? Compile-time
typechecked html? Compile-time typechecked CSS? This sort of thing is already
being done (see Play!) and is pure awesome as a user (i <3 the type-checked
HTML templates) but the implementation involves a huge amount of mysterious
extra compilation steps and magic that macros could help simplify and codify.

------
vrotaru
I seems to be a rather common tension in open source projects developed by
research institutions.

The goal of users of those projects is immediate usability which clashes with
goals of developers, research.

I have some doubts where one can get PHD (publish an article with high
citation index) by improving a compiler error messages. As opposed to adding a
new cool feature to the language/compiler, cryptic error messages be damned.

~~~
CookWithMe
True.

I still think it is a shame that neither Red Hat nor Jet Brains has joined
forces with Scala/Typesafe. Both of these companies know how to ship products
and make developers happy. This is something that Typesafe has to learn now.
On the other hand, Red Hat and Jet Brains need to learn how to do language
research... (which may actually be harder, once they go beyond fixing the
obvious Java flaws).

~~~
cageface
As much as I personally like Scala I understand why a lot of people are taking
a wait & see attitude. I think it's not too unreasonable to imagine that a
less ambitious Java++ like Kotlin might find the traction that has so far
eluded Scala.

------
lysium
The listed points are valid, but they are by far not as easy as the author
thinks they are. I don't say they are not improvable, but they are not 'little
things' that need to get fixed before new features come in. For example, the
jars are large because every closure needs its own class (thank you jvm!).
Compile time is high because of type inference. Error messages and stack
traces are cryptic, because you do high-level programming but get low-level
diagnostics. etc.

------
baconserker
The author doesn't even know what macros are: "I understand macros to provide
two things (1) forced code in-lining and (2) a kind of non-strict semantics or
lazy evaluation of arguments"

~~~
smanek
What else do you think macros provide?

As a former professional Common Lisp dev, those two features strike me as the
real difference between a macro and function call.

~~~
mindslight
The other difference is (of course) what the macro returns - more code to be
compiled, which enables things like binding of new symbols/types/etc. The most
important part is that the generated code is adjacent to user code, so the two
are lexically coupled. But in CL land, you probably take the codegen for
granted and the lexical coupling doesn't matter as much.

Although, Scala specifically, I ran away from. The language itself isn't
terrible, it just forces one to confront the endgame of the bankrupt Java
philosophy. The stdlib is basically the cross product of pre-optimized
category theory with noun-hell, and idiomatic scala is far too overabstracted
through abuse of first-class modules (which, admittedly, was a design goal).

~~~
gclaramunt
what do you use when have to use the JVM?

~~~
mindslight
The emergency exit. (sorry, couldn't help it)

I tried to use the JVM for a long time, I really did. I thought the advantages
of multiple implementations, platform flexibility, and a common type system
were fantastic. The fundamental problem is that no matter the skin, when you
want to peek through the abstraction you're still fundamentally writing Java.
I absolutely detest the Java language itself, as it's basically what should
come out of a compiler, not be fed into one (I realize this criticism is about
15% unfair given that Java did actually introduce some new concepts to
mainstream programming but implemented them in the VM rather than the
compiler, but it has really failed to keep up with the times).

At this point if I were using the JVM, it would most likely be to make use of
some hypothetical _damn good_ libraries. In that case, as I'd be gluing
together things that are already typed, I'd use whatever dynamic language
seemed prudent. The Java type system is an everpresent fact of life on the
JVM, and I'm now of the opinion that trying to use a second type abstraction
concurrently is foolish (especially if one wants to run their code on anything
other than Hotspot). I think Clojure is an amazing piece of work, and would
still be using it, but its lack of importance on typed data structures really
kills my ability to reason while developing new abstractions (the lack of
well-supported algebraic data types and pattern matching especially). Taken
together, those last two points imply that a statically typed JVM language has
to utterly commit itself to the Java type system. If I absolutely needed to
write a library on the JVM, I would look into and most likely go with
Ceylon/Kotlin (or Gosu if they ever actually did a source release), but
failing those probably write java-in-kawa for the straightforward class
definitions with macros to ease the pain.

In any case, I'd be damn certain of exactly what I was writing beforehand. The
Java ecosystem is absolute _garbage_ for prototyping.

------
rbehrends
The post raises a few good points -- those are real problems in using Scala --
but the problems are mostly a result of the compiler targeting the JVM as its
primary backend. Obviously, a JVM-based language can leverage a rich
infrastructure of libraries, but it also will have to deal with the fact that
the JVM is primarily designed for Java, not as a more general backend (such as
the LLVM or the CLR).

(That the compiler is slow has other reasons, but is also something that
they've been working on simultaneously for 2.10, so it's not as though that
has been pushed aside by macros.)

------
billjings
I'd like to single out this line from the linked article:

"Unlike Java, type signatures in Scala don’t really explain idiomatic usage to
mortals very well."

This is a great diagnosis of one of the big issues with Scala's style of
static typing. Everyone knows that the type descriptions take a bit to grok
compared to a C-style languages. Why? Well, in C-style languages, a function
definition is itself a representation of idiomatic usage. In Scala and its
brethren, you have to translate from one to the other.

I'm sure that's a commonplace observation, but it had not jumped out at me
like that before.

~~~
sausagefeet
Ii have not used Scala but I have found the opposite to be true in Ocaml and
Haskell. The type definition of a function is quite often sufficient
information to infer the usage. I think this is because of the type variables.
Looking at the definition of fold in Ocaml, for example, it becomes very
obvious what is the input output init and the signature of your function from
the type where such information generally seems to be less clear in C-like
languages.

~~~
billjings
The point is not whether you have sufficient information to infer the usage.
Do you have to infer the idiom at all? Usage is homomorphic with declaration
in C, so this step is unnecessary.

~~~
sausagefeet
I don't follow what you mean, I'm saying a list of function types contains
vastly more useful information than a list of C function types.

------
baconserker
For those curious about what Scala macros actually are: "This facility allows
programmers to write macro defs: functions that are transparently loaded by
the compiler and executed during compilation. This realizes the notion of
compile-time metaprogramming for Scala." <http://bit.ly/zeMjOb>

<http://scalamacros.org/usecases/index.html>

------
eloisant
Macros are useful in Scala. Mostly not for application developers, but for
library or framework developers. It enables you to do things while keeping the
typesafety nature that makes Scala so great.

In Play, we have generated code for forms validation and json serialization.
This enables the compiler to see code that can't work, while a more dynamic
approach would lead to an exception at runtime.

------
draegtun
For those interested the next release of Rakudo will be the first to
(partially) implement Perl6 macros.

ref: [http://strangelyconsistent.org/blog/macros-progress-
report-d...](http://strangelyconsistent.org/blog/macros-progress-
report-d1-merged)

------
soc88
I think many people share some concerns about macros, especially the need for
good error messages. A far as I know Scala's language designers have been
against including macros for a long time. So the proposal for macros has to be
damn good if they change their opinion.

Macros are supposed to be a replacement for writing compiler plugins in many
cases (compare with Java annotation processing vs. Java compiler plugins). The
API is easier and more straightforward, semantics are easier to understand and
don't require separate setup step.

In the meantime work has of course not stopped:

\- Syntax has been simplified and several confusing bits have been
removed/deprecated (Octal literals, FP literals without digit after dot, ...).

\- The jar size of the library has been reduced, while adding fixes and
functionality.

\- Value classes have been integrated, which provide the performance of
primitives with user-defined classes.

\- Inlining has been improved a lot.

\- Compilation speed has been improved a lot.

\- Some collection classes have been performance optimized and are now a lot
faster (and sometimes faster than the Java implementation).

\- Play 2.0 and Akka 2.0 have been released, both with huge amounts of
documentation.

\- Documentation has been improved.

\- ...

My personal impression is that while Scala has roots in academia, solving
real-world issues is the focus of the team.

~~~
anonymoushn
As far as I can tell, value classes provide the performance of primitives with
primitives. Is there a way to make a value class that contains more than one
primitive, so that you could have semantics similar to those of a struct in C?

