
Reasonable Scala Compiler - adamnemecek
https://github.com/twitter/reasonable-scala#
======
joefkelley
My opinion of this changed after reading this comment by the developer:
[https://www.reddit.com/r/scala/comments/6ubuix/twitter_annou...](https://www.reddit.com/r/scala/comments/6ubuix/twitter_announces_reasonable_scala_compiler_an/dlrxs6m/)

Sounds like they are intending it as a research effort and hope to make
progress with the entire Scala community, not fork it.

------
addisonj
It will be interesting to see what the _subset_ is.

I have been writing scala for 2 years, mostly in spark and flink and as much
as I love parts of it, overall, the language is just not the most
approachable. Some of the features are super valuable every once in a while,
but I might gladly give some of them up for a streamlined language that is
easier to get people started on (while not just writing imperative style Java
with Scala syntax)

My other curiosity is compatibly. If I can compile a subset of my project much
faster that doesn't need all of Scala, that seems like an okay place to be in.

~~~
adamnemecek
Check out Kotlin. I picked it up in like half a day. It wasn't idiomatic
Kotlin but w/e. It can compile to JS and you can use just about any JS lib
from Kotlin (which is insanity). You'll be able to write ios apps with it soon
enough.

Scala can be too hard sometimes.

~~~
alextheparrot
He's using Spark and Flink... Given that I'm also in that big data space, I'm
certain the tooling provided by Kotlin is strictly worse than Scala for those
applications.

------
bearforcenine
I am a bit concerned that between the Scala, Dotty, Typelevel, and now the
Reasonable Scala Compiler, that the Scala community might become fragmented.
Hopefully the rsc's claim of "making our findings available to the Scala
community at large" holds true.

Edit: should have keep reading before commenting. In their related work
document:

>we are definitely not declaring Lightbend Scala dead. To the contrary, we
view ourselves as complementary to Lightbend Scala and hope that our results
will create new opportunities for the official compiler. We are planning to
keep in contact with Lightbend to discuss our findings and facilitate
technology transfer.

~~~
foolfoolz
scala and dotty will be one and the same in the next major version. the rest
are niche tools. no one should be afraid to develop their own compiler or tool
chain for fear of fragmentation. the lessons learned can only help the trunk

------
Others
I'm interested in what they said about reducing the compiler to 4 passes.
Dotty takes a really interesting approach to this with 'miniphases'
([https://infoscience.epfl.ch/record/228518/files/paper.pdf](https://infoscience.epfl.ch/record/228518/files/paper.pdf)).
I think it'll be interesting to compare these two methods when both compilers
are production ready.

------
majidazimi
This truly means we still lack a programming language with:

    
    
        1. reasonable support of object oriented programming.
        2. reasonable support of functional programming.
        3. solid concurrency features.
        4. runs comparable to native code.
        5. simple to learn.
    

Even though it is not difficult to create such a language but we see a new
language poping out every now and then and none of them try to solve these
issues.

~~~
m12k
What I'm looking for in a language:

1\. Solid mix of functional programming and Object oriented programming
(though FP is more important - I'd gladly give up inheritance if I get good
support for interfaces and traits instead)

2\. Static type checking

3\. Good support for efficient immutable data structures.

4\. Solid concurrency features

5\. Easy cross-compilation to and interaction with JS

5a. Cross-compilation to iOS and Android would be nice but it's not a
dealbreaker

6\. Good tooling (i.e. IDE-support, package manager)

7\. A solid standard library

8\. An active open source community around it

9\. Said open source community should contain a good web MVC framework (or
sensibly pluggable modules to the same effect)

10\. A REPL

11\. Sensible compilation times

So far it looks to me like only Scala and Haskell are candidates - and it
sounds like Scala really struggles with the compilation times. If Reason/ocaml
gained more traction it might become a contender. And if I gave up on static
type checking then Clojure would be an option. And if I settled for a
sprinkling of functional programming instead of having that be the basis, then
Kotlin or maybe C#. Any other suggestions?

~~~
Matthias247
Besides the options you mentioned I really think Typescript is a high
contender on that list. Even though it's just Javascript with typing it ticks
most boxes you mention. Depending on how you value some features (e.g. nr. 5)
it might even be the best option.

~~~
lmm
Its FP support is even more limited than C#'s, no?

~~~
Matthias247
Imho they are about the same, or maybe TS is even a little bit better, since
you can really have freestanding functions (C# has either (static) member
functions or delegates). The Javascript ecosystem probably also leans a bit
more towards FP style in the mainstream (with lodash, ramda, redux, etc.) than
C# does. What are you missing in particular? I guess the main thing for both
is tail recursion.

~~~
lmm
Does TypeScript have any kind of "do notation" or "comprehension" syntax? LINQ
is not the best by any means but it does seem like a big improvement over
nothing.

~~~
Matthias247
No, it doesn't. I guess that's because monads are not considered idiomatic
typescript. And for the special case of async programming there's async/await
support.

------
jankotek
> _Identify a subset of Scala that can be compiled with reasonable speed_

I am afraid of Scala fragmentation. There is no way new compiler can be 100%
compatible. Hopefully it will be tested on major projects, or there will be
some official Scala Language spec.

It would be nice to add a switch into old scalac, which downgrades its
features, to make it compatible with new scalac.

I believe Scala has too many features, and it will be very difficult to fix
compilation speed. But there could be decend boost just by rewritting compiler
from scratch. If anything we will get better error messages.

Compilation speed was major reason why Jetbrains started Kotlin.

Disclaimer: I worked with Scalac / IDE back in 2009. Kotlin fanatic.

~~~
harpocrates
> There is no way new compiler can be 100% compatible. Hopefully it will be
> tested on major projects, or there will be some official Scala Language
> spec.

I, for one, think that an official language spec would do the language a lot
of good. As it stands, it is sometimes difficult to separate bugs from
features in scalac...

~~~
dtech
I am not sure how complete/correct it is, but there is one for 2.9 [1]

Martin Odersky & co have been focusing most of their effort the last few years
on Dotty (aka Scala 3), which succeeded in giving in giving Scala a proven
theoretical basis in exchange for a few esoteric typesystem features they
couldn't prove. [2]

[1]: [http://www.scala-
lang.org/old/sites/default/files/linuxsoft_...](http://www.scala-
lang.org/old/sites/default/files/linuxsoft_archives/docu/files/ScalaReference.pdf)

[2] [http://www.scala-lang.org/blog/2016/02/03/essence-of-
scala.h...](http://www.scala-lang.org/blog/2016/02/03/essence-of-scala.html)

~~~
simon_o
There are also specs for later versions, it's just that they are pretty
outdated and unmaintained. (Like pretty much everything related to
documentation.)

I think Jetbrains has some experience using that spec to implement the IDE's
Scala typechecker. Long story short: years later, it still doesn't work.

I would be kind of concerned about the esoteric new features they added to
Dotty, looks like the lesson has not been learned yet.

~~~
dtech
They removed the esoteric features because they could not be theoretically
proven sound, not added them.

~~~
simon_o
Just off the top of my head, esoteric additions:

\- "Numeric harmonization" which makes some of the existing issues with
implicit numeric conversions even worse.

\- Multiversal equality, which adds even more special rules to the implicit
resolution algorithm, complicates the mental model of the language, is either
extremely invasive or severely limited when applied to real-code, and falls
apart when dealing with existing language features of Scala like variance.

\- Addition of inline, which has absolutely no reason to be a new keyword.

\- The tentative idea of adding T? for T|Null (as it appeared on some slides)
which makes it obvious that no thought has been given to how the lessons of
handling nulls apply to Scala.

Then we have enums, which are just poorly designed and executed, repeat the
mistakes made with both case classes and scala.Enumeration, do not address the
problems it is supposed to solve, fail to address actually valid existing
problems, all while introducing not one, but two additional syntactic
constructs which are unlike anything we had before.

On top of that we got incompatible additions in minor releases of 2.12:

\- Additional places where commas can be added, which ignores one of the most
common complaints about Scala: Too many syntactic variations to express the
same thing.

\- The addition of @showAsInfix which is a solution in search of a problem.

But at least unsound type projections got restricted, I guess. (That's the
only major thing I can think of that will have a larger impact...)

Probably the removal of forSome and cleaning up existentials?

------
needusername
I'm a bit confused. I have been told multiple times by Scala users that Scala
compile times are a non-issue.

~~~
nicktelford
I've seen a lot of complaints about Scala's compilation speed (I believe it's
either the 1st or 2nd most common complaint), but I've not really had any
issues myself.

Could people who have had issues with compilation speed explain their
workflow, I'd like to better understand the problem.

For example, here's my general workflow:

1\. code in IntelliJ until the feature/fix is complete, and there aren't any
red-wavy lines in my source tree

2\. run `compile` in an already running sbt session.

As a result, I only find myself "waiting" for a compile a few times a day. And
even then, because I'm using the incremental compiler it's usually only a few
seconds.

The only reason I run the sbt compile at all is because the IntelliJ compiler
is known to be a bit buggy, especially around things like implicit resolution
and some of the more advanced features used for abstract programming.

I have seen CI builds take some time, but even then, in every project I've
worked on, the complete compile-time is dwarfed by the time taken to run
tests.

I'm not trying to dismiss problems that others have; I would love to learn
more about them!

~~~
harveywi
Here is one that I run into frequently. You're building a CRUD app, but want
some semblance of sane FP interaction with your database. So you use Doobie
[1] because free monads and all.

So you type your query using a Doobie SQL string interpolator...

    
    
      sql"SELECT ItemId from dbo.Item WHERE IsOnSale = TRUE"
    

...but then you forget all of the
".query[Int].vector.transact(tx).unsafePerformSync" crap that you are supposed
to tack onto the end of the string to actually run your query. What are your
options?

1\. Type a '.' character in your IDE and watch it crash while the presentation
compiler thrashes around, sifting through a combinatorial explosion of
implicit values (or whatever it is that makes it so slow). You quickly learn
to stop using autocomplete. In a sane world, this should take milliseconds and
you should quickly get a list of methods to guide you down the right path.

2\. Type a '.' and try to recompile. Wait 30 seconds. While you're waiting,
what work can you possibly do??? None.

3\. Stop what you're doing, open up The Book Of Doobie [2], try to remember
which page has the syntax that you need for doing this one simple thing, go to
that page, navigate to the right part of the webpage, read the crap, think a
bit, copy/paste it into your code. This FEELS more productive than (2), but is
it really?

This is just one example, maybe not terribly compelling, but it happens.
People (well at least me) have a limited amount of memory for administrivial
crap to hold in our heads while we try to get our work done. Tooling is
supposed to help with that, but with Scala it really starts to get in the way.
Flow state just isn't possible.

I have heard anecdotes of developers running two or more instances of scalac
so that they can work on more than one feature at a time in order to not
completely waste away their time. Imagine the context switching there, and
having to pay attention to the little oven timer and switching context when
one compiler is finally done recompiling the same code it already compiled
hundreds of times.

[1] [https://github.com/tpolecat/doobie](https://github.com/tpolecat/doobie)

[2] [http://tpolecat.github.io/doobie-
scalaz-0.4.2/00-index.html](http://tpolecat.github.io/doobie-
scalaz-0.4.2/00-index.html)

~~~
nicktelford
Thanks! I haven't used doobie yet (though I intend to), I had no idea it's API
caused IntelliJ such an auto-complete headache.

I'm curious how a faster scalac would help in this case though. Perhaps I'm
missing something, but scalac's errors don't really facilitate this kind of
API exploration.

I completely agree that exploring an API (because who wants to memorise the
standard library and API of all your dependencies) in IntelliJ is often quite
cumbersome. One trick I've come to lean on a lot is to type '.ensur' to
determine the type of the current term; this causes IntelliJ to present the
method signature for 'ensuring', which is available (implicitly) on everything
(except Nothing), showing the current type in its return type.

------
ece
Overall, I feel like with the success of Go and Kotlin, having an easy to
learn language definitely wins. Faster runtime speed might be a bigger
'market' than wanting fast compile times, which matters if you have a large
enough codebase. I do wish Scala native was further along, and if the language
was going to get simpler, it would focus more on native speed than compilation
time. Quite a few Scala compiler releases have made big strides in compilation
times, to the extent that java vs scala compilation times I feel are the same
or well worth the cost, so this has been and will be an ongoing effort. The
Java group is also working on being native, so we will likely see Java native
when it gets there.

~~~
danarmak
> The Java group is also working on being native, so we will likely see Java
> native when it gets there.

I haven't heard about this. Can you give a reference please?

~~~
ece
They are working on AOT compilation:
[https://www.youtube.com/watch?v=Xybzyv8qbOc&index=241&list=W...](https://www.youtube.com/watch?v=Xybzyv8qbOc&index=241&list=WL)

~~~
danarmak
IIUC, this is about AOT but still using the JVM. This is a fix for slow warm-
up, but in longer running processes the eventual behavior isn't different from
JIT. Which is important in its own right, but very different from a
hypothetical 'Java native' in the sense of 'Scala native', without a JVM at
all.

~~~
ece
I believe you are right. Azul has released a JIT compiler based on llvm:
[http://www.infoworld.com/article/3193078/java/azul-
introduce...](http://www.infoworld.com/article/3193078/java/azul-introduces-
llvm-compiler-to-java-runtime.html) Also:
[https://www.excelsiorjet.com/](https://www.excelsiorjet.com/)

and there seem to be open source projects like vmkit,
[https://github.com/ReadyTalk/avian](https://github.com/ReadyTalk/avian) that
are somewhere in between.

------
jgalt212
So Clojure is slow to start and Scala is slow to compile. Is Scala also slow
to start? As compared to a Java program to performs the equivalent functions.

~~~
iLemming
Clojure doesn't have slow start-up, Leiningen and Boot-clj make it feel slow.
But once you start a REPL you can keep it running. I keep mine running for
weeks.

~~~
jgalt212
I know. I've often heard this, but for me that's a bit tricky b/c I'm worried
that my current definition of "foo" is not what it is in my .clj file b/c I
may have monkey-patched it in the REPL and forgot I did that.

------
wpietri
Huzzah! Maybe this is already in the works, but I would love to see a compiler
that stays hot in RAM and works incrementally.

RAM is cheap nowadays, and the number one reason I want a fast compiler is to
speed up my feedback cycle between making a change and evaluating the effects
of the change. So I want something that squats on a large chunk of RAM and
works in parallel with me, updating with every edit I make.

~~~
walkingolof
SBT always runs Scalac in the same VM, so it will be hot and JIT compiled
after a few passes.

Intellij implemented a SBT Shell in 2017.1, so it will keep the same SBT
session going between passes also.

~~~
wpietri
That's a good start, but won't the compiler still be batch-oriented? By which
I mean doing a ton of I/O and activity only when I hit go?

I'm talking about something much more real time. I want the parser to keep
everything hot, updating every time I hit a key. And whenever I get to
anything that the parser approves of, downstream stages, which also keep
everything hot, dynamically update all of their structures, revising object
files as they go.

Batch orientation is great, and I understand why compilers started there. But
I want something that moves beyond that. It's sort of the difference between a
batch-oriented rendering system like TeX or early word processing versus a
modern WYSIWYG word processing setup, where we burn resources prodigiously to
give users much faster feedback loops.

~~~
speedkills
The Dotty compiler is going down the route you are looking for.

------
harveywi
I'd like to see a Scala compiler that works more like a proof assistant like
Coq. I want the compiler's job to be to help me quickly write correct code.
That should be first-class functionality. Batch-compiling the same source
files over and over again should be just a neat trick that the compiler can do
on the side.

------
jmnicolas
Doesn't Scala run on the JVM ? Are they talking about JIT times or are there
other Scala 'environment' that are not semi-interpreted ?

~~~
maemre
This is the compiler from Scala to Java bytecode. It does type checking, some
optimizations (like inlining, removing some instances of boxing, or converting
tail-recursive functions to loops), and converts all Scala-specific features
to some Java constructs (e.g. traits become interfaces, classes etc.; lazy
values become methods and accompanying values; lambdas get lifted to the class
they are defined in).

The JIT works on Java bytecode. Java also has a similar compiler (javac) from
Java -> bytecode.

------
amelius
Will they use this compiler only for prototyping/debugging/testing? Or also
for production work?

------
Throaway786
Twitter is not performing well as a business. It is also at almost all time
low. Should it be working on a new language/compiler?

~~~
adamnemecek
It's because their devs are spending too much time compiling their code /s

~~~
monksy
or maintaining their monorepo

------
edem
Writing a reasonable compiler would need a reasonable language, no? Scala had
its chance in the last 13 years now it is time for this abomination to go.

------
hota_mazi
Regardless of Scala, it's an absolutely insane idea to develop your own
language to write your apps.

Either stick to the real Scala or switch to Java or Kotlin, but writing your
own language is a failure waiting to happen.

I give this project a few more months before it gets canceled.

~~~
carlosdp
That's... not true. You mention Kotlin, that was developed by JetBrains as a
better language to develop Intellij and plugins.

Hack/HHVM was developed by Facebook to modernize PHP.

There are plenty of cases where organizations successfully developed their own
languages/compilers that gave them advantages. It's insane only if you have no
good reason to do it. This sounds like they have a need: compile times are
slow.

~~~
alexandercrohde
you think compile times are so slow that they're saving time by writing a
whole language? I'd like to see the numbers on that...

~~~
carlosdp
We're talking about a large organization that probably does 1000s or 10000s of
builds a day, so yea it might

