
Building Scala Projects: Maven vs. SBT - rdubz
http://www.hammerlab.org/2017/04/06/scala-build-tools/
======
chickenbane
One of my responsibilities on a medium-sized Scala team was managing the
builds. Naturally, this meant learning the wonderful world of SBT. SBT is the
"Simple Build Tool", but it's misnamed; in a way, I think SBT is the
quintessential Scala program:

* SBT features its own DSL (Scala is, after all, the Scalable Language), which also includes infuriating operators such as "<++="

* There's a affinity towards elegance at the expense of basic understandability. A good example to illustrate this is the design of SBT is recursive; that is, your SBT build is an SBT program that must be built before your program is built. Then you may write a meta-SBT-build program to build your SBT build, and so on. I'm very sure SBT developers are quite proud of this architecture, but my team members universally thought this was pointless and confusing.

* It's slow, and trying to speed it up is an engineering spike in and of itself.

* As Scala doesn't bother to maintain binary compatibility across point releases, the SBT and plugin ecosystem has to use an older version of Scala. So now your SBT plugin code is probably using a different compiler than your Scala application code.

* In exchange for the steep learning curve, you are rewarded with a powerful feature set that was definitely innovative: incremental, parallel, and continuous builds, a REPL, a great SBT plugin community, etc.

After learning and using Make, Maven, and now Gradle, I have mainly learned
one lesson: never write a build system.

~~~
rdubz
post author here, this is good perspective, thanks.

* The "<++=" operators are much-maligned but I've seen them ~0 times in 6 months; I may have come in to SBT-land just as they were being finally phased out, I think I did see them in the beginning a bit.

* Agreed that the recursiveness of the build is presented somewhat confusingly and navel-gazingly!

* Speed-wise, I imagine you're talking about reloading projects, either in the SBT CLI or IntelliJ? That's pretty fair, I've had to adjust my IJ workflow a bit to not trigger full-project-refreshes as often (e.g. disabling auto-import). If you're talking about e.g. compilation or other tasks being slow, I'd be surprised and interested to hear more, though.

* re: SBT plugins being stuck on 2.10, I was also a bit daunted to read that early on, but at this point I've written several SBT plugins and it's basically never been an issue or introduced any friction; curious to hear if it's caused problems for you.

* Agreed that there is a steep learning curve and some payoff that reasonable people may value differently, and also to not write a build system :)

------
220
> operator-overloading enabling eDSL-creation

This is almost certainly in the negative column for me. SBT has the dubious
distinction of being the only build system that takes me hours to figure out
how to change a setting. Inevitably I have to fall back to reading the source,
and given all the macro magic that can be difficult to unwind.

I've been using Pants, mostly because it's polyglot and has better multi-
project support. The oss community is a lot smaller, and it has a fair number
of problems around bugs, but I find the code much more approachable.

~~~
rdubz
(disclaimer: post author)

where are you using Pants?

I would really love to be able to use Pants as well, but alas, once I got
running on SBT and was faced with a bit of rough-edges in Pants public-facing
adoption/documentation story, I had to abort :)

~~~
220
My current startup and my last one, both smaller orgs. I setup both build
systems, and initially used SBT in the previous one. In both cases I think it
helps that we had engineers from larger companies familiar with a working
monorepo; if you've seen one done well you have some aspiration as to what to
shoot for with pants, even if it's more than what's currently available.

I'd use SBT again for a locally contained, single project setup. Once you
learn the arcana, it works well, has a ton of plugins, and the repl is nice. I
don't think it scales well with new engineers or number of projects though.

~~~
rdubz
All very reasonable, thanks.

I've used Blaze at Google and then Pants at Foursquare, so I've seen this sort
of thing done, but yea, Pants seemed like it was going to take a higher level
of commitment to set up / maintain, due to smaller OSS ecosystem around it, so
I had to short-circuit that thread.

Also, putting everything in a monorepo is not really an option in my current
OSS-focused setup, and I've come to have grave doubts about its desirability
overall, after years of believing that it was the ideal way, but that's
another discussion :)

------
evdev
Just a note to people who are encountering this for the first time: adopting
SBT, deciding it was a mistake, and going back to Maven is also a common
narrative.

"Build/Package/Release logic is complicated, and deserves sophisticated tools
and abstractions just as much as the code it builds."

This seems correct until you realize that the #1 priority for your build
system is to not be spending time screwing around with your build system. The
benefits of using Maven to stay with the herd ends up dominating any benefits
power you get to come up with clever build mechanisms. Then, yes, maybe you
have an ugly shell script you have to run in your CI--but for all the
crappiness of that, it's still composed on top of the thing that you basically
never touch.

I've heard first hand from one big marquee Scala company (you could get it in
a couple guesses) that this was basically why they were going to move away
from SBT. That was years ago, so do with that what you will.

~~~
neeleshs
Agree with this. Build tools should remove complexity, not add it. We tried
adopting sbt and we found its way too complex and switched back to maven.

~~~
rdubz
There may be some subjectivity around what constitutes added/removed
complexity.

The ability to factor out repeated logic and configuration in SBT, by dint of
using Scala instead of XML, removes a lot of complexity. Of course, arbitrary
Scala can get more complex than arbitrary XML.

As some points of reference, not saying these clearly argue for one side or
the other:

Here is the Maven POM for ADAM's core module:

[https://github.com/bigdatagenomics/adam/blob/adam-
parent_2.1...](https://github.com/bigdatagenomics/adam/blob/adam-
parent_2.11-0.22.0/adam-core/pom.xml)

and its parent POM:

[https://github.com/bigdatagenomics/adam/blob/adam-
parent_2.1...](https://github.com/bigdatagenomics/adam/blob/adam-
parent_2.11-0.22.0/pom.xml)

For comparison, build.sbt from my fork of ADAM:

[https://github.com/hammerlab/adam/blob/e4bee3227b5979e65a73d...](https://github.com/hammerlab/adam/blob/e4bee3227b5979e65a73d7c2051518640545dc36/build.sbt)

And a plugin that it inherits that factors out a lot of functionality I reuse
across projects:

[https://github.com/hammerlab/sbt-
parent/blob/1.7.5/src/main/...](https://github.com/hammerlab/sbt-
parent/blob/1.7.5/src/main/scala/org/hammerlab/sbt/ParentPlugin.scala)

Spark's POMs (e.g.
[https://github.com/apache/spark/blob/v2.1.0/pom.xml](https://github.com/apache/spark/blob/v2.1.0/pom.xml))
vs. SBT project folder
([https://github.com/apache/spark/tree/v2.1.0/project](https://github.com/apache/spark/tree/v2.1.0/project))
are another point of comparison.

It's not hard for me to imagine how someone used to staring at one or the
other of these kinds of configs, maybe for years, would feel that it made more
sense than the other, but having spent some significant time with both, I
think the points about having the option to express more sophisticated logic –
that SBT provides – are pretty important, and the lessons extrapolate-able
beyond building Scala projects :)

~~~
neeleshs
Indeed, sbt has its advantages. We found it it too complex for our use cases
where maven provided an out-of-the-box build process, and sbt did not.

~~~
hocuspocus
On the contrary, I believe the out-of-the-box experience is where SBT shines.
Even for a relatively simple project with a few sub-modules, the amount of
boilerplate is simply insane with Maven.

------
60secs
In my own experience gradle is far simpler than SBT, especially when you
factor in the binary incompatibility between scala versions and the technical
debt this adds to your build plugins.

Plus compilation with SBT is ridiculously, laughably slow. Modifiying your
build.sbt essentially causes a rebuild of your whole project. The complexity
and expressiveness of SBT is a BAD THING in terms of compilation and
troubleshooting.

Maven suffers from a different issue. Build processes are fundamentally
procedural, not declarative so sooner or later you have to shoehorn your shell
commands into a maven plugin section in your xml instead of just executing
them as a line from within your build script.

~~~
djsumdog
I would take SBT any day over Maven. The amount of tedious XML, over-complex
poms, etc. etc. Every project I've worked with in Maven must have been setup
incorrectly because they were all terribly painful to build.

That being said, I've tried Gradle (knew a guy who worked for them too) and
it's really nice. I could see using it over SBT. The trouble is sbt has a lot
of nice plugins (sbt-native-package, sbt-docker) which, last time I checked,
didn't have equivalences in the Gradle world (they might now).

~~~
barrkel
The biggest pain point I've had from Maven is trying to compile the sub-tree
that uses SBT. YMMV, but Maven integration in IntelliJ makes it extremely
pleasant to use.

~~~
SureshG
Gradle now supports kotlin ([https://github.com/gradle/gradle-script-
kotlin](https://github.com/gradle/gradle-script-kotlin)) for writing your
build scripts and the IDE (Intellij) integration is very nice there too.

------
doug1001
version 1.0 of sbt should be released soon (latest stable is at 13.13 which
has been out for several months, and 1.0 has been the parallel dev release).

we have noticed major improvements in stability, in the api, and in the
community plugins from say 0.11 to now.

shortcomings aside, there are a few things sbt does much better than maven and
gradle, and which we rely for our builds.

cross-compilation: One of them relates to binary-incompatability between major
scala versions, hence the need to cross-compile. Sbt lets specify this
direction in the build definition ('crossScalaVersions' is a built-in
setting). The dependencies for each version automatically get picked up

quicker development cycles using sbt's interactive environment; compiling
Scala source is time consuming; this latency can be pretty substantially
reduced by working interactively, eg, one way to take advantage of this is to
enter the sbt console mode and then preface tasks with a tilde '~' which will
re-execte the task when the source)changes.

------
bassman9000
> Build/Package/Release logic is complicated, and deserves sophisticated tools
> and abstractions just as much as the code it builds.

Disagree. It should not be complicated. We dumped Maven for Gradle. Huge
mistake IMO. Instead of trying to simplify build and get rid of the million
build exceptions and different conventions, we persisted on them, now with a
more powerful tool.

We used to chop our legs off with a knife. Now we have a chainsaw.

------
framebit
I enjoyed Frank Nothaft's talk about ADAM at Spark Summit East, cool to see
y'all in the wild :)

FWIW I'm at a Scala shop where we use SBT exclusively. There's things about it
that suck, it's definitely symbol-happy, but for our purposes it sucks less
than Maven. Haven't looked into Gradle, all the other comments here have made
me curious to try it out.

------
jcdavis
(Hey Ryan!)

> operator-overloading enabling eDSL-creation

As far as I'm concerned, this is a bug not a feature. At least sbt seems to
stick to ascii characters ;)

~~~
rdubz
hiya jackson :)

yea… that's a longer discussion. I've been influenced on this by some
coworkers (cf. [http://www.hammerlab.org/2015/01/21/introducing-
ketrew-0-0-0...](http://www.hammerlab.org/2015/01/21/introducing-
ketrew-0-0-0/)) and conference talks (including one at NEScala 2wks ago,
[https://github.com/nescalas/proposals-2017/blob/master/long-...](https://github.com/nescalas/proposals-2017/blob/master/long-
shechtman-katrin-can-dsl-be-human-.md) ); one way to think of it is that there
are blurry lines between a DSL and any plain old API… they're kind of the same
idea, though conventionally DSLs have tended to involve more punctuation
marks, and I agree that that can be taken too far, and that SBT did in fact
take it too far in the past!

On the other hand, XML with a given schema is a kind of DSL as well, and where
it's possible to make e.g. Scala DSLs overly concise to the point of
inscrutable trickiness, XML APIs are frequently considered to be inscrutably
verbose, as we've all heard and experienced ad nauseum.

However, given Scala's powerful type-checker and flexible syntax, more sane
middle grounds can be explored. Maybe a less controversial example in SBT is
the "/" operator for constructing filesystem paths
([https://github.com/sbt/sbt/blob/v0.13.13/util/io/src/main/sc...](https://github.com/sbt/sbt/blob/v0.13.13/util/io/src/main/scala/sbt/Path.scala#L13)),
which I incidentally just mimicked in a java-nio-Path-wrapper library this
past weekend ([https://github.com/hammerlab/path-
utils/blob/1.0.2/src/test/...](https://github.com/hammerlab/path-
utils/blob/1.0.2/src/test/scala/org/hammerlab/paths/PathTest.scala#L16)). The
alternative here is Java's Path.resolve(), which is a bit clunky to say the
least.

Also some of the JSON-DSLs we dealt with at 4sq come to mind as nice bits of
ad-hoc syntax that still benefit from all of Scala's type-checking
wondrousness :)

(edit: fixed broken link)

------
fwefwwfe
Is there a good book for SBT, like SBT in Action? Amazon reviews for that
aren't glowing.

------
jakozaur
I would disagree about plugin parity. There are a lot of maven plugins, much
less snt ones.

~~~
rdubz
(disclaimer: post author)

Any specific examples you have in mind?

As I mentioned in the post, the first… 5 Maven plugins I looked for all
existed in about as good or better form in SBT-land.

Come to think of it, the dependency plugin in SBT,
[https://github.com/jrudolph/sbt-dependency-
graph](https://github.com/jrudolph/sbt-dependency-graph), seems to be missing
some wildcard features that I used to like in maven-dependency-plugin. OTOH,
it has some things that afaik don't exist in maven-dependency-plugin, like
dot-graph output.

------
lacampbell
continuous compilation isn't just a nice to have in scala, it's mandatory
given how slow the compiler is. Does maven solve this problem?

~~~
rdubz
scala-maven-plugin allegedly supports this: [http://davidb.github.io/scala-
maven-plugin/example_cc.html](http://davidb.github.io/scala-maven-
plugin/example_cc.html)

That said, I remember trying it and not sticking with it years ago, but I also
don't use it in SBT. I remember if it was particularly broken in any way that
caused me to abandon it in Maven.

