
What's in a Build Tool? - lihaoyi
http://www.lihaoyi.com/post/WhatsinaBuildTool.html
======
xyzzy_plugh
Regarding make, the author states:

    
    
      Parallelize different commands: Okay: There's no automatic way to parallelize different targets, though you can still parallelize a single target using Bash.
    

This is wrong, make has supported the -j flag since 1988:

    
    
           -j [jobs], --jobs[=jobs]
                 Specifies the number of jobs (commands) to run simultaneously.  If there is more than one -j option, the last one is effective.  If the -j option is given without an argument, make will not limit the number of jobs that can  run
                 simultaneously.

~~~
microcolonel
I can't believe anyone could use make in the last ~12 years without the -j
flag. We've had dual cores at least for ages.

~~~
Ace17
Actually, I think make should default to parallel builds.

~~~
MereInterest
The problem there is that it would break makefiles that do not handle their
dependencies correctly. With non-parallel builds, the order of execution is
deterministic, and so you can get away with sloppiness in your dependencies.
If it works the first time that you test it, it will continue to work.

I can understand why they would be hesitant to change the default, as they
would rather that old, tested scripts continue to work without modification.

~~~
_pmf_
> The problem there is that it would break makefiles that do not handle their
> dependencies correctly.

Serious question: In what way are Makefiles that are not specifying their
dependencies correctly ever to be considered non-broken?

~~~
MereInterest
It's the exact same argument that comes up whenever gcc improves their
optimization algorithms by exploiting undefined behavior, making some code no
longer work. In both cases, the original code was fundamentally broken from
the start, and the change in tooling only revealed the brokenness, not causing
it.

I would completely see such makefiles as being broken.

~~~
slavik81
The impact of those changes is rather different. In the case of make, your
build would probably break. In the case of gcc, your program's behaviour would
silently change.

~~~
MereInterest
In the case of make, with missing dependencies, it can result in a file not
being re-compiled when it should be. If you are compiling C, this can result
in the definition of a function definition being different in two different
compilation units. When one of those compilation units calls a function
defined in the other, your program's behavior breaks. All due to a change in
the build tool.

~~~
slavik81
That's true. I qualified my statement with probably because there are
exceptions. Protecting against those sorts of errors is why my release
candidates are done with a clean build and newly fixed bugs are reverified on
that package before release.

------
herge
It's a pity that rake is not included in the table at the bottom.

One huge advantage of rake is the ability to easily debug what is happening. A
quick `p [filename, task]` lets you know what is happening. Even, if needed,
you can quickly do

    
    
        $ irb
        >>> require 'rake'
        >>> p FileList["foo/*"]
    

and actually play around with the libraries. If you know anything about ruby,
and especially if you are looking to build anything other than a c/c++
project, it is definitely worth looking at.

------
luckydude
I thought this was going to be a rant against autoconf. When that rant comes
I'll post our 186 line shell script that does what autoconf does for pretty
much any platform. Handles AIX, Solaris, FreeBSD, HP-UX, IRIX, Linux, NetBSD,
OpenBSD, SCO, SunOS, Windows, MacOS.

I hate autoconf but that's not part of this thread. I'll wait.

~~~
Ace17
I'm curious about this shell script ; this looks to good to be true!

Having automated the build of dozens of FOSS packages, I can definitely say
that autoconf-based packages are _by far_ the easiest ones to build ;
especially if you want to do funny stuff like out-of-tree builds or cross-
compilation (can your 186-line script do this?).

Some projects (x264, ffmpeg) provide 'configure'-like custom scripts, trying
to imitate what autoconf does ; but in practise you can't automate their build
the same way you did with autoconf-based projects.

~~~
atilaneves
Have you tried CMake?

~~~
Ace17
Yes.

Cross-compilation of existing packets has proven to be a nightmare with cmake.
I have to override manually CMAKE_C_COMPILER, CMAKE_CXX_COMPILER ...

Have a look at zenbuild, which is a toy-project of mine trying to homogeneize
the build interface of many FOSS projects. Each packet get its script (ala
PKGBUILD), it can be seen as a list of hacks around the quirks of each
project's build system.

Projects using autoconf are a breeze to build:
[https://github.com/Ace17/zenbuild/blob/master/zen-
libxau.sh](https://github.com/Ace17/zenbuild/blob/master/zen-libxau.sh)

But projects using cmake are another story:
[https://github.com/Ace17/zenbuild/blob/master/zen-x265.sh](https://github.com/Ace17/zenbuild/blob/master/zen-x265.sh)

------
codyps
ninja? tup?

Article doesn't note the property of being able to depend upon the entire
command that generates an output (ie: re-generate when compiler flags change).
This is something that makes doing reliable builds much easier (when present).
It's notably very hard to do in make (and even then is very inefficient).

Also, on "download" the author seems to presume that one takes the naive
approach in each tool. In most cases, if one spends a bunch of time on it the
downloads can be done fairly efficiently (especially in make, without even
much work there). Most of these build systems are fully programmable, so the
rating should probably focus more on the level of difficulty to do well (with
some requirements specifying what "well" is)

~~~
thelema314
To depend on compiler flags, I do this:

## compiler_flags file keeps a copy of the compile flags
$(builddir)/compiler_flags: force mkdir -p $(builddir) echo '$(CPPFLAGS)
$(CFLAGS)' | cmp -s - $@ || echo '$(CPPFLAGS) $(CFLAGS)' > $@

##rebuild all objects if compile flags change $(LIBOBJECTS) $(RTLLIBOBJECTS)
$(OPTLIBOBJECTS) $(TESTOBJECTS) $(builddir)/init_qt_workdir:
$(builddir)/compiler_flags

I"m pretty happy with the results.

------
falsedan
I think the author does a great disservice lumping together deploy, test, and
dev sandbox tools under 'build tools'. They are different scenarios which all
happen to require a built copy of the code.

There's no fundamental reason for them to use the same tool, except the
inertial tug of whatever build tool your project happens to use.

------
djKianoosh
Gradle is a glaring omission if maven, ant and sbt are included.

------
jschwartzi
I'm curious what the author thinks of CMake. It reminded me a lot of Maven,
but for C/C++ projects.

~~~
imron
For c and c++, I've recently starting using meson.

It's a breath of fresh air compared to some other build tools, and the config
files are both expressive and concise.

~~~
sa2ajj
For those curious (like me): meson's home page --
[http://mesonbuild.com/](http://mesonbuild.com/)

------
Too
Am i the only one thinking that doing a simple compile is something that
should be handled fully by the compiler and not the build tool. The compiler
should automatically detect which files are being #included and allow for
incremental compile based on this. The Typescript and SASS compilers are good
examples of this, you just invoke it like 'tsc main.ts --watch' and it will
automatically know that main.ts included other files and these will also be
watched.

If your compiler does not support this, the only way to build a proper
dependency tree is 1. if the compiler can generate one in a standard format
(gcc -MMd), 2. your buildtool knows as much about the language as the compiler
does, 3. you write the dependency tree manually.

1\. Is bad because it requires the compiler to support generating a wide range
of dependency-graph formats, or every build tool must support the make syntax.

2\. Is bad because the build-tool becomes one massive monolith and there is no
way they can reliably support every language in the world.

3\. Is bad because humans are lazy and make mistakes.

For the build tool, this leaves it's responsibility to doing things before and
after compilation (generating/preprocessing code, copy assests, run tests,
deploy to staging), which usually means invoking arbiatry and custom shell-
commands, the order of this is always so application-specific that #3 is
inevitable. If doing these things are complicated, your build tool sucks, and
sadly i found most build tools to fall into this category because they try #2
and want to be the center of the universe. Even doing a simple thing such as
copying a file requires you to learn a new awkard xml-syntax, install the
"copy-file-plugin" or write your own copy-file-plugin in the awkward syntax.
What i would want is somthing with the simplicity of Gulp or Make, but with a
bit more widespread syntax and more batteries included for file-manipulation,
like Python would be perfect. Rake looks promising, maybe it makes it worth
learning Ruby?

~~~
lmm
Disagree. Compilers, like anything else, should be small composable libraries.
Compilation and dependency parsing should perhaps be separate, and the same
dependency-tracking library should be usable by the compiler, wider build
tools, and IDEs or similar.

------
lasermike026
I was just writing a Rakefile and have a good time of it. I like using make
but rake gives me more functionality.

~~~
shoover
Rake is a pleasure for small projects, like generating HTML from a handful of
org-mode files or zipping up a Chrome extension. I've not tried to take it
much farther than that, though there is a series of blog posts I can't find
attached the moment in which the author used rake for real C projects with
GCC.

------
gkya
The OP mistakes what a build tool is. A build tool generates the executable
from sources. Use other tools for other tasks. I once wrote a tool for running
commands on file system events, but I lost it. But for example I write files
rather often, a build on each write would be annoying for me. It's a personal
thing, so give me the build script and let me use it the way i like.

------
dlandis
Ant and Maven are fairly obsolete at this point in the Java world having been
superseded by Gradle, and so aren't really worthy of much discussion. But
Gradle is quite good and I think it would stack up very well against the
others including sbt. Like sbt, I think it fuses a task based and
configuration based approach and also does dependency management.

~~~
pkd
Definitely not Maven. Maven is ubiquitous in Java projects. Gradle has been
pushed hard by Android, but I know that a lot of developers aren't convinced
that putting code into configuration is a good idea. Maven even after all the
XML crud remains insanely flexible and powerful.

~~~
twic
Maven is really not insanely flexible. I have a build in which i wanted three
distinct test phases (unit, functional, and API contract). A simple
requirement. But i couldn't have that without writing a plugin, or breaking
out a dummy module for one of them. We ended up making the contract tests a
special case of the unit tests, selected by passing in some flags; a
consequence of that is that it's impossible to run both sets of tests in a
single build.

tl;dr: Maven is the exact opposite of flexible.

~~~
tdurden
creating profiles specific to each might help

~~~
twic
I think i still couldn't run all three sets of tests in one build though,
right?

~~~
tdurden
You can activate multiple profiles per build if needed.

~~~
twic
But i can't run a goal twice under different profiles, can i?

~~~
tdurden
No, but you can tie multiple plugin executions to a single phase.

------
davidpronk
After reading this article on NPM as a build tool I immediately gave up on
Gulp and Grunt. Highly recommended! [http://blog.keithcirkel.co.uk/how-to-use-
npm-as-a-build-tool...](http://blog.keithcirkel.co.uk/how-to-use-npm-as-a-
build-tool/)

------
manaskarekar
A nice summary of some build tools:

[https://bitbucket.org/scons/scons/wiki/SconsVsOtherBuildTool...](https://bitbucket.org/scons/scons/wiki/SconsVsOtherBuildTools)

------
shoover
This is helpful and uses a good matrix to compare the tools in question. I've
been wanting to see a version of this for cross-language tools: Bazel et al,
msbuild, scons, and do not forget the quiet but powerful waf.

