
Benchmarks of various C++ build tools - ingve
http://retropaganda.info/~bohan/devel/wonderbuild/benchmarks/time.xml
======
to3m
Which cmake generator was used? The links seem to be dead, so it isn't clear.

I mention this because I'm using cmake for a current project, and backend
choice seems to be somewhat important. I found a noticeable improvement when I
switched from using a make-based backend (NMake on Windows, GNU Make on OS X
and Linux) to the Ninja one (see [https://github.com/ninja-
build/ninja](https://github.com/ninja-build/ninja)).

Based on my notes from the time, full build times were basically unaffected on
a per-core basisi - but no-op builds were markedly quicker. My notes don't say
anything about builds where only one or two files were changed, but my
recollection is that that case was noticeably improved as well.

For whatever it's worth, figures from my notepad from a few months ago.
Apologies for the scrappiness; these were just brief notes I took at the time,
all rather random, just to determine whether the improvement was ~0, or >0...
no more than that :/

All times in seconds. All tests performed on my laptop - 3.1GHz dual-core i7.

Windows in a 3-core VM:

    
    
        nmake full = 143.72 (1 cpu)
        nmake no-op = 2.22
        ninja full = 140.41 (1 cpu)
                     79.06 (3 cpus)
        ninja no-op = 0.135
    

OS X native/Linux 4-core VM (unsure which - probably Linux VM):

    
    
        GNU make full = 46.81 (4 cpus)
        GNU make no-op = 0.93
        ninja full = 49.36 (4 cpus, I suspect - by default Ninja uses all available cpus)
        ninja no-op = 0.03
    

(Other notes: cmake works on Windows, and has a reasonably-sized ecosystem,
but... OMG. The scripting language is fucking insane. Still worth a look
though.)

~~~
piokuc
After maybe three years of experience with cmake I must admit its scripting
language is just disgusting. Makefile provided a nice, well thought out,
simple DSL, with just three concepts: targets, dependencies and actions, and
while it didn't provide enough support for handling multiple compiler
toolchains, etc. I think replacing it with the cmake shell like 'scripting
language', sporting one data structure (string; lists are really just
semicolon separated strings...), is an example of very, very bad engineering.

~~~
to3m
Yes. If they had used something like Python, it would probably be heaps
better. (Though to be honest I'm not really sure I trust the cmake developers'
collective judgment. I'm sure they could manage to make it awful.)

But, even though it's horrid, I do find it CMake easier to work with than make
on a day to day basis. CMake's scripting language is at least
straightforwardly imperative; CMake has at least some support for working
around certain platform/toolchain differences (though this support is rather
half-hearted in some key places); it has a couple of handy features that make
working with C/C++ libraries quite easy; it has builtin support for test
suites. All more convenient than doing it all by hand in GNU make, and you've
got support for multiple backends too. Oh, and it supports Windows, too.

Though I admit it only through gritted teeth, it provides value.

------
lorenzhs
This is pretty old and the title should reflect that. The versions of the
software used are ancient by many people's standards: Linux 2.6.24 (released
six years ago, in January 2008), ext2 file system (ext3 was introduced in the
2.4 series, seven years before that kernel release), gcc 4.2.4 (also 2008, 4.2
series is from 2007), cmake 2.4.7 (2007). Overall it seems like these undated
measurements were performed in early 2008, nearly six years ago. A lot has
changed since then and I don't see how this article in its current form should
still be relevant today.

~~~
bjohan
I take note that I should put a date on the page! I actually ran again some of
these benchmarks again over the years (last time about one years ago), on the
same hardware and updated software, and results didn't change significantly.
FWIW, at the time these results were first published, I had my working dir on
ext2 because it had higher performance than ext3.

------
bjohan
Hi, I'm the author of these benchmarks, and yes, they are getting old. One of
Scons' maintainers kindly asked me a few days ago to try again with current
Scons version, so I've started to update and run these benchmarks again with
current software and hardware (SSD). It takes some time to do it right (blame
API breakage!) and collect all the results, so I won't be able to publish them
before at least a month or so. I will include the meson+ninja and cmake+ninja
combinations, and the new CBS tool (an experimental build system from Qt).

------
mianos
Nice to see that WAF, being python is competitive. It goes to show that smart
can beat brawn.

~~~
berkut
It's only really SCons which gives Python as a build system a bad name in
terms of speed in my experience.

~~~
waterhouse
And judging by the "number of system calls" section, SCons's performance
problems don't appear to be due to the Python runtime.

~~~
bjohan
I'd say Python's runtime only has a small fixed startup cost that prevent it
from beating natively-compiled tools in no-op rebuilds (when no source file
has changed). Some tools like tup or ninja reach insanely low times that are
below the startup time of Python itself. But at such low times, it doesn't
matter anymore whether it's 10ms or 100ms.

