
Looking for help improving performance of GNU Make - motiejus
http://lists.gnu.org/archive/html/help-make/2014-10/msg00000.html
======
adekok
I've had luck using
[https://github.com/dmoulding/boilermake](https://github.com/dmoulding/boilermake)

It's a GNU Make framework which allows you to create non-recursive Makefiles.
I've re-worked it a bit, and used it in my pet project:
[https://github.com/freeradius/freeradius-
server/](https://github.com/freeradius/freeradius-server/)

100K+ LoC, ~400 source files, multiple subdirectories, building libraries,
executables, etc.

Most of the slowness of Make is due to people creating recursive Makefiles.
Running endless shell programs is very slow. Make by itself is surprisingly
fast.

The other benefit of non-recursive Make is that the builds can now go in
parallel. What used to be a ~90s build is now ~7s on a multi-core machine.

I also replace libtool (shudder) with jlibtool. That was taken from Apache,
and modified to fix bugs work better, etc. That sped up the build a lot, too.

Maybe I'm crazy, but these tools have worked across all Unix-y platforms I've
tried. And I needed a migration path from recursive Make / libtool to
something better. Dropping the build system on the floor and hoping for
something better wasn't an option. This tool suite works much like the old
one, but it's simpler, faster, and doesn't require additional packages
(python, etc.) to do the build. Everything outside of Make is self-contained,
and is included in the FreeRADIUS distribution.

My projects:

[https://github.com/alandekok/boilermake](https://github.com/alandekok/boilermake)

[https://github.com/alandekok/jlibtool](https://github.com/alandekok/jlibtool)

[https://github.com/alandekok/boilermake](https://github.com/alandekok/boilermake)

~~~
chubot
If anyone wants to do non-recursive make for big projects, I would take a look
at what the Android platform does. They have this elaborate Android.mk
makefile fragment system.

At the beginning of the build you go and find all the fragments, then
process/concatenate them into one huge makefile, and do a single make
invocation. It results in a very parallelizable build.

It seems kinda crazy and hacky but it definitely works better than recursive
make.

For some reason the best docs seem to be on this Korean site:

[http://www.kandroid.org/ndk/docs/ANDROID-
MK.html](http://www.kandroid.org/ndk/docs/ANDROID-MK.html)

~~~
maxhou
Except parsing all the Android.mk and computing the dependencies is slow as
hell.

Try running make at the top of an AOSP tree, it takes about 60 seconds, and
that part cannot be parallelized.

If you are looking for alternatives, have a look at
[http://martine.github.io/ninja/](http://martine.github.io/ninja/)

~~~
chubot
Yeah definitely if you were starting from scratch, I would consider other
build systems.

But if you already have a huge mess of recursive make, it might be easier to
port it to a non-recursive Android-like system then completely rewrite it with
gyp/gn/cmake + ninja.

It would speed up your full build times; not sure what it would do to
incremental build times. It is true that the Android.mk thing has a large
startup cost. Then again, you're building a whole OS with one make invocation,
which is kinda cool.

------
jordigh
I really hope someone takes the job offer, fixes the problem however possible,
and then publicises the process.

This is what free software is supposed to be about: you can hire anyone to fix
it. Let's put theory to practice now and actually hire someone to do it.

Edit: You know what, I'm now determined to now headhunt this person. It's the
principle of the thing. Once I get the original poster's permission, I am
reposting this ad on freedomsponsors.org, bountysource.com, and on the GNU job
list: [http://www.fsf.org/resources/jobs](http://www.fsf.org/resources/jobs)

------
teddyh
Summary:

> _What we see is that if we have more than 30 or so clients connected to the
> build Make is often unable to generate tasks fast enough to keep them busy._

[…]

> _Our management doesn 't want us taking time to learn the internals of make
> and see if we can improve it ourselves, but they are willing to throw money
> at the problem, either by offering some sort of donation/bounty for
> performance improvements or directly hiring an expert on a contract basis;
> any improvements achieved would of course be submitted back to the community
> as patches._

------
anjbe
Last month at EuroBSDCon, Marc Espie gave a neat talk describing some of the
performance improvements he’s made to OpenBSD’s Make (and some positive words
about Ninja as well) – [http://www.openbsd.org/papers/eurobsdcon2014-parallel-
make.p...](http://www.openbsd.org/papers/eurobsdcon2014-parallel-make.pdf)

------
andrewchambers
Make has so much legacy cruft handling, people just moved to other systems
that are simpler.

Most developers don't even know most of the features make, and people don't
really want to learn such an old program through and through.

~~~
norswap
Mmh, which build system do you think are simpler than Make?

Make has a lot of quirks, but at its core it is a simple system.

~~~
hendzen
Cmake/ninja is becoming the standard for new C++ projects.

~~~
davexunit
Cmake is a fucking nightmare to work with. Autotools are light years better
and more robust than cmake.

~~~
bodyfour
I have a love/hate relationship with cmake.

Writing non-trivial functionality in cmake's language is possible but you're
fighting the language the entire way. It would have been so much better if
they'd just started with something like Lua and extended it instead of
building their own barely-adequate scripting language. Some of the grossest
code I've ever written is in cmake macros.

On the other hand, the results you get from cmake are fantastic. Want to use
ninja instead of make? Just pass a compile flag. Need Visual Studio project
files? No problem. Xcode? Kdevelop? You're covered.

Honestly I couldn't imagine ever going back to autotools at this point.

------
deadgrey19
If they're building C/C++ with it then this is a good option:
[https://github.com/Zomojo/Cake](https://github.com/Zomojo/Cake) \- which is
really just a front end to make, but produces really fast makefiles.

It was initially developed to allow interactive speed iteration of compiling
large C/C++ projects for data analysis.

It does require that your source code comply with a few basic rules (just that
c/cpp files and headers have the same name). But beyond that it runs
automatically and quickly.

------
sklogic
Unfixable. Use ninja instead.

~~~
onli
You might actually be right. Even though I didn't use ninja, given their
description a specialized build system for fast build that circumvents the
problems they mention that make their build slow might be the solution.

But are they not giving way not enough information to seriously assess their
situation? That - combined with the shortness of your comment and that people
could think it's a joke - would explain your downvotes.

~~~
sklogic
At least they've explained that dependency resolution seems to be the
bottleneck, and Ninja solves this particular source of congestion most nicely.

