
Make your own build system - jstimpfle
http://jstimpfle.de/blah/buildsystem/buildsystem.html
======
RcouF1uZ4gsC
> Although it adds to the code that must be maintained as part of the project,
> I think that having a custom build infrastructure that can grow with the
> project is a good idea for medium-sized and larger projects.

Please don't do this for open source projects. Other people may want to use
your code, but they generally don't want yet another build system. There are
already a lot of build systems that are building very complicated projects.
Surely, one of these existing, widely used build systems can be made to work
for your project.

~~~
jstimpfle
TBH I don't have many fond memories of open-source build systems. Any widely
used build system (that probably <10% of the developers know very well,
anyway) is not going to help if the build realized with it is broken (too
flexible, too confusing, too optimistic about others' environments). And
frankly most uses of those build systems in the wild are "give up control and
pray".

If it's about making it easy for uninitiated collaborators to add new source
files, then the listed build description format is pretty self-explanatory to
use. And the build script just needs to be run. It takes no arguments, so I
don't think there is a problem with that.

~~~
Ace17
> If it's about making it easy for uninitiated collaborators to add new source
> files, then the listed build description format is pretty self-explanatory
> to use.

This misses the point IMHO. In practice, whatever complex the build system is,
it's generally easy to add a new source file: simply grep for an existing
source file name, find the build files that refers to it, and insert a similar
line for your new file.

> And the build script just needs to be run. It takes no arguments, so I don't
> think there is a problem with that.

What about: cross compilation? ccache injection? verbose builds? parallelism
control? installation prefix? Try being a package maintainer for a GNU/Linux
distribution, where all of the above needs to be kept under control. You will
_despise_ custom build systems.

Established build systems (like GNU Autoconf (e.g gcc), and, to some extent,
custom Autoconf (ffmpeg, libvpx)) handle all these cases pretty well, and
provide _fixed_ commande-line options (--prefix, --host) to control them. This
means the project author doesn't get to decide how the option will be named
and how it will work.

This make a huge difference if you're trying to packages for your distrib
dozens of projects from dozens of various authors.

With a custom build system, the best case scenario is you have to re-learn all
these options for each project. The worst case scenario is: one project
doesn't support some use case (e.g cross-compiling), so you have to maintain a
patch that hacks into the build files to add the behavior your need.

~~~
crdoconnor
>With a custom build system, the best case scenario is you have to re-learn
all these options for each project.

The _best_ case scenario would be that it handles all of those scenarios in A)
a highly readable way and B) has high quality documentation that demonstrates
the use of all the necessary features in context.

Make's approach appears to be to use cryptic flags and settings and the best
case documentation appears to be "have you tried stack overflow?". It is not
the be all and end all of build systems.

Additionally, for many of the non-C++ workflows I want to accommodate, the
'make' approach appears to be hack around its deficiencies with bash
scripting. Stringing together hacky bash scripts is not what I would describe
as a best case scenario in any project.

I don't think of this guy's project is a meaningful competitor to make/cmake
but equally, I can envisage other projects doing a better job than make on its
home turf.

~~~
Ace17
> I can envisage other projects doing a better job than make on its home turf.

Of course it's possible, GNU make is far from being perfect ; it's a pity we
can't slowly evolve the language for backward compatibility reasons. Autotools
also are far from being perfect.

\- As a project author, I like writing custom simple makefiles and hate
anything related to editing "autoconf.ac".

\- As a package manager, I love projects using autoconf (or exposing
autoconf's options, such as --host): they're the easiest to build because they
all have the same "build interface" (alas, by default, there's no equivalent
to "./configure --host" for cmake). And I despise projects using "custom
simple makefiles", because it means that I'm going to have to learn yet
another parochialistic BS (for build system) and have yet another special case
for this specific project.

Maybe one day we will have a build system that's loved by both sides? :-)

~~~
crdoconnor
I think the ideal build tool would be a framework written in a decent,
readable high level scripting language.

------
HaoZeke
I think for most projects, contributions are hard enough to come by without
having to factor in a new build system.

Honestly even bazel isn't built with bazel. (it can be, but they do that as a
test)

It's like rolling a whole new project into your existing ones... For no clear
reason.

Perhaps it's a learning experience, however there's no reason to hobble a
project with it.

Thankfully, I doubt most established projects would suddenly switch to a
custom build system.

Also there are plenty of auxiliary tools developed for build systems which are
at best, difficult to reproduce.

If one needs to improve a build system... Try a pull request?

~~~
ndh2
Yeah, about that last sentence. Are you actually a contributor to an existing
build system? CMake? Make? Bazel? Anything?

~~~
maccard
I agree with the sentence. I've contributed to both cmake and gradle to fix
very specific pain points that I've experienced. It's definitely been less
work than writing an entire build system because I don't like how big and
complex the existing systems are.

------
evincarofautumn
I once used Shake[1], a Haskell library/EDSL for expressing and running build
rules, to make a build system for work. I highly recommend it over rolling
your own, if you need custom build tooling at all.

Our use case was a developer tool that would take in project files, sources,
and assets, and bundle them for various platforms, push code to devices, or
launch simulators for you. Notably, Shake made it fairly easy to ensure that
we never needed to implement a “clean” command—all dependencies were correctly
tracked.

Having strong static types and the Haskell library ecosystem available was a
huge win for convenience and correctness over something like Make. Shake makes
it easy to do concurrent builds with full or limited parallelism, minimise
incremental build time, write multi-stage rules where e.g. you need to run the
compiler to calculate the dependencies of a file, and properly handle commands
that generate multiple output files. Nowadays it even supports progress
reporting, profiling, and some linting of build rules.

[1]: [https://shakebuild.com/](https://shakebuild.com/)

------
loriverkutya
Build your own build system sounds like reinventing the wheel. As a former
build build engineer and a nowadays devops, a custom build system would be
enough for me not to take a job. Because investing time to learn a custom
build system is just wasted time, because you never going to use it again, on
the next project or next job or whatever.

Also for a project, a custom build system means, that you cannot hire anybody,
who knows the build system, however if you are using any established build
system, you can.

And also, software developers tend to forget, that they are not the only users
of a build system, and usually it needs to be integrated into a build&release
pipeline.

------
MatzeBraun
Nice to see a minimal version of a system.

In my experience the biggest (or most annoying?) problem though is not setting
up the dependencies and executing the steps along the DAG but doing the
configuration part of a build: What libraries are installed on the system,
giving the user ways to opt in/out of build-time features, figuring out how to
build shared libraries on the target OS, how to crosscompile, there’s tons of
of these that may need solving depending on your project.

ninja does a fantastic job about handling the core and leaving all the other
annoying things for someone else to solve (like cmake with the ninja backend
:)

------
Awtem
This is awesome! Instead of getting all mixed up in the discussion about which
of the existing build-systems is best, and instead of trying to figure out how
to work around all of the (mostly arbitrary from user perspective)
restrictions, it is a very rational and pragmatic choice to just get it done
and make it do what you want.

~~~
jstimpfle
Let me be honest, the last step of doing all myself was quite a time sink over
the last 3 weeks. But in the end I've learned something that will be of use in
the future. I don't think this applies equally to the time I've put in off-
the-shelf systems.

~~~
Awtem
I completely agree! All the hours (more like days/weeks) I already spent
trying to make CMake do what I want... It only strengthens my dependency on
CMake in the end :-/

------
evmar
This is neat! If you're interested, Ninja ([https://ninja-
build.org/](https://ninja-build.org/)) is designed exactly to provide the
dependency-graph execution underneath a program like yours that has a project-
local higher-level description of the build goals.

~~~
AstralStorm
Ninja is a very specific tool that does only one thing and that only somewhat
well enough, though it is fast. Even cmake has trouble employing it for
certain generated file cases.

------
zaarn
I've implemented build/task systems in two different ways in the past.

The first is a bash script that ends in

    
    
        $1 ${2:}
    

or something in that region. Then I can define functions and call them
directly from the command line. It's rather protable (if you have bash or I
bothered to write POSIX sh compatible)

The other was a simple DAG similar to how the author describes it. Each task
was a root node and subtasks were nodes connected to it. To figure out which
tasks to run I'd simply scan the entire task graph for what has no subtasks
and spawn a thread/process for each runnable task. Once a task finished, it's
removed from the DAG. Successfully used to run database upgrade tasks (update
multiple tables at once if they queries don't depend on eachother)

The only thing I ask of a custom build system is that it A) includes the
classic "all" and "clean" options B) it doesn't have any dependencies beyond
what I can expect from any random distro, C) autodetect libraries in common
locations, not the locations on the original dev machine and D) don't require
root permissions. ever. (outside "install" and maybe we can insert a rant in
here how a lot of packages don't have a "uninstall" or "remove" option, making
me track down the files it shat onto my disk)

------
sushisource
So he re-built Gradle, or Buck, or Bazel, or any number of roughly similar it-
runs-a-dag build tools? Not sure I buy that his problem was real. Gradle's the
only one I'm meaningfully familiar with and making custom commands over a dag
is a breeze.

~~~
jstimpfle
The "run the DAG" part is very small and simple (as you can verify). I don't
think there is a good argument against "rebuilding" that.

As I explained my motivation was primarily keeping my build description in my
own datastructures (see the text database). IMO that's a very worthwhile goal.
I then went on to see what it takes to NIH everything, and it turns out that
detecting what files need to be rebuilt is hairy, especially considering
transitive C includes (which requires a dependency cache). But there is no one
solution to these problems, and I do think there is some value in controlling
that logic.

And if you get the concepts right, it can be done. I've divided the code into
loosely coupled building blocks. That hopefully makes it easier to understand.
It should also be possible to recombine these blocks to make different build
systems.

~~~
sushisource
I suspect you could (continuing the gradle example) write gradle plugins that
would accomplish what you want to do as far as using your own data structures
goes. Seems to me though the majority of what's in that textual database is
standard C compile/link stuff that'd be handled fine by
[https://guides.gradle.org/building-c-
executables/](https://guides.gradle.org/building-c-executables/)

~~~
jstimpfle
Yep - currently only my data structure generator is in the build. But I have
plenty more to add. I have a pretty poor asset pipeline that I currently run
manually. This stuff is very costly, so you want to have good control and
manual intervention as far as possible. I also want to add more packaging
workflows for a basic "component entity architecture" which means collecting
information from various modules and reorganizing them into a more global
database. These things are tricky and I'm pretty positive I simply want to do
them myself.

------
archi42
Seems nice from a "been there, done that" point of view. Usually rolling your
own solution helps in better understanding the underlying problem.

But, as many people pointed out, having a custom build system calls for
trouble (eg cross compilation, prefixes, feature control) and often produces a
lot of work to maintain or cater to new needs.

Source: We roll a package based, distributed build and test system. Most build
packages wrap around cmake. A core dev created it, and while he works on the
build system, he can't work on the product. And if he ever decides to switch
company or has a severe accident, well... (The code is mostly well written,
but still the code base is huge since it was adapted to changing needs in the
past decade; and while we're not tiny, we're small enough to lack the
resources of having him train a second person in the guts of the system - I
know probably 70% of it and am regularly lost).

If it was up to me, we'd take a look at bazel.

~~~
nrclark
Our company is in the same boat - we have this weird, custom package
management system that's a franken-hybrid of shell scripts, vanilla Python,
Scons, and CMake. Every time I have to do anything even slightly outside of
the norm with it, it breaks badly.

And there's this huge sunk-cost that makes it functionally impossible to
suggest replacing with anything else. It mostly works but it's getting
crustier by the day.

So I feel your pain.

------
nrclark
I'll probably get some flack for this, but I'd like to put in a plug for
autotools (if you don't need to target Windows).

Autotools has really come a long way since the bad old days back in the early
2000s. It's got some crusty corner cases, but I like it more and more as I use
it.

With CMake, everything seems like it's easy to bury yourself in custom macros.
And before you know it, you have a build system that's complex, hard to debug,
and impossible to re-target. Scons is similar (although at least you can step
through the code in Python).

I'd encourage anybody to try it out. At least, if your codebase is C or C++
and you don't need to target Windows.

------
ndh2
The biggest problem I see with the article is that you didn't say exactly what
you're doing with this build system. And this gets people's imagination going.

What is the scope? Is this an open source project? How many people are working
with it?

To me it looks like this is a personal project that only you are working on. I
don't see any problem using this approach. I mean, I use a batch file as my
build "system" for my personal projects. But for anything else it would be
worth discussing the trade-offs versus using one of the available build
systems.

I didn't understand how this works: _scanCmd = 'scanincludes-%s' %(oPath)_.
What is this scanincludes thing?

~~~
jstimpfle
It's just supporting my own project, and I'm the only person working on the
build and the project. The listed build description is all I build with it
currently.

> scanCmd = 'scanincludes-%s' %(oPath)

I have each node in the DAG identified by a unique string, including command
nodes. This one is a name I make up for the node that is responsible for
rescanning the dependencies of a particular compilation (the compilation is
identified by the .o path)

------
jstimpfle
Oopsie. Must have todo with me enabling ipv6 at request from another guy. Will
fix this later when i come back. You can get the descriptio from
[https://github.com/jstimpfle/learn-
opengl/blob/master/build....](https://github.com/jstimpfle/learn-
opengl/blob/master/build.wsl) as well

------
lolikoisuru
>its configuration syntax is awkward

This argument and others similar to it are often repeated but I've yet to see
any of those people describe what makes it awkward.

>and it is restricted to filesystem files and shell commands

It'd be pretty easy to make a makefile that depends on say a specific row on a
SQL database using dummy files. Also changing the language for recipes is
trivial and painfully obvious. If you can't do something as simple as `SHELL
:= /your/own/interpreter' then why do you think you are qualified to make a
build system?

~~~
jstimpfle
It's very easy to explain and there are many examples of bad Makefiles on the
internet. Magic variables, unintelligible string replacement routines,
unportable extensions. Almost nobody understands the finer points like = vs :=
and execution order. Using a simple scripting language with well known
semantics can be relieving. Maybe try this?
[http://nibblestew.blogspot.de/2017/12/a-simple-makefile-
is-u...](http://nibblestew.blogspot.de/2017/12/a-simple-makefile-is-
unicorn.html)

Creating dummy files is inefficient, inelegant, and unmaintainable. Changing
SHELL does not solve the inconvenience of having start a new process and
having to go through a command-line interface each time.

Btw. watch your language, please.

~~~
nrclark
Agreed on dummy files, that's one of my least favorite things about Make. If
you structure everything right though, you can usually figure out how to do it
with the actual input and outputs.

GNU Make's documentation is very well written though. Like, shockingly well.
Don't be afraid to read through it if you see some weird syntax that you can't
figure out.

~~~
jstimpfle
I know Make quite well, but I don't like the clever compressed syntax. My
ideal is a good structure it so that descriptive function names and clear
semantics don't lead to unmaintainable verbosity.

------
boris
Writing your own, general-purpose build system is a hard, multi-year project
(spoken from experience[1]).

[1] [https://build2.org](https://build2.org)

~~~
adrianratnapala
Ahh, but you don't need a general-purpose one do you?

------
Annatar
From these articles, one can always tell who just hit the “need to really
master make” stage in their project.

~~~
gmueckl
You mean the one build system that doesn't know the first thing about header
files? The one build system that fails to detect that a header file changed
and thus a certain set of source files must now be rebuilt?

If there's a case for an existing build system here, the system in question is
not make. There are much better systems out there (scons, cmake...). Just my
two cents.

~~~
lolikoisuru
>You mean the one build system that doesn't know the first thing about header
files?

It's a general purpose DAG runner. It's not really feasible to add parsing for
all the different possible languages that one might want to use imports in.
Luckily at least GCC and Clang both have the option -MMD and any sane compiler
for other languages should have something similiar.

~~~
gmueckl
C/C++ has text-based inclusion, which is just not a sane form of importing
stuff from other modules. This is why we can get crazy stuff like broken
partial builds. Most other languages are saner to build and some even come
with tailor-made build tools.

Using a saner tool than plain make for C/C++ is worth it every time in my
book.

make can still be very useful for other stuff, though.

------
ndh2
Here's Casey Muratori on saying "reinvent the wheel". This is in the context
of game development, but it sure applies to build systems for C/C++.

[https://youtu.be/fQeqsn7JJWA?t=140](https://youtu.be/fQeqsn7JJWA?t=140)

------
daemin
Actually the harder thing is building your own data/content build system,
where not only the assets change, so does the compiler and tool chain used to
build them.

------
reacweb
I work on a project where a huge build system has been developed. It uses the
same source to build a dynamic library, the same library in static and many
test programs. It handles well dependencies whatever the directory structure.
It is developed in gnumake. I think you will be less criticized if you use
gnumake instead of python. It has all the features needed for dynamic
generation of dependency rules. And if someone want to improve the build
system, a good knowledge of gnumake seems a reasonable prerequisite. The main
defect is that it needs 10s before starting any compilation.

~~~
jstimpfle
However, a few disadvantages to Make: Only files and timestamps means that the
granularity of your project's modules needs to be at least as fine as required
by the build (I think that's one reason why C builds are slow: Too many files
written). Pattern matches are not enough IMO. The scripting language is bad
for advanced things. It's like implementing a large project in shell code.

Having the full power of Python really is something else. So I was considering
generating simple repetitive Make rules. But I wanted to explore what it takes
to do the rest myself, as well. Setting up dependencies for Make is not nice,
either. Having the code under control means being able to adapt to any
requirement. It would be easy for example to convert this into a build server
kind of thing.

Having a 10s delay is not acceptable to me. I didn't experience that myself
though.

~~~
reacweb
The 10s delay is caused by the correct handling of dependency in a full source
tree (around 5000 source files). It is the price of perfectionism.

There is no simple repetitive make rules in the build system of the project I
am on: all the rules are generated using macros. gnumake is a full scripting
language that is dedicated to timestamp dependencies. It can manipulate easily
lists of string (as long as they do not contain spaces). What are the advanced
things that requires a better language ?

~~~
jstimpfle
> The 10s delay is caused by the correct handling of dependency in a full
> source tree (around 5000 source files)

There is something awfully wrong if processing 5000 files and cached
dependendencies takes 10s. It should take a few ms at most, even if written in
a scripting language like Python. I would not be surprised if this is because
that build system heavily misuses GNU Make's variable substitution features,
instead of being written in a language that features proper data structures
(e.g. set, map)

~~~
AstralStorm
Heck, even cmake does this better then without fancy data structures. My bet
is this either does many shell calls or wildcards multiple times through file
system.

