
Show HN: Minimal build system using just /bin/sh - akkartik
https://notabug.org/akkartik/basic-build
======
jwilk
> This repo illustrates a way to manage a project using just plain Bourne sh,
> available on all Unix systems since 1977.

No, this code uses functions, which didn't exist in the original Bourne shell.
They were introduced only in 1984.

Source: [https://www.in-ulm.de/~mascheck/bourne/#variants](https://www.in-
ulm.de/~mascheck/bourne/#variants)

~~~
jperkin
It also uses the "local" keyword, which is non-portable.

People like to bash the autotools, but they exist for exactly this reason -
they already spent decades working through all these gritty details to ensure
the scripts generated are portable. To paraphrase Spencer "Those who do not
understand autotools are doomed to re-invent them, poorly."

~~~
rossy
That's the argument for autotools, but I wholeheartedly disagree. Untested
code can be assumed not to work, so if a build script doesn't work on a
certain platform, what's the likelihood that the C program it's building will
work on that platform? When you see a poorly-written autoconf script checking
for basic things like size_t, you have to wonder if the program it's building
really supports pre-ANSI C. This wastes real time on not-so-uncommon platforms
like Cygwin and MSYS2, where spawning subshells is slow and running
./configure can take minutes.

So if you've never tested your program on a platform where /bin/sh doesn't
understand _local_ , feel free to use _local_ in the build script. Bashisms
like _local_ are safer anyway. Variable scoping is good. (Though I guess the
author's claim of it working in a 70s era Bourne shell is still wrong.)

~~~
jancsika
> This wastes real time on not-so-uncommon platforms like Cygwin and MSYS2,
> where spawning subshells is slow and running ./configure can take minutes.

That's true, but in my case it turned out to be insignificant. While changing
all the autoconf junk in > 100 plugins could move the total build time from 50
minutes down to 40, I'd need an order of magnitude decrease in build time to
warrant the work. I say order of magnitude because its only at a 5 minute
build time that the entire dev process would see a paradigm shift-- at that
point I could, with a straight face, stop shipping Linux binaries and just
suggest that users compile the software themselves. (Of course that's not
great usability for a whole class of Ubuntu users, but at build time == 5
minutes some other dev could easily come along and package things up with
ease.)

Also, the extra wasted time under Windows isn't the fault of autoconf. Try
doing `git any_subcommand` in msys2 and you'll get latency so high you'd think
it were a practical joke.

Edit: added parenthetical

------
edejong
I seriously wonder if the author ever worked on a project with more than
around 10k LOC, multiple languages and tools, automatic testing, deployment,
version control (merging this build script is going to be a huge PITA).

No, you have to realize you're working on one side of the graph (source code),
but your build produces something on the other side (your artifact). Without
automatic dependency resolution of you DAG, you're constantly traversing from
one side to the other, making the build 'work'.

~~~
brudgers
There are a lot of projects that don't meet those criteria. Scale is
bidirectional. Tools designed for big projects can have features and designs
that don't scale _down_ well. Tools designed for big projects often depend on
institutional type knowledge where expertise is spread among multiple
individuals...at Google, there teams of engineers managing the single code
tree is what allows a monorepository.

Alternatives are not replacements.

~~~
edejong
Yes, it's a team of engineers. But interestingly, Google _started_ with a
mono-repo. All through their explosive growth they maintained (mostly, except
Android and Chromium) their vision of mono-repo. It clearly shows that their
vision of mono-repo works both for small companies, as well as large ones.

A build-script without dependency resolution clearly does not scale. It was
the reason to design and implement Make, CMake, Maven, SBT and many others.

~~~
moosingin3space
To add to this, Google spent a lot of time and effort on building a custom
build system (Bazel) to make this sort of dependency tracking in a mono-repo
easy. It's pretty great, and I'd rather write BUILD files than Makefiles/shell
scripts any day, but the OP appears to have been focusing on extremely simple
projects, where shell scripts are okay.

~~~
brudgers
Right, "works for me" doesn't mean "you must do it.".

------
vog
I applaud the effort, but is "pure /bin/sh" really a good feature in today's
systems?

To my knowledge, today every Unix system has Python, usually at least 2.7, if
not Python 3. Some even replaced most of their system shell scripts with
Python scripts. (Formerly, Perl had this status, but never really caught on in
that area.)

Python is a much cleaner language than any shell language, especially with
regard to error handling. (yes, I'm aware of "set -eu") Also, if you
occationally need some more elaborated data structures or algorithms, you can
express these in a straight forward way, rather than a series of complicated
(and usually very fragile) text processing steps.

~~~
adtac
I'm having a hard time thinking of a use case for complex data structures in a
build script. Can you give a real world example?

Array is the only one that I have found lacking, in my experience.

~~~
vog
If you aim for slightly larger projects and efficiency, you'd need at least a
DAG (because that's the fundamental nature of a dependency graph).

You can find a nice explanation in the "tup" build tool, although there are of
course many others: [http://gittup.org/tup/](http://gittup.org/tup/)

And yes, you can express DAGs with text and/or filesystem operations (plus
symlinks), but not efficiently and writing code that operates on that level is
cumbersome.

~~~
svckr
Regarding DAG: tsort (1) sorts input lines by interpreting them as a list of
edges. I guess that's what you where referring to when you said "cumbersome"
:)

    
    
        $ cat > f
        a c
        a d
        b c
        ^D
        $ tsort f
        a
        b
        d
        c

~~~
vog
The problem is not tsort, it is adding, removing, updating items in such a
text file. (not to mention nasty corner cases like filenames containing
whitespaces, or the security implications of filenames containing newlines)

------
boomlinde
This leaves the topological sorting of the tasks to the programmer? This is
neat, but that's one very useful feature of make that's missing.

~~~
akkartik
Yes, that's true. I'm still noodling on that, primarily to enable parallel
builds.

Outside of parallel builds, however, I actually prefer an explicit sequence of
steps to so-called "declarative syntax". The frequency with which I find bugs
in the dependency graph of `Makefile`s [1][2][3] suggests that people are
starting from a sequence of commands anyway. And it's better for the reader as
well to see the commands laid out in order, when they need to debug breakage
on their setup. People should stop pretending to be Vulcans and just write a
script for building their project, and then sprinkle in `older_than`
directives as an optional optimization. Building from scratch should be _the_
case to optimize for readability, rather than some exceptional situation.

[1]
[https://github.com/zetavm/zetavm/commit/4e5e2e9d8b](https://github.com/zetavm/zetavm/commit/4e5e2e9d8b)

[2]
[https://github.com/tekknolagi/stackx/commit/5999202423](https://github.com/tekknolagi/stackx/commit/5999202423)

[3] Fun fact: when you try to build OpenBSD's userland from source, the top-
level Makefile always unconditionally calls `make clean`. I've seen this
pattern a few times: large projects find themselves up shit creek in the build
system, and it's difficult enough to test and debug the dependency graph that
they give up all performance and retreat to just recompiling everything, just
to be safe.

~~~
vog
_> The frequency with which I find bugs in the dependency graph of `Makefile`s
[1][2][3] suggests that people are starting from a sequence of commands
anyway_

This is why the more elaborate build systems recognize dependencies
automatically. For Makefiles, usually some sub-Makefile is generated
automatically and then included. For language-specific build systems like OPAM
(OCaml) or Cargo (Rust), this is one of their core features: As a programmer,
don't care about maintaining dependency definitions! Whenever a source file is
changed, the build system recalculates the relevant part of the dependency
graph as well.

------
zimpenfish
How does this compare to e.g. @apenwarr's minimal `do`?

------
ejholmes
A true build system really needs to build out a DAG of dependencies to be
efficient and fast. Compose this with
[https://github.com/ejholmes/walk](https://github.com/ejholmes/walk) and
you're good.

~~~
qznc
Thus ninja [https://ninja-build.org/](https://ninja-build.org/)

------
pskocik
I'm also using (for now) something based on these time checks (also pure posix
+ local, but local is extremely portable (tests well on 7 shells), though
mine's a little more involved (automatic build job timing, controlled
concurrency, robust failure/cancellation with autodeletion, fancy logging,
filterable implicit dependencies, dependency + counting). It scales well to
about 1000 source files. Then the lag starts to feel noticable.

------
reacweb
One of the benefits I see for this approach is the possibility to handle
spaces in filenames (AFAIK very difficult with gnumake). Sadly the provided
function does not accept spaces in filenames.

------
sgt
Somehow I find it slightly amusing that I see ./a.out in 2017.

------
lauretas
"notabug.org"... interesting.

Anyway, how does this compare, features-wise, with GNU Make? For example with
make you have `./configure` files, do I also have them here?

~~~
gardnr
Looking as the sh code, it appears that it will work without tab characters.
[https://stackoverflow.com/a/2131227/1144060](https://stackoverflow.com/a/2131227/1144060)

So there's that.

~~~
akkartik
As it happens, I was moved to create this standalone repo showing off my
current build setup after writing this comment:
[https://www.reddit.com/r/vim/comments/6l8z34/vim_betrayed_me...](https://www.reddit.com/r/vim/comments/6l8z34/vim_betrayed_me/djsaxwr/?context=3)

So yes, eliminating tabs (or rather the execrable _requirement_ of using tabs)
was absolutely a prime goal.

~~~
Symbiote
Put this in ~/.editorconfig

    
    
        [Makefile]
        indent_style = tab
    

And install the plugin, which is available for most editors and IDEs.

[http://editorconfig.org/](http://editorconfig.org/)

------
jlebrech
has anyone tried tup? [http://gittup.org/tup/](http://gittup.org/tup/)

------
TheDong
> The build script is easy to repurpose to your needs

No, it's not. This isn't under a free or open source license. The lack of
license means that only an idiot would use it for anything.

