
Polyglot Makefiles - sciencerobot
http://agdr.org/2020/05/14/Polyglot-Makefiles.html
======
brandmeyer
That's interesting.

Author probably wants to use `private` for those target-local variables,
though.

For example,

    
    
        R: .SHELLFLAGS := -e
        R: SHELL := Rscript
        R:
            greeting = "bonjour"
            message(paste0(greeting, ", R!"))
    

Everything that target `R` depends on will also have SHELL and .SHELLFLAGS
over-ridden. If `R` depends on some data generated by another program, it
probably wants to be built and executed with the default SHELL (or another
shell, perhaps).

    
    
        R: private .SHELLFLAGS := -e
        R: private SHELL := Rscript
        R:
            greeting = "bonjour"
            message(paste0(greeting, ", R!"))
    

Now, `R`'s dependencies will be generated with the makefile's defaults.

Usually I prefer to build up richer stages like this using the system shell
anyway, though. Build a target which in turn is executed by the shell normally
to traverse the next edge in the graph. But I can see how this mechanism has
its uses.

See also
[https://www.gnu.org/software/make/manual/html_node/Target_00...](https://www.gnu.org/software/make/manual/html_node/Target_002dspecific.html)

~~~
sciencerobot
I didn't know about private. Thanks for the tip.

------
gregwebs
Make was designed for building dependencies. I think it is always problematic
to use it as a command runner (for example there is no standard way to list
out the available commands).

[just]([https://github.com/casey/just](https://github.com/casey/just)) is a
tool that feels similar to make but is designed explicitly for the purpose of
running commands.

I think of this as a simple CLI for your project workflow. You still really
want to avoid putting code into a Justfile and put it into scripts. But the
Justfile helps provide a slightly nicer UX and automatically invoke
dependencies.

~~~
dima55
The tool you want is remake:
[http://bashdb.sourceforge.net/remake/](http://bashdb.sourceforge.net/remake/)

This is GNU Make + a few patches. So it's 100% compatible. And you get an
interactive debugger, and lots more stuff. For instance, to list out the
commands:

    
    
      remake --targets
    

No idea why this hasn't been merged upstream.

Your larger point really stands, though: if you're just running commands, you
shouldn't be using Make. But it is abused in that way often, so...

~~~
jackewiehose
> remake --targets; No idea why this hasn't been merged upstream

I would think because it's useless. The targets in a Makefile are very often
just internal and aren't always meant to be run by the user.

~~~
rixed
Many shell autocompleters would read the makefile to complete target names
though, suggesting it is not useless.

Anyway, one could always have a 'help' target that prints a short
documentation. This also avoid listing internal targets.

~~~
jackewiehose
Yes, I remember using zsh and in my experience this was barely useful since
most Makefiles are auto-generated with hundreds or thousands of targets.

> one could always have a 'help' target that prints a short documentation

Sure, that's fine. But the point is that if you have an unknown Makefile you
can't (or shouldn't) just execute it without knowing what it will do.
Makefiles should be treated as individual programs just like any other
executable and there's no guaranteed standard way to get help from it.

------
epistasis
For complicated pipelines that I want to reuse multiple times, I have turned
Makefiles into executables by putting this at the top:

    
    
        #!/usr/bin/make -f
    

And then putting them in my $PATH. I run them with arguments like:

    
    
        $ process-data.mk INTSV=a.tsv DB=largefile.gz OUTDIR=finished
    

This makes me feel like I've sold my soul to the devil, and that I'm just
living on borrowed time until it all fails and falls apart. It hasn't yet,
however...

~~~
asveikau
I can see two areas in which it might break.

I might put _#! /usr/bin/env make -f_ in case it's somewhere in else in PATH.

Also some systems (BSD, old commercial Unix) have non-gnu-compatible make and
sometimes call their gnu make port "gmake" or "gnumake".

~~~
hawski
AFAIR on Linux shebangs only support single argument so it would fail in this
case. One can overcome this treating the file as a shell script:

    
    
      #!/bin/sh
      # make ignores next line \
      set -e
      # make ignores next line \
      exec make -f "$0" "$@"
    

Make treats slash-escaped new lines as a continuation even for comments, shell
does not.

~~~
asveikau
Wow. Impressive.

------
ainar-g
Note that this article (like many, many others) assumes GNU Make. POSIX Make
has neither .ONESHELL nor local macros. Neither do most built-in Make
implementations in other OSes, like OpenBSD's bmake.

~~~
ddevault
POSIX shell is also notoriously obtuse and difficult to use. As a big advocate
of POSIX as a target, I don't blame anyone for using GNU make - or perhaps BSD
make is a better lowest common denominator.

Personally, I try to use POSIX Makefiles, but I often find that they're most
useful as a target for Makefile generators (in my case, these are usually a
shell script called configure).

~~~
ainar-g
One person's obtuseness is other person's simplicity :-) . In all of my
personal and some of my work projects I used nothing but portable features in
Shell, Make, Sed, etc. Checking with multiple implementations where possible.
As long as you use the right tool for the right job, there shouldn't be any
problems.

The most common mistake of that sort that I've seen is people trying to do
complex conditionals inside their makefiles when they clearly would be better
off in a Shell script. (I'm looking at you, fans of ifeq.)

~~~
wahern
You can implement conditionals semi-portably. See
[https://github.com/wahern/autoguess/blob/master/Makefile.gue...](https://github.com/wahern/autoguess/blob/master/Makefile.guess.sub),
which works with GNU Make, NetBSD/FreeBSD make, OpenBSD make, and Solaris
make. Alas, it doesn't work with AIX's native make.

Once POSIX standardizes "!=" then POSIX-portable conditionals will be possible
using the same technique as above, replacing, e.g. OS = $(shell
$(OS.exec))$(OS.exec:sh) with just OS != $(OS.exec). Though, you'd need to
wait for Solaris, AIX, and macOS gmake[1] to add support for !=.

Alternatively, if you add an extra level of indirection using .DEFAULT to
capture and forward make invocations, you can simply pass OS, etc, as
invocation arguments. Indirection solves everything, though, so that's
cheating.

[1] Apple's ancient GNU Make 3.81 predates != support. :(

------
IshKebab
I wish someone would write a modern alternative to GNU Make. I've looked and
there don't seem to be any. The closest is Ninja but it doesn't seem to be
intended to be hand written.

~~~
boris
[https://build2.org/build2/doc/build2-build-system-
manual.xht...](https://build2.org/build2/doc/build2-build-system-
manual.xhtml#intro)

~~~
brandmeyer
> We believe, paraphrasing a famous quote, that those who do not understand
> make are condemned to reinvent it, poorly.

Alas, I think the developers have in fact done just that. Make is not a build
system. Make is a batch shell.

Build2 didn't produce a better make, they produced yet another C++ build
system.

~~~
boris
We did start with C++ thinking (correctly, IMO) that if we can build C++, we
can build pretty much anything. But build2 is a general-purpose build system,
for example:

[https://build2.org/build2/doc/build2-build-system-
manual.xht...](https://build2.org/build2/doc/build2-build-system-
manual.xhtml#module-bash)

[https://github.com/build2/libbuild2-rust](https://github.com/build2/libbuild2-rust)

~~~
brandmeyer
I think these examples support my position, they do not refute it.

If you have to write plugins to describe the rules that walk the edges of the
DAG, then you haven't captured the essence of Make. It isn't just the DAG, its
also the ability to walk graph edges with a generic shell alone. Here's some
examples of things that we're using it for:

\- Compile C, C++, and FORTAN on a common DAG for 5 unique ABIs.

\- Parallelizing and sequencing atmospheric analysis with orbital mechanics
programs.

\- Post-processing our regression test suite.

\- Executing and verifying SystemVerilog tests.

\- Generating documentation with Doxygen and LaTeX.

\- Generating linear flash images and a compressed initial filesystem.

\- Transforming said initial filesystem into a linkable object.

Make does all of things without any prior knowledge of any of them, because it
just uses the shell to express how the edges are walked. In some cases, we
build the program that traverses an edge and express that as just another
dependency in the chain.

If you have to write and compile plugins into build2 to do things like that,
then you haven't re-implemented Make. You've just created another purpose-
dedicated build tool. That's fine if its what you set out to do. But that also
means statements like "We believe, paraphrasing a famous quote, that those who
do not understand make are condemned to reinvent it, poorly." do not belong in
your documentation. Because I don't think you understand Make.

------
adelarsq
I added this to my list about Make at [https://github.com/adelarsq/awesome-
make](https://github.com/adelarsq/awesome-make)

Pull requests are welcome.

------
mehrdadn
I think this might have repercussions? Like if you do "make foo bash bar" then
can you predict what that SHELL is used for?

~~~
sciencerobot
The shell is specific to each target. So doing `make ruby bash python docker`
works. It even works in parallel if you do `make -j`.

Edit: I'm the author.

~~~
mehrdadn
And what happens if there's overlap between the targets?

~~~
rhencke
Define 'overlap'?

~~~
mehrdadn
What if they have a common prerequisite?

------
purplezooey
loving the python3 makefiles

