
GNU make insanity: finding the value of the -j parameter - jgrahamc
http://blog.jgc.org/2015/03/gnu-make-insanity-finding-value-of-j.html
======
snoopybbt
I don't know how many hours the author has spent/wasted on this topic (and how
many people are now wasting their time coming up with other solutions that in
the end do NOT fix the real problem), but...

Quite frankly, I find this whole "look i did a horrible hack and let's see who
can make the best worst horrible hack" thing quite stupid and silly.

GNU Make is free software released under a free license, my opinion is that
instead of doing that crazy thing, the author could have just written a patch
for GNU Make in order to make it export a "JOBS" environment variable to all
its child processes.

Oh but yes, "I felt this feature was missing so I added it" is way way way
less cool than "geez the gnu make folks are insane lollerplex they have no way
to know how many jobs they're running".

ALSO: [http://www.catb.org/esr/faqs/hacker-
howto.html#believe2](http://www.catb.org/esr/faqs/hacker-howto.html#believe2)

Here it is a patched version of GNU Make, providing a #J variable that holds
the number of jobs as passed via -j/\--jobs:

    
    
        http://santoro.tk/~manu/gnumake.png
    
    

As can be seen in the screenshot, it can be passed as environment variable to
programs called by make

Source: [https://github.com/esantoro/make](https://github.com/esantoro/make)

~~~
LukeShu

        > I don't know how many hours the author has spent/wasted
        > on this topic
    

The author, John Graham-Cumming, has been extremely active on the GNU Make
mailing list, is the author of the "GNU Make Standard Library" of supplemental
functions for Make, has written at least 2 books on GNU Make, and has
developed commercial products that integrate with GNU Make (namely, "Electric
Make").

That is to say: He is one of the world experts on GNU Make.

I'm confident that he did this more "for fun" than because he actually wanted
to get the number of jobs.

~~~
jgrahamc
Yes. It was 'for fun'. Someone at CloudFlare asked me how to get the value of
-j and I figured there must be _some_ way :-)

~~~
jordigh
Do you know if this person hangs out in IRC? I asked this precise question 10
days ago in #emacs, and I was curious to see my precise question answered,
almost the way I phrased it.

~~~
jgrahamc
Too funny. I think it's just random chance. Although he is an emacs user.

------
jwatte
Those who don't learn make are doomed to reimplement it. Poorly.

Also: never, ever, build a make system that relies on recursive make
invocations.

Automake is a particularly egregious violator is this rule (and other sanity
rules) and has probably hurt make more than anything else.

~~~
santaclaus
Someone should implement make without the silly tabs.

~~~
babuskov
And I thought I was the only one burned by this ;)

~~~
burstmode
At least every 2nd C programmer working on a non-unix platform has been burned
by this at least once. It's like a 10ft deep pothole on Broadway that won't
get fixed because traditionalists see it as a 'national monument'.

------
Quanticles
Okay, I'll bite. This is my attempt at it:

\----------------------

PID := $(shell cat /proc/$$$$/status | grep PPid | awk '{print $$2}')

JOBS := $(shell ps -p ${PID} -f | tail -n1 | grep -oP '\\-j *\d+' | sed
's/-j//')

ifeq "${JOBS}" ""

JOBS := 1

endif

all:

echo ${JOBS}

\----------------------

quanticles@glados $ make -j8

8

quanticles@glados $ make

1

\----------

edit: for silly mistake

~~~
anon4
Note: you can forego the ps and simply read /proc/$PID/cmdline, which is a
\0-delimited list of strings. So,

    
    
        grep -oP '\-j\x00\d+' /proc/$PID/cmdline
    

Edit: and no need for sed when a simple cut -b4- does the same job.

~~~
protomyth
Does that work on anything other than Linux?

~~~
myhf
Not in exactly the same way, but every unixy system has some way of finding
the parent pid and looking up its command line.

~~~
protomyth
Is Quanticles's code portable to the other unix systems?

~~~
myhf
It should work on most systems that have a /proc filesystem:
[http://en.wikipedia.org/wiki/Procfs](http://en.wikipedia.org/wiki/Procfs)

If you don't have /proc but do have ps (1), you could get the parent pid like
this:

    
    
        PID := $(ps -p $$$$ -o ppid=)

------
JoshTriplett
It's an amusing exercise, but for actual practical use, the right solution is
to share make's jobserver. If you have a build rule that wants to run multiple
things in parallel, put + in front of the command, and then your command will
receive the file descriptors for the jobserver pipe on the command line. Then,
when you want to start a parallel job, do a blocking read of a byte from the
read fd, and when you're done with that job, write a byte to the write fd.

~~~
chaosfactor
This does not generalize to arbitrary purposes. If you just want to detect
that the user has run more jobs than your build system can possibly benefit
from and print a warning, there is no reason to start messing with the
jobserver. This is not the Ockham solution :)

~~~
JoshTriplett
Why print such a warning? For most projects, make -j65536 will be effectively
equivalent to make -j, both of which are perfectly fine. Suppose you're on a
system with more CPUs than you have build rules in your build system; you
shouldn't print a warning just because someone does make -j$(nproc).

------
moonbug
Carp all you want about make, I'll take it over any of its ostensible
"replacements".

~~~
Negitivefrags
I don't think anyone has a problem with make. The thing you need to replace is
autotools.

~~~
davexunit
As crufty as autotools is, nothing comes close to being a more robust build
system. It does everything. Learning to use the autotools is difficult, but
what's most frustrating is trying to build software that uses clever Makefiles
with a bunch of hardcoded assumptions about the platform you are building on
that you have to hack to get to work.

~~~
saboot
"As crufty as autotools is, nothing comes close to being a more robust build
system."

.. CMake? [http://www.cmake.org/](http://www.cmake.org/)

~~~
michael_h
I wish this were true. Getting cmake to do something off the wall like
building static executable binaries is...frustrating.

I had high hopes for scons, but that turned into a quagmire. Any build system
that's going to unseat autotools will need to obviate the need for almost all
custom scripts. AKA, specify the 'what', and let the user apply a 'recipe' for
the 'how'. Or something...if I knew what it was supposed to look like I'd
build it.

~~~
MereInterest
Out of curiosity, what are the quagmire-ish aspects of scons? I've been using
it for a while, and it hasn't seemed too bad, which probably means that I
haven't run into them yet.

~~~
michael_h
Scons itself isn't bad, but it's power is also it's downfall: the build script
is an unrestricted Python file. I almost always find that something 'clever'
has been stashed in the SConscript. Plus, everybody writes wrapper functions
for the env.* stuff - meaning just about every project has a second, hidden
codebase that you need to learn first.

~~~
MereInterest
Ah, got it. I'll admit that I tend to make build scripts that are a bit too
complicated, due to the freedom of having a full language, but it is quite
nice to have when needed. It does end up with build scripts that are far more
flexible, though, not having read build scripts that others have written, I
don't know how much of a cost there is to it.

------
foxhill
i think the true insanity lies within the mind of the developer whose build
system depends on the number of jobs being ran in parallel..!

interesting academic exercise though. unless i'm mistaken, this will only
detect up to 32 build jobs? i'd hate to see the Makefile that works for
arbitrary values..

~~~
chaosfactor
It makes some sense to want to know this. You might know that your build
process doesn't benefit from more than 4 jobs, and want to alert this to the
developer who tries to use more.

~~~
moonbug
Not all problems need a solution in code.

~~~
nsajko
Awesome! This goes in my fortune file :D

------
raindrop777
Reading this only pops one question: What is the best replacement of make in
2015? :)

~~~
vezzy-fnord
tup [1], redo [2] and ninja [3] are probably the build systems with the most
potential.

redo is the simplest, which is unsurprising given that it was originally
proposed by djb. Alternatively, you could look into mk, which is the make
replacement used by Plan 9, Inferno and derivatives.

[1] [http://gittup.org/tup/](http://gittup.org/tup/)

[2]
[http://homepage.ntlworld.com/jonathan.deboynepollard/FGA/int...](http://homepage.ntlworld.com/jonathan.deboynepollard/FGA/introduction-
to-redo.html)

[3] [https://martine.github.io/ninja/](https://martine.github.io/ninja/)

~~~
raindrop777
Are any of these cross-platform like make?

~~~
nsajko
mk has been ported to UNIX-likes, see plan9port.

~~~
smorrow
mk originated on Unix, and so did almost everything else in Plan 9.

------
msie
I need more information as to why it's required to get the value of that
parameter. Can't it be simultaneously be passed in as an environment variable?

~~~
rcthompson
There is no reason your makefile should ever depend on the parallelism level,
with the possible exception of requiring no parallelism at all (i.e. "-j1").
That's why make doesn't bother to expose it. This little hack is just for fun.

~~~
jordigh
I was the one who asked the original question. My purpose was so that
Mercurial's test suite could specify the amount of parallelism from the
Makefile. The "test" rule in the Makefile invokes "run-tests.py". This latter
script itself accepts a -j option. It's possible to invoke it directly, but
this misses the second part of the "test" rule in the Makefile, which executes
some tests that run-tests.py can't do.

------
chaosfactor
This probably won't work if you just pass -j. If I needed to solve this
problem, I would have just grepped the output of `ps' to get the make
invocation. If I wasn't sure which invocation was correct, I would have used
heuristics like the output of `pstree' to figure it out.

~~~
Someone
I would think a process launched by make can easily obtain the process ID of
its parent process. If so, no heuristics are needed to find the correct
instance of make.

On the other hand, reading the command line isn't sufficient. The _-j_
argument does not require a job count, defaulting to "as many as possible".

Also, you would have to check your OS, too since the -j argument is ignored on
MS-DOS
([https://www.gnu.org/software/make/manual/make.html#Options-S...](https://www.gnu.org/software/make/manual/make.html#Options-
Summary))

------
tobik
For FreeBSD users: FreeBSD's make provides the value of -j through
${.MAKE.JOBS}

------
joeyh
The real insanity with make is that it has no interface to check whether a
Makefile contains a given target.

Best you can do is to run make -n target to probe, but it's possible to write
Makefiles that run code even though -n is used, which will defeat such probes
at distribution scale. It quickly becomes an exercise in heuristics and output
parsing.

~~~
chaosfactor
Write a target that greps the Makefile itself and extracts the targets.

~~~
TheDong
As shown in the post, targets are not always grep-able. The post's makefile
contained the target "par-30" even though that string appears nowhere on the
page or in the makefile.

You need to write a full parser to discover all the targets.

~~~
Hello71

        $ make -pqrf <(curl https://gist.githubusercontent.com/jgrahamc/2cc6df7fd5cd61c4d93f/raw/parallel) | grep '^[^ ]*:' | cut -d : -f 1
        par-%
        par-22
        par-30
        par-5
        par-27
        par-31
        all
        FORCE
        /proc/self/fd/13
        par-13
        par-9
        par-23
        par-19
        .SUFFIXES
        par-15
        par-7
        parallel
        par-29
        par-28
        .parallel
        par-4
        par-25
        par-6
        par-14
        .DEFAULT
        par-24
        par-0
        par
        par-1
        par-10
        par-16
        par-2
        par-11
        par-17
        par-20
        par-8
        par-21
        par-12
        par-26
        par-18
        par-3

~~~
TheDong
Right, you used 'make' as the parser. He said "grep the makefile". You're not
grepping the makefile. Nice solution though!

~~~
Hello71
> You need to write a full parser to discover all the targets.

I don't think grep '^[^ ]*:' really counts as writing a parser.

~~~
TheDong
Running "make -p" is running a full parser which someone else wrote. Grep is
not extracting the targets from a makefile there, it's extracting the targets
from an intermediate representation of a makefile.

~~~
Hello71
> The real insanity with make is that it has no interface to check whether a
> Makefile contains a given target.

>

> Best you can do is to run make -n target to probe, but it's possible to
> write Makefiles that run code even though -n is used, which will defeat such
> probes at distribution scale. It quickly becomes an exercise in heuristics
> and output parsing.

~~~
TheDong
We're arguing different things.

The comment I replied to, chaosfactors, specifically specified grepping the
makefile. You're arguing that you can find all the targets and I never said
you can't. I simply said you can't find them all by only grepping the
makefile.

------
raverbashing
If you change the maximum value (I put 1024) on line 10 it prints:

319 on Mac OS X 10.8

337 on Centos 6

------
cafebeen
On a semi-related note... do any of the make gurus out there know if there's
anything similar to "-j" that instead sends parallel jobs off to a cluster,
e.g. Sun Grid?

~~~
adrusi
That wouldn't be appropriate as a specialized component of make. You could
write a script that sends a command off to a cluster and just set

    
    
        CC=script/send_to_cluster gcc
    

Then specify the number of jobs you'd like to run simultaneously on your
cluster with the "-j" flag.

~~~
cafebeen
Thanks for the thought. That sounds fine for simple compiler makefiles,
although I'm using it for general purpose work (data analysis and
visualization). I'm hoping for something that wouldn't require each command to
be wrapped up, as there are many.

~~~
TylerE
For that sort of thing, maybe roll your own script using something like Fabric
([http://www.fabfile.org/](http://www.fabfile.org/)) for doing all the
communication with the remote hosts?

~~~
cafebeen
Thanks, I'll check it out

------
mempko
GNU make is free software. If you are a capable programmer, you can make a
change and send a patch.

------
sillysaurus3
Why do you need to know the value of the -j parameter in a makefile?

~~~
saboot
It will reduce total compile times for large projects by running make in
parallel. Files dont compile faster, but make will act (compile) on N files at
the same time together.

~~~
babuskov
I didn't downvote you, but I assume why it happened: What you wrote is
correct, but does not answer the question.

