Not a bad introduction. I spent years working on GNU make stuff (including a complete emulation) at Electric Cloud and wrote an "Ask Mr Make" column. Everything I wrote about GNU make can be found here: http://blog.jgc.org/2013/02/updated-list-of-my-gnu-make-arti...
[blatant advertising] And also I have a new book on GNU make coming out this month: http://www.nostarch.com/gnumake if you really want to get into make deeply.
If anyone wants to get into GNU Make deeply, one would think that the obvious thing to do would be to read the official manual for GNU Make.
Anyone who writes their own, not freely available, manual to a free software program, is dividing the world into two halves: Those readers who can afford this new book, and will pay the author (not the writers of the software, mind you) for a nice manual, and those readers who can’t afford to pay for this new manual, and will have to do with the inferior¹ official manual. Those readers who can afford the new manual will not be incentivised to improve the official manual, since they have the new, better, one. Therefore, this act (of creating new manuals which are not freely available) is reducing the number of readers who can be expected to help improve the official manual to those readers who cannot afford the new manual. These effects, i.e. reduced funding and reduced help to the original software authors, is not, to me, worth it to have a new, even if better, manual.
① Presumably, by their own standards, since they wrote a replacement manual.
The entitlement of your post is out of control, and you're making a lot of assumptions in what is really just a knee-jerk reaction to someone, somewhere making money.
> If anyone wants to get into GNU Make deeply, one would think that the obvious thing to do would be to read the official manual for GNU Make.
There is a lot of room for both "user guide" type publications in addition to the official manual. In particular the former may be useful to individuals who have had difficulty learning how to use make from other resources. Are you saying that all of the "Learning Basic Linux" books should be pulled since people can just read the Linux manuals?
> and will pay the author [of the manual] (not the writers of the software, mind you) for a nice manual
You act as if writing a manual is somehow a less valuable act than writing the software, when in fact high-quality documentation is sorely lacking in almost all parts of the free software world.
I doubt you have ever written, edited, and had published a book before, because even for something as "simple" as a guide to a software tool, it is an intense, laborious process that involves lots of research and fact-checking in addition to the actual writing and editing. Its incredibly entitled to suggest that all of this effort should not be compensated at the author's discretion.
> Those readers who can afford the new manual will not be incentivised to improve the official manual, since they have the new, better, one.
They were never incentivized to improve the official manual in the first place.
> reduced funding and reduced help to the original software authors
There was no expectation of funding to the original software authors in the first place, and to imply that they deserve the money for what would be spent on this book, a separate creative work, is disingenuous.
Well, jgrahamc did post a link (http://blog.jgc.org/2013/02/updated-list-of-my-gnu-make-arti...) to 38 articles that provide free information on GNU Make usage. Reading his post suggests that the new book is a properly edited and formatted edition of the older book with the same content updated to Make 4.0.
Copyright law is pretty crappy on the whole, but jgrahamc did write the articles and has been giving them away for free. Why is it unreasonable for him to package, edit, update, and release them years after they were written, with the originals still available for free?
I do not get the impression that the “The GNU Make Book” has any substantial similarity with the freely available “Ask Mr. Make” articles – I get the impression that the book is a wholly new work, aimed at being a general manual for GNU Make. The title is especially clear in giving me this impression, but also the text describing the book on its home page gives me no reason to doubt my conclusion.
Anyway, whatever other good things jgrahamc may have done is immaterial, since I was not only speaking about him, but also about the general case and its consequences. Also, I did not say it was “unreasonable” of anyone to do this. I said that when anyone does something like what jgrahamc did, they take away some unknown amount of human and monetary resources from the authors of the actual software and the official manual, and I don’t think that this drawback is worth it to have a new & improved, but non-freely licenced, manual.
Why would any software program be limited to one book which tries to serve all possible audiences? This is never the case with commercial software. Why would free software be limited to one book? Different audiences have different needs and they are free to vote for the authors who meet their unique needs.
There are certainly good arguments for the existence of books for specialized purposes and audiences, like, say, “Python for Data Scientists”. However, this case was simply about “if you really want to get into make deeply”, and for that specific purpose I would think that the obvious thing to do would be first to go to the GNU Make official manual.
What a shitty argument. On the contrary, the fact that somebody writes a non-free book about a piece of software that comes with a free manual with the expectation that he can actually compete with the free manual tells you much about the quality of the free manual. Everything else is just normal, healthy competition in a free market.
You seem to pre-suppose that all-rights-reserverd copyright on a manual for a free software program is OK. I do not agree, mainly for the reasons outlined here:
No, I don't. I merely think that having a free manual and a non-free alternative to it is perfectly fine, because the user will decide which one they pick. No need to force any kind of specific ideology upon anyone.
Since the title is "the GNU make book", a consumer could easily mistakes the book as made by the author of GNU make or/and published by the GNU project.
Edit: Missread the article. It was blizzard that got sued for filling the DMCA, not the other way around. A more relevant article is the Harry Potter Lexicon case where a guide to the harry potter works was deemed not protection by fair use. (http://writers.stackexchange.com/questions/8243/can-anybody-...)
The copyright argument does not seem very plausible, and regarding your linked example: Unofficial guides and such should really be allowed, and I don’t see any consensus forming around disallowing them.
Your point about trademarks is much more interesting, though; how many people have paid money for this “The GNU Make Book”, thinking that they have somehow supported the GNU Make project? Isn’t this exactly what trademarks are supposed to prevent?
Agree, except that the underpinning is about about the freedom to improve and share, which generally means there's a freely available version on the internet, but by no means means you can't or shouldn't buy electronic or paper copies for money to support the author. ie. the gnu make manual: http://shop.fsf.org/product/gnu-make-version-381/
GNU make is one of the (many) things that I've meant to get a grasp on. I picked up an electronic copy of the book. Very nice! It's a lot more example-driven than the GNU Make Manual.
I'm currently using Make as part of a data-building pipeline. It's nice that it's a build system that doesn't assume my goal is to build a binary or a JS file, and that it's remarkably easy to parallelize.
One deficiency I've come across, which seems to be well-known, is the M:N problem -- where one step takes M input files and has N output files. Make rules seem to expect to have only 1 output, and the workarounds like .SECONDARY prevent some features of Make from working correctly.
I've also seen this limitation in many of the fancy new build systems that get posted here on HN.
Is there a build system, a modification to Make, or anything out there that does keep track of builds with multiple outputs? Not an I/O-guzzling MapReduce framework, please.
FWIW, my tool Ninja (was just on here again this week here: https://news.ycombinator.com/item?id=9282539 ) was designed to be more or less the same idea as make with some semantic fixes, and that includes supporting multiple outputs.
(Within Ninja this manifests concretely as needing to represent the build graph as a bipartite graph between files and commands. Though I don't know make's internal representation, I imagine the straightforward implementation is as a graph between files and I can imagine why it'd be difficult to make multiple outputs work in that case.)
I really like the minimal and solid design of ninja; I'd agree that if make isn't cutting it, ninja + (your own script to generate all the correct rules) is probably the best way to go.
Is there some fundamental reason why the approach used in Ninja couldn't be used to accelerate a substantial subset of Makefiles? If you dropped Make's built-in rules, more complex functions, VPATH, conditionals, and similar items the Ninja manual describes as slowing down Make (all of which are non-portable features of GNU Make), while mostly keeping compatibility with the portable subset of Make, is there some fundamental reason that couldn't be accelerated to the same degree as Ninja?
No, they should be equivalent. The Ninja manual goes so far as to encourage you to continue using Make if Make is fast enough for you.
However, another way of writing your question is "if you removed the features of make that ninja doesn't have and then optimized the remainder for performance, wouldn't they match?" and that is maybe tautologically true -- that description is roughly what Ninja is, after all.
Because we cared more about performance than other things we maybe were a little crazier about shaving off milliseconds than other systems, so it might be hard to make make fast enough. But (as the Ninja manual suggests) this may only matter for very large (Chrome-sized) projects. You can read a chapter about some of the performance work done on Ninja here: http://www.aosabook.org/en/posa/ninja.html
> However, another way of writing your question is "if you removed the features of make that ninja doesn't have and then optimized the remainder for performance, wouldn't they match?" and that is maybe tautologically true -- that description is roughly what Ninja is, after all.
Not exactly. I'm asking if an implementation of the portable subset of Make people actually use (pointedly excluding GNU Make extensions) could be made nearly as fast as Ninja. That would have the advantage of running existing makefiles, as long as those makefiles weren't written specifically for GNU Make.
When you write "could be made nearly as fast as Ninja", I thought you meant you would change the code of Make or write a new tool. If you don't write a new tool and just use a subset of Make's functionality for your own Makefiles I'd expect those to be faster than your standard Makefiles, sure.
But "existing makefiles" typically use a mixture of GNU make-isms and other things that are not GNU-specific but still slow (e.g. from a skim of the BSD make manual I see lazy variable expansions and control structures). So if you're talking about implementing a subset of Make you're talking about something that's unlikely to be compatible with Makefiles seen in the wild. And then if you're not compatible with Makefiles seen in the wild you've effectively written an incompatible faster subset of Make, which brings us back to what Ninja is. (In case it's not clear, in Chrome's case the same code was used generate Makefiles and Ninja files with some relatively minor output differences -- the tools are that close.)
Perhaps there's some "common" subset of Make that covers some high percentage of build files seen in the wild that could be made faster. That could be valuable to organizations who want faster builds without changing anything. I vaguely recall seeing some commercial software that did this even -- as I recall, their value add was that they had tooling that would figure out how to run your Makefiles in parallel; though Make supports parallel execution, apparently this business found enough customers by just targeting those with underspecified Makefiles that weren't parallel safe and then fixing them!
Anyway, all of this is kinda moot because for 99% of projects Make is plenty fast. I even use Make myself for all my personal projects! In Chrome's case (what we wrote Ninja for) there's so much code that the build files themselves are over ten megabytes of text, so to parse that quickly you're at the level of worrying about avoiding memory allocations in the lexer, which is beyond what most people would care about.
There are plenty of shell scripts out there that are written for /bin/sh without using bash features; similarly, many makefiles just use the portable subset of make, such as POSIX make. I agree that you'd likely need to write a new tool rather than attempting to optimize GNU make or BSD make. But you wouldn't need to introduce a new syntax; you could just use the POSIX subset of make.
I was partly asking because a fast make-compatible build tool seems easier to switch to, and partly because I wondered how much of Ninja's performance depends on dropping the slow features of make versus adding new capabilities or new syntax.
Pretty cool. The fact that I'd have to write my own tool to generate the list of dependencies, instead of using pattern rules, is kind of a downer, but--
Okay, now I'm wondering if I could get Make to generate a Ninja file.
Part of the reason Ninja doesn't support pattern rules is that it would require matching patterns while choosing what to compile. In projects like Chrome (with hundreds of thousands of source files) that can take a while.
IIRC, I got around the M:N problem with GNU make by defining a pattern-matching rule to do what I want. For example, I was using yacc, so I did this:
%.tab.h %.tab.c: %.y
stuff
Using pattern matching forces make (GNU make) to correctly treat this as one rule which outputs multiple files, instead of the default of it being multiple rules which each output one file. It's not perfect, but it is a simple fix for some of those issues.
That's actually one rule which applies to multiple targets.
If you run without parallelism (the default, -j1), it'll work fine, since make will run the rule for one target, and then when it comes to the second one, make will notice that it's already up-to-date (based on file timestamp).
If you run with -j4 or something, it make will likely run the rule twice, simultaneously. That may or may not be a bad problem depending on how the rule writes the output file.
EDIT: man I must have really rushed reading your post... as you say, this does appear to be a heuristic for GNU Make when it's a pattern rule. clearly make is tricky and I end up learning stuff every time I use make...
The automake manual has a description of how to get the dependencies right for multiple outputs: http://www.gnu.org/software/automake/manual/html_node/Multip... . (Disclaimer: I'm no expert, and the Make book mentioned elsewhere in this thread may have a better solution.)
Multiple outputs are tricky, what you going to do if you run a step and it somehow produces 2 out of 3 expected outputs? I mean, you need a way to decide whether whole thing failed or not, easiest way is indeed to have dummy targets.
Anyway, take a look at https://github.com/spotify/luigi, it is basically make-like tool geared toward data pipelines. We are experimenting with it for our data pipeline (which is quite sizeable, ~30TB/day gzipped, thousands of files, a bunch of processing steps) and while it lacks some things, especially in the UI, this approach seems to work quite well.
You can actually articulate that relationship in GNU make, as long as your outputs can be coerced into a pattern rule (like "%.c %.h: %.y"). Otherwise there's no way to do it that's reliable in the face of both incremental and full builds. There's been discussion of adding syntax to GNU make for this purpose on-and-off over the past several years, but nobody can seem to agree on what the syntax should be.
> One deficiency I've come across, which seems to be well-known, is the M:N problem -- where one step takes M input files and has N output files. Make rules seem to expect to have only 1 output, and the workarounds like .SECONDARY prevent some features of Make from working correctly.
Jeez, you know, this is the sort of thing that I would hate to get somewhat deep into using make, only to then discover.
I often see hackers lamenting that a plethora of build tools exist instead of everyone just using make, but I'm not so sure that this state of affairs is less preferable. Maybe it's better if the build tool understands some things about the platform it's building for.
Have you read the Make pattern rules examples [0]? There's the specific example of yacc/bison doing something similar to as your describing (at least for 1:N case).
I've used SCons for processing audio files recently and it has worked pretty well for me; multiple targets are handled just fine.
I did have one percieved speed problem, but this scons calculating file checksums to see if it needed to re-build targets; this is usually a good thing, but if you're working with large files it might be better to use the MD5-timestamp Decider instead.
I usually create dummy targets for that kind of thing, and then include loops to create dependencies of each of the N on the dummy task. Alternatively, you can create a directory for the output and use that as a target, although you need to ensure the target directory only exists once the M:N task has finished.
I'm pretty sure everything you're suggesting here is a workaround that breaks the intended features of Make.
If a multiple-output step fails, I want the outputs to be deleted, and the rest of the dependencies to keep working so I don't have to rebuild everything from scratch. Whatever you're suggesting with dummy targets sounds like it's certainly not going to track the dependencies correctly.
I already know that directories aren't tracked correctly as dependencies.
make is a good idea gone bad. It's supposed to be dependency-based, but it has no mechanism for finding dependencies. It has to be told about them manually. So kludges have been written to find dependencies, but they have to overwrite makefiles. Its analysis of what has changed is purely based on time ordering, so it can get confused. This leads to too much "make clean". Although it wasn't intended as a macro processor, it's turned into one.
Then, of course, there's "./configure".
The alternatives tend to be bundled with some giant IDE, or are language-specific. The trend seems to be towards the latter; Go has "go", and rust has "cargo".
In my current very huge (takes hours to build) project at work there is a lot of voodoo around the amount of parallelism you can use. "make -j 4" seems OK, but "make -j 20" fails.
Of course, nobody wants to work on improving the build system.
The reason I love make so much isn't because it's a super reasonable way to build projects, it's because it has the best interface for building projects in a way that can apply to all languages.
I wrap lots of other build tools in make, so that the way things happen always follows
make setup && make build && make (deploy || install)
No matter the underlying tools. It just makes getting started with one of the many dozens of repos we worth with easier.
Incremental and parallel computations, like building software in a development environment, is better done using some kind of purely functional system, imho.
Nix & NixOS - http://nixos.org/
But, in practice it is used more on the level of packages and configuration than as a build system within a package. I remember reading an article demonstrating its use for building C code but I can't find it.
[blatant advertising] And also I have a new book on GNU make coming out this month: http://www.nostarch.com/gnumake if you really want to get into make deeply.