Redo lacks features baked into Drake, especially the Hadoop integration, but I believe it would be easier to incorporate custom functionality into redo versus hacking Make or writing a custom build system. I haven't used Drake, so I would be interested in a small but complicated Drake script which tackles an intractable problem in Make. I don't claim redo can provide a cleaner solution than a purpose-built system, but I think it will be unexpectedly simple.
The most crucial thing that Make lacks is multiple outputs and precise control over execution. When you're debugging/developing a large and expensive workflow, you absolutely must have the ability to say things like:
- run only this step, I'm debugging it
- I've changed implementation of this step, re-build it and everything that depends on it
- build everything except this branch, it's expensive and I don't need to rebuild it that often (example: model training)
Other examples of intractable problems in Make would be timestamped dependency resolution between local and HDFS files. If Make can't look at HDFS, it can't say if the step needs to be built or not. I don't think you can fix it with external commands.
But generally, search for intractable problems is a futile one. Remember, everything you can code in Java, you can code in a Turing machine. :)
So make's default behaviour "make somefile.csv" is to build the whole tree of dependencies. To force rebuild of everything, run "make -B somefile.csv". It then assumes everything is out of date.
To force rebuild of one step, just delete its output or run "touch" on one of its dependencies before running make. Then that step will get redone.
I like to have generated data in a separate folder, say "output/" which you can then snapshot, blow away, or do what you like with. Basically though, I keep it separate from data and code inputs.
Thanks! This much I know. But it doesn't answer my question. Let me repeat it: could you please give me a command to re-build a particular target and everything that depends on it?
Aboytsov wants to rebuild the target and everything that depends on the target, not rebuild the target and everything that the target depends on. He wants to walk the dependency tree in the opposite direction.
No, make -B mytarget rebuilds either mytarget only or mytarget and everything mytarget depends on. A more common scenario is when you need to rebuild mytarget and everything that depends on it. Without rebuilding other parts of the workflow that you don't need.
This is a really weird request. make won't rebuild things that haven't changed, so the default make all rule will only rebuild the things depending on mytarget. Every time you change mytarget, just run make (all) and everything that depends on mytarget (and only those things) will be rebuilt.
This is not a weird request, this is one of the most common things we do when we're developing a workflow. You need to do this every time you make changes to code and you want these changes to propagate.
You can't run "make all", because it literally builds everything. You might be working on a specific branch of the workflow, and the overall workflow could be huge. And out-of-date in a lot of places. Or it could contain steps that are very expensive, but not necessary to build for your development purposes (for example, generating a model). This is why exclusions are also important, and make also does not support them.
Make also does not support multiple outputs, and I gave you a prooflink before. And a lot of other things which we think are important, too (I could make a list. I did, actually).
If you like Make, you should continue using it. I think it is a little arrogant on your part to try to explain to us that we simply wasted our time. We built the tool to address the problems we were facing. If you do not face similar problems, by all means, use Make.
Sorry, I didn't mean to imply "you're doing it wrong". Didn't even realize you made the tool. Oops. Personally, if large chunks of my output are out of date, I don't like the idea of commingling them with new stuff, but obviously I don't know a whole lot about what you're doing.
Now imagine you're not the only one working on it. You may have even never run it in its entirety, since it takes 10 hours. Imagine there's a branch which you, a developer, is currently working on. This branch depends on some other files in the workflow. Let's say, generate synonyms from the sentence dataset. Or, some complicated cleaning of some intermediate data. This is not a small task and you will spend a couple of days doing it, re-running your code dozens of times in the process.
You don't care about other parts of the workflow. You only care about what you're developing and how it propagates. Does it propagate? Does it break something down the road? What is the final output? Did all this synonym collection help? Did the changes you made in learning code improve the results?
When you're done, you may commit your code and somewhere else somebody will build a nice new dataset, but while you're working on it, you really need to be able to run any target individually, with dependencies or without, as well as forcibly rebuild all steps down the tree to see the final result.
This is basically the case where you don't (or it's infeasible to) capture the dependences fully, so you want to rebuild everything from target onwards after some change.
Haha, well pointed out. I'm clearly having trouble parsing today.
"rm target; make" can work, but only if you're using a pattern for data pipelines where there is only one default set of downstream targets. If the one Makefile supports a range of downstream targets, then this won't work.
I concede, make doesn't support that operation out of the box :)
