Pardon the plug, but I thought I'd point out that I think SciCommander [1], or the ideas behind it, could be very relevant for this stack.
It does capture a provenance graph for any ad-hoc shell commands executed in the shell mode, or if prepended by `sci run`, by tracing all new files created from commands, and creating an accompanying json file for every output with metadata, which can later be assembled into a graph specific to any output file, using the `sci tohtml` or `sci toshell` commands (producing an HTML report with an SVG graph, or a reproducing shell script, respecively).
I'm quite bullish of the possibilities with this approach.
And, this was in fact also born out of the thinking to "rip apart" workflow tools, and build them up again using small, well-defined tools that do one thing well.
It does capture a provenance graph for any ad-hoc shell commands executed in the shell mode, or if prepended by `sci run`, by tracing all new files created from commands, and creating an accompanying json file for every output with metadata, which can later be assembled into a graph specific to any output file, using the `sci tohtml` or `sci toshell` commands (producing an HTML report with an SVG graph, or a reproducing shell script, respecively).
I'm quite bullish of the possibilities with this approach.
And, this was in fact also born out of the thinking to "rip apart" workflow tools, and build them up again using small, well-defined tools that do one thing well.
[1] https://github.com/samuell/scicommander