I keep seeing people mention "a mess of shell scripts", and then use replacements which are dozens if not hundreds of lines of custom-configuration files.
Shell scripts, when you're not dealing with autotools generated script, aren't really that big, or even that complicated.
Not to mention that the ability to chain disparate tools and subshells can be incredibly elegant and powerful.
I agree mostly. But when you're in the node ecosystem, often times you need to run JS as part of the build. For example, you might need to create some JSON data as part of the build.
Windows offers shell scripts as well. Powershell, instead of Bash/Zsh/..., but they still offer many of the same opportunities.
Considering that teams frequently homogenize on one type of machine - bet it windows or Mac or Linux, shell scripts are typically portable enough for these use cases.
Indeed. For JS projects these days I just use npm as the command-line tool. npm scripts gives you an easy way to create commands for building, watching, etc.
...I would like the task system to provide an abstraction for pipelining tasks. (Perhaps gulp helps with this?)
Hmmm, perhaps someone needs to look at any gulpfile ever written, anywhere.
gulp.src('src')
.pipe(this())
.pipe(that());
Maybe it's frustrating for anyone who spent "18 months" hacking grunt, but gulp really is better. Both tools are built on node for Pete's sake; async really shouldn't be that hard.
Yes while gulp > grunt, in many cases npm by itself is what you want. gulp.watch(), in particular, seems more difficult to use than the watch facility built into many (but not all) node modules. Also something like Browserify must be somewhat shoehorned into gulp. It might be worth it to look at what gulp has that npm run is missing? Maybe it doesn't need to worry about streaming files, but the notion of some tasks depending on arbitrary sets of other tasks (a generalization of npm's "pre-" and "post-") seems pretty handy.
How are all of you using any npm-based system combined with a continuous integration or clean build environment? It seems that a simple yeoman/grunt setup that just preprocesses some HTML and JS takes... a very long time. A minute or two just for the actual work. Plus 5-15 minutes for npm to make a thousand requests and write 10K+ files to the filesystem.
I hack around it by keeping node_modules around as a special resource on a custom build server and symlinking it in, but that wouldn't work with a proper "start from zero" build system.
1) We have our own private npm that can optionally include a caching layer. This has been invaluable since the beginning of our company to manage internal dependencies.
2) We use StriderCD, the self-hosted CD/CI, and it too supports caching node_modules. 2 minutes is a 'long' time for us from git push to autodeploy.
3) We use gulp and npm always. Grunt and bower are a mess.
EDIT: Also, we npm shinkwrap projects. It avoids numerous issues and makes module caching a natural part of the process.
grunt has its issues, but you are correct that bower is truly a mess. "Let's do one of the many jobs that npm (the tool we use to install bower) already does, but not as well!" No thanks.
> It seems that a simple yeoman/grunt setup that just preprocesses some HTML and JS takes... a very long time.
I have not experienced this. I built a 15k+ line CoffeeScript app. Every time a file changed it would compile it to JS and run thousands of unit tests in ~1 second.
As for the clean NPM... I never deleted that directory. I believe I read an article somewhere that you should actually commit your module dir. Maybe that's no longer in vogue though?
Why does it matter if your build system takes a while? You shouldn't be pushing it to there until you are happy it works locally, at which point you can move on to the next task and forget about it.
Yeah, I guess you can get make to do the exact same things as gulp, but gulp already does what a web developer needs, in a language they already know, with a massive plugin ecosystem you don't have to re-implement in shell.
Yes, I acknowledge in the article that Grunt is not designed for this, and if you use Grunt for what it is designed for, it works pretty well. However, it wasn't obvious to me the using Grunt to build an SDK was outside of its scope when I started the project, so I figured it may not be obvious to other people as well.
I started with grunt, then moved to gulp because I like the plugin support more and it felt less magical to me.
Then, I found that working with many different projects, gulp wasn't DRY enough for me. I kept copying any pasting my tasks across projects and forgetting to go back and improve them as I went. I also disliked the chain of require statements and devDependencies in my package.json. I was spending too much time coding my build.
So, I started a project [1] to move gulp tasks into re-usable functions and hide the devDependencies. Now I'm pretty happy with the result, my gulpfile [2] is now mostly configuration and very little code.
The inability to specify config asynchronously turned into a disaster. Our primary need for async config was specifying temp directories for tasks to operate in. We used tmp, which does not have a synchronous api ...... I would like the task system to provide an abstraction for pipelining tasks
Is this abstraction not normally just called a program? Where you can do something like:
val tmp = makeTempDir()
generateFoo(outputDir = tmp)
generateBar(inputDir = tmp)
etc?
How much of this complexity is fundamental and how much is people attempting to work around Node's inherent limitations and fully asynchronous APIs? Perhaps the author should check out something like Gradle instead?
I probably just don't have enough experience. I like the idea that Grunt and Gulp are written in JS so using JS to build JS seems like a win.
But...., aren't build systems supposed to do dependency checking and just built the minimal amount of stuff? Maybe few JS projects are big enough for it to matter or maybe I missed where the dependencies are checked but AFAICT the default is to build everything always.
Grunt doesn't do dirty-file dependency checking. Gulp does, sort of, but it's awkward.
There are many, many build tools in JS-land. Grunt and Gulp happen to be the most popular. (This isn't really due to technical merit. They're popular mostly because they're popular.) My personal favorite is Jake, which is a classic Rake-like (or Make-like) tool.
Your grunt or glup file can run a command to check for those dependency and update it via bower and npm.
Bower are for front end dependency check and NPM are for the backend dependency check.
Grunt and Glup are more of a task management system where you run tasks. Such as transpiling your jade/haml/sass, minifying your js, running your test cases. You can run these tasks in specific orders.
There's also gruntfile too but it's been a while since I've use these things. I think gruntfile are for grunt specific module for tasks though, so it'll dependency check grunt modules?
I'm a recovering frontend guy, well at least I did front end in a few gigs that were suppose to be fullstack.
greggman is talking about dependencies between build targets (such as files or directories or projects), eg: "lib.js depends on lib.coffee" + "lib.coffee's last modified time > lib.js's last modified time" = "must run task that regenerates lib.js"
First the build tool needs to load the build files that declare the dependencies, so it knows which dependencies to check. For big, distributed projects, this step can be slow.
I have had similar problems with my team, and I went with the "meta-project" approach. I created an external project with all default configs[1]. It works well-ish. Main complaint is speed, grunt really is dumb to not lazy load...
I'm a bit confused, what are they using grunt for? We're using gulp, but it's basically the same software. It just compiles our coffeescript, tags some assets and moves some files around, perhaps run a minifier somewhere.
In what situation do you need your build toolchain to do more than that?
I think I found myself in a similar situation as the OP. We have multiple different, but architecturally similar, products which each get their own Gruntfile. If we stop here, we're solid, perfect. Unfortunately, each product has multiple flavors which each need to build into their own separate output directories. For us, that's where the Grunt patterns break down and get awkward. It's actually been good enough for almost 2 years now, but I can definitely see the cracks.
Yes, that sounds very much like the scenario we see ourselves in. For us, we have too many projects for copy/pasting the Gruntfile between them to be a viable option, and the Gruntfiles themselves would have been too complex.
I'm not convinced the shell scripts are a good alternative, though. (Maybe they are for your scenario.) I would like to write a follow-up post talking about this.
I'm willing to believe the first two comparisons even though I haven't used webpack, but I have tried to use make on multiple occasions over the course of decades. (That is, I've attempted to write new Makefiles and to modify existing ones to fit a purpose; I'm not talking about just using a Makefile someone else already wrote.) make is not nice. I admire anyone who can use it!
To be specific, what advantages does webpack have over gulp that webpack doesn't also have over make? (I would also accept advantages that gulp has over grunt that gulp doesn't also have over make.)
Shell scripts, when you're not dealing with autotools generated script, aren't really that big, or even that complicated.
Not to mention that the ability to chain disparate tools and subshells can be incredibly elegant and powerful.