

No more `grunt watch` – faster builds with the Broccoli asset pipeline - joliss
http://www.solitr.com/blog/2014/02/broccoli-first-release/index.html

======
munificent
This is slightly related and I don't want to sound like I'm trying steal its
thunder, because this looks really cool. I work on the asset pipeline that
comes with the Dart SDK. It has many of the same principles as these.

Any transformation step can read in many input files and produce many output
files. The built-in dev server tracks the entire asset dependency graph and
only rebuilds the assets that are dirtied by a source file changing.

We have a plug-in system, and it's built on top of the same package management
system that the SDK uses, so you can get transformer plug-ins as easily as you
can get any other dependency.

We still have a lot of work to do to fully flesh things out, but it already
does a lot, including supporting complex scenarios like transformers whose own
code is the output of a previous transformer.

More here: [https://www.dartlang.org/tools/pub/assets-and-
transformers.h...](https://www.dartlang.org/tools/pub/assets-and-
transformers.html)

~~~
pekk
Why can't Dart reuse (or provide) a tool that also works for things other than
Dart?

Why does every language need to have its own everything unique to itself?

~~~
munificent
This is a great question. There are two main reasons:

1\. The asset build system is built on top of a bunch of policies and
conventions specific to our language's package manager. The build system needs
to be able to locate plug-ins somehow, and it's hard to define that "somehow"
without some kind of assumptions for how dependencies are located and
organized.

2\. Like other platforms, we want to minimize external dependencies. If we
reuse grunt, then every user has to have node and npm installed. There's
nothing wrong with that, of course, and many users do already, but we'd like
to avoid _forcing_ that dependency.

3\. Since the build system is plug-in based, we need a way for the build tool
and plug-ins to communicate with each other. We want that API to be simple so
that it's easy to create plug-ins. Since they're sending entire assets through
that communication channel, we need it to be efficient too. It's hard to do
that in a language-agnostic way.

4\. We want to make it as easy as possible for people to contribute to the
build system and various build plug-ins. Since we already assume users know
Dart, then when those components are also written in Dart, it increases the
chances that they are able to pitch in and help.

I don't like walled gardens or feeling like we're reinventing the wheel
(though I'll note that before this announcement, I wasn't aware of many other
build systems that were as many-to-many based as ours), but there is value in
having things be internally consistent.

------
JangoSteve
_Broccoli is a new build tool. It’s comparable to the Rails asset pipeline in
scope, though it runs on Node and is backend-agnostic._

This first line is a little disingenuous. Technically, it's not backend-
agnostic, since it depends on Node being installed on the backend (in the same
way that Sprockets [1] depends on Ruby). The Rails asset pipeline is a
framework-specific integration of Sprockets. In much the same way, you could
more closely integrate Broccoli with Rails if you wanted and call it a new
Rails asset pipeline.

The project itself looks great, just the first line was confusing since they
started the docs off by comparing apples to oranges.

A better comparison would probably be, "It's comparable to Sprockets (which
powers the Rails asset pipeline), but runs on Node instead of Ruby."

[1]
[https://github.com/sstephenson/sprockets](https://github.com/sstephenson/sprockets)

~~~
JangoSteve
Someone commented below (but their comment is dead, so I can't respond
directly):

 _The Rails asset pipeline requires a JS runtime, so Broccoli doesn 't have
any more dependencies than asset pipeline in my mind._

Technically they're right that the Rails asset pipeline does require a JS
runtime by default. But it's important to note that the Rails asset pipeline
doesn't strictly require the JS runtime; it's only required for the coffee-
script gem, which happens to be included by default for new Rails apps. If
you're not using coffee-script for any of your assets, then no, it doesn't
require a JS runtime.

~~~
Wintamute
But isn't Broccoli more of a frontend build tool? The primary motivation seems
to be to speed up frontend development in terms of the save-file-build-wait-
view-in-browser loop? I don't know much (anything) about the Rails asset
pipeline, but isn't it used by the actual backend framework? As far as I can
tell Broccoli is used to compile assets before the frontend files get anywhere
near a backend ... so in this way its backend agnostic.

~~~
byroot
Sprockets _is_ a frontend build tool. It works basically the same way.

The main difference between sprockets and grunt / lineman / broccoli, is that
sprockets do not watch files. It's a web application that compile your assets
on the fly[0][1] when you need them.

So it's lazy, and blocking. Meaning that you do not have to guess if your
assets have been recompiled yet. Your browser just ask for them, and get them
as soon as they are ready.

[0] In development mode of course [1] In fact it cache the output, but
whatever.

~~~
JangoSteve
Bingo. But keep in mind that Sprockets doesn't have to be used just as a Rack
server to compile on the fly, it can also be used to precompile to static
files. In fact, you could then combine it with something like the watcher gem,
and you now have automatic precompiling whenever you save your source files.
In fact, there's the guard-sprockets gem which does all this for you [1].

[1] [https://github.com/pferdefleisch/guard-
sprockets](https://github.com/pferdefleisch/guard-sprockets)

------
Nemcue
Not often I see tools announced with such a thoroughly researched article.
Great stuff!

I guess the problem these build tools are facing is the amount that people
have invested in Grunt. There are just /so/ many grunt tasks at this point.

~~~
_puk
I've also been playing around with gulp, and whilst still in the 'trying to
use it like grunt' kind of mindset, I have seen some merit in using these
stream based build tools.

Have to wonder why this all requires new build tools to achieve though.

It's obviously technically possible (npm shows grunt-gulp which does exactly
what you'd think), so is the grunt architecture so firmly rooted in files that
streams could not be leveraged, or is it something that will come in time?

Personally managed to get a nigh on perfect build process using grunt-watch,
so no rush to move away.

Nice to see build tools still moving forward though.

~~~
AdrianRossouw
grunt takes an almost entirely declarative approach, which becomes really hard
to reconcile with situations that the order of execution really matters.
because you are basically trying to map a flat(ish) array of commands onto a
tree-like structure to trick the grunt internals to work the way you need them
to.

You can either do this explicitly (by setting up aliased), or implicitly (by
all the temporary file watch jiggery and pokery).

It's unlikely that I will start any new projects with grunt, even though I
know it a lot better than gulp right now.

~~~
iamstef
ya, grunt tends to suffer from an massive explosion of complexity

------
iamstef
Having used literally ever alternative, Broccoli has been a joy to use so far,
can wait to port all my projects to it.

It manages complexity really well. I have thrown many known failure scenarios
at it, and it handled them all without a hitch.

~~~
filearts
As the author mentions, the real insight seems to be to switch from the file-
based unit to the tree-based unit. This means asset building can be made more
intelligent.

Now, I wonder if the same approach can be taken in gulp using vinyl-fs?
Otherwise, I wonder if it might be worth plugging this tool as a specialized
asset pipeline into existing grunt/gulp files.

~~~
pmllr
I have been working on replacing the whole production build pipeline as well.
And since Grunt isn't a thing you can ignore these days I created a simple
grunt wrapper that collapses the entire 200+ lines of config down to two
config options: [https://github.com/Munter/grunt-
reduce](https://github.com/Munter/grunt-reduce)

------
mikewhy
Not sure what the pros are against Brunch. The author states:

> Brunch also tries to do partial rebuilding rather than caching; see section
> “Caching, Not Partial Rebuilding”

But the end of that section seems to imply that it's still something that
needs to be implemented per-plugin:

> Plugins that map files n:1, like Sass, need to be more careful about
> invalidating their caches, so they need to provide custom caching logic.

I'm excited to see what comes of it, but still prefer the idea of simply
running `npm install (sass|jade|less|stylus|coffee-script)-brunch --save`

------
matteodepalo
I'm happy to see this reach beta version, it's a great step in the right
direction. Grunt is too generic as a tool and we've all seen Gruntfiles reach
enormous lengths, to a point when it's really hard to figure out what is
processing what.

One thing that has room for improvement though is the syntax, which in my
opinion doesn't reveal the intention behind some methods and is a bit too
coupled with the implementation. What does `makeTree('lib')` mean? If it's
taking a folder and its files then why not rename it to something like
`broccoli.requireFolder('lib')`? Also another thing that might improve
usability would be chaining compilers instead of calling them directly with
the tree as parameter.

These are just minor things anyway, I'm sure the library will improve over
time. Congrats joliss, great fan of your work!

------
clhodapp
I want to note that this headline is a wonderfully hilarious if read without
js-programmer context.

~~~
krsunny
Broccoli, the laxative your JS needs.

------
jonaldomo
I would like to request renaming brocollifile.js to brocolli.js. I believe
brocollifile.js is too long for a standard build file name. Gruntfile.js
always bothered me. Compare it to pom.xml, build.xml, package.json and it
feels out of place.

~~~
joliss
I named it after the `Makefile` pattern, since it contains a build definition.
We have `Rakefile`, `Cakefile`, `Gruntfile.js`, etc.

~~~
applecore
How about "Brocfile.js"?

------
sonnym
This looks like a solid project.

I want to mention mincer[1], which I have used in the past for compiling
assets, and it has been an entirely painless process. Definitely take a look
at it as an alternative, which has been around longer and has seen assistance
from the folks behind sprockets[2] (according to the README) for creating a
similar API.

1\. [https://github.com/nodeca/mincer](https://github.com/nodeca/mincer) 2\.
[https://github.com/sstephenson/sprockets](https://github.com/sstephenson/sprockets)

------
ChikkaChiChi
There are at least 5 separate build tools referenced in these comments.

Obligatory XKCD: [https://xkcd.com/927/](https://xkcd.com/927/)

------
Natsu
The fun part is when your non-technical coworkers ask what you're reading so
intently and you read something like this to them with no context: "Run
broccoli serve to watch the source files and continuously serve the build
output on localhost. Broccoli is optimized to make broccoli serve as fast as
possible, so you should never experience rebuild pauses."

------
Kiro
What's the difference between a build tool and a task runner?

~~~
machty
Tasks are a more general concept; one of the many possible tasks you could
program would be to build your project.

~~~
roryokane
Other possible tasks including initializing the database (for a new
developer), regenerating some artificat that isn’t normally regenerated during
the build, and launching a local web server running the project.

------
SippinLean
Can someone please build a Grunt GUI that I can drag-and-drop project folder
to, to watch multiple folders?

Prepros is the closest software out there now, but it's not extensible like
Grunt.

------
malandrew
Is this a beta build system like tup? Sounds like it. I ask because I've
recently begun work on the same.

------
jakswa
Anyone know of any resources for writing plugins?

~~~
grayrest
Read the source for the ones there. The ES6 module plugin is the most
involved.

------
stefan_kendall
Yet ANOTHER build tool. I've started placing bets on when repositories will
flip to NEW-HOTNESS-BUILD-TOOL at the cost of actual product development time.

Engineers will constantly run toward shiny baubles at the expense of
everything else.

