

Watchman: Faster builds with large source trees - patangay
https://www.facebook.com/notes/facebook-engineering/watchman-faster-builds-with-large-source-trees/10151457195103920

======
evmar
If I'm reading that graph right, the red line is at 10kmsec, i.e. 10 seconds?

Incremental build times are near and dear to my heart; I spent a lot of time
making the Chrome incremental build fast, resulting in this tool:
<http://martine.github.io/ninja/> . In developing Ninja I was surprised to
discover that Linux stat() with a warm disk cache is very fast -- well under
100ms to stat the ~40k source files Chrome uses in its build (see the "node
stat" lines here: <https://github.com/martine/ninja/wiki/Timing-Numbers> ). At
its best point I think we got the one-file-changed build/compile/link cycle of
Chrome (a ~70mb C++ binary) to around 5 seconds.

Of course, Facebook's problem is surely very different -- their scale could be
many more files, and perhaps the programs their engineers run while developing
cause their disk caches to flush more frequently. Just found it interesting to
worry about the cost of stats.

~~~
groby_b
And of course, there's OSX. See the 58s cold build stat for ninja/Chrome on
OSX. And the fact that OSX's cache gets cold whenever you sneeze.

------
norswap
What is the difference with guard (<https://github.com/guard/guard>), except
the "service" aspect?

------
yid
Not sure how this is different from inotify+md5+make?

~~~
_mpu
Not sure how inotify is different from while+stat?

~~~
yid
Your snark is unwarranted. While+stat in a loop is a poll loop in userspace,
inotify is a push mechanism in kernel space.

My point was that the additional functionality of this significantly-sized
package beyond running inotify+md5+make in a shell script was unclear.

------
hildegard
I was just looking into inotify to reload nginx and various other services on
conf changes. I guess I'll go with Watchman so I don't have to write a series
of shell scripts.

I considered guard (<https://github.com/guard/guard>) and Nodemon
(<https://github.com/remy/nodemon>), but Watchman has less dependencies
(doesn't require Ruby/Node).

There's also Supervisor (Python) (<http://supervisord.org/>), but I think that
is more process management. I'm not sure if it can do file-watching as simply
as Watchman.

~~~
Game_Ender
Not directly applicable but if you are using Salt to manage your servers you
set it up to restart a service when certain files change with the watch
requisite [1].

[1] -
[http://docs.saltstack.com/ref/states/requisites.html#require...](http://docs.saltstack.com/ref/states/requisites.html#require-
in)

------
rejuvenile
The tup build system also does this.

------
thelarry
Compiling while editing. Intredasting.... I assume there is some crazy
dependency algorithm.

~~~
kevingadd
Traditional build systems are all dependency algorithms. The interesting thing
here is that they run a service to track file modifications so it's not
necessary to stat() the entire hard drive to find out what has changed, so you
can do incremental rebuilds very cheaply.

------
b0b0b0b
This reminds me of Eclipse's support for native hooks to detect file updates
(in case something was changed from outside the eclipse session). Watchman
would then be like a deconstructed ide.

------
livingparadox
I read their problem statement, describing the features they needed. Seemed
like Git could solve their problems, with a few well written hooks.

~~~
raylu
Only if you want to commit on every iteration.

