
Show HN: Fast `git status` (git core.fsmonitor in Rust) - jgavris
https://github.com/jgavris/rs-git-fsmonitor
======
avar
If you're optimizing for speed use BSER instead of JSON to talk to watchman:
[https://facebook.github.io/watchman/docs/bser.html](https://facebook.github.io/watchman/docs/bser.html)

~~~
jgavris
Fantastic suggestion! I will give this a go soon.

[https://github.com/facebook/watchman/tree/master/rust/serde_...](https://github.com/facebook/watchman/tree/master/rust/serde_bser)

------
sigi45
At first i thought i don't understand what your little rust code is doing,
than i got it.

Its a watchman wrapper written in rust.

But in that article you linked, there is mentioned that git ships with a
sample watchman hook already?!

~~~
jgavris
It is not enabled by default yet, since it depends on an external tool to
provide repo file update events. Git has one bundled that uses Facebook's
watchman (with homebrew if you've installed it like so), but you have to
configure each repo (or globally) to enable it. I was just curious how much
faster a Rust implementation could be over a Perl script, and wanted to learn
a bit of Rust.

~~~
Phrodo_00
I wonder how much faster it would be to use the OS' inotify equivalent (if
Linux, inotify itself) directly from rust rather than using an external tool.

------
benpeart
Very cool! I wrote the original (with help) in Perl as it has the advantage of
being installed with git but I love the speed advantage of your Rust
implementation. Also helps shows the advantage of making core.fsmonitor a
pluggable interface.

------
pitaj
I wonder what kind of gains could be seen by using a fully rust-native bundle
with something like notify [1]

[1]: [https://crates.io/crates/notify](https://crates.io/crates/notify)

~~~
avar
This is not at all comparable to what the OP submitted and would be at least a
many thousands of lines worth of effort.

What watchman is doing is doing inotify-as-a-service for you, in order for any
program to use the Linux inotify API it needs to be running constantly so it
can keep consuming events from the kernel, and in the case of the use case
watchman fulfills, maintain an internal database of how the current state
differs from various times in the past.

There's lots of very nasty edge cases you need to deal with. E.g. if you don't
consume the kernel events fast enough (or the buffer is too small) new events
will be dropped.

You'll get told about this by the kernel, but now the only way to re-build
your in-memory representation to match reality is to recursively walk the FS
with readdir(), stat() etc, but _at the same time_ consume new inotify events
and update your in-memory structures while doing both.

Needless to say this is a lot more complex than just using watchman off the
shelf.

It's also pointless to optimize this area, watchman usually returns in 2-3
milliseconds even on huge trees, but because of bugs / misfeatures in the
current git-side implementation it can take hundreds of milliseconds until we
make sense of all of that. This is because git doesn't really "know" about
watchman/inotify, it just uses it as a replacement for walking the FS, but on
big repositories other stuff can still be expensive.

To OP: I highly encourage you to submit this to the Git mailing list for
comment. I've been involved in this area (helped write the Perl hook) and
there's people who'd be interested in having this be e.g. a part of git's
contrib/ directory so it can be compiled with git.

~~~
jgavris
Thanks for the explanation, indeed watchman does all the heavy lifting! I will
submit it to the mailing list, I'd love to see it in git core.

------
ufmace
Cool idea, but I tried it on one of my team's bigger repos, and it's actually
slower than stock.

------
mwean
What about using entr[1] instead? I've had great success using it for
rebuilding ctags on file changes.

[1]: [https://github.com/clibs/entr](https://github.com/clibs/entr)

~~~
jgavris
I have a 'git aware' bash prompt, so my primary use case is making sure my
prompt is ready with the git status _really fast_.

------
bluejekyll
How does this differ from the standard fsmonitor?

This appears to mainly be a lightweight wrapper of watchman.

~~~
jgavris
In my benchmarks it's about twice as fast as the reference Perl script
monitor. Also just a quick exploration for me into Rust.

[https://github.com/git/git/blob/v2.16.0/templates/hooks--
fsm...](https://github.com/git/git/blob/v2.16.0/templates/hooks--fsmonitor-
watchman.sample)

~~~
glandium
You should make it clearer that the benchmark is perl fsmonitor vs. rust
fsmonitor. Without reading the comments here, I would have thought it was
without fsmonitor vs. with.

~~~
jgavris
Good suggestion! Without the fsmonitor enabled at all that looks more like
400ms on my iMac Pro.

