
What to do about NFS and SIGBUS - deafcalculus
https://rachelbythebay.com/w/2018/03/18/nfs/
======
drewg123
Back in the 90s, before package managers for Linux/BSD and when disk drives
were fairly pricey, it was common to have say, a network of commercial *nix
workstations with nfs mounted binaries for stuff like gcc, latex, emacs, X
utilities, etc.

This was typically done by a network of symlinks. As best I can remember,
you'd have /opt/local exported from the server, with all the open source
goodies. It was arranged something like

/opt/local/emacs-19.1.7/bin/emacs

/opt/local/emacs-19.1.8/bin/emacs

/opt/local/emacs -> /opt/local/emacs-19.1.7

<....>

All the clients would have a local /usr/local with another symlink,
/usr/local/bin/emacs pointing at /opt/local/emacs/bin/emacs

When you upgraded from 19.1.7 to 19.1.8, you'd just move the symlink on the
server. You'd then wait a week or two, and garbage collect the old version
(after rsh'ing around to the clients to make sure none had the old version
open, if your users were lucky).

The local network of symlinks into /opt/local was maintained by something like
cfengine, which ran at install and nightly to update configs (eg, for newly
supported stuff, or new binaries in new versions of existing software).

If you screwed this up, and replaced a binary out from under a user, they were
rather unhappy..

~~~
js2
There was actually a tool for managing such symlink trees, “depot” from CMU:

[https://www.cs.cmu.edu/~help/unix_linux/software_collections...](https://www.cs.cmu.edu/~help/unix_linux/software_collections/local_depot.html)

~~~
predakanga
Interesting - I wonder if that inspired GNU Stow (similar tool I often see
used for managing dotfiles)

~~~
js2
The CMU depot was presented at LISA '93:

[http://static.usenix.org/publications/library/proceedings/li...](http://static.usenix.org/publications/library/proceedings/lisa93/full_papers/wong.pdf)

Stow apparently started development in 1996 and was initially named depot:

[http://git.savannah.gnu.org/cgit/stow.git/tree/doc/ChangeLog...](http://git.savannah.gnu.org/cgit/stow.git/tree/doc/ChangeLog.OLD)

So probably?

------
notacoward
As a developer and operator of a distributed file system, and specifically one
that might have caused Rachel some pain, I kind of agree with her. If you can
run with local code and data, do so. Even if you think you can't, think again.
Only run from data on the servers if you have no choice, e.g. if the dataset
is too large to copy or you need simultaneous access (better be read-only by
that point) from multiple clients. Not only will you be happier, but so will
the people who have to operate that distributed filesystem in the face of many
users all abusing the hell out of every operation in the POSIX/VFS API.

------
greglindahl
Rachel's advice, "Kick NFS to the curb", is good. My last startup, blekko
(2007-now), managed to have no NFS usage, ever. Didn't miss it. Peak of 50
devs, 1500 servers, no NFS.

Even before that, in a supercomputing context (2002-), when writing MPI
"wrapper" software that would be used on end-user clusters with all kinds of
unfortunate network setups, I avoided a lot of trouble by copying the
executable to the local node before running it.

This isn't a new issue. And the traditional solutions work.

~~~
lmm
I've found that even for unix-unix, samba is just more reliable with more
reasonable error behaviour.

~~~
WHoWHo
To me samba has just too much magic, especially when mixing unix, OSX and
windows clients.

~~~
pnutjam
magic is just technology we don't understand. I use cifs over nfs as well.

~~~
WHoWHo
With excel locking, os x trashes, etc.? What good configuration guide would
you suggest for a mixed Linux, OS X and Windows environment with MS Office?

This magic only solves have the problems

    
    
        fruit:resource = xattr
        fruit:metadata = netatalk
        fruit:locking = none
        fruit:encoding = private
        delete veto files = yes
        veto files = /.snapshot/.windows/.mac/.TemporaryItems/.DS_Store

------
mattbillenstein
rsync, ln -s, mv -T always worked for me -- NFS always presents so many
problems, I almost never deploy it anymore given copies of everything are so
cheap via various cloud storage solutions.

------
peterwwillis
NFS has plenty of good uses. Maildir works fine, sharing large datasets works
fine, sharing static assets works fine, it's widely supported. No need to
throw the baby out with the bathwater. Just stop putting the baby in boiling
water.

NFS is a way to share files, not provide a reliable application platform.
There are lots of options you can enable to make it more stable, but it simply
isn't designed as an application platform, so it's all going to be hacks.

And keep in mind that NFS is not secure. Supposedly it can be secured, but I
have never seen this in a production environment. Running an application over
it is just a bad idea from a security standpoint. Home directories over it are
a terrible idea from both a security and stability standpoint.

Finally, don't try to change a file and expect stability. If you need a new
version of a file with some change to it, make a new file and use that new
file path.

~~~
xenadu02
Except CIFS proves it is possible to have a network file protocol that isn’t
quite such hot garbage

~~~
peterwwillis
I'm not saying there aren't good network filesystems, I'm just saying that
while NFS is hot garbage, it is totally useful hot garbage when used in the
right way.

------
al2o3cr
My favorite NFS story: many many years ago (mid 90s) I was part of the staff
for a physics department's IT group. At the time, affordable Alpha
workstations had just started appearing on the market, while most users still
relied on "semi-dumb" X Terminals/thin clients. Everything was attached to our
10Base2 network, capable of a BLAZING 10Mb/s.

We started seeing weird whole-display freezes on bunches of the X Terminals
one day. It took a while to correlate that these happened at the same time one
of the newly-installed Alphas in a far-off office was doing a lot of file I/O
over NFS - saturating the entire coax segment and freezing the displays...

IIRC the "upgrade to 100BaseT" project got scooted WAY up on the priority list
right after that. :)

------
aargh_aargh
Previous post for context:
[https://rachelbythebay.com/w/2018/03/15/core/](https://rachelbythebay.com/w/2018/03/15/core/)

