

What filesystem to use? - jacquesm

I'm setting up a system to store hundreds of millions of files distributed across many servers. Total storage is several 10's of terabytes, output is tens of thousands of requests per second.<p>Modification rate is on the order of several hundreds of files added and or deleted per minute, deletions can be totally random.<p>The system will use varnish on top of apache to serve out the content via http.<p>What filesystem would you recommend to achieve these goals ? I've so far looked at XFS and EXT3 (this all runs on linux, redhat 4), with varying results.<p>A trial system has been built using XFS, but
I can not get it to perform well, even after
reading lots of tuning guides and tips.<p>Especially deletions are very slow, but archiving data is slow as well (4 Mb/sec is not unusual on an otherwise unloaded machine).<p>Pohmelfs looked promising but I could not get it to work reliably (and it would still need an underlying filesystem), ditto glusterfs.<p>Any information you could give on this subject to guide me in the right direction would be much appreciated.
======
blasdel
XFS is always dog-slow at deletions, and is pathologically slow[1] in the most
common use case -- "rm -R". It has a huge amount of synchronization overhead
for deletions, made much worse by the fact that rm is listing directories
while it deletes things, and XFS is trying to do these things in parallel.

If you build the list of files to delete up-front, it is almost normal.

[1] I have repeatedly run into cases where it takes much longer to delete
something than it does to write it, especially when your disk has bad latency
spikes.

------
jm4
Here is some good info on the advantages and disadvantages of various file
systems:

<http://www.debian-administration.org/articles/388>

I've had good results with JFS, but with much less data than what you're
dealing with. Chances are most information you find will not apply to such a
large amount of data so you're probably going to be stuck doing your own
benchmarks.

------
blasdel
You have an advantage with a distributed system -- it's totally feasible to
migrate between filesystems / tunings after you get started.

I highly recommend using netboot and an NFS root filesystem for the storage
nodes.

RHEL4 is going to be a thorn in your side if you ever want to try ext4 or
btrfs. Even if you stick with the mature filesystems, VFS improvements in
newer kernels will give you a performance boost.

