I know virtually nothing about SANs, other than that they are frighteningly expensive. Can anyone clarify how they compare to systems such as GFS, HDFS, MogileFS and other systems that attempt to store data in a safe, redundant manner across lots of cheap servers?
SANs aren't filesystems. They're block devices that run over the network instead of over a SCSI cable. They connect to servers by special HBA cards which look like SCSI devices, but actually run the SCSI protocol over the network to a rack full of disks. In a SAN configuration, the admins of those disks carve out a volume of storage for you, give it a name, and you configure it on your server; now it looks like you have a couple terabytes of storage directly connected. You format it like any other disk.
Backup works like any other system; you do it app layer. In high-volume high-sensitivity apps, the people running the disk server also back it up, by running protocols that mirror block-level changes to a second rack full of disks somewhere in Tennessee.
(yes, yes, or IDE or FC, yes yes, or software initiator, etc etc).
I'm not familiar with MogileFS, so if my comments below are incorrect when applied to MogileFS, then I'm sorry.
GFS/HDFS are officially called filesystems, but they're about as much as a filesystem as a Samba server: they expose files to the outside world, with some extra layers of abstraction you can mount the filesystems under Linux, but in essence they live on top of existing filesystems themselves and have nothing in common with the traditional filesystems, where you operate on a block device. A SAN, otoh, is such a block device and traditional filesystems operate on them.
This means that you can run a RDBMS on a SAN, while you can't on GFS/HDFS. This covers a common use case where you have a central expensive database server, with dozens of terrabytes of storage in a SAN, which you see in a lot of large corporations.
Cloud storage systems like S3 are often inferior in raw latency/response performance to a SAN which has VERY expensive adaptor silicon in it--but they are far superior on built in automatic, systematic, redundancy and global fail over and dispersal.
