I worked for several years in VFX/HPC. 30k+ cpus and 15pbs of storage.
Firstly with storage its very rare that people want actual block storage (unless you are hosting VMs, but thats so 2007.....) Yes, I know, openstack, but that's just fucking horrific, seriously just use netboot and be done with it. I've seen people do it inside new clustereing systems, but its really not fun to do, especially if you consumer is prone to disappearing without warning. (FSCK is a terrible mechanism for fast recovery)
Most apps, unless they have bought into the "shove everything over HTTP and pay the penalty", want a posix file system to store anything of importance. (yes, yes database, but where is that writing the data to?)
Now, there are three ways you can do this:
o use a clustered file system
o Use NFS (with or without a clustered filesystem underneath)
o Fuck about with iscsi/SAS/FC and dynamically map block dynamically.
Using a clustered filesystem spread over many clients is begging for trouble, mainly because one client can fuck it up for everyone. Some FSs are dynamic and sexy, but they have a habit of fucking up in new and interesting ways that even the authors can't figure out.
The common ground is having storage nodes attached directly to a pack of big fat disks(for streaming IO) or NVME/SSDs for random IO. They then serve out NFS traffic. Now, you can either have a clustered file system underneth, or not. (Having stand alone servers can be advantageous, if you can map your filesystem out hierarchically)
Now, unless you have a Storage area network, then the last option is just begging for shit performance. You really don't want IO traffic fighting with network traffic. However, if you want raw throughput, this is the way to go, but be warned, you won't get any friendly help if you accidentally disconnect a disk.
Basically, kubernetes/HPC and storage is a solved problem ducks no really, just map in NFS shares and be done with it. If its exotic, its probably going to fail hard, and in ungoogleable ways. More importantly only a few people are going to be able to help, and they may or may not still employed at your company.
I find your offhand dismissal of clustered filesystems, on which literally every supercomputer relies, to be a little strange. They might not have worked well for you, but "googleable" is not the bar that is generally set for the HPC problem space.
What I was angling at is unpartitioned clustered filesystems are fragile. Dirty nodes cause lots of problems. Storage area networks which then interface via another means is much easier to look after, for little/no performance hit. (it can be a lot faster because there is less coordination/chatter)
Would I use Lustre for long term storage, with 100% uptime requirement? no. WOuld I use it as a linearly scalable scratch space? yes.
Would I use GPFS (or whatever its been rebranded to) instead? yes, probably. If I was hosting lost of small(<10tb dataset) disparate apps, would I put them all on the same namespace? fuckno.
Would I wire all my clients directly into the same clustered namespace? not if I could avoid it.
The issue is this, in HPC, or any other multi-node scheduled based system(basically mainframes but without the documentation, or the error checking) nodes die in new and interesting ways, If you have shared memory then its surprising how well interesting problems propagate.
In most HPC senarios, you can, if you are desperate stop and restart from a known good point. If you are serving public things, you don't get that option. So its in your best interest to partition.
Now, as for the googable bar, Unless everyone running the cluster is intimately familiar with the filesystem, including the interesting ways its fucked with VFS, How the metadata server handles stale locks, sudden bursts in lost clients or what ever, you need google. Even if you are a master, you might forget.
Why isn't a fourth option being explored: local storage with async replication? Seems like it'd be fairly simple and fast, and no worse than non-clustered NFS regarding data integrity.
I'm just talking from ignorance, so am I missing something?
I wonder if you could have something better, closer to streaming replication of databases. A few weeks ago I found zrep, which sounds more like what I had in mind: http://www.bolthole.com/solaris/zrep/
I worked for several years in VFX/HPC. 30k+ cpus and 15pbs of storage.
Firstly with storage its very rare that people want actual block storage (unless you are hosting VMs, but thats so 2007.....) Yes, I know, openstack, but that's just fucking horrific, seriously just use netboot and be done with it. I've seen people do it inside new clustereing systems, but its really not fun to do, especially if you consumer is prone to disappearing without warning. (FSCK is a terrible mechanism for fast recovery)
Most apps, unless they have bought into the "shove everything over HTTP and pay the penalty", want a posix file system to store anything of importance. (yes, yes database, but where is that writing the data to?)
Now, there are three ways you can do this:
o use a clustered file system
o Use NFS (with or without a clustered filesystem underneath)
o Fuck about with iscsi/SAS/FC and dynamically map block dynamically.
Using a clustered filesystem spread over many clients is begging for trouble, mainly because one client can fuck it up for everyone. Some FSs are dynamic and sexy, but they have a habit of fucking up in new and interesting ways that even the authors can't figure out.
The common ground is having storage nodes attached directly to a pack of big fat disks(for streaming IO) or NVME/SSDs for random IO. They then serve out NFS traffic. Now, you can either have a clustered file system underneth, or not. (Having stand alone servers can be advantageous, if you can map your filesystem out hierarchically)
Now, unless you have a Storage area network, then the last option is just begging for shit performance. You really don't want IO traffic fighting with network traffic. However, if you want raw throughput, this is the way to go, but be warned, you won't get any friendly help if you accidentally disconnect a disk.
Basically, kubernetes/HPC and storage is a solved problem ducks no really, just map in NFS shares and be done with it. If its exotic, its probably going to fail hard, and in ungoogleable ways. More importantly only a few people are going to be able to help, and they may or may not still employed at your company.