raid5 sucks hard when a drive fails, you are right. But, see, for me, what I need is something that can be mounted as a block device by a xen DomU. Something that performs reasonably well.
On top of that, I am kindof dumb compared to the sort of person I'd want writing my block device drivers. I want to take well-tested, open-source software components and plug them together in a clear manner. I don't know of anything off the shelf that will give me a filesystem with reasonable performance on a '3 jbod' system, unless you mean running md in a raid1 with 3 drives. (I'm actually doing that on my more remote servers; the idea is that I can wait longer before replacing a bad disk.)
The system I proposed above is basically that you can export drives that may fail to clients, who can then do their own redundancy. Because the drive is specified as 'may fail' write-back caching may be utilized. (God, ram has become cheap.) The client, in my case, will be the Dom0 of the DomU that wants the space, but if I was selling this space to random people on the internets, it seems that the client could be some box running md that treats it's iscsi devices syncronously, meaning it waits for the write to return from both MD devices before it returns it's write. If the intermediary client device did no caching at all, it seems that I may be able to setup IP failover. (though that part... sounds dangerous. I'd need to be very careful with 'fencing' it or what have you, so that the two nodes were not active at once.)
Yea I see your point. I think in the end it still comes down to scale, for a couple of racks it's not worth the risk and effort while on a larger scale it might be worth brewing your own meta-filestorage system.
Yeah. that's my thought. I'm at a couple racks now, but I want a solution that will scale beyond that. Thing is, to be honest, the coraid crap will probably be more reliable (and cheaper if you include my time) than stuff I build myself for the first few units, until I learn what I'm doing, but long-term, I could be significantly better off if my idea works as planned, as my units are potentially much cheaper and have a whole lot more cache (and cache redundancy) than the coraid.
On the other hand, if I handle the 'beta' poorly, like taking it out of beta early, there won't be a long term to worry about.
On top of that, I am kindof dumb compared to the sort of person I'd want writing my block device drivers. I want to take well-tested, open-source software components and plug them together in a clear manner. I don't know of anything off the shelf that will give me a filesystem with reasonable performance on a '3 jbod' system, unless you mean running md in a raid1 with 3 drives. (I'm actually doing that on my more remote servers; the idea is that I can wait longer before replacing a bad disk.)
The system I proposed above is basically that you can export drives that may fail to clients, who can then do their own redundancy. Because the drive is specified as 'may fail' write-back caching may be utilized. (God, ram has become cheap.) The client, in my case, will be the Dom0 of the DomU that wants the space, but if I was selling this space to random people on the internets, it seems that the client could be some box running md that treats it's iscsi devices syncronously, meaning it waits for the write to return from both MD devices before it returns it's write. If the intermediary client device did no caching at all, it seems that I may be able to setup IP failover. (though that part... sounds dangerous. I'd need to be very careful with 'fencing' it or what have you, so that the two nodes were not active at once.)