Some points I think bear some clarification wrt current zfs state:
- current versions of zfs (ie: openzfs) error (unless forced) when adding non-redundant devices to a redundant pool, and have device removal to handle the case where someone bypasses the error.
- pools can grow in zfs (ie: the things one allocates storage out of). what is probably being referenced here is that one can't reshape a zraid vdev to add more disks to it (basically the reshape operation in classic raid setups). This does make expanding in some cases less flexible that old-style raid setups.
Also, I really have to agree that ARC behavior is still not great under load, and I don't think that's that much of an edge case.
But of course there are peculiarities that have 'bitten' people (softly) in the past:
- dedup has hardly any use case, novices get enticed to try something that doesn't do what they think it does
- ZIL and SLOG sound like "a simple cache" when in reality they are much more complex
- You can add single devices to a (redundant) pool, at which point you've lost redundancy, perhaps without knowing.
- No defrag
- No re-balance
- No growing pools
- ARC still surprises people (also, I've had kernel crashes under heavy load when the ARC didn't free ram fast enough ... edge case maybe, but still)
That said, of course, ZFS is awesome and I encourage everybody to give it a try.