I have had the opportunity to work with an Epyc server with a full load of NVMe/U.2 direct connected to all those tasty PCIe lanes. Any given drive worked wonderfully, and writing simultaneously across many drives with DD showed very good performance. Unfortunately any attempt to use Linux native software RAID showed performance barely in excess of a single drive, even with a simple stripe. I don't blame the NVMe, but something in Linux RAID just doesn't scale in performance. I spent days tweaking it, ultimately it was a great disappointment.
> I don't blame the NVMe, but something in Linux RAID just doesn't scale in performance. I spent days tweaking it, ultimately it was a great disappointment.
Yeah, it's a real mess to get it up to speed and if it involves parity calculations (RAID 5) then there is a CPU bottleneck. Used to be single-core only until a couple years ago, no idea how the situation looks today.
