They state "The disks were attached as JBOD devices to a ServeRAID M5210 controller (with a stripe size of 512)". I interpret the stripe size to mean a CEPH stripe size.
So the fancy RAID controller can bite you in the ass even if you try to lock its risky functionality away in a closet.
Maybe it's hard to buy name-brand server hardware with lots of disk bays and a safely dumb controller?
Ah, good catch, I somehow missed that they had it in "JBOD" mode.
As you point out, though, it doesn't seem like it's "really" JBOD mode. It seems like that card is bonding the disks together somehow into a larger logical "stripe". Weird.
They weren't attached as JBOD: JBOD disks have no stripe size. The firmware changelog talks about VDs, which implies this was a RAID volume.
So I'm pretty sure what they did is the usual trick of a 1-disk "RAID0" to get "fake JBOD" mode, which is the only way to do it on crappy MegaRAID cards with old firmware without JBOD passthrough. Except, as they found out, this involves all the mess of RAID code in the controller, leading to bugs and other problems.
That controller does supposedly support proper JBOD passthrough mode though, so this was a configuration mistake.
Either way, the real lesson here is that RAID cards largely suck, and you want to stay as far away from any RAID features as possible if you're trying to run disks for object storage. For these MegaRAID cards, the best option is to flash IT (HBA) firmware if you can, or IR (HBA with "simple" RAID support) firmware. If you have to use MegaRAID firmware (the full fat RAID thing which uses a completely different driver), then get a version that supports true JBOD passthrough. Only use the RAID0 trick if you really, really have no other option.
So the fancy RAID controller can bite you in the ass even if you try to lock its risky functionality away in a closet.
Maybe it's hard to buy name-brand server hardware with lots of disk bays and a safely dumb controller?