Disclaimer: I work at Backblaze. We use the boot drive just because it's a tiny bit "cleaner" to boot off a separate drive (and configure swap on that drive) and have all the "data" alone on the 45 drive array. It allows you to boot and THEN reassemble a failed raid array that is on other drives, stuff like that. You could do it off of USB sticks, but I'm not sure about how well that would work with swapping? Maybe it would be fine? In the end it hasn't been worth focusing on yet, the price difference would be really super small. We spend most of our brain cells trying to find cheaper hard drives which account for 80 percent of our costs.
Netbooting -> we're actively looking into that as an option. But you still probably need a local swap drive.
You don't normally need a swap-drive on a storage node (if it needs to swap then something is wrong to begin with), but of course I don't know the details of what your nodes may do beyond dealing out files.
The main advantage of PXE-booting would be maintainability (rolling out upgrades by a simple reboot, etc.) but I assume at your scale you have that already figured out in one way or another.
Either way, thank you for all the insights that you keep sharing with the public!
These hands-on blog posts are priceless both in entertainment and education value. :-)