In fact I am not sure why... I am just reporting the experience. Maybe because of architectural coverage, because while bugs also happen in architecture-independent optimizations, more bugs happen in, say, PowerPC backend.
Edit: Another possibility is difference between how often Rust and Swift update to LLVM trunk. Both have LLVM fork pinned to specific version updated from time to time and I guess the first to update gets all the bugs.