Nothing in your SLA says you have to have servers 1, 2 and 3 available. It says you have to have service X available. You can do that with out taking a downtime hit.
It's an "SERVICE LEVEL AGREEMENT" not a "SERVER LEVEL AGREEMENT".
Sounds like your experience is the only one that matters, then.