"Independence Of Irrelevant Alternatives" is thought to be particularly problematic. There definitely are thought experiments that demonstrate that IIA is not as desirable a criteria (to rule out a voting system) as you would want. If a voting ballot runs afoul of IIA, it can point more to a confused voting population than a flawed voting method.
For instance, the Condorcet Criterion runs afoul of IIA. The Condorcet Criterion states that if a candidate would beat every other candidate one-on-one, that candidate would be the winner. Apparently it is possible to introduce additional candidates (that don't change the relative rankings between pre-existing candidates) such that a Smith Set is instead produced. However, it is not possible to introduce additional candidates such that a different Condorcet Winner is produced.
But let's think about what that says. A Smith Set is not a problem in itself. It means an indecisive or conflicted voting population. It is interesting that additional candidates can expose a conflicted voting population, a confusion that actually already was there; just covered up by the lack of candidates. In other words, if IIA changes a result from a Condorcet Winner to a Smith Set, it is an indication that the population was not given enough choice in the original ballot.
In that sense, IIA is actually useful, and not a flaw. It does not point out a flaw with the Condorcet voting mechanism; it points out a flaw with the limited choice of candidates (and time to research/understand those candidates) for that vote.
This is part of the problem with voting theory, is that they make unwarranted theoretical assumptions, such that a vote will always offer sufficient choice to a voting population. At any rate, I believe IIA is a flawed criterion, and therefore the Impossibility Theorem, while true, is less useful than it is celebrated to be.
But, as you say, Arrow's Theorem is simply too broad with it's definition of "fair". It's entirely reasonable for the voters to have a list of candidates they hate and a list of 3 candidates they would prefer, and for there to be a rock-paper-scissors situation among those top three candidates. It's entirely possible to have a voting system that only chooses from among the rock-paper-scissors options, and indeed every Condorcet system will make such a choice. Those options are the Smith set.
The tragedy is that Arrow's Theorem is often used to justify systems (like plurality or range voting) that often won't choose from among those top three, and might even choose the Condorcet loser! I'm not sure what a good definition of "fair" is, but I'd settle for not picking the candidate that would lose to literally every other candidate in a one-on-one election.