Without swap you have two modes of operation: working fine, and not working at all (killed by the OOM killer). With swap, you introduce a third mode: working very slowly. The more modes of operation, the harder to reason about.
This is sort of fair, but working very slowly occurs in more than one dimension.
Swap on means applications may be slower due to page faults requiring swapped out memory.
Swap off may mean file system operations are slower due to less opportunity to file system cache frequently used objects.
There is still a trade off here on performance balanced against use case.
