I had a similar perception with Solaris on SPARC. It was slower overall but more responsive when interacting on the shell. Sometimes I had high load (10 or more) and did not notice when using a shell in a terminal.
I think Linux got much better since then. I don't feel the difference on current FreeBSD, Solaris or Linux installations.