I see this sentiment a lot, but I disagree with it. What are "iTunes, Xcode, Safari/Chrome, and a mail client" doing now that they weren't doing four years ago? Is it enough to justify their latest versions feeling less responsive than their versions from four years ago?
iTunes: layers of software for dealing with Wi-Fi sync, Ping social network, iTunes Match, and so on
Mail Client: totally agree with you there...
But I still pretty much agree with you overall...
Things change. My old 2006 iMac core 2 duo feels a bit clunky sometimes these days, but it runs a lot of stuff fine and is actually just as good a machine as it ever was.
Lot's of things. XCode was rewritten and does live AST syntax completion, background compiles, etc.
Safari/Chrome have several more features --did Chrome even exist 4 years ago?
Expectations are certainly a big part of the perceptual speed equation. But with OS X, don't underestimate the benefits of keeping your disk less than 90% full. With all the caches, iPhone and iPad backups (over 40 GB in my case), Xcode, sleepimages and swapfiles, installers (Adobe!), SyncServices, etc., a 160GB SSD fills up in no time. When things get slow, getting back below 90% works wonders.