We've gained a few trolls but I don't think we've suffered from evaporative cooling yet so it's not too late. I support removing down voting and switching to flagging specific offenses e.g. spam|offensive|troll to start.
Alternatively [just a crazy thought] how about we open up and show what each user has down voted. Perhaps as a courtesy we could only show down-votes starting Halloween 2010.
If we assume that a user's total score is a good indicator of their value to the community, then the HN leaderboard would agree with you.
I'm not so sure, though. IIRC, pg has mentioned in the past that he spends less time here now, and I rarely see anything from Reddit's founding user group. Another HN user that I've been following for a while -- DaniFong -- who's doing some pretty interesting stuff and has a tendency to post fairly intelligent comments, has had a really obvious decline in activity here.
Although I've been a part of this problem as often as not, I've also found that the discourse here isn't worth participating in any more.
But, maybe there are alternative explanations for all of that.