So much this. Downvotes are not for disagreements, they should be used for trolls, shills and bigots. If the comment adds a view you don't agree with, write an answer or ignore it. Downvoting in this case doesn't adds value. Reddit suffers the same problem.
Which is why it would make sense to write a reason why you're downvoting. A lot of people wouldn't have energy to write an explanation when downvoting on a knee jerk reaction.
People in general care about karma points and don't like the score to go down. People also take downvoting personally, and when someone comments and explains why they downvoted, they're susceptible to "revenge downvoting" by several accounts (owned by or related to the one who was originally down voted) on many of the downvoter's comments across threads. This in turn would cause distress to the downvoter.
As long as HN has a number assigned to everyone that people interpret as significant, I'm completely for downvoting without any explanations. The fact that both upvoting and downvoting are anonymous (to the one getting the votes) ties directly with this, IMO.
There's another site similar to hackernews and their downvote button requires a reason.
One of: off-topic, incorrect, me-too, troll, spam.
I bet though a lot of people would still use incorrect for disagreements.
I always thought slashdot had a great moderation system where +5 was the max a post could get and everyone only had a few votes every few weeks so you had to be careful. The upvotes also had reasons like (insightful, funny, interesting ..) so a user could sort by those.
This could be a neat experiment, especially at lower karma/patience levels.
Drive by downvoting is often by people who aren't willing to pause and share their why.
It might not be easy to reflect on whether one is as open minded or comfortable to look at their own views, or openly entertain a viewpoint that isn't their own.
Innovative and disruptive thought may not exist in agreeable form even for a place as forward thinking as HN may imagine itself to be.
