For a site that is ostensibly focused on intellectual curiosity, this is antithetical to that goal. A robust, informed discussion requires exposure to a variety of perspectives, especially ones that seem challenging or uncomfortable.
If you do not agree with a comment--so what? Either refute it or move on. Just because you are unable or unwilling to refute it does not mean that someone else will not come along after you and do so. Downvoting should be reserved for comments which are truly useless, not comments that you merely disagree with.
And, yes, I am aware that downvoting for disagreement has been promoted and accepted by some on HN, but it's wrong--at least, it is if your actual goal is to have robust, informed discussion of intellectually interesting topics.