In the "old days," there was more-or-less agreement that the vote was for quality of comment -- feedback about the value of the participant to the community -- and agreement/disagreement was voiced (if necessary) in comments. As a rule I (and others) would upvote comments we disagreed with as long as they were thoughtful. There were exceptions, of course, as you have aknowledged, but this was a pretty good guideline.
Now I think things have turned around a bit, and votes have started signifying agreement/disagreement more than quality of comment. I think this takes the community in the wrong direction.
You seem to be thinking that this is largely trollish behavior. I suspect, however, that most of it comes from ignorance of the expectations of the community. I'm not sure there is a fix for the trolls. But for noobs that watch reality tv shows and want to come here and be the acerbic judge I think we can stop it by keeping the behavioral expectations in the forefront.
The flaw with the idea isn't what you say. I think it's important to expect decent, reasonable behavior. The flaw is, 'ok, we have two buttons. Now what?'
I would start with having the quality score 'float' the article and pretty much ignore the agree/disagree score in karma metrics. But I would probably keep that a secret.