Here's a parable: imagine that you and your friends are all playing a pen-and-paper RPG with a focus on storytelling, getting along, quite happy to collectively have interesting things happen to all the characters. Then, imagine that a new guy joins, and he's a munchkin—he plays the game to win, getting all the other players angry. He doesn't like his character to "get screwed" to tell a good story, and so on. The other players realize this, but they put up with it, hoping, for a while, that he'll eventually adapt. But he doesn't. So, eventually, the other players begin to leave—and new players join that play in much the same way the munchkin does. Eventually, the group is no longer story-focused, but is rather completely focused on collecting imaginary points and tokens.
See what I mean? Although karma doesn't matter objectively, if it's important within the "shared delusion" of the majority, it can still do damage. That something was voted high or low wasn't the basis for my statement—it was that it seems, lately, that the majority shares the delusion; that this place is a munchkin group.
witch brings us to the point: You are right and the majority is wrong. But without good explanation, you still deeply care about what other people think about you("it still can do damage").
When I played pen-and-paper RPG, the storyteller used to favor her friends-interest over anyone else(the majority), so maybe that is the real objective: Impose your way of thinking over the majority, If not, you don't play.
That's not quite what I meant by the phrase. It "does damage" because others in the majority accept that it does—for the same reasons that I am "in debt" when a number on a machine reads negative. I might agree or disagree with the conception of money as a fiscal instrument, but it is supported by the group in a way that prevents me from participating in the group unless I acknowledge it and treat it the same way that they do. To not acknowledge money (or karma) is, in effect, an act of self-alienation. You "reject yourself" from the group by doing it, because the game they are now playing is a competitive one—and if you do not acknowledge the table stakes, you cannot ante.
This is why it is important to do some research on social game design (or consult a game designer) before you code a game mechanic into your social webapp. Years of research into MUDs and MMOs show that different mechanics reward, and thus encourage, different types of player-styles, and thus shift the user-base in different directions.
Here's the most famous paper on the subject, just to get anyone interested started: http://www.mud.co.uk/richard/hcds.htm . It would be awesome if someone would write an "Interpreting Bartle for Web 2.0" article and submit it back here; I've been meaning to do it for months now but it's still a ways down on my to-do list.