Unfortunately there is always a temptation with karma systems to use them for "social proof". Social proof is implemented by downvoting opinions you disagree with and vice versa. The only ego-preserving way of dealing with this is to view your minority opinions as "ahead of your time" and resolve to politely keep up the thought leadership.
EDIT: I bit and looked at your comments. Suggestions on the thought leadership appear in another message.