I'd rather have a system where instead of simple up/down vote you have an elaborate system of 'Offtopic', 'Poor taste', 'Incorrect' as mentioned in a comment somewhere above. But also, I'd like to add positive halfs of these such as 'Insightful', and maybe one or two other labels. The idea being that they can select only one out of the total, and each label should secretly have different values. So maybe flagging a comment as Incorrect isn't as penalizing as putting one as Off-topic or Poor taste. Insightful may have different positive value as compared to maybe something labelled as 'Excellent advice' or something.
With a lot of options, I don't think newbies would actually click randomly on any button without knowing the meaning and context of each button. Anyway, the buttons need not have full names, simple short letters like P/OT/I/EA/X may be enough. Of course, this automatically requires to have a way to retrieve your vote, if you accidently click on the wrong button, but it should still allow only one option to be used.