When it happens to me it is because I have replied to that comment. I assume that it is because if you replied to a comment you DO think it is worth reading it, and it surely contributed to the discussion, therefore no sense in downmodding it
There is also some sock-puppet detection as well - this is to prevent you creating accounts and upvoting yourself. Presume it also prevents you downmodding every comment by someone you don't like too...
I've seen it occasionally go awry - you'll upvote and the vote just disappears.
Well I cannot downmod comments I have already replied to. Up/downmodding does NOT mean you agree or disagree with that comment, it means you find it useful/interesting or that it has added something/contributed to the discussion
Well, it looks like I was looking at comments older than 24hrs (can downmod comments I have replied to, in the last few minutes): so I am wrong on this :-)