Do you ever notice that you'll lose two points on a comment at the same time, then later they'll come back? I see that relatively often (once every couple weeks), and on completely uncontroversial comments, which is how I started noticing it. I wondered if it was some vote manipulation detection, where they've linked two accounts voting similarly and nullified their upvotes, but later clear them so they votes are reapplied. In any case, it's always two together (which is why I don't think it's an accidental downvote), and seems to be close to the same time if not the exact same time.