You can still collect this data anonymously. Rather than giving people a fixed number of votes, you have them vote for everyone, but use random methods to conceal the source. (This has the effect of hiding single complaints, though)
For example, when downvoting: flip a coin, if it comes up heads, downvote that person. If it comes up tails, only downvote that person if you think they deserve to be downvoted.
Doing it this way, each person can be expected to get roughly 50% of people downvoting them. If the number is enough higher than that, then that person probably deserves the downvotes, but you can't tell who they came from.
For example, when downvoting: flip a coin, if it comes up heads, downvote that person. If it comes up tails, only downvote that person if you think they deserve to be downvoted.
Doing it this way, each person can be expected to get roughly 50% of people downvoting them. If the number is enough higher than that, then that person probably deserves the downvotes, but you can't tell who they came from.