Note expecially section 6, "Personalized PageRank", where they replace the random jump with a jump back to some basis pages. Your own up votes/downvotes probably make a pretty good set of reference points, so a PageRank-like algorithm could be heavily weighted according to your own judgement.
To be totally clear, I'm not suggesting everyone get personalized results, but rather that HN is the same for everyone, one view "personalized" according to your judgment (and obviously that propagates through to include the judgement of the people you respect, and they respect, etc).
Yes this is a handwave, and Google has gone lightyears beyond this early paper. But intuitively it may help you think about the problem. It's really going to be some iterative algorithm that flows through all the upvotes and downvotes to determine the authority of each vote, perhaps in a similar way to how the authority of each link on the web can be calculated.