People who have been longer in the community and have a good kharma would have the power to vote +1|-1, while people who have less kharma or have joined the community recently would be able to vote, say, +0.1|-0.1, and there would be a lot of people in between (+0.3|-0.3, +0.5|-0.5, etc.)
It just makes sense that your upvotes/downvotes should have more voting power than the ones coming from, say, myself.
As far as "who is voting" for arguments sake a member of SCOTUS joins HN and decides to cast a vote. Or a member of SCOTUS joins HN and after a period of 2 years has low karma, because, they don't have much time on their hands. I'm not exactly sure why someone who has not been a member for a long time or doesn't have a high karma score isn't to be trusted with voting. And why someone who decides to spend a lot of time on HN should be given more privileges.