Remember the BBS forums and forums of old days when you didn't have points and rankings? Yeah. The forums often degenerated there too. Points didn't seem to help or matter. But moderated forums worked well. If you want high quality, you'll need a walled garden. But walled gardens rarely work and stifle creativity. However if you think a point system is a substitute for moderated comments. Well, then, the results speak for themselves.
So, points and voting be gone, maybe? I doubt the quality would change much. Even PG admits that it's not the voting per se, but the entrance of a larger volume of people.