Google is the best example to learn from. Search is a misnomer, Google is about ranking. They've put many PhD-centuries of effort into deciding which of the three million matches to put first (ranking). Choosing the three million matches (search) isn't where Google adds its value.
Lots of lessons have been learned there, I'm sure HN can tap into that pool of knowledge by opening up data to the right people.
HN is an awesome resource. I often read the comments before the articles because I expect them to be more useful.
It's completely worthwhile to invest a lot of effort in maintaining that greatness as it scales.
Sorry I couldn't help myself.
EDIT: (total rewrite)
The cool thing about using metrics and machine learning is that the results speak for themselves and opinions and factions are less important. Intead of guessing what matters in advance, you get to discover what matters after the fact.