Let's be realistic, even when the number of articles coming in is really high, you're talking about 100-150 posts for an hour.
The HN algorithm has temporal-gravity to help pull down old content that wasn't hugely popular. Due to that algorithm if the content doesn't take off in the first hour, it likely won't take off.
Those 2 points point to a far simpler solution. Just show most of the items from the last hour on the new page. So instead of 30 items, you show 100 items on the new page, which should accommodate most of the recent posts without requiring any clicks.
Well, the first step is identifying what the problem is. To me it's not obvious and a concrete statement of what we want to solve would help.
That is, we know it's that "articles that should be on the front page get overlooked"—but votes are the main metric we have for what should be on the front page. How can we tell an article that fell off /new because it was bad from one that fell off because nobody saw it?
One way of evaluating this would be to exercise editorial discretion, pick out articles you (or some set of people) believe are good and compare how they progress with different versions of the page. However, this is rather labor intensive and risks biasing the experiment based on who picks the articles out.
I can't think of other approaches off-hand. Coming up with a concrete statement of the problem would probably help here.
I agree. That's why I asked how to measure whether it works—it requires defining "works".
Upvotes alone don't determine HN's front page. If they did, HN would be dominated by controversy, gossip, and fashion—those get the most reflexive upvotes, and there are more reflexive upvotes than reflective ones. Coutervailing mechanisms include flagging and moderation.
The fundamental difficulty is that HN cares most about quality and one can't easily measure that or automate it. Human curation is needed, plus humans don't always agree about what counts as high-quality.
It might be interesting to be able to follow other users (who you know will tend to like the kinds of stories you want to see) and you can share saved stories or something. Or have related content or per-account filters that change the frontpage.
If, as it seems, there's no objective standard for quality (since it can't be measured or defined) then maybe subjective filters are the way to go.
I think there's value in HN having a shared single view of the front page and the threads. It means we all can't do the standard internet thing of self-selecting into subworlds that don't interact. This leads to friction in the short term—we're all exposed to things we don't like, some of which is uncivil and shouldn't be here, but some of which is merely disagreeable because it contradicts what we believe or like, and that's unpleasant. In the long run, though, my bet is that having a single whole community provides something deeply valuable.
On the question of how objective vs. subjective HN's notion of quality is, I don't think it's completely subjective. Everyone has their personal tastes, but HN's standards are reasonably well-established too. So I still think we may be able to come up with a story curating mechanism to supplement upvoting that is open to any user who wants to put in the effort. The big question, again, is how to score the effort. It can't be just "vote for whatever you like," because we already have that.
Determine if a nontrivial percentage of upvotes for a story are by users who are reading /new and the ranking of the story on /new at the time of the upvote is 30 or greater.
How about making the front page more sophisticated? Three buckets: popular, upcoming, new. The popular bucket is what you have now, and gets 25 slots. Upcoming gets 3 slots, and new gets 2. "New" is the two most recently posted links. If they receive an upvote or two, they get into upcoming. I think that'd help worthy links make it to the front page.
The HN algorithm has temporal-gravity to help pull down old content that wasn't hugely popular. Due to that algorithm if the content doesn't take off in the first hour, it likely won't take off.
Those 2 points point to a far simpler solution. Just show most of the items from the last hour on the new page. So instead of 30 items, you show 100 items on the new page, which should accommodate most of the recent posts without requiring any clicks.
(minimaxir's point is also very valid.)