#standardSQL
SELECT id, title, url, score
FROM `bigquery-public-data.hacker_news.full`
WHERE timestamp > '2019-01-01'
AND REGEXP_CONTAINS(url, '.pdf|arxiv.org')
ORDER BY score desc
One of the huge issue on HN is lack of good algorithms for ranking. Currently all posts suffer from insufficient initial exposure to receive enough votes and thus false negatives are extremely high on HN. Just go to new page right now, take a look at most recent posting and you will find that majority of good genuine posts would never make it beyond their initial single upvote. Very small subset gets re-posted few times and only randomly some will get attention. Unfortunately papers are probably the biggest victim so you could have as well have collected all the links, did kind-of-random sort and you might still get fairly decent results. The bottom line is that never rely on HN's flawed ranking. Use that as weak signal, combine with Reddit, citations, tweet counts etc if you want to work around vast sea of false negatives that HN currently is.
I think this is what most social news sites get wrong about their points systems. Someone who submits an article gets points for each vote it gets. But the few heroes that waded through the submissions to initially upvote them get nothing. Even though they likely did a lot more work than the submitter.
I'd think a better algorithm would be that if you upvote an article in the "new" section that eventually makes it to the main page, you'd get at least as many points as the submitter.
That incentivize upvoting everything, since there is no punishment for upvoting posts that don't make it to the frontpage. That could be solved by making votes a limited resource, but that approach has many downsides.
Back when pg used to run the community [0], he'd deeply think about problems [1] and then later solve them [2] but not without controversy [3]. news.yc was better for it with such a strong personality at the helm who'd withstand the backlash and hold on to their convictions. I guess, we need someone of that caliber, someone to step up to usher in to the post-pg era [4], as it were.
What I do to find my share of interesting links is to search on https://hn.algolia.com with different keywords and sift through those discussions, follow links in the comments, follow the commentators previous submissions and posts, follow other submissions from the website, gather new keywords and search again... it is kind of a rabbit hole once you start. Though you'd quickly begin to realise that it is great folks continue to submit links that don't end up hitting the front-page, as the aggregation of links is valuable even though the discussion is missing [5].
That said, I have stumbled upon a lot of interesting things aren't simply shared at all on news.yc, though, I guess this is by design. I just hope folks don't stop sharing links just because their previous submissions didn't generate discussion or upvotes on news.yc.
Team is doing a fine job. It's much harder than the early days. Many want to game the system so it's a SEO vs. anti-spam job to keep this place running smoothly.
Getting to the front page isn't predictable, it's like creating a hit song. Tons of great music gets unnoticed. There's bias and well known names have an advantage.
Could you add links to the relevant threads on HN to this listing e.g. by linking the HN score there? This would give access to both the PDF for those who just want to read the thing as well as to the discussion for those who wonder what made it reach such dazzling heights on HN.
Thanks for the suggestion. I went ahead and made the change to link the score to the original discussion. It will take a few moments for the CDNs to propagate the changes.
One caveat is that some stories have multiple submissions; I just linked to the one with the highest score for now, but will need to iterate a bit to better handle multiple submissions.
A good suggestion. For those that want to look up the conversion before that suggestion is (hopefully) implemented, you can use the search of HN to search by submission URLs.
A link at the top for 2019 would be helpful, maybe change text color for year being displayed? When navigating years returning to 2019 isn't intuitive.
Thanks for the suggestion. I added a link for 2019, and the style for the active year is now bolded. I didn't want to deviate too much from the default color scheme of the theme I'm using, but that may change in the future.
That sounds like a fun and useful follow up. My initial thoughts would be to do some unsupervised topic modeling with LDA. Happy to consider other suggestions.
https://docs.google.com/spreadsheets/d/1he6ca0BBYbj2ZEOEpu8L...