

Ask HN: Why the HN Nginx error for articles below 9M? - bbcbasic

https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=8999999 not working
https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=9000000 working<p>The pages I have tried above 9000000 works and below that number doesn&#x27;t.<p>Frustrating when using Google to search for HN articles!
======
dang
We turned old articles off a few hours ago as an emergency measure because the
site was being crawled aggressively and our poor single-core Racket process
couldn't keep up. Previous solutions that have served us well until now, such
as IP-address rate limiting, appear no longer to suffice. If you noticed HN
being unusually slow earlier today, that was why.

Sorry for the inconvenience! We've turned everything back on for the time
being.

Our interim solution will probably be to cache the html for old articles,
since nothing about old threads ever changes. This is not a perfect solution
because, for those older articles, logged-in users won't see their usernames
in the top right, nor will they see which items they voted for or the orange
asterisk beside their own posts. But everything else should be there. Edit:
oh, and those of you with a custom top color won't see that (on older items)
either. :(

We have a better solution in mind that will restore the per-logged-in account
details to the old threads, but that will take longer.

~~~
bbcbasic
Thanks for the detailed answer. Caching is a good idea. Another idea is
provide the content as static JSON to crawl over. I presume there is a lot of
crawling from other nerds wanting to make their own HN interface or search.

I'm impressed it runs off a single core it's a very popular site!

~~~
kogir
We offer a lot of data (in JSON and XML) via our Firebase powered API:

[https://github.com/HackerNews/API](https://github.com/HackerNews/API)

------
kogir
Sorry, we went down under load and figured it was better to prioritize current
discussion. Working on resolving the problem.

