Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Same goes for JS files. There is only on page that I'm serving. I also know that there is 10% of visits are returning "customers". So there should be around 90 css/js files served per 100 html serves but it's not.

Few minutes after releasing the app on HN I saw a swarm of "...bot" user-agents in my logs and lots of 405 errors also for the "...bot" user-agents. That made me think that the disproportion might be cause by robots.



After looking carefully at the image you linked to, the first 405 seems to show your web server is configured to disallow HEAD requests (and probably others). The failed HEAD requests by "bot" User-Agents (along with what looks like some bots using forged User-Agents) is interesting since a HEAD request, if successful, does not actually load the page data and should not count in your page load stats. In other words, the bug is in the counting of pages loaded.

The tough questions is, why are you getting hammered by bots?


I'm not sure. It seems that the route is:

  HN -> Twitter -> A swarm of bots sitting on Twitter
When I re-posted my app on Sunday to get extra stats I saw the same thing. The Terminator Judgment Day happened but just on Twitter.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: