Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Your calculation of claimed percentage of robots is deeply flawed. Not only do web browsers aggressively cache CSS files, but internet service providers also aggressively cache them.

Though I'm a rare exception, my web browser is configured to ignore CSS completely, including both in-page styles and all additional loadable files. A configuration like this is very rare because it's difficult to do and the result often breaks sites. The only place where it is common is for people with visual impairments.

A well behaved bot will (typically) announce itself by its User-Agent string. A securely configured browser (IMO) will never send a User-Agent string or Referrer, and will reject cookies as well as ignore javascript --Needless to say, this breaks a whole lot of sites, but suffice to say, HN does have a higher percentage of people wearing tin-foil hats than the regular populace.



Same goes for JS files. There is only on page that I'm serving. I also know that there is 10% of visits are returning "customers". So there should be around 90 css/js files served per 100 html serves but it's not.

Few minutes after releasing the app on HN I saw a swarm of "...bot" user-agents in my logs and lots of 405 errors also for the "...bot" user-agents. That made me think that the disproportion might be cause by robots.


After looking carefully at the image you linked to, the first 405 seems to show your web server is configured to disallow HEAD requests (and probably others). The failed HEAD requests by "bot" User-Agents (along with what looks like some bots using forged User-Agents) is interesting since a HEAD request, if successful, does not actually load the page data and should not count in your page load stats. In other words, the bug is in the counting of pages loaded.

The tough questions is, why are you getting hammered by bots?


I'm not sure. It seems that the route is:

  HN -> Twitter -> A swarm of bots sitting on Twitter
When I re-posted my app on Sunday to get extra stats I saw the same thing. The Terminator Judgment Day happened but just on Twitter.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: