Hacker News new | past | comments | ask | show | jobs | submit login

There’s 100x more bots that don’t run JS than do, so the presence of a running JS environment is a helpful filter. User agent is not a good substitute.



So, so many bots. 100x might be hyperbole but only just.

We host websites and the bots are super annoying, because even the well-behaved ones throttle requests per domain, which means they just hit all of our customers at once. If our cache architecture were a little more rotten, like I’ve seen on other jobs, then bot-driven evictions would get ugly, instead of just spiking our traffic, increasing our overhead, and making it harder to get clear metrics.




Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: