Hacker News new | comments | show | ask | jobs | submit login

> Since Google uses multiple IP addresses to crawl it’s also difficult to block these type of GET flood

It wouldn't be too hard to block by User-Agent: Mozilla/5.0 (compatible) Feedfetcher-Google; (+http://www.google.com/feedfetcher.html); if you notice the traffic.

Feedfetcher does not fetch robots.txt though; so you'd have to do something in your server config.

[edit: fixed a typo, and agree with the update]




Thanks, I've updated the post.




Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | DMCA | Apply to YC | Contact

Search: