Hacker News new | more | comments | ask | show | jobs | submit login

May be they should visibly put that FeedFetcher can crawl your website can incur 1TB bandwidth in couple of hours ? May be apache and other httpd should block these crawlers by default ? How else would anyone know about a certain Google feature that could be disaster for them ?

Applications are open for YC Summer 2019

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact