Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Misleading headline: the disallows affect all crawlers equally.

Missing context: what is someone, expert at reading robots.txt or not, expected to find interesting in the target link?

Bad submission.



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: