
Please don't block everything but Googlebot in robots.txt - luu
http://danluu.com/googlebot-monopoly/
======
podgib
Is there a technical reason that other crawlers can't just follow the
googlebot rules? It's great that bots seem to obey the wishes of site owners,
but I'm genuinely surprised that the yandex bot doesn't just follow the Google
bot rule and ignore the general disallow rule.

~~~
Volundr
No technical reason. Following robots.txt is purely self-enforced, crawlers
don't even have to read it if they don't want to. It's just part of being a
good netizen.

~~~
podgib
Thanks. That's what I suspected - I'm just surprised that yandex is behaving
like a good netizen :)

