

What Google Won't Let MS Click - byrneseyeview
http://www.google.com/robots.txt

======
gojomo
Misleading headline: the disallows affect all crawlers equally.

Missing context: what is someone, expert at reading robots.txt or not,
expected to find interesting in the target link?

Bad submission.

------
Goladus
robots.txt is not a security feature. It's a courtesy that saves resources and
makes life a lot easier for everyone who is willing to play by the rules.

------
rainsill
This is a bad submission. False headline. How do you vote an article down?

~~~
rms
Making a comment saying it is a bad submission is great, just please don't say
"This article should be on reddit!!!!!!!!"

------
chengmi
Does this even matter? How many bots actually respect robots.txt?

