

Bostonglobe.com (mistakenly) tells search engines to F off. - donohoe
http://bostonglobe.com/robots.txt

======
cube13
Or they actually don't want search engines to spider them, and instead of
suing, they're using the preexisting technical solution.

~~~
binaryorganic
That's the official rec. from Google, but it doesn't seem to be the case as
it's been changed to allow all. What was it when this was posted?

