

Is it time for a web crawling code of conduct? - jdrock
http://www.readwriteweb.com/archives/is_it_time_for_a_web_crawling_code_of_conduct.php

======
gojomo
Upstanding crawlers already follow this and more; rogue crawlers won't be
influenced by such a code-of-conduct.

Furthermore, unless a crawler is trying really hard to disguise itself, site
owners have the ultimate 'self-help' available to them: just block bad
crawlers entirely.

Any formal code-of-conduct beyond what are already customary practices (like
following robots.txt rules) thus strikes me as superfluous.

~~~
jdrock
My hope in making a code of conduct accepted by legitimate web crawling
companies is that it lets customers and users more easily decide which
products to use.

It also helps in distancing legitimate companies from illicit ones.

