
An Analysis of the World's Leading Robots.txt Files - fauigerzigerk
http://www.benfrederickson.com/robots-txt-analysis/
======
PaulHoule
Baiduspider, Yandex, and Bing all generate a lot of robot activity but often
seem to send little or no traffic.

