

Songkick's robots.txt has a few special rules. - nintax
http://www.songkick.com/robots.txt

======
dibarra
It's interesting they're disallowing 008. I work at a popular webhost, and
we've had problems with this crawler too (It's distributed, and can crawl
things a little too fast).

~~~
nintax
Did you notice the bottom 3 rules? Maybe I should resubmit and change the
title...

