

Google boutiques.com has no robots.txt - techvibe
http://www.boutiques.com/robots.txt

======
ithkuil
Well, you can try it with:

wget -O- <http://www.boutiques.com/robots.txt> -U "Mozilla/5.0 (compatible;
Googlebot/2.1; +<http://www.google.com/bot.html)>

and you get the same error.

I tried a couple of other bot user agent strings from
<http://www.useragentstring.com/>, same result

------
AndrewDucker
Why would it?

------
andrenotgiant
they have NOINDEX, NOFOLLOW robots META tags that tell search engine crawlers
to go away.

Maybe it's their way of playing nice with all of the merchants they are
listing.

------
yread
Could it be user agent filtering?

------
mooism2
So?...

