

Ask HN: mad search bots? - xentronium

I've got a groupblog-like site (somewhat niche, I must notice) which got articles and tags. Watching the stats I found out that about 80% of time and like 250+ reqs/minute application serves are tag pages for searchbots (from different search engines).<p>Am I doing something wrong?<p>I don't mind search bots since they hit the cache, but they seem to crawl the same content over and over again. Isn't that weird?
======
JoachimSchipper
Microsoft's bot used to do that kind of thing - not sure if it's been fixed
yet. AFAIK, it's just stupid and there's no way to fix it.

You can set crawl-delay in robots.txt if you want to reduce load, but you
don't appear to need it. Note that, again, MS' bot doesn't honour this.

It may be sensible to make the tag pages noindex, and a sitemap.xml may help
these bots. But I have no other ideas.

