

Tell HN: Increase your crawl rate in Google Webmaster Tools - ig1

I found that only a small percentage of my site was getting included in the Google index even after submitting a sitemap, however after putting the crawl rate upto the maximum (i.e. allowing Google to hit my server hard) my entire site was pulled in.<p>In theory changing the rate should have no impact on how much of your site is crawled, but in practice it does.<p>I'm guessing Google allocates crawling your site a block of wall time and then stops crawling after that period, hence increasing the rate Google can crawl at allows it to reach more of your site.<p>The end result for my startup CoderStack was dramatic, search traffic increased by an order of magnitude (roughly 10x).
======
bobf
From my experience, Google does actually seem to do a reasonable job of
ensuring they don't over-crawl your site with the automated crawl rates.
Increasing the crawl rate above the automatic figure can certainly be helpful
- just make sure you know what you're doing.

------
tzm
Thanks for sharing. I wonder if others experienced similar numbers.

------
nbhat
I am also facing same problem

~~~
nbhat
I increased the crawl rate and now I see considerable increase in the indexed
pages.

------
aonic
Where is this option?

~~~
thinkalone
Here's Google's page with instructions and further info -
[http://www.google.com/support/webmasters/bin/answer.py?hl=en...](http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=48620)

