

Lessons Learned While Crawling the Web - jennita
http://moz.com/ugc/lessons-learned-while-crawling-the-web

======
alecsmart1
The title is misleading. It should be "Lessons learned about web crawlers
while crawling the web"

~~~
jgmmo
I think it should be called 'web scraping 101'

Very thin article.

~~~
billrobertson42
I found certain aspects of it really interesting. In particular, the way that
the author interacts with the crawling software to gain insight into the
structure of a website. Not just the process, but the way he interacts with it
is interesting too.

