
Crawling is harder than it looks - prakash
http://glinden.blogspot.com/2008/05/crawling-is-harder-than-it-looks.html
======
okeumeni
Great document!

I architected and implemented a crawler, reading this documents really
confirmed problems encountered and taught me a thing or two on how to improve.

This document goes the same scope as: Why Writing Your Own Search Engine is
Hard - by Anna Patterson, Stanford University
([http://www.acmqueue.org/modules.php?name=Content&pa=show...](http://www.acmqueue.org/modules.php?name=Content&pa=showpage&pid=143))
for those interested in search engines.

------
hobbs
That probably explains why most humans prefer to walk, once they've developed
the necessary motor skills. It's a much more efficient way to move and puts a
lot less strain on the wrists and knees.

(Just in case someone mistakes this for seriousness, I did RTFA)

