

How We Built Our 60-Node (Almost) Distributed Web Crawler - netvarun
http://blog.semantics3.com/how-we-built-our-almost-distributed-web-crawler/

======
netvarun
OP here. This post is an expansion of my comment
<http://hackerne.ws/item?id=4369590> on the excellent post 'How to crawl a
quarter billion webpages in 40 hours' ( <http://hackerne.ws/item?id=4367933>)

Don't hesitate to ask me any questions.

