
Ask HN: Crawl the entire web? - nullundefined
If you were tasked with crawling the entire web, how would you go about doing it?
======
Eridrus
I'd grab the CommonCrawl data and keep moving on what my actual problem is.

