

Why CommonCrawl is a Disruptive Force in Big Data - edmarferreira
http://myeverwrite.com/commoncrawl-disruptive-force-search/

======
asharp
Wasn't the major problem with commoncrawl that most of the index data was too
old to be useful?

Besides, search index crawling is comparitively cheap compared to the data
processing required to make it useful for actual search.

