I would use this for my people search application. This would definitely bring costs down significantly. The economics actually work the same if you want to do a really good job (be comprehensive) with your search application, you need to crawl the entire web to find the documents (part of the vertical) your search application is interested in. You could use a white list of sites, but then that would not be comprehensive.
So, building a good vertical search engine is really hard, you have to crawl the entire web first to find documents that look like the ones you are interested in. At 100 billion documents, 20k bytes each:
Bandwidth = 100 Billion documents * 20k bytes each = 1.8PB
To download the documents at wire speed (gigabyte speed) it would take over 230 days. And of course processing, analysis etc. would take more compute time.
It would cost over $1M a month to process all web data on AWS. This is assuming we go at wire speed and ignore any kind of politeness.
If Yahoo were to open it's crawl, we would be able to just write our specific applications on top of Yahoo's crawl without needing to download the entire web on our servers.
So, building a good vertical search engine is really hard, you have to crawl the entire web first to find documents that look like the ones you are interested in. At 100 billion documents, 20k bytes each:
Bandwidth = 100 Billion documents * 20k bytes each = 1.8PB To download the documents at wire speed (gigabyte speed) it would take over 230 days. And of course processing, analysis etc. would take more compute time.
It would cost over $1M a month to process all web data on AWS. This is assuming we go at wire speed and ignore any kind of politeness.
If Yahoo were to open it's crawl, we would be able to just write our specific applications on top of Yahoo's crawl without needing to download the entire web on our servers.