So,using one of those crawlers presented in that article would be enough? I read something about Nutch,Lucene,Sphinx but I'm not sure how the process of building an agregator works: go crawl some websites,get the data in the database or disk, analyze the data, display the data according to some specified criterias.
Are those crawlers spidering RSS feeds or a special crawler is needed for that?
A diagram would be helpul...
Aggregation and Search are two different components. Full-text search ( http://en.wikipedia.org/wiki/Full_text_search ) will help you index the aggregated data, so that it can be searched through a UI.
