Hacker News new | past | comments | ask | show | jobs | submit login

If you are able to crawl that volume as cheaply as you say, you should definitely be offering it as a service.



Its a very single purpose system, so probably not amenable to general purpose crawling (though we do have a separate system that is based of the design of our core system that is a general purpose web crawler/indexer)




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: