Hacker News new | comments | show | ask | jobs | submit login

We've actually never crawled out of Amazon (mostly because it was expensive to do so), so the crawl blocking stuff is unrelated, at least for us. As Sarah noted, it's really been costs, service, and support issues.

I'd redact my suggestion then if I could. Thanks for answering so directly and promptly! (and great seeing you on HN)

Thanks dude! Totally fair question & assumption BTW :-)

What type of hosting infrastructure works best for crawling?

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact