A web crawler that can handle any number of websites | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		A web crawler that can handle any number of websites (mixnode.com)
		17 points by mixnode on April 12, 2017 \| hide \| past \| favorite \| 2 comments

spoondocz on April 12, 2017 [–]

What's the largest crawl you can handle? meaning your upper limit...

mixnode on April 12, 2017 | [–]

We've done crawls of over 250TB and 3 billion web pages. If you need to crawl more than this you can simply shoot us an email and we'll provision resources for your mega crawl.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact