

Dead simple web crawling (at scale) with Zillabyte - jakequist
http://blog.zillabyte.com/2014/08/04/crawl-entire-domain-one-line/?

======
gamegoblin
So just two hours ago the other post by OP
([https://news.ycombinator.com/item?id=8143000](https://news.ycombinator.com/item?id=8143000))
was flagged to death. Presumably because quickly after posting, two eerily
generic and positive comments showed up (one from a new account).

That being said, as far as the service goes, some comments:

1\. The popup "join our newsletter" after a minute was a little annoying.

2\. Without guarantees on capacity and utilization, pricing by "machine hour"
is useless to me. I'd prefer something more concrete.

------
kumarski
Why hasn't someone built this before?

Maybe I'm missing something. I figured you could do this already using a few
mashable API's.

~~~
sjarvie
There have been efforts. Looking at the product, I see that the component
builds on an open source project.

Doesn't passing data through multiple APIs raise concerns of data
polution/cost overhead? At least this uses a flat,pay as you go cost.

------
markitos144
useful tool

