
Ask HN: how much does it cost to crawl 1M sites? - visox
What is the cheapest api or other solution ?
How much more does it cost to crawl sites with javascript ?<p>thx
======
visox
I think i made an unclear/wrong question. I wish to crawl about 1M pages
(rather then sites)

so 1M http get requests.

I was hopping there would be some solution for 100$ lets say

~~~
segmondy
$0. Crawl from your computer at home.

------
bufferoverflow
Depends on which sites. Some of my websites have 200K pages. Some have just
three.

~~~
bryanrasmussen
Also depends on what you want to do when you crawl them. I guess if it's a
million sites it is going to be median distribution of number of pages. But I
don't know that information is available anywhere. I think the http archive
used to tell you the number of pages on sites, but now it just tells you page
sizes.

[https://httparchive.org/reports/state-of-the-
web?start=lates...](https://httparchive.org/reports/state-of-the-
web?start=latest&view=list)

of course page sizes might also be important dependent on how you crawl and
what you do with the data.

