
Running a scraping platform at Google Cloud for as little as US$ 0.05/month - decimalrandom
https://medium.com/@irio/running-a-scraping-platform-at-google-cloud-for-as-little-as-us-0-05-month-6d9658982f04
======
chumanak
great article, the problem I found in those cases is that some pages like
immobilienscout24 sometimes take a lot to load, and that increases the cost.
Also in my case I found some websites (not immobilienscout24) blocking my
requests, so I ended up implementing proxycrawl to unblock the requests, that
allowed to bypass the blocks of google cloud but still I faced the loading
times problem of some of the websites

------
hermanradtke
Neat article. My only gripe is that the cost is a function of how much content
you scrape. If 3 minutes turned into 30 minutes it would cost more, regardless
of the tech stack. Also, since web scraping is IO bound the only thing that
really matters is how fast GCP compute servers can download the content.

