

Ask HN: How do companies like Moz get search data without getting banned? - antjanus

I know that Google bans scrapers outright and anyone trying to shadily get around the system gets caught at one point or another. So how is it that large companies like Moz, Ahrefs, and others are able to get all the data they need, quickly, without getting banned, and I swear that I&#x27;ve seen Matt Cutts even link back to Ahrefs in the past.
======
cynusx
Ahrefs crawls the internet just like Google does. SeoMoz does the same:
[http://moz.com/help/pro/what-is-rogerbot-](http://moz.com/help/pro/what-is-
rogerbot-)

Every service that builds a backlink database uses their own crawlers

So to answer the question on how they get their data so fast, they OWN the
data and they use big hadoop farms to crunch it and prepare it for api
retrieval.

------
caruana
They probably pay for it. Or rather google gives them a paid account but
doesn't charge them.
[https://support.google.com/customsearch/answer/72334?hl=en](https://support.google.com/customsearch/answer/72334?hl=en)

~~~
dangrossman
That's pricing for Google Site Search, a Google-hosted search engine of your
own site. It's not a way to buy SERPs for the entire Google web index which is
what SEO services need.

------
staunch
It's not hard to setup a bunch of machines on various networks with hundreds
of IPs. You can't really block people with resources.

------
jotato
I worked for a competitor to Moz. We used a service like Anonymizer to hide
it. My guess is they do as well.

------
moron4hire
I would also like to know.

