

Ask HN: How to scrape Google's similar searches? (challenge) - gbachik

Alrighty guys! I&#x27;ve been at this for about 4 hours straight with no dice!<p>I&#x27;m hoping some genius here can help me out.
Open a new tab and type in a band name like: &quot;All Time Low&quot;<p>You&#x27;ll notice a box on the righthand side with more info about the band.<p>If you click the down arrow you&#x27;ll see a &quot;People also search for&quot; section.<p>My goal is to get those names.<p>I&#x27;ve tried everything I could possibly think of to do it.
The only thing I got working was phantomJS and the time it took to scrape just one page was over 5 seconds. 
Thats way too long...<p>Anyone got a better solution than me?
======
nostrademons
So, I know the folks who do DOS protection for Google, and...well, good luck.
Scrapers get blocked, and the folks in charge of that are very, very good at
what they do. Your best bet is probably to put up with the slow query rate and
mimic ordinary user traffic. You really, really do not want to end up on
Google's bad side.

~~~
Jake232
Google isn't _that_ difficult to scrape. I've worked on numerous projects
doing it and it just takes a little thinking outside the box and a few
proxies.

