Ask HN: How to scrape Google's similar searches? (challenge)

[gbachik]

Alrighty guys! I've been at this for about 4 hours straight with no dice!

I'm hoping some genius here can help me out. Open a new tab and type in a band name like: "All Time Low"

You'll notice a box on the righthand side with more info about the band.

If you click the down arrow you'll see a "People also search for" section.

My goal is to get those names.

I've tried everything I could possibly think of to do it. The only thing I got working was phantomJS and the time it took to scrape just one page was over 5 seconds. Thats way too long...

Anyone got a better solution than me?

[nostrademons]

So, I know the folks who do DOS protection for Google, and...well, good luck. Scrapers get blocked, and the folks in charge of that are very, very good at what they do. Your best bet is probably to put up with the slow query rate and mimic ordinary user traffic. You really, really do not want to end up on Google's bad side.

[Jake232 → nostrademons]

Google isn't that difficult to scrape. I've worked on numerous projects doing it and it just takes a little thinking outside the box and a few proxies.