
Ask HN: What do you scrape the web for? - cirowrc
Hey folks,<p>It&#x27;s not rare for me to see myself scraping the web for both side projects and personal needs - checking whether the prices of the groceries are good or not based on the local supermarket&#x27;s website; gathering some emails to cold-email about ideas ...<p>It seems to me that just by having the ability to quickly write some code to gather information from a government web page or anything that shows stuff on the web is a <i></i>huge<i></i> power.<p>What about you? Any great web scraping stories? Have you ever got in trouble by doing so? Made an entire company of it?
======
byoung2
I just wrote a crawler for a client who needs to check vendor licenses
(plumber, electrician, etc). Given a license number, state, and trade, it
looks up the appropriate licensing agency in that state and pulls the license
info (issue/expiration dates, biographical information, infractions, etc).
Luckily most of these are old school sites so they're easy to crawl, but a few
have captcha (easily solved with OCR) or are single page apps.

~~~
cirowrc
yeah, that's been my experience too: old (or in my case, gov websites) are
much much easier but at least here I never needed to solve captchas - maybe
it's just that scraping is not a big thing where I live (Brazil).

------
beld_pro
I think some of the most real-world use cases are in the
realestate/hotels/coworking-spaces sector. The second space, email gathering
(guess what, all the time people have to use <blabla> at <something> dot com
to try to not have their emails scraped).

~~~
cirowrc
totally agree! I've never been exposed to realestate stuff but it definitely
seems a place where that makes a lot of sense.

Did you have some experiences with it yourself?

------
PaulHoule
I made the State of Delaware change its search form so that you can only look
up a company if you know both the name and the number.

