
Ask HN: 37% of people are using Python for web scraping. What do they scrap? - ggregoire
Just saw this stat in this survey: https:&#x2F;&#x2F;www.jetbrains.com&#x2F;lp&#x2F;devecosystem-2019&#x2F;python<p>I&#x27;m not familiar with web scraping. Wondering what are the most common applications of it?<p>I can think of a few like public users data collection, prices comparison, news aggregation… Are there some other obvious ones I am missing?
======
princess445
You can use something like this [https://github.com/proxycrawl/proxycrawl-
python](https://github.com/proxycrawl/proxycrawl-python)

To crawl literally anything. For my last project I got more than 10 million
linkedin profiles to build a social network for entrepreneurs.

So data can give you power to build anything

------
mjhea0
Most people are probably using scrapers and crawlers to collect data on sites
that either don't have an open API or the API is difficult to use.

 _Examples:_

Ancestry.com scraper - [https://github.com/mjhea0/ancestry-
scraper](https://github.com/mjhea0/ancestry-scraper)

Indeed job scraper - [https://github.com/mjhea0/indeed-
scraper](https://github.com/mjhea0/indeed-scraper)

Craigslist housing scraper - [https://github.com/mjhea0/craigslist-housing-
scraper](https://github.com/mjhea0/craigslist-housing-scraper)

[https://github.com/ThaWeatherman/scrapers](https://github.com/ThaWeatherman/scrapers)

 _What can you then do with the data?_

Monitor competition or your own brand

Sentiment Analysis

Gather sales leads

Machine learning

Generate content (blog posts, building a custom job board) Find cheap flights

