
Ask HN: Web-scraping – Do patterns/recipes exist for common scraping targets? - kisamoto
I&#x27;m fairly familiar with web scraping&#x2F;crawling however I was wondering if there is a company&#x2F;tool that has re-usable modules for scraping common websites?<p>Examples could include: scraping article texts from news websites; extracting recipes from Good Food etc.<p>Rather than rewriting what others have - is there an existing library of these scrapers&#x2F;crawlers to use &#x27;out of the box&#x27;?
======
abarrettwilsdon
Not exactly what you're looking for, but there's a OSS Chrome Extension that
allows you to record your actions in browser and transcribes them into
Nightmare.js code:

[https://github.com/segmentio/daydream](https://github.com/segmentio/daydream)

Probably the best you're going to get - most things worth scraping are worth
money, and as such are not freely available

------
mendelevium
For extracting news articles:
[https://newspaper.readthedocs.io/en/latest/](https://newspaper.readthedocs.io/en/latest/)

~~~
kisamoto
While not general purpose templates, for news articles that seems
exceptionally useful

------
atmosx
Try: [https://www.scrapinghub.com/](https://www.scrapinghub.com/)

~~~
kisamoto
I know ScrapingHub actually, has some useful tools but as far as I know they
don't have a library of off-the-shelf scrapers for popular websites?

------
thedevindevops
What you're looking for sounds very open ended but the closest thing I can
think of is the Huginn project on github?

------
kilroy123
This is sorely needed IMO.

