
Scraping Hacker News on a Schedule with TaskPipes - f292
http://blog.taskpipes.com/2015/09/22/scraping-hacker-news-on-a-schedule-with-taskpipes/
======
tynman
What Open Source projects are similar? It seems like data transformation tasks
like this are a universal problem. I'd just do map/reduce, but that's the
advantage that comes with being a developer. Great tools for non-developers to
do limited programming tasks are incredibly useful.

------
juanescobarcom
Check out Diffbot @ [http://www.diffbot.com/](http://www.diffbot.com/) \-- you
can setup repeat crawls and extract data into .csv/Excel file format or JSON
with Crawlbot API, extract data automatically with automatic APIs, or Custom
API Toolkit. You can get a free trial account at
[https://www.diffbot.com/plans/trial](https://www.diffbot.com/plans/trial) to
try it out, and you can get paid plans at
[http://www.diffbot.com/pricing/](http://www.diffbot.com/pricing/).

------
fulafel
This general category of tool seems very useful. Apparently there are many
Yahoo Pipes followers. [http://www.makeuseof.com/tag/12-best-yahoo-pipes-
alternative...](http://www.makeuseof.com/tag/12-best-yahoo-pipes-alternatives-
look/)

Support for email and HTML tables seem to be some distinguishing features of
this one.

~~~
giarc
I thought I remember reading that Yahoo was discontinuing the pipes tool?

------
chdir
I've seen something similar more than once on HN but my google-fu is failing.
What other similar services are out there that'll let you smartly scrape a
page into a spreadsheet.

In the past, I've tried Google spreadsheets with "ImportXML" option but got
frustrated after a bit and resorted to python.

~~~
dhruvkar
Kimonolabs.com

Allows you to scrape then import that data into a google spreadsheet. But its
not the only way to consume the data.

~~~
chdir
Thanks, that's the one I had in mind. Also found : import.io, scrapinghub.com
& 80legs.com

------
draugadrotten
Looks pretty but I can't find pricing information.

~~~
bold
I was looking for prices as well, or any other indicator for that matter that
the tool will be available for a reasonable time at least.

~~~
iag
+1

I love these tools but before I invest more of my time leveraging this in my
workflow I'd want to know that it'll be around for another few months at
least...

------
johnong
I tried NewsBlur as an alternative to Yahoo Pipes. Like it because it has an
Android app as well.

