

Ask HN: Web crawler generator - villagefool

I am looking for a web crawler generator, that I could prototype what I would like to crawl using a visual dom explorer type of interface, is there such a thing? anything recommended? Thanks
======
skram
Not sure if you're looking for a hosted solution (there are less of those) or
a library to build upon (google "scraper"/"crawler" and the programming
language of your choice).

Either way, check out ScraperWiki: <http://scraperwiki.com>

Hope this helps.

~~~
villagefool
Interesting service, never heard of it. I am actually looking for a
library/framework rather than service, I was hoping there is some "visual
editor" for defining what to scrape/crawl and simplify the building process.

~~~
skram
Well like I said, google or even a targetted search on Github/Sourceforge will
reveal a TON of libs. As for a GUI... I don't recall any free ones coming up
in my search a few months back.

