There is a quite powerful crawler and scraper engine. We developed it for internal needs at happystayapp.com. We used it as a hotels reviews meta search and hotels aggregator from sites like booking.com, expedia, etc.
But I'm feeling it's a bit underused and I'm thinking how else it can be used?
It supports many of features modern web scraping tools have. The engine is scalable and easy extendable.
The technologies:
- MongoDB
- C#
- Microsoft Azure
- JSON
Where and how else the technology can be used? Any ideas?
