
Ask HN: Web crawler/data product ideas? - MechanicalTwerk
Is there a data set you would love to have but you just don&#x27;t have time to compile?<p>Is there a data set that you currently compile manually that you can&#x27;t live without?<p>Would it make your life easier if you could avoid opening all those browser tabs every day&#x2F;week&#x2F;month and instead purchase a neat file (csv&#x2F;google spreadsheet&#x2F;something else?) with everything you need? Which browser tabs would this eliminate?
======
JamieLewis
Probably not what you are looking for, but I would love to see more real-time
data curation - I would certainly be willing to hand over cash for some
decently priced real time data feeds - stock tickers, sports events, raw
traffic data, headlines, currency, flights - many of these are available in
pull formats, but it would be great to get a push feed.

~~~
MechanicalTwerk
Any good examples of a push feed you currently like?

~~~
JamieLewis
Twitters streaming API used to work fairly well when I was playing with it.
[https://dev.twitter.com/docs/streaming-
apis](https://dev.twitter.com/docs/streaming-apis) that was a few years ago
though.

------
dalek2point3
one big market for you is researchers. if only i had a penny for each time
some economist came to me asking for a list of xxx in yyy, where x could be
anything from flight prices, to businesses, to startups and yyy could be city
/ state/ coutnry /world. The internet is really good for current snapshots,
but what reserachers most often need is repeated snapshots of the same thing.
that helps us analyze how things change over itme. if you're a data company,
please, please dont overwrite your tables to "update". save the old data. its
really really useful. for research. and more importantly, for you.

~~~
MechanicalTwerk
Thanks. Will definitely look deeper into this. Know of any good sites where
researchers make such requests?

------
bobosha
Have you looked at [http://enigma.io/](http://enigma.io/)? They won TC Disrupt
last year

------
Dnguyen
Priconomic just refocus to this business model

~~~
MechanicalTwerk
Priceonomics looks like more of a professional service at the moment. I'd like
to sell actual data sets in more of an ecommerce/subscription model. Kind of
like
[http://www.cbinsights.com/datastore/](http://www.cbinsights.com/datastore/)

