for some good lists of available stuff.
I always ask there if I can't find what I'm looking for.
Here are more and more data sets. These are general data sets. Email me if you have a specific data set in mind (e.g. web-as-corpus, spam, images, social, reviews, etc.). I have a big file of information.
http://ckan.org [Comprehensive Knowledge Archive Network]
http://infochimps.com/collections/datamob (and the other collections on the site)
Thanks for these, iisbum. I wish more public data was available in db, xml or similar structures - too often I find myself scraping government sites or pdfs to get the tables I need
If you're up to something in the economic data space we'd love to talk. Happy to take this to email (email@example.com) if anyone's interested.
Things that would be very cool would be 1. financial statements in a database format. I know you can scrape this but I don't know if they are available legitimately?
2. Historial Implied volatilities and historical observed volatilities.
Is it your site? Are you going to add yield curves?
The Comprehensive Knowledge Archive Network! Pretty sweet resource really.
Edit: Although the use of this database is not free, I believe for personal use is just fine to download and experiment...
This data set, contributed by Google Inc., contains English word n-grams and their observed frequency counts.
Some free, some paid.
some free, some paid
AIS Data (Marine Traffic)
And there's a great list of sources on Quora
Geonames : http://download.geonames.org/export/dump/
OS Open Data (UK Specific) : http://www.ordnancesurvey.co.uk/oswebsite/opendata/
http://data.govloop.com/ has data and lots of pointers to local government data.
Also I'm surprised no one mentioned Carl Malamud's site: http://public.resource.org/ - Lots of US gov't and legal data in friendly formats.
bookmarked and shared this thread.
Something like satellite imagery, medical imaging, semiconductor masks and wafers photos or CAD files, etc.
Some other applications, example CT Colonography
Anyone know of any publicly available song lyric databases?
All the datasets I found on www, are Huge (in double digit GBs..).