Ask HN: best way to become a "Numerati"?

[Tichy]

Not sure if "Numerati" is the best word. I already have a maths degree, but did little with statistics so far. I am capable of reading a book on statistics, but it is not the same as getting some practical experience.

I guess I could just start with randomly selected data sets, but somehow that doesn't seem to be enough motivation (like stock market data: seems unlikely to discover something new, with all the Quants already working on it). So I guess what I mean is, among other things: how to find worthwhile statistical problems for starting out? I am not into sports, so betting is kind of out. Also missed the elections, since I am not from the US. I look forward to the book by the 53something website guy, though.

[mechanical_fish]

You need to ask a biologist, a sociologist, or an epidemiologist.

Or you could take up statistical process control. Useful skill, and applicable to many things besides semiconductor manufacturing. Here's a hilarious quote that I've remembered for years, from http://www.pbs.org/cringely/pulpit/2003/pulpit_20030925_0004...

"I taught over 300 courses for industry where we designed cars and electronic devices, but it wasn't until one day I took over my wife's kitchen and used Taguchi to perfect my recipe for vanilla wafer cookies that I realized how broadly it could be applied," Kowalick recalls. "It took 16 batches, but by the end of the afternoon I had those wafers dialed in."

Go forth and bake some cookies.

[jbert]

Read some good sources:

http://www.badscience.net/2008/11/you-are-80-less-likely-to-...

http://www.google.co.uk/search?hl=en&q=the+tiger+that+is...

Perhaps take some public health scares and look up the papers behind them (if they exist). Try and work out if the scares are justified or not, based on the evidence.

Perhaps do some meta-analyses of studies to see if you can determine evidence of publication bias: http://en.wikipedia.org/wiki/Publication_bias

Find something that interests you and Do Science To It.

[ntoshev]

http://netflixprize.com

Statistical language processing is also cool:

http://nltk.sourceforge.net/index.php/Main_Page

http://norvig.com/spell-correct.html

[hugh]

If numerati were a word, it'd be a plural. I'm reasonably sure the correct singular forms would be literatus (male) and literata (female).

Oh, as for your actual question? I can't think of anything.

[davatk → hugh]

Nitpicking here, but wouldn't it be numeratus and numerata?

[hugh → davatk]

Ummm, yes.

Drat.

[helveticaman]

I would still use stocks, like you said, for starting out. I don't know if you're unlikely to find new things, but it'll probably be more interesting than sports.

[dronethebone]

Look at some fun stuff like the Monty Hall problem, and Benford's law. These are delightful examples of how counterintuitive probabilities and statistics can be. If either of these things surprise you, you are in the great majority, and unlikely to perform practical statistical work without first doing a great deal of studying. Nothing to be ashamed of, not even Gauss was born knowing this stuff.

[Tichy → dronethebone]

Well I am familiar with these. I just never got my hands on a real world data analysis problem.

[fbbwsa]

i think there are still plenty of fun things to find in stock market data. most quants are looking at the same problem. i play with market data a lot and you'd be amazed how uneducated the general public seems to be.

also, i second ntoshev's recommendations for netflix prize -ish applications. collaborative filtering/clustering seems to be an inadequate science at best as far as i can tell.

[sammyo]

If you're a hacker you might look for a niche in the R project and do some development.

[scott_s]

Look at graduate programs in statistics.

[Tichy → scott_s]

That occured to me, too, but is it really necessary? Also, will a graduate program lead to practical applications?

[scott_s → Tichy]

It's never necessary, but it might be the easiest place to find interesting problems to work on, and the place that is most conducive to study. It's also probably the best lead-in to finding a job in the area.

You don't have to do a PhD. Look up some Master's programs, which typically take two years.

Disclosure: I'm a Computer Science PhD student.

[hugh → Tichy]

I'm sure it's not necessary. You haven't said why you want to learn statistics. Is it just for fun, or do you have some eventual practical application in mind?

Personally, I just learn the bits I need, as I need them.