Hacker News new | past | comments | ask | show | jobs | submit login

I have a simple criterion for a summary judgement of the reliability of results:

a) Is the data made available? b) Is it a Bayesian analysis? c) Has a power study been offered?

As a statistician, I have a keen awareness of the ways that p-values can depart from truth. You can see Optimizely's effort to cope (https://www.optimizely.com/statistics). You can read about it in The Cult of Statistical Significance (http://www.amazon.com/The-Cult-Statistical-Significance-Econ...). This Economist video captures it solidly (http://www.economist.com/blogs/graphicdetail/2013/10/daily-c...).

The key component missing is a bias towards positive results. Most scientists only have two statistics classes. In these classes they learn a number of statistical tests, but much less how things can go wrong. Classic, "just enough to be dangerous."

In order to cope, I have a personal set of criteria to make a quick first sort of papers. It's a personal heuristic for quality. I assume some degree of belief (Bayes, FTW!) that those that offer the full data set along side conclusions feel confident in their own analysis. Also, if they're using Bayesian methods, that they've had more than two stats classes. Finally, if they do choose Frequentist methods, a power study tells me that they understand the important finite nature of data in the context of asymptotic models / assumptions.

I'd suspect that other statisticians feel this way, because I've heard that privately --- what do you think of my criteria?




These are reasonable criteria.

I also tend to be very sensitive to failure to correct for multiple hypotheses, as this is something I see all the time, particularly when people start sub-setting data: "We looked for an association between vegetables in the diet and cancer incidence, but only found it between kale and lung cancer." This happens all the time, and people report such associations as if they were the only experiment being run, whereas in fact they have run some combinatorically huge list of alternative hypotheses, and unsurprisingly have found one that looks significant at p=0.05 (which is a ridiculously lax acceptance criterion.)

I also pretty much categorically reject case control studies: http://www.tjradcliffe.com/?p=1745 They are insanely over-sensitive to confounding factors. They can and do have legitimate uses to guide further research, but should never be used as the basis of policy or action beyond that.

There's also a sense one gets from many papers that the researchers are black-boxing their statistical analysis: that they have plugged the numbers into some standard package and take the results at face value. While I appreciate that maybe not everyone can have a solid technical grasp of this stuff, it always bothers me when I see that because it is far too easy to generate garbage if you don't understand precisely what you're doing.

[Disclaimer: I am an experimental and computational physicist who has never taken a stats course, but believe myself to be competently self-educated in the subject and have spent part of my career doing data analysis professionally using primarily Bayesian methods.]


I've encountered way too much "It must be good because its Bayes!", too much "It's Bayes because I used MCMC, ignore my flat uninformative prior..." etc. to put much stock in that as a metric.

I'm also involved in enough medical research where data just can't ethically be made available that, well...you and I clearly disagree.


R A Fisher is now promoting Bayesian analysis???

https://www.google.com/?gws_rd=ssl#q=Fisher+%22prominent+opp...




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: