
Ask HN: Tools to evaluate scientific claims? - pgt
What statistical knowledge and training do I need to evaluate the legitimacy of scientific papers, esp. in the medical field?<p>Whenever I read about any scientific claims, I ignore the press and go straight to the original paper cited (if there is one, often it is misquoted). I then read the abstract and the testing methodology. If I can spot any issues in the methodology I usually stop reading, e.g. small sample size, obvious confounding variable, blatant causation&#x2F;correlation errors. But if all seems well, that still doesn&#x27;t tell you if the study&#x27;s claims match the test results or if the threshold parameters make sense.<p>Given a basic stats background, how can I obtain a deeper, intuitive understanding  of things like p-values (which seem to be outdated anyway) and other sample sizes. Thanks, HN.
======
veddox
Get a good book on statistics?

About p-values: they aren't exactly outdated, but they are the subject of a
pretty fierce controversy. Many, if not most scientists still use them when
doing statistical analyses because they are simple to apply and to understand
and provide a quick metric for measuring the significance of a study's
results.

Other scientists say that they are _too_ simple and don't convey important
information about the actual data (such as the spread). Apparently, there are
more modern statistical procedures that do a better job than p-values do. (I'm
not a statistician though, so don't ask me what these procedures are...) Also,
p-values are all too often subjected to "p-hacking" \- massaging the data
until you get a statistically significant result (p <= 0.05). In fact, the
very concept of significance is problematic. Originally, the p <=
0.05/0.01/0.005 significance limits were just approximate guidelines to help
scientists interpret their data. Nowadays, they are often treated as definite
boundaries of "truth". ("If my data gives a p-value of 0.049, the result is
significant therefore my hypothesis must be true. If p=0.051, it is not
significant, therefore my hypothesis must be wrong - or I must tweak my data
until I get p=0.05.") This is obviously nonsense, yet a surprisingly common
attitude (though not always as extreme as in my example).

As far as I personally am concerned, the real problem is not the actual
p-values as such, but perhaps a lack of understanding of statistics by many
scientists. (Coupled with the pressure exerted by journals that only want to
publish "significant" results and so indirectly encourage p-hacking.)

~~~
pgt
Can you recommend a good stats book?

~~~
veddox
Sorry, I'm afraid I can't :-/

I got my own statistics knowledge from a lecture series, a book on R, an
ecology textbook and various articles...

Check your local university library if you can or google around. I'm sure
you'll find something.

------
mbrock
Just curious, isn't this kind of basic legitimacy verification supposed to be
done by the peer reviewing process? Are scientific journals really publishing
papers so obviously illegitimate that laymen with a bit of statistical
knowledge can spot obvious errors or problems?

~~~
dagw
Things like a small sample size and confounding variable, doesn't in any way
invalidate a paper. Especially in fields where collecting data is difficult
(only a small number of people have a certain obscure disease for example, or
it's practically impossible to isolate one variable from another) those are
simply facts of life that you have to deal with. Sometimes you all you can do
is say I only managed to find 4 cases of X, but they all had Y. Was not able
to control for Z, still I wasn't expecting that and found it kind of
interesting. As long as you aren't faking data (like you actually found 12
cases of X but threw out the ones that didn't fit your conclusion), then that
is absolutely worth publishing.

It's always worth getting results out there even if they don't/can't live up
to statistical certainty one might hope for. Every paper doesn't have to stand
on its own and come to some undeniable ground breaking conclusion. Over time
you'll hopefully collect enough published data to be able to do some more
statistically powerful and useful meta-studies down the road.

~~~
veddox
Absolutely. Especially in the life sciences it can be very hard to get
statistically solid data.

None of which means there isn't a lot of statistics crap floating around,
though.

