
It’s time to adopt modern Bayesian data analysis as standard procedure - snth
http://www.indiana.edu/~kruschke/AnOpenLetter.htm
======
hessenwolf
1\. Your list of references proves that Bayesian statisticians have been
writing papers across a variety of disciplines.

2\. Bayesian methods are not readily computed with today's hardware and
software, and my desk is a counter-example.

2.1 Last time I fitted a Bayesian model, I had 600 processors with infinibandy
things joining them up and left it for a week.

2.2 None of the software I use does Bayesian by default.

3\. In practice, a lot of statistics in industry can be done by barcharts. I
know that that is hard to hear. It is a big leap from there to Bayesian;
bigger by far than from there to chi-sq tests. Data are so sparse, expert
judgement so rich, and time so short...

4\. Priors introduce subjectivity - no doubt about it. However, so do utility
functions, pretty much cancelling it in my opinion. It is inappropriate to use
p-values as a decision making framework for various reasons, but a lot of
scientific papers are about recording experimental observations, not decision-
making. Policy-decisions should use utility functions and priors; but I am
happy with my science papers frequentist.

~~~
nazgulnarsil
priors _acknowledge_ subjectivity.

~~~
Sniffnoy
I think about this point someone should link to [share likelihood ratios, not
posterior beliefs]([http://www.overcomingbias.com/2009/02/share-likelihood-
ratio...](http://www.overcomingbias.com/2009/02/share-likelihood-ratios-not-
posterior-beliefs.html))... the summary is that you can do your Bayesian
analysis without specifying the prior and just report the resulting likelihood
ratios, telling everyone, "Here are the likelihood ratios, update your beliefs
appropriately." Though that may lack some practicality.

------
yannickmahe
Could somebody explain what is Bayesian analysis and how it works or point to
an article explaining it ?

The wikipedia article regarding Bayesian analysis is cryptic to me.

~~~
hessenwolf
Frequentist Statistics:

1\. Start with a model that you think describes your data.

2\. Find the parameters of the model that make it fit the observed data most
closely.

3\. If the model really does not seem to fit the parameters, reject the model.
The criteria for it not to fit is usually the p-value, which is the
probability of data at least as unlikely as your observations occurring if
your model is actually correct. If the p-value is less than .05 (by an
arbitrary and accidental custom) the model is rejected. Wrap your head around
that, if you dare.

4\. If the model is not rejected, use it to predict future outcomes with the
best-fit assumptions, ideally allowing for some uncertainty in the values of
the parameters. Decisions are made on this basis. The uncertainty in the
values of the parameters is quite difficult to allow for in practice.

Bayesian Statistics.

1\. Start with a model that you think describes the data.

2\. Then get some models to model the parameters of your model, called prior
models, describing your uncertainty about the parameters of your model. You
get the initial parameters for the prior models from your own head, or the
head of an field expert.

3\. Use the observed data to refine your estimates of the parameters for the
prior models. So you started with educated guesses of these 'prior'
parameters, and then you made the guesses better using your observations. In
practice, your guesses can become irrelevant very quickly as you add more
data.

4\. Predict future outcomes using your model, where the uncertainty in the
model parameters is modelled explicitly using your prior models as described
in 3.

5\. Put those predictions of future outcomes into a utility function to make
decisions.

Summary: The main thing is that Bayesian statistics allows you to specify
models for your parameter uncertainty, provided you are okay with the educated
guesses.

~~~
tel
Because hessenwolf and I are philosophically opposed here, I'll give my take.
Balance your impressions between us as you choose.

\---

Frequentist statistics attempts to answer the question "How will my
experimentation appear knowing that there is some hidden, unknown truth to the
world which generates it?" The methods then proceed to use a variety of clever
arguments to show that seeing a certain experimental result (considering _all
possible experimental results_ ) constrains the possible underlying reality
and gives you a good guess at to what it is (and allows you to estimate how
much it might vary).

Bayesian statistics asks the very different question "How does this
observation I'm making affect my current knowledge of the world?" It is pretty
difficult to look at the methods without seeing an interpretive nod toward the
process of learning. To do this update step, Bayesians consider the relative
likelihood of _all possible underlying realities_ given that they've seen said
experiment.

It's not clear to me that these two methods are at all asking the same
question. In particular, they each consider (marginalize, integrate) vastly
different properties and their results have different interpretations.
However, since both of them fit into the space of quantifying the effect of
observation on the parameters of a model of the world they end up in constant
conflict.

Moreover, it's easy to construct Bayesian arguments which correspond to
exactly the same algorithms as some Frequentist arguments. Bayesians argue
then that their path to reach that algorithm is more interpretable and clear,
especially to non-mathematician. This method collision serves to further
conflate the two methods as enemies.

\---

tl;dr: Bayesian statistics is an average over possible realities, Frequent
averages over possible experimental outcomes. It's not clear that these are
comparable at all, but since they often try to answer the same questions we
compare them anyway.

~~~
hessenwolf
Nope. I agree with what you are saying; and would like my research presented
as summary statistics under different models (of which the p-value is one) for
sciency stuff, and expected utility values under different utility functions
and priors for decision making. I think the priors actually really become a
moot point after you tack on the utility function.

~~~
tel
I think that choice of model (even nonparametric or empirical distributions)
and choice of priors are linked. Both are assumptions based on prior knowledge
and analytical approach. Both are overwhelmed by the data in a fertile
experiment.

Utility functions are a different beast though. They don't have an update
procedure and can wildly affect your decision. I'm also convinced they're the
best tool we've got so far, so I take it as an illustration that making
informed decisions is just _hard_.

Presentation of summary statistics is fine. I prefer presentation of full,
untransformed, unpruned data as well when feasible. It's, of course, often not
feasible. I also demand justification for why you think those summary
statistics are meaningful and under what kinds of situations they would fail
to capture the conclusion presented. Not saying that this isn't done in a
frequentist setting, but I think it's harder.

~~~
hessenwolf
You demand, eh?

Honestly, really, truly, honestly, i never bothered with p values as a
statistician except for two cases. The first is when performing a test for
somebody else to go into a standard article format. The second is when
automating reports on complex data.

P values are for people who you do not trust to make decisions. Graphs and
arrays of summary statistics fro. Several differentodels are for
statisticians.

Also, i disagree that model choice will be overwhelmed by the data.

~~~
tel
Hah, I should really write these with more care. I'd feel entitled to demand,
but more meekly expect that there's a bit more trust and convention in
scientific publication. Though that can be taken too far.

You're right that model choice can still break your analysis given large
amounts of data. I was thinking more in terms of a whole inquiry where large
amounts of data will help you to locate a model that extracts the maximal
information from your observations. If we're able to keep experimenting
forever, we pretty much assume we'll eventually get highly accurate maps of
the world.

The primary difference was in utility functions where no matter how long you
experiment they remain exogenous and static.

------
nycticorax
I more-or-less agree with the OP: I'm a neuroscientist, and it would be nice
if I could use Bayesian analyses in my papers without it being a point of
contention with reviewers.

But it seems like one of the big advances in frequentist statistics in the
last fifty years is the introduction of nonparametric methods, which don't
require you to make strong assumptions about the distribution of your data. My
understanding is that the field of Bayesian nonparametric inference is still
in its infancy.

Also, this paper seems relevant:

[http://stat.stanford.edu/~ckirby/brad/papers/2005NEWModernSc...](http://stat.stanford.edu/~ckirby/brad/papers/2005NEWModernScience.pdf)

(Bradley Efron is in the running for Greatest Living Statistician.)

------
araneae
While I like Bayesian statistics it is NOT a substitute for maximum likelihood
in many situations.

Additionally, some of his criticism of NHST is unfair because he criticizes
weaknesses Bayesian also has. In the part where he gives the example of the
pollster and how uncertain the p value would be because you have to
incorporate sample design- well, you have to do that in Bayesian stats too.

Statistics is a big field. Obviously people should expand their tool set, and
maybe Bayesian is underused, but that doesn't mean Bayesian stats are right
for every experiment and experimental design.

~~~
jules
Maximum likelihood estimation falls out of Bayesian statistics with the right
utility function. In practice though, that's often not the utility function
you want, hence general Bayesian statistics.

