
Empirical Bayes for multiple sample sizes - csaid81
http://chris-said.io/2017/05/03/empirical-bayes-for-multiple-sample-sizes/
======
phillc73
Although tangentially linked to in the article, David Robinson's Introduction
to Empirical Bayes[1] is also an excellent resource. It deals primarily with
beta-binomial distributions.

[1] [http://varianceexplained.org/r/empirical-bayes-
book/](http://varianceexplained.org/r/empirical-bayes-book/)

~~~
csaid81
It's an excellent blog post, although it's worth emphasizing that it is
designed for the binomial case, where you wish to compute the fraction of
occurrences within some events, such as batting averages. For continuous
variables, however, it makes more sense to use one of the methods described in
the original post.

TL;DR: One blog post is for Rotten Tomatoes and the other is for Metacritic.

~~~
phillc73
Absolutely, and thanks for better defining the distinction.

I really just wanted to point out another solid Empirical Bayes resource, as
there's not that many about. Yours and David's make a good combination
covering different cases.

------
nl
That definition of symbols! So good!

~~~
nerdponx
I know! I love when math books and papers do that.

------
robterrin
Stan is great! Glad to see it on HN. Nice write up too.

------
pps43
Technical term is overkill. Just use
[https://en.wikipedia.org/wiki/Bayesian_average](https://en.wikipedia.org/wiki/Bayesian_average)

~~~
apathy
Thank you. As a statistician, the fact that mixed effects models (e.g. does
this rater tend to rate high?) are overlooked is, IMHO, a death sentence. Too
much nomenclature, too early (link to the table within the text, please, and
omit needless words), and with too little attention paid to the value of an
external citation.

Also, MCMC for ratings? Surely you jest. If the author had touched on mixed
models, then maybe it would make sense. But given the sample sizes involved
here, and the noise in the variance estimates, I recommend that the author
investigate mixed models _tout suite_ if they do in fact care about the
sources of shared and unshared effects on variance. Because that is what mixed
models do.

~~~
csaid81
Author here. Please see the section on mixed models in my post. As I mentioned
there, I would love if an expert could expand on the relationship between
mixed effects and Empirical Bayes.

Regarding MCMC, one of the things I try to emphasize throughout the post is
that the best solution depends on your needs (for example if you want a full
posterior). In fact, most of the post is devoted to quick and simple methods
-- not MCMC -- because they are good enough for most purposes. I welcome your
feedback though on how I could make this point clearer.

~~~
apathy
> Author here.

Alright, I'll put on my Reviewer Number 3 hat and say that I learned some neat
things from your work, including that the National Swine Improvement
Federation. I'll try and do a halfway decent job here.

> I would love if an expert could expand on the relationship between mixed
> effects and Empirical Bayes.

A real expert? Here you go:

[http://statweb.stanford.edu/~ckirby/brad/LSI/monograph_CUP.p...](http://statweb.stanford.edu/~ckirby/brad/LSI/monograph_CUP.pdf)

Read it, all of it, but particularly chapter 1, section 2.5, and chapters 8,
10, and 11. Why does testing, effect size estimation, and high-dimensional
analysis have anything to do with anything? Because...

1) independence is largely a myth 2) you are likely to have multiple ratings
per reviewer on your site, whether your generating distribution is nearly-
continuous (0-10, mean-centered) or discrete (0/1, A/B/C). If you discard
this, you are throwing away an enormous amount of information, and failing
utterly to understand why a person would estimate not just the variance but
the covariance _even for a univariate response_.

The second point is the one that matters.

Also, "empirical Bayes" is in modern parlance equivalent to "Bayes". What's
the alternative? "Conjectural Bayes"? (Maybe I should quit while I'm ahead,
pure frequentists may be lurking somewhere)

> I welcome your feedback though on how I could make this point clearer.

For starters, edit. Your post is too damned long.

Think about where you are getting diminishing returns and why. Is there ever a
realistic situation where your ratings site would not keep track of who
submitted the rating? (It's certainly not going to be an unbiased sample, if
so; the ballot box will get stuffed) So if you have to keep track of who's
voting, you automatically have information to decompose the covariance matrix,
and everything else logically follows.

A univariate response with a multivariate predictor (say, rating ~
movie*rater) can have multiple sources of variance, and estimating these from
small samples is hard. When you use a James-Stein estimator, you trade
variance for bias. You're shrinking towards movie-specific variance estimates,
but you almost certainly have enough information to shrink towards movie-
centric and rater-centric estimates of fixed and random effects, tempered by
the number of ratings per movie and the number of ratings per rater.
(Obviously you should not have more than one rating per movie per rater, else
your sample cannot be unbiased).

I think you will return to this and write a much crisper, more concise, and
more useful summary once this sinks in. I could be wrong. But you'll have
learned something deeply useful even if I am. I do not think you can lose by
it.

~~~
rcthompson
> Also, "empirical Bayes" is in modern parlance equivalent to "Bayes". What's
> the alternative? "Conjectural Bayes"?

My understanding of the difference, as a frequent user of empirical Bayes
methods (mainly limma[1]), is that in "empirical Bayes" the prior is derived
empirically from the data itself, so that it's not really a "prior" in the
strictest sense of being specified _a priori_. I don't know whether this is
enough of a difference in practice to warrant a different name, but my guess
is that whoever coined the term did so to head off criticisms to the effect of
"this isn't _really_ Bayesian".

[1]:
[https://bioconductor.org/packages/release/bioc/html/limma.ht...](https://bioconductor.org/packages/release/bioc/html/limma.html)

