
How to Avoid Idea Averaging - andrewxhill
http://andrewxhill.com/blog/2016/01/04/idea-averaging/
======
mizzao
Idea averaging is not an example of "the wisdom of crowds". It's a
manifestation of groupthink.

More specifically, it's an example of when a group of people will tend to
agree on something that is the least objectionable on average. This means that
it's less likely that an unconventional idea will be accepted.

This concept has been studied for a long time in the social psychology
literature. For a survey of related topics:
[http://chicagounbound.uchicago.edu/cgi/viewcontent.cgi?artic...](http://chicagounbound.uchicago.edu/cgi/viewcontent.cgi?article=1165&context=law_and_economics)
(see specifically "Hidden Profiles and Common Knowledge").

~~~
p4wnc6
Since most unconventional ideas tend to fail (because people have thought for
a long time about what would be good, and over time those things have often
become conventional), it is a useful heuristic to go with conservative,
conventional ideas. I think this totally fits in with "wisdom of crowds".

Of course, this also means you won't generate that once-in-a-million super
great _unconventional_ idea. But then again, I think that's the point. If you
try to generate the great unconventional idea, the base rate of failure is so
high that you should not have so much hubris as to believe you actually can do
it, and most often you should admit that what you think is a great
unconventional idea really probably can't do any better than whatever
conventional wisdom would choose.

I think my view on this is not popular because everyone wants to believe in
the idea underdog -- the idea that was spurned by conventional wisdom but
proved itself regardless and made someone famous for executing upon it.

That's great and all, but in the same way that putting all of your money on a
single stock is a bad strategy, failing to average across ideas embraced by
conventional wisdom is also (often, but not always) a bad strategy.

On a different note though, there is a lot of value in your comment even
connecting it back to the "wisdom of crowds" stuff. One of the issues with
"wisdom of crowds" is that the aggregated average tends only to be better
whenever there is a sufficient variance of opinions in the crowd. This way,
there is not systematic bias in everyone's thinking, meaning that the average
too will have that bias. The more variation, the more that each participant's
particular way of being wrong will contribute to "canceling out" the errors,
leaving the average in a good position.

To me it suggests that when unconventional ideas do hit it big, it is because
they are highly related to some source of demonstrable bias in conventional
wisdom, and by removing that bias, such ideas can actually improve upon
whatever is the current state of conventional wisdom.

But most of the time, it seems we only hit upon these bias-based improvements
by accident because tons and tons of people keep trying (and failing, often to
their personal detriment) to commit to unconventional ideas that aren't
articulated in terms of any demonstrable bias, and it ends up basically being
an idea lottery to see who happens to hit the "jackpot" idea. That's what
makes conventional wisdom appealing as an alternative, steady expected return
with low risk, but it's also what means it will never lead you to the next big
thing.

~~~
meric
Ironic, then, conventional wisdom is a useful heuristic because of all these
unconventional ideas individuals do attempt, and so the answer to keep moving
humanity forward is, yes, go ahead and try that unconventional idea, or you
will contribute to systemic bias.

I would expect the "wisdom of the crowds" to have a different result if, when
asking for each person how many peas are in a bottle, each person is first
told of the current running average or median. I think it would achieve a
better result if everyone made their own guess independent of the "crowd".
Otherwise, the random bias that occurred from the first 20 people's guesses is
going to percolate because everyone who is guessing after will have their
guess anchored when they're told of the group opinion.

------
p4wnc6
Idea averaging is a wonderful thing, and you should probably do it way more
than you already do, and it should probably be your default mode when looking
to choose between ideas.

A lot of this is spelled out in the literature about why ensemble methods tend
to be superior to individual methods in statistical inference, which has a lot
of ties to why averages of non-expert opinions tend to be as accurate or even
more accurate than small pools of expert opinions.

In the limit, when you reallocate credibility to a set of ideas in a manner
that is fully derived from your prior beliefs and the evidence you have, it
becomes exactly Bayesian reasoning. A posterior distribution is exactly an
"idea average" where each idea (each potential outcome) gets as much (and no
more) credibility as it deserves, according to the prior and the probability
model at hand.

Here's somewhat of a popular account of it:

< [https://www.washingtonpost.com/business/is-it-better-to-
trus...](https://www.washingtonpost.com/business/is-it-better-to-trust-the-
best-expert-or-the-average-of-a-group-of-
experts/2013/11/15/59cc716e-4b01-11e3-be6b-d3d28122e6d4_story.html) >

There is also the recent stuff under the marketing buzzword "Wisdom of Crowds"
e.g. <
[http://onlinelibrary.wiley.com/doi/10.1111/j.1551-6709.2011....](http://onlinelibrary.wiley.com/doi/10.1111/j.1551-6709.2011.01223.x/abstract;jsessionid=58C236728B81FFF49A4E84DF1E3B65CB.f02t04)
> or the Wikipedia article on it too (and many sources that point out
potential problems and corner cases that are also important).

In general though, I think I have to disagree with the article's premise.
Unless two distinct ideas truly are mutually exclusive (most things aren't),
then it's better to have a diversified portfolio over the space of all the
ideas than to put all your eggs in one basket, and averaging ideas is sort of
a humble heuristic that seems to work in a lot of areas.

~~~
davmre
This is a very confused post. A posterior is not an "average", it's a
probability distribution. That distribution may _have_ a mean value, but
that's not a meaningful quantity in general.

For example, you're chasing someone and you come to a fork in the road. You
know they must have taken the left or the right branch, so your posterior
distribution on their location has two modes. It also has a mean value, in the
middle of the forest between the two branches, but unless you have specific
evidence they've abandoned the road, this is a very low-probability state.

The type of averaging prescribed by Bayesian decision theory is averaging of
_utilities_ , not of beliefs or actions. You take the action having the
highest expected utility, where the expectation is taken over your posterior
distribution. Assuming you actually want to catch your suspect, the expected
utility of following either the left or the right fork will be much higher
than the utility of dropping the pursuit to poke around in the middle of the
forest.

~~~
p4wnc6
I think you are mistaken. Yes, a posterior is a probability distribution,
which is exactly taking each possible outcome in accordance with its
probability. I didn't say anything about collapsing the distribution down to a
single point estimate like the posterior mean, or the MAP estimate, or any
other single statistic. I am saying that I hold in my head a bunch of beliefs
all at the same time (thus they are "blended" or "averaged" together), each in
accordance with the amount of credibility it deserves. I think it's totally
fine to speak about this as a type of uncollapsed "averaging" process, and
indeed when you do something like hierarchical models where you supply a
metaprior distinguishing between two alternative models, we widely talk about
such things as _model averaging_ even in the statistics literature. It seems
like a rather misguided nitpick to me to insist that the English word
"average" cannot be invoked unless it specifically coincides with the
statistics word "mean".

Further, you're simply wrong about Bayesianism being about averages of
utilities. The best account of Bayesian probability as a mapping of
plausibilities to the unit interval is in Jaynes' _The Logic of Science_ , but
there is also a brief account of it in David Mumford's essay _The Dawning of
the Age of Stochasticity_ and e.g. in the introduction of _Bayesian estimation
supersedes the t-test_ by Kruschke [1] (where he explicitly describes it in
terms of reallocation of credibility). Further, a la Jaynes, I think the right
way to understand probability at all is in a mind-projection fallacy sense of
the term: it describes your state of ignorance about the uncertain item.

It's not about utilities of actions -- utilities can be modeled with
probabilities if the utility of certain actions is uncertain, but that is
different than the base concept of probability being about which outcomes are
more valued.

[1]
[http://www.indiana.edu/~kruschke/BEST/BEST.pdf](http://www.indiana.edu/~kruschke/BEST/BEST.pdf)

~~~
davmre
I read the original (andrewxhill's) post as making a point analogous to the
"fork in the road" example: it's fine to maintain a full posterior on ideas,
but if you need to choose, you should put in the effort to choose the best
idea, not blindly try to merge them into a single idea that might combine the
worst of both worlds. The latter would be like searching the forest between
the two roads instead of just choosing a branch. Under this reading the point
seems quite sensible to me.

You seem to have read the post quite differently, in a way that causes it to
seem totally wrong.

I actually do think it's just a fact that "average" in both colloquial _and_
mathematical usage means something like "to collapse multiple values to a
single typical value" (even Bayesian model averaging collapses a distribution
over probabilities into a single probability). But even if it were genuinely
ambiguous what andrewxhill meant, you generally get a lot more out of a post
by choosing the reading that allows it to be insightful over the reading the
causes it to be nonsensical.

~~~
p4wnc6
First, model averaging is a bit different than when you describe it as (at
least in my reading, but maybe you mean "single probability" differently than
I read it):

> even Bayesian model averaging collapses a distribution over probabilities
> into a single probability

as I mentioned in the child comment to the other reply to this comment, model
averaging creates an average value, but the type of that value is _a
distribution_. That is, you have a distribution _over distributions_ (each
coming from a different model) and the effect of averaging does not reduce you
down to a point estimate, rather it reduces you down to just one distribution.

This is why it's totally fair to say a posterior can be an average. It is the
average of a bunch of other distributions. I think if I had said it that way
in my first comment, it would have removed some confusion.

But it is important, because the criticism that "a posterior isn't an average"
is very wrong. A posterior most certainly can be the average of some other
stuff, if that other stuff was itself a bunch of _distributions_ \-- and
that's exactly what I am trying to talk about.

But to your other point, about the 'sensible' vs. 'not sensible' readings, I
mostly agree. However, the problem is who gets to decide when two ideas fall
into the "knife-vs-spoon-clearly-exclusive" category, or when it's more gray
than that, and the choice is not so black and white, and there is not a need
to over-commit to just one approach?

The reason the OP post strikes me as problematic is that it seems like a
matter of opinion, or in the worst case a matter of bureaucratic/dictatorial
mandate, as to when ideas are of the type that can be averaged vs. when they
are not.

I'd generally like people to be more humble about it and tend to believe that
conventional wisdom and model averaging are better, at least as a first
heuristic, than deeply committing to just one thing. That way there might be
less urgency to rush into the claim that some debate is "knife-vs-spoon".

I sort of see the whole "knife-vs-spoon" thing as a kind of Godwin's law of
brainstorming. Once you invoke the "knife-vs-spoon-so-we-can't-average" claim,
it's like game over and all useful intellectual discussion dies and everyone
just either picks Team Knife or Team Spoon and then the political battles
start. Unless it's _really_ mutually exclusive, I'd rather that doesn't
happen.

------
cheetos
Also known as design by committee:
[https://en.wikipedia.org/wiki/Design_by_committee](https://en.wikipedia.org/wiki/Design_by_committee)

------
erikrothoff
What would you call this topic or category of theory? I'm interested in
learning more. Any books or literature that's recommended?

I've become the "product" guy in the startup I'm in, and would love to learn
more about helping a group innovate and push forward.

~~~
andrewxhill
Not sure... what part of it interests you? Teamwork? Innovation? Decision
making?

I was putting together a "best read of 2015" type post but never finished it.
There were a couple on my list that I think are related and maybe you'd find
interesting.

This one from HN a few weeks back, [http://justinkan.com/three-
stories](http://justinkan.com/three-stories)

Higher level company organization stuff I thought these two books gave at
times contradictory and at times very complimentary thoughts on what made
teams work. In reality, neither were exactly about making teams work though :)

Playing to Win [http://www.amazon.com/Playing-Win-Strategy-Really-
Works/dp/1...](http://www.amazon.com/Playing-Win-Strategy-Really-
Works/dp/1491528796) is on creating strategies

Reinventing organizations [http://www.amazon.com/Reinventing-Organizations-
Frederic-Lal...](http://www.amazon.com/Reinventing-Organizations-Frederic-
Laloux/dp/2960133501) is about holacracy but talks a lot to people's
motivations in a company.

Hopefully I have a couple more interesting things to say on andrewxhill.com
too, so keep an eye on it!

------
snowwrestler
The rotating image carousel seems like an example of idea averaging.

\- "What should we put on our homepage?"

\- A bunch of people have different ideas.

\- "I know, we can put them all up with a carousel!"

Even Apple has succumbed to his, which is perhaps not surprising. Carousels
often come out of collaborative cultures (everyone wins!), which has been a
point of emphasis for Tim Cook.

Avoiding carousels requires that a single person is willing to make a bunch of
colleagues unhappy by picking only one thing at a time to feature.

------
sharemywin
I think something the article missed is you should probably test your
ideas(cheaply if possible) because facts are better than opinions(apparently
drunken' ones in the authors case).

