
Does Diversity Trump Ability? An Example of the Misuse of Mathematics [pdf] - todd8
http://www.ams.org/notices/201409/rnoti-p1024.pdf
======
nostromo
I was shocked to read the original article. I can't believe that such
overarching conclusions about human nature were drawn from a little toy box
computational experiment. You might as well publish a psychology paper after a
weekend playing The Sims.

[http://vserver1.cscs.lsa.umich.edu/~spage/pnas.pdf](http://vserver1.cscs.lsa.umich.edu/~spage/pnas.pdf)

~~~
kenjackson
It also seems to defy common sense. How many top ACM programming teams
randomly pick programmers from the college? How many great bball teams are
randomly picked from players? There are few activities, where excellence is
easily measured, where randomly picking participants results in superior
results.

~~~
DanBC
Moneyball shows us that all that knowledge and experience used to pick a
baseball team was wrong and that people should ignore their "common sense"
(which is usually just another word for "cognitive bias") and should use hard
data.

In the absence of hard data just picking a team randomly froma pool of
qualifying players seems to me like it would be as good as pickping players
based on the team-picker's experience of the game and "cultural fit" and other
bullshit.

~~~
scythe
>Moneyball shows us that all that knowledge and experience used to pick a
baseball team was wrong and that people should ignore their "common sense"
(which is usually just another word for "cognitive bias") and should use hard
data.

Nothing of the sort. Moneyball showed that an actuarial method was better than
other methods in picking a baseball team. This does not mean that the previous
methods were no better than random chance. In the case of baseball there was a
large corpus of data which had good correlations to players' future
performance. That's a rather unique situation.

In this case assuming that whatever previous methods are immediately wrong was
on a _much_ weaker basis: just the dataset from one particular toy challenge.
A comparison with Moneyball is misleading.

------
amathstudent
As someone who knows it intimately, I would like to say that this kind of
thing is endemic across the social science literature. What is frustrating is
that only very few people seem to have the ability to understand why it is not
sound.

~~~
jev
Probably because the mathematicians don't seem to have the ability to explain
why it is not sound without devolving into obscure symbols and jargon.

~~~
dxbydt
If this was html & written to the point instead of stuffed in a pdf & written
up in this roundabout manner, it would be on top of HN with 100+ votes :) I
read the whole pdf but here's the TL:DR; anyways -

Randomization improves algorithms. This is so obvious to CS folks its taught
in basic cs101 - the ones i'm familiar with are where you pick a random pivot
element for quicksort, or say the one where you draw a circle inside a square
& pick n random points inside the square, four times the number of points
inside the circle will equal pi as n becomes large, or you resort to the
miller rabin test for primality when you are doing those rsa calculator type
problems where you pick keys for encryption, which will randomly sample some
number of possible witnesses and call the number prime if none turn out to be
witnesses, or using monte-carlo methods to compute integrals for functions for
which no closed-form formula exists etc....there's like tons of examples where
you introduce a little bit of randomness & it'll speed up your algorithm.

So this Page+Hong wrote a paper where they want you to randomly hire employees
( who they call problem-solvers or agents ), because diversity trumps ability
?! So if you have 1000 applicants, instead of hiring based on some metric like
ivy-league/test-scores/IQ/github/whatever, you just hire randomly, because the
randomness introduces diversity which trumps ability ?!! To prove this
nonsensical point, they introduce an artificial math problem where agents
proceeding randomly obtain the right answer, & not doing so gets you stuck.
Ergo, diversity > ability. Sheesh.

~~~
yummyfajitas
This is in Notices of the AMS, which is about as mainstream (together with
SIAM Review) as mathematical publications get.

 _Randomization improves algorithms._

This is a contentious claim. Random choices are actually very rarely the best
- they are often good enough and versatile, but they are rarely the best.

For example, consider Monte Carlo integration. You get O(N^{-1/2})
convergence. If you use a deterministic set of points explicitly designed to
have low discrepancy (aka "Quasi-Monte Carlo"), you can get O(N^{-1 +
logarithmic stuff}) convergence.

[http://www.chrisstucchio.com/blog/2014/adversarial_bandit_is...](http://www.chrisstucchio.com/blog/2014/adversarial_bandit_is_not_statistics_problem.html)

Eliezer Yudkowsky also wrote a great critique of this issue, though I can't
find it right now.

~~~
zwegner
I believe the article you're referring to is:
[http://lesswrong.com/lw/vp/worse_than_random/](http://lesswrong.com/lw/vp/worse_than_random/)

Choice quote: > As a general principle, on any problem for which you know that
a particular unrandomized algorithm is unusually stupid - so that a randomized
algorithm seems wiser - you should be able to use the same knowledge to
produce a superior derandomized algorithm.

An interesting counterexample to this, an example which to me illustrates a
very powerful aspect of randomness, is the monte-carlo revolution in computer
Go (AI for the ancient Asian board game). For years, computer progress
stagnated, as the techniques were mostly focused on encoding human knowledge
into code. While most of this human knowledge is generally correct, it
introduces significant bias in the way positions are evaluated. The way these
rules-of-thumb interact is very hard to predict, and tree search algorithms
are quite good at finding positions that are incorrectly evaluated. Because of
this, the worst-case behavior of an evaluation function is much more important
than it's best-case or average behavior.

Computers started making lots of progress when a new technique was used:
rather than try to evaluate positions by a large set of heuristics, they were
evaluated by playing random games. Go positions are quite hard to evaluate
from simple rules of thumb, and this random game approach gives a much more
balanced, long-term view of positions. And most importantly, there is much
less bias.

Obviously, an entirely random game, with a uniform distribution over possible
moves, is easy to improve upon. But computer go programmers noticed an
interesting phenomenon: while certain types of knowledge incorporated into the
random move distribution (to make it "more intelligent", as judged by a human)
were helpful, others were not (even after taking into account the
computational cost of adding the knowledge), and it wasn't always clear why.
The same observation about heuristic evaluation noted above applied: having a
balanced distribution of move choices, with a reasonable probabilistic lower
bound of effectiveness, is more important than making an intelligent choice
that is usually correct, but has unpredictable, extreme worst-case
performance.

So we see that randomness does have an important property: it avoids the
downside of "knowledge" that generally seems correct but can go horribly wrong
in unexpected ways.

I don't know of a good writeup of this phenomenon. My understanding of it is
mostly assembled from following informal discussions on the computer-go
mailing list for several years. In a quick search through my gmail archives I
can't find much on the subject, but here's an interesting post about related
topics in the computer chess world (that incidentally doesn't talk about
randomness, but illustrates well the benefits of avoiding bias):
[http://www.talkchess.com/forum/viewtopic.php?topic_view=thre...](http://www.talkchess.com/forum/viewtopic.php?topic_view=threads&p=135133&t=15504)

~~~
yummyfajitas
I don't know a lot about this Go example, but using a uniform distribution for
the evaluation function sounds surprisingly similar to another phenomenon I
observed.

Suppose you have a linear evaluation function - h(x) is the value of
something. Suppose also the coefficients of h are all positive. Then you'll be
right 75% of the time (averaged over all possible h, drawn uniformly from the
unit simplex) if you just approximate h=[h1,h2,...] by u=[1,1,...,1].

[http://www.chrisstucchio.com/blog/2014/equal_weights.html](http://www.chrisstucchio.com/blog/2014/equal_weights.html)

So I agree with this claim - uniform distributions are fairly robust to
errors. But I don't think that's particularly related to randomness - Monte
Carlo is only needed to integrate the distribution.

It's also worth noting that adversarial situations (like Go or Chess) are
considerably different than most other cases. In a true adversarial problem,
there is no probability distribution - the opponent is omnipotent. The purpose
of randomness is simply to reduce the power of the adversary's intelligence -
in a completely random world, intelligence is useless.

~~~
zwegner
That's not too different from a pretty old observation in the chess world: the
presence/absence of an evaluation term is more important than the weighting
given to it.

> So I agree with this claim - uniform distributions are fairly robust to
> errors. But I don't think that's particularly related to randomness - Monte
> Carlo is only needed to integrate the distribution.

Ah, that's an interesting distinction, thanks. I'll have to think about this
some more. But given a situation where exact integration is intractable (like
chess or Go), I'm not too sure what the difference really is, because it is
those cases (on first thought) where the uniform distribution is useful--if
you can see to the end, you don't need to care about bias, right? I mean,
"randomness" in the strictest sense is not really necessary; all these
programs I speak of used deterministic pseudorandom generators of course. It's
really just about ensuring lack of bias given finite sampling. I'm happy to
hear your take on it though--you definitely seem to have a lot more knowledge
of math/statistics/etc. than I do.

(That does remind me of another fascinating tidbit from the Go world:
programmers noticed that using a low-quality PRNG, like libc's LCG rand(),
produced significantly weaker players than more evenly-distributed PRNGs, even
though it would seem that playing lots of random games of indeterminate length
(with the PRNG called at least once per move) would not correlate at all with
the PRNG's distribution.)

The adversarial-or-not issue is also good food for thought. I'm not convinced
that it explains much in this case, though, since I believe most of these
observations were made by playing computer-computer games with each program
using very similar algorithms, or with old hand-tuned programs against the
newer Monte-Carlo based programs.

~~~
yummyfajitas
_But given a situation where exact integration is intractable (like chess or
Go), I 'm not too sure what the difference really is, because it is those
cases (on first thought) where the uniform distribution is useful--if you can
see to the end, you don't need to care about bias, right?_

Put it this way - suppose I can cook up a deterministic quadrature rule, e.g.
quasi monte carlo or an asymptotic expansion. I assert that the quasi monte
carlo will work just as well as monte carlo, probably better if convergence is
faster.

If I'm right, this is a situation of "yay for uniform distributions". If I'm
wrong, it's a "yay randomness" situation. It's nice to know which situation
you are in - if I'm wrong, there is no point cooking up better deterministic
quadrature rules.

Incidentally, LCG is known to be useless for monte carlo due to significant
autocorrelation. So it's quite possible that people using LCG are incorrectly
estimating their evaluation term.

Also for me, it's nice to know these things just for theoretical purposes and
to enhance my understanding.

------
bayesianhorse
In finance it has long been known that diversity in a portfolio can often
trump the performance/ability in a particular stock.

Turns out, you can see portfolio theory popping up outside of financial
markets, not only in assembling a team, but also advertising campaigns, ant
colonies and bacterial colonies.

And yes, I know that the linked article debunks a paper that abuses math...
It's just that portfolio theory is at least an analogy to understand why
diversity can trump ability. Within reason.

~~~
Guthur
In my opinion the reason "portfolio theory" works in the cases that you
mentioned is that it most perform well across a time series within a highly
dynamic environment. But if one is allow to pick an optimal team for each
discrete problem then it will surely outperform.

In my mind diversity will help when you have little to no a priori knowledge
and then for can not predict what attributes you will need.

~~~
bayesianhorse
Have you never seen a software development project which ran into
unanticipated problems?

What about "knowledge work" is not a highly dynamic environment?

------
4bpp
The response paper seems to harp a lot on how the circumstance that typically,
N_1 and N >> k, precludes any realistic applicability of the original theorem
-- but it admits a fairly self-consistent and altogether much more sinister
interpretation if you call the different \Phi "groups" (or "races" for maximum
creepiness). The assumptions then say something to the effect of "every
group's approach to problem solving produces fundamentally predictable
results, and no group can be the best at solving each type of problem", which
is a meme that in some form has been floating around in diversity activism
circles for a long time.

The "mathematical theorem" then simply captures the triviality that under this
sort of worldview, you want your team to contain Zorblaxians (sorry, SMBC)
because there are some Zorblaxian problems that Zorblaxians have a natural
affinity towards, and nobody without such an affinity could possibly make
progress on. Since this is not stated explicitly, the statement becomes
invokable even in settings in which otherwise a large number of people would
raise eyebrows at the smell of exoticist quackery that the idea of "different
ways of knowing" exudes.

~~~
QuantumChaos
Your post captures the problem much better than the linked article.

I would summarize the problem as being that the result is tautological, but
the gloss of mathematics gives it the appearance of not being tautological.

If people said "diverse teams are better because people from different
backgrounds are good at solving different kids of problems" then that would be
accurate, but leave room for debate about whether then assumption is true. If
they say "diverse teams are mathematically proven to be better", this would be
inaccurate, but give great ammunition to argue that science supports
progressive views.

------
cjdrake
Richard Feynman has some relevant thoughts on this subject:
[https://www.youtube.com/watch?v=IaO69CF5mbY](https://www.youtube.com/watch?v=IaO69CF5mbY)

------
nether
One of the authors, Scott Page, majored in math at the University of Michigan.
He also teaches the course "Model Thinking" on Coursera:
[https://www.coursera.org/course/modelthinking](https://www.coursera.org/course/modelthinking).
Perhaps one to avoid...

~~~
dxbydt
Hey that's a great course. Pls do not avoid. Highly recommend signing up. The
reason he wrote this famous paper & the book that evolved from that paper, in
his own words -
[http://vserver1.cscs.lsa.umich.edu/~spage/thedifference_inte...](http://vserver1.cscs.lsa.umich.edu/~spage/thedifference_interview.html)

He says - "progress and innovation may depend less on lone thinkers with
enormous IQs than on diverse people working together"

Not too sure about that. At all.Especially not in math, the subject he majored
in.

~~~
eropple
_> He says - "progress and innovation may depend less on lone thinkers with
enormous IQs than on diverse people working together"_

 _> Not too sure about that._

Why would that not be a thing?

I mean, I am not the smartest person on my team. The list of things I know
very little about is _long_. But I occasionally ask questions that are really
obvious to me that reorient the discussion because people didn't think of them
--problems that were Too Obvious To See up close, you know? And, similarly,
when working in stuff that I _do_ know a lot about, I find myself ignoring
things that are to me so obvious and basic that my brain just goes right by
them.

Innovation isn't like Civilization, you aren't just generating lightbulbs.
Diversity of thought process leads to some inefficiencies, but it helps you
reach new global maxima.

~~~
jrapdx3
I'm not so sure diverse (divergent?) approaches to a problem are _always_
better than one leader's effort.

For one thing, stand-out genius or talent shows up _randomly_ in populations,
it's not really predictable when or where it will occur. But on rare occasion
when a natural leader does emerge, suppressing diversity (of problem-solving
approaches) is likely the better strategy.

The true maxima of human effort have generally followed this pattern, born of
singularity and not diversity. Once an innovation is widely enough known, it
attracts a diverse range of followers who improve the idea and nurture its
maturation. Diversity is useful for the "aftercare" of innovation, but not its
creation.

The other major point is that value of diversity (however defined) vs.
uniformity depends on context. In the context of one's home, there may be many
ways to arrange furniture in a room, and with few exceptions, one way is as
valid as another. By contrast, in a shared household over time diverse
opinions are likely to converge to an arrangement using the space optimally.
Diversity probably leads to a better workflow solution vs. one person's
choices.

OTOH there is not a great diversity of "valid" ways to remove an inflamed
appendix, that is, regardless of other considerations, a qualified surgeon is
required and procedural diversity is constrained by the anatomical realities.
This situation demands uniformity not diversity.

Finally, "diversity" has an indeterminate number of definitions, and
applicability of the term is entirely dependent on context. On second thought,
that would seem to drain it of specificity or meaning, on that ground, I
should have a policy of using "diversity" carefully and sparingly.

------
HelloMcFly
If anyone is interested in better research on diversity and team outcomes, I
recommend looking into research on "faultlines" in teams. It is an interesting
way of operationalizing diversity and predicting its effects.

Link to abstract:
[http://amr.aom.org/content/23/2/325.short](http://amr.aom.org/content/23/2/325.short)

I can't find a direct link to the PDF at the moment. That's the original
article. There's been a lot of research since testing the theory, typically
supporting it (though I'm not 100% up to date on this topic).

------
Russell91
Am I missing something or is the author of this pdf's proof equally flawed.
They assume each function is idempotent and injective ... meaning that it is
the identity function. And the proof doesn't follow. A much more natural way
to have "fixed" the original proof would be to require that V: X -> R be
injective.

~~~
4bpp
Assuming by function you refer to the "agents'" \Phi, where do they make the
assumption that it is injective? I see nothing to the effect in the text, and
the functions in the counterexample on page 6 are not injective.

~~~
Russell91
Oh, oops. I was reading their statement: This interpretation is incorrect
without the additional hypothesis that V (x) is a one-to-one function. as -
all of the phi functions must be one-to-one. Looks like they made the same
assumption that I would have.

This was bothering me though so thanks for your response.

------
rlvesco7
There is one positive aspect to all this that I would like to point out.

Most social science does not use math, so it makes it hard to argue against
ill-defined concepts and ideas. So the fact that these mistakes were found is
actually a positive thing and shows one reason why math is useful. That's not
to say math cannot be used to obscure arguments and assumptions, it can, but
words can do that too.

------
pervycreeper
That was surprisingly entertaining. Any recommendations for other academic
papers in that vein?

------
nether
> Page’s work on diversity has been cited by NASA, the US Geological Survey,
> and Lawrence Berkeley Labs, among many others

Oops.

------
im3w1l
Has there been any attempt at quantifying the "wishful thinking bias" in
social science?

~~~
amathstudent
Here's something you might find interesting:
[http://vserver1.cscs.lsa.umich.edu/~crshalizi/weblog/698.htm...](http://vserver1.cscs.lsa.umich.edu/~crshalizi/weblog/698.html)

There have are also loads of replication projects, some currently ongoing, of
both experimental and statistical papers, with generally dismal results.

