
The wisdom of smaller crowds - mazsa
http://www.santafe.edu/news/item/Galesic-wisdom-smaller-crowds/
======
geoelectric
The article touches on it, but the clear differentiator I see here is whether
the "crowd" has expert knowledge of the domain.

If it does, then the noise of experts clashing (resolving conflicts between
equally informed ideas) starts outweighing the benefits of them contributing
the ideas to begin with. Basically you get a too many chiefs issue.

If it doesn't, you're seeking to average variably-educated wild ass guesses
and taking advantage of the fact that WAGs have a tendency to form a bell
curve around the actual answer if there's any ability to estimate whatsoever.
Since there's otherwise no special knowledge, that ends up being the best you
can get but requires a lot of people to get right.

Without having read up on it, my guess is the latter is related to central
limit, with each person's WAG forming the first term of the mean of means; a
WAG is itself sampling your own mental model and boiling it to a number. Given
enough of those you get a normal that you can average.

~~~
vegabook
this may be why the prediction markets were so terrible at guessing brexit.
Each participant was a kind of expert in said domain - not impassionate. Not
random. Moreover, it's possible that the crowd view was self re-inforcing.

~~~
eutectic
The chance of Brexit was hovering around 25% just before the vote; I wouldn't
describe that as 'so terrible'.

~~~
vegabook
as of 10pm in London last night, when one could reasonably expect it to be
getting more accurate, it was at 11%. And 25% was in any case a very poor
prediction considering these markets are supposed to be more accurate than
polls.

~~~
smeyer
>And 25% was in any case a very poor prediction considering these markets are
supposed to be more accurate than polls

If the markets predicted 75% A vs 25% B and A always turned up, that would
make them a poor prediction. Predicting 25% B may mean they're not making much
of a prediction (since they're not so far off from 50/50), but it doesn't mean
that B being the outcome is indicative of a the prediction being deeply wrong.

~~~
vegabook
yes I know how sampling works. However as the uncertainty decreased, the
prediction markets went in the wrong direction, rapidly on the last day and
signficantly in the last week. This strongly suggests that there is an element
of "herding" going on by what are supposed to be rational, atomic, individual,
actors (the first two of those three assumptions were violated by profit
incenctive, and a non-secret ballot, respectively, it seems). Hence I stand by
my view that the prediction markets turned out to be terribly poor. You will
admit that as final polls all hovered around 50%, if accurate (and they
seemingly were), prediction markets should have been wildly swinging between,
say 25/75 and 75/25\. They were not. They were trending consistenly lower for
brexit.

------
kennon42
The primary thing that determines overall group performance (assuming no
interpersonal biases like peer pressure or coercion, etc) would be the
probability that any individual chooses the "right" answer. If this is greater
than 0.5, then according to the Condorcet Jury Theorem, the limit of the group
as a whole choosing the right answer is 1 as the group size increases.

Perhaps having a smaller size allows the "noise" in the group to still produce
the "right" answer more often than it would in a larger group?

~~~
tedsanders
No, that's incorrect in two ways. First, you must also assume that the
individual opinions are _independent_. Second, the 0.5 probability threshold
is only correct for binary situations; in real life where they may be a range
of opinions, the threshold is much lower.

~~~
wutbrodo
> Second, the 0.5 probability threshold is only correct for binary situations;
> in real life where they may be a range of opinions, the threshold is much
> lower.

The study from the article did focus on binary situations. From the article:

> Where previous research on collective intelligence deals mainly with
> decisions of ‘how much’ or ‘how many,’ the current study applies to ‘this or
> that’ decisions under a majority vote.

------
nickff
Crowds are wise only if their errors are at least pseudo-stochastic; if they
are skewed in some direction by a pervasive bias (high/low, left/right, etc.),
their predictions or choices will be driven by that bias.

~~~
Hermel
Even if biased, crowds can be the best option if all others are more biased.

~~~
geoelectric
Depends on whether you agree with their bias, really.

~~~
sverige
Not if it's a binary decision, as in the article.

------
rxm
The timing of this post is uncanny given how some large crowds voted.

~~~
ktRolster
Maybe they were right?

I personally think the brexit was a mistake, but now that it's over, I hope
the best for them, and hope they manage to prove me wrong. I certainly don't
think that my opinion is necessarily the correct one.

------
dredmorbius
There are a few elements here, one of which geoelectric addresses in an
earlier comment: does the crowd have expert knowledge.

Another is whether or not the question at hand _requires_ expert knowledge.
There are times when you're better off handing off the helm or pilot's seat to
a qualified pilot than trying to average inputs of a large crowd, or to allow
an electrician or plumber to address a problem within their skill scope.

A third aspect though might concern what the _negative cost functions_ of a
crowd might be.

The wisdom of crowds concept generally assumes the larger the crowd, the
better it will be at arriving at some truth. In reality, various biases,
distortions, and manipulations can emerge, to the point that the crowd's view
is far _worse_ than other options. Aristotle drinks hemlock. Trump is
presumptive nominee. Brexit.

An element of networked systems, including decisionmaking systems, is what
their _cost functions_ are, in the sense of imposing _negative_ results on
those participating. I've been arguing for a year or two now that there _is_
such a cost function, and that you can estimate that by noting the maximum
size an effective network can grow to. Conversely, you can increase (or
decrease) the effective size of a network by addressing that cost function.
Increase it and you'll make large-scale aggregation less viable. Decrease it,
and you can increase the size of effective aggregation.

As examples, a village is constrained in total size not only by its ability to
secure necessary inputs (especially food and water), but in its ability to
_dispose of_ wastes and noxious emissions. London of the late 18th century had
a mortality rate _above_ its natural birthrate, and the only way the city
could maintain its population was through net in-migration from the
countryside (or foreign lands). This wasn't materially addressed until
revolutions in water provision and sanitation, including the first modern
sewerage system around 1850, addressed such concerns as cholera epidemics
which were killing as many as 50,000 people a year.

In programming, Fred Brooks' _The Mythical Man Month_ notes that few
programming teams scale well beyond about 6-12 developers. The inter-personal
communications costs make larger groups not only inefficient, but _less
effective, net_ than smaller ones. To produce larger teams, _you 've
effectively got to split them into smaller ones_. That's among the things that
a highly modularised development process as is common in Free Software
projects achieves -- see Apache, the Linux Kernel, or the Debian Project as
examples (Gabriella Coleman, now of McGill University, wrote her dissertation
on this topic, it's fascinating reading).

Computer chip design essentially removes the space and resistance costs of
crowding high densities of electronic gates in small spaces. Again, the cost
function is reduced.

In email and traditional (POTS and mobile) phone service, increasing amounts
of spam are _increasing_ cost functions, reducing the appeal and utility of
the network to all involved. POTS has been shedding subscribers for some time,
my expectation is that mobile phone service itself will be as well, more
especially if interconnects, _and filtering_ of VOIP alternatives (including
iChat, Google voice chat, Skype, etc.) are further developed. Those networks,
as a Long Island friend of mine some time back said, "gotta leahn to _tawk_ to
each othah! _

------
hristov
After the whole brexit fiasco i do not want to hear about the wisdom of
crowds.

