
A group of 3,000 citizens is making better forecasts than CIA analysts - sizzle
http://www.npr.org/blogs/parallels/2014/04/02/297839429/-so-you-think-youre-smarter-than-a-cia-agent
======
joshuahedlund
This Economist post[0] addresses some of the many comments about statistical
outliers:

> The big surprise has been the support for the unabashedly elitist “super-
> forecaster” hypothesis. The top 2% of forecasters in Year 1 showed that
> there is more than luck at play. If it were just luck, the “supers” would
> regress to the mean: yesterday’s champs would be today’s chumps. But they
> actually got better. When we randomly assigned “supers” into elite teams,
> they blew the lid off IARPA’s performance goals. They beat the unweighted
> average (wisdom-of-overall-crowd) by 65%; beat the best algorithms of four
> competitor institutions by 35-60%; and beat two prediction markets by 20-35%

[0][http://www.economist.com/news/21589145-how-sort-best-rest-
wh...](http://www.economist.com/news/21589145-how-sort-best-rest-whos-good-
forecasts)

~~~
TrainedMonkey
Could this be simple selectivity? I do not know how CIA selects analysts, but
I doubt it is by having large pool of applicants that are evaluated on their
predictions. On the other hand we have group of 30 people, selected for their
guessing from amongst 3000 population. When put that way, I would expect 30
people to be consistently better.

~~~
akira2501
These people are also isolated from the internal agency narrative, I imagine
that can help quite a bit.

~~~
negativity
In other words, what if the "taxpayer-funded" forecasts we're permitted to
know about are "on the books", so to speak, and so much more is "off the
books" and memory-holed beyond even the typical "black magic marker, and white
out" form of redacted classification that some times surfaces into the public
eye?

Truths so brutal, and predictions so harsh that no one dares write them down.

But now we enter the information age, and certain realities are essentially
undeniable, and can no longer concealed from the civilian panopticons like
Google.

Sort of like how TV Networks (and even basic cable channels to a degree), with
their Standards & Practices departments, waste time sanitizing television
destined for the public air waves and radio spectrum, while media on the
internet operates with an almost total lack of restraint.

------
lostcolony
I need to know more. If the questions are all yes/no questions ("Will there be
a significant attack on Israeli territory before May 10, 2014?"), and the
sample set is large enough, you would -expect- there to be some outliers who
mostly got things right, by pure chance. And even outside of that, I want to
know what sort of average the CIA agents were batting; if they were hitting
just 50%, I would -expect- nearly any sample size to have an outlier who did
better.

That is, to borrow the old get rich quick scheme, take a list of 1024
addresses. Send to half of them "Stock X will be higher on (date) than today",
and to the other half "Stock X will be lower on (date) than today". Repeat
with the set you sent the correct 'predictions' to, and continue repeating
until you're down to four people who you've sent the correct prediction to
every time. Suggest they sign up for your premium stock tips newsletter.

All you're doing is playing statistics, you're not actually demonstrating any
predictive ability.

~~~
jerf
"and the sample set is large enough, you would -expect- there to be some
outliers who mostly got things right, by pure chance."

According to the article, "In fact, she's so good she's been put on a special
team with other super forecasters whose predictions are reportedly 30 percent
better than intelligence officers with access to actual classified
information."

If they mean that _after_ the team was formed, the predictions are still 30%
better, than random chance ceases to be an acceptable explanation. If they
mean that the team that they formed was 30% better on the predictions used to
grant entrance into the team, then your objection still holds. Unfortunately,
I don't think the article makes it clear which is the case.

That said, there are definitely some mechanisms that could lead an outside
group to higher accuracy on these sorts of things. The two biggest are: 1. The
insiders are simply that bad, for whatever reason, and random chance can
easily beat them, and 2. The failures of the insiders are correlated and
overwhelm their theoretically superior information, and I'd especially finger
potential ideological biases there, which could affect both information
filtering and the analysis. (It is one thing to believe the world should be a
certain way as a result of one's ideology... it is quite another to believe
the world _must already be_ a certain way as a result of one's ideology, even
when one is staring evidence in the face that it is not. One is noble and
laudable... the other somewhat less so.)

~~~
Ovid
As a guess, I would suspect that the researchers who set up the experiment are
more careful about their analysis than NPR is with its reporting. NPR needs to
popularize its information, thus ensuring that we, the casual reader, can get
a grasp of what's going on. That necessarily dumbs down the content.

~~~
vitoreiji
Why did they go half way, though? If you're going to dumb down things, why
don't just say "scientists found that" instead of giving a distorted view of
the experiment, popularizing a wrong idea of what real science actually looks
like?

------
lostcolony
Now the fun question; are there any anti-forecasters? People who guess wrong
so consistently and powerfully they're as good as super forecasters?

~~~
cryoshon
This is a really interesting question, actually.

In my undergrad ecology class, we were assigned a scent-based laboratory
experiment which involved all the male and female members of the class wearing
the same white t-shirt for 48 hours in order to accrue bodily odors-- heavy
workouts, perfume, showering, etc were all prohibited. At the end, everyone in
the class sniffed all of the t-shirts and guessed which t shirt had been worn
by what sex.

Most people had a 50-50 chance, and a few guessed slightly better than chance.
I was far worse than chance, guessing only 10% correctly. In a way, anti-
forecasting is a type of forecasting, too. If someone had asked me for my
guess and then guessed the opposite, they'd have had superior guessing power.
This paradigm falls apart quickly when the chances of events aren't so clear,
however.

~~~
adambard
Ah, the old Costanza maneuver.

------
babs474
For a prediction market anyone can join checkout
[https://scicast.org/](https://scicast.org/)

Whats neat about SciCast is you can make predictions based on the assumption
of how other questions will turn out.

eg I think that if the price of bitcoin exceeds 1000 by 2015 at least one
presidential candidate will accept bitcoin donations. If btc ends up being
below 1000 I make no prediction.

~~~
tedsanders
Just to add, the SciCast project came out of the DAGGRE project, which was a
competitor to Good Judgment in the IARPA prediction challenge.

------
cottonseed
Pure speculation: random, untrained and disinterested parties might be better
than CIA analysts at making geopolitical predictions because their results
aren't politicized. "Are there WMDs in Iraq?"

~~~
chflamplighter
I was having the same thought, at what point/how do you factor in the, "if I
make this predication, which is contrary to my boss's expectations, will I
still have a job". Surely there is a huge amount of pressure on "rowing in the
same direction" towards a predetermined group think (which is terrible, of
course).

~~~
zeidrich
There's also the "if I make this prediction and I'm wrong, will it affect me?"
bias too.

The more involved you are, and especially the more accountable you are held
for your decisions, the more you're likely to second guess yourself. There's
normally a reason your second guess wasn't your first guess.

------
Robin_Message
It's totally unclear what they are actually doing.

Are they averaging everyone's answer (wisdom of the crowd)? Are they looking
back at who got lucky and declaring them "super forecasters" (and then seeing
large regressions to the mean)? Did they have separate periods of testing to
try and control for this? Did they have a control group of similar size just
flipping coins?

The way this is written it sounds like someone used a Gladwell book as their
entire science education.

~~~
nbouscal
There are multiple experimental groups in the project, which use different
methods. One of the methods is a prediction market, where you can buy or sell
shares of any given outcome, with the market price being the group estimate of
the probability. For example, if the group thinks something will happen with
35% probability, you can buy shares for $35 (fake money). When the questions
are eventually resolved, correct shares are valued at $100, and incorrect
shares are valued at $0.

~~~
tedsanders
There were multiple experiment groups at the beginning, but I'm not sure if
any are left besides Good Judgment. DAGGRE, the prediction market group, is
out.

Edit: Oops, I misunderstood. Apparently Good Judgment also has a prediction
market group inside of it. Thanks nbouscal.

------
jljljl
They may be good at prediction, but who writes their questions?

It's probably much easier to reach a successful prediction when a lot of the
framing has already been done for them. If you take the Ukraine question, for
example, there's already a lot of intelligence encapsulated, such as the City
and Date when the CIA is expecting a Russian invasion. As a post mentions
below, the questions may even be phrased/biased towards the right answer.

I wonder if they would be as good at prediction if they didn't have the
resources and intelligence of the CIA providing them with the correct focus,
prioritization, and framing.

~~~
kazagistar
The CIA is also providing focus, prioritization, and framing for themselves,
and is being outperformed.

------
tghw
Now I'm really confused. The title of this post was originally "So You Think
You're Smarter Than A CIA Agent ", which is consistent with the title of the
article. Now it's been changed to editorialize the content? Isn't that the
opposite of HN policies?

~~~
kissickas
Is this policy written somewhere? The only trend I've noticed is that I always
agree with anyone complaining about a changed submission title. (I don't think
you're complaining, are you? The NPR title is pretty cheap IMHO.)

~~~
Noelkd
You can read the guidelines here:

[http://ycombinator.com/newsguidelines.html](http://ycombinator.com/newsguidelines.html)

This may fall under:

" Otherwise please use the original title, unless it is misleading or
linkbait. "

~~~
kissickas
I would call it linkbait, cool. Sad to see coming from NPR.

Thanks for the link, it's been a few years since I read those rules.

------
nnq
Now actually selecting the outliers, the people that, on average, made really
good predictions despite access to limited information, and data-mining their
features set (psychological traits, information diet, background etc.), _THAT_
would be an interesting study...

~~~
nbouscal
They are doing a bit of that too. They have found correlations with IQ and
other metrics.
[http://www.cis.upenn.edu/~ungar/papers/forecast_AAAI_MAGG.pd...](http://www.cis.upenn.edu/~ungar/papers/forecast_AAAI_MAGG.pdf)

------
damian2000
Highly recommend this book, which is on the subject ...
[http://en.wikipedia.org/wiki/The_Wisdom_of_Crowds](http://en.wikipedia.org/wiki/The_Wisdom_of_Crowds)

~~~
nswanberg
This is a fantastic book and explains most of the questions currently in the
thread about the circumstances where large groups of people produce more
effective answers than any one of its members and the cases where they do not.
If this article was interesting it is worth reading.

The book also briefly covers the Policy Analysis Market, a program to
experiment with using a futures market to help predict terrorist attacks. As
this program used real money, and offered the opportunity to make money off of
terrorist attacks, some members of Congress were outraged and the program was
immediately cancelled
([http://news.bbc.co.uk/2/hi/americas/3106559.stm](http://news.bbc.co.uk/2/hi/americas/3106559.stm)).
This version discussed in this NPR article seems more publicly palatable. In
fact, it looks like the similarity was already reported:
[http://articles.latimes.com/2012/aug/21/nation/la-na-cia-
cro...](http://articles.latimes.com/2012/aug/21/nation/la-na-cia-
crowds-20120821)

By the way, can anyone recommend a more rigorous book than _Wisdom of Crowds_
on this subject?

~~~
jaynos
This one [1] is recommended by the guys behind the Radio Lab podcast (specific
episode [2]). I left the referral tag on it since it'll go to help them
produce more great radio.

[1]
[http://www.amazon.com/exec/obidos/ASIN/0385503865/wnycorg-20](http://www.amazon.com/exec/obidos/ASIN/0385503865/wnycorg-20)

[2] [http://www.radiolab.org/story/91502-the-invisible-
hand/](http://www.radiolab.org/story/91502-the-invisible-hand/)

------
sailfast
I am a big fan / proponent of prediction "markets" or similar otherwise
incentivized crowd prediction tools. Happy to see someone execute it within
the government while avoiding the terrible branding and public relations faux
pas of past "terrorism market" endeavors.

Hopefully this will scale more broadly, continue to improve its incentives,
and increase our capability to prepare for future events.

------
pserwylo
The issue of crowd sourcing answers to questions is a very interesting one.
I've recently been reading research in this area with a colleague, and it is
far more involved than the article makes it sound.

If you are interested, one of the first people to work on the problem were
Dawid & Skene [0] back in 1979. They were trying to solve the issue of
clinicians talking to patients and asking how they are. Each time they asked,
the patient may give a slightly different answer. The goal was to estimate the
"true" answer from the number of various "noisy" answers (sound familiar after
reading the OP?)

The general intuition for most future research into the topic is that if you
tend to agree with others more often than not, your score goes up. If your
score is higher, your answers are trusted more.

Then there are a whole bunch of interesting additions to this principle, such
as factoring in question difficulty [1] (e.g. questions which have the most
disagreement can be considered more difficult, and hence you may want to ask
more people about these questions to get a clearer picture). It is also
helpful to seed certain questions for which the answer is "known" (may or may
not be possible, depending on the project). Then you can use the answers to
these questions to help score people even more accurately.

An even more interesting approach is to take into account people who answer
adversarially, that is, they always answer incorrectly (either due to
malicious intent, or they consistently misinterpret the questions in a
backward fasion). If you have somebody who answers incorrectly 90% of the
time, then they provide almost exactly the same information as someone who
answers correctly 90% of the time. All you need to do is flip their vote once
you've identified this.

The whole field is very facinating, and there seems to be more and more
publications coming out with interesting approaches to the problem.

[0]
[http://scholar.google.com/scholar?cites=15953840037809938294...](http://scholar.google.com/scholar?cites=15953840037809938294&as_sdt=5,39&sciodt=0,39&hl=en)

[1]
[http://scholar.google.com/scholar?cluster=144091158652582285...](http://scholar.google.com/scholar?cluster=14409115865258228521&hl=en&as_sdt=0,39)

------
probablyfiction
One thing that I really enjoyed about this article was its description of "the
wisdom of crowds." Typically that phrase shrouded in marketing bullshit that
just boils down to the idea that groups are always smarter than individuals
because magic. That idea on its own is really counterintuitive and hard to
accept.

It was a good call to talk about the scientific basis for the idea; I wish
more places that relied on crowdsourcing would do the same thing.

~~~
Noxchi
> That idea on its own is really counterintuitive and hard to accept.

Not really... For example, have you heard of the Cicada puzzle? That is solved
by groups of people, and probably wouldn't be solved if it was just one
person.

Multiple people with specialized knowledge > one person with specialized
knowledge.

We didn't evolve to be social animals because it would make us stupid...

------
dang
A note on the title change:

When the story title is linkbait and there are no good subtitles, we sometimes
draw on the first sentence of the story. In this case I used the description
under the photo. The key thing is to use only language that's already there.

------
jmzbond
Doesn't this article seem to contradict itself? It cites superforecasters'
amazing ability, then talks about how better forecasting is the result of
averaging the noise from a crowd of diverse opinions...

~~~
dean
That's what jumped out at me as well. First the article extols how the Wisdom
of Crowds is smarter than the experts. Then it goes on to say how some of the
crowd are superforecasters, (essentially experts), and the rest of the crowd
makes no difference.

It would be interesting to see how small the group of superforecasters would
need to get before their accuracy falls below that of the CIA analysts.

~~~
jmzbond
Good test!

------
nswanberg
This is a great topic for a site that uses some "wisdom of crowds" mechanisms
as part of the story and comment ranking. Though HN is different in some
significant ways: in the experiment, subjects are asked carefully worded
questions and give answers without being shown any information from others,
whereas here there are several signals available before voting on a story or
comment, like prior vote count and previous comments, which can bias a vote.

------
logfromblammo
They started off reasonable. Ask ordinary people to assign probabilities to
potential world events that can be verified. The next step is also reasonable.
With a large enough crowd, the estimation noise around the "true" probability
will cancel out. That's how the crowd mean estimated the ox weight so closely.

But then they turned left onto stupid street. They took the people who,
perhaps by chance alone, predicted results closest to the crowd mean, and put
them on a "team". What the heck is the point of that? It just reduces the size
of the crowd and shows a fundamental misunderstanding of statistics.

Past performance does not necessarily predict future results.

Start with 10000 coins, and flip them all. Keep the ones that land heads. Flip
those, and keep the ones that land heads. Repeat until all coins land on
tails, or one coin remains as the "champion of heads." What is the probability
that when it is flipped again, it comes up heads? Still 50%. It isn't a
supercoin. You just managed to select it from a giant group by making it the
sole outlier from a series of independent trials.

Though you should probably check to make sure it actually has a tails side,
just in case.

~~~
nzp
> It just reduces the size of the crowd and shows a fundamental
> misunderstanding of statistics.

Why do you assume that what's in the NPR article is all there is to it, that
there's no info, technical or other, left out?

Of course they thought of that, and checked. There's no regression to the
mean, the elite group further improved, their estimates were 65 % better than
the overall crowd.[0]

[0] [http://www.economist.com/news/21589145-how-sort-best-rest-
wh...](http://www.economist.com/news/21589145-how-sort-best-rest-whos-good-
forecasts)

~~~
logfromblammo
That article is _much_ more informative. Kudos to you, nzp.

~~~
nzp
Heh, actually, kudos to joshuahedlund[0], I found it in his comment.

[0]
[https://news.ycombinator.com/item?id=7515263](https://news.ycombinator.com/item?id=7515263)

------
joosters
How does the 'crowd' adjust for bias in sources? e.g. if they are all picking
the answers to the surveys based on similar methods, e.g. google searches or
from listening to the news, how does the crowd avoid the biases from news
sources? You can't just 'average them out' if there are systematic skews.

e.g. just because half of the people are watching (say) CNN and the other half
are watching (say) Fox News doesn't magically mean that the average of their
opinions will be correct. What if both news sources overplayed the likelihood
of terrorist attacks? How would the crowd adjust to this?

------
doctorpangloss
>First, if you want people to get better at making predictions, you need to
keep score of how accurate their predictions turn out to be, so they have
concrete feedback.

>But also, if you take a large crowd of different people with access to
different information and pool their predictions, you will be in much better
shape than if you rely on a single very smart person, or even a small group of
very smart people."

Both points are clearly true. But it's obvious that the CIA, and probably
anything political, has problems "keeping score," not getting more voices.

------
psikorsky
Obviously there are some very smart people working in government intelligence
agencies, including the CIA, MI6, etc. However, in my experience working in
the private intelligence sector, I found that many former government employees
lacked creativity and relied too heavily on 'classified sources' and their
network. The best analysts were extremely lateral thinkers who used Google and
public databases, but would never have passed a government test on account of
their quirks.

------
diminoten
> In other words, there are errors on every side of the mark, but there is a
> truth at the center that people are responding to, and if you average a
> large number of predictions together, the errors will end up canceling each
> other out, and you are left with a more accurate guess.

This just reminds me all too much of the Twitch Plays Pokémon phenomenon that
took place last month. The fact that they actually are completing games is,
frankly, astounding.

------
prestadige
It seems to me that all professional experts are subject to groupthink.
However, could anonymous lay forecasters still make such excellent predictions
_without_ those groups in place?

Anyway, I think all this argues in favor of Robin Hanson's concept of
"Futarchy":

[http://hanson.gmu.edu/futarchy.html](http://hanson.gmu.edu/futarchy.html)

------
njharman
Maybe the founding father's "fear" of the ignorant masses and choice of
representative government needs to be reevaluated. Go back to direct actual
democracy. But with ranges instead of yes/no. So as to capture "average is
true signal".

Maybe first step is chance voting of representatives from yes/no to 1-100 how
much you wan this person to win.

------
lifeisstillgood
So could I suggest that CIA forecasts are affected by internal politics,
perceived needs of political establishment, and afflicted by biases in top
secret material that cannot be easily counter-balanced.

Whereas if they put these super-forecasters in the CIA and said "if you dont
predict what i want its off to Afghanistan with you" they will soon regress to
the mean.

------
kriro
Most people in the world only have access to public information so if you
think most geopolitical events are brought about by "the masses" instead of by
some Platoesque elites (which is of course very much debatable) it may well
make sense to base your predictions on the data that is available to most
players.

------
mtrn
Hearing the article, I felt thrown into PKD novel. One of the recurring theme
in his work are precogs, normal people with special abilities to forecast the
future. Or also to mispredict the future very accurately. Large corporations
use groups of precogs to plan marketing campaigns, product designs, etc.

------
ballard
The Wikipedia page on prediction markets storied history and interesting bit.
Curious that USM had a prototype that was shutdown for being a "terrorism
futures market."

[http://en.wikipedia.org/wiki/Prediction_market](http://en.wikipedia.org/wiki/Prediction_market)

------
epeterson19
"The wisdom of crowds is a very important part of this project, and it's an
important driver of accuracy," Tetlock said

This part threw me off. But overall, crowdsourced estimates could be a useful
tool, provided everyone is actually trying to guess correctly (ie national
pride, monetary reward, etc)

------
ahi
I went to undergrad with quite a few future CIA analysts. You should not
overestimate the average intelligence of CIA analysts. No doubt they have some
brilliant people, but you shouldn't be surprised that amateurs with Google
News outperform careerists inside a giant bureaucracy.

------
user24
That article felt like it ended far too early. I was left with a "is that it?"
feeling. Not that I was underwhelmed by the message of the article, but that I
couldn't find a message in it. If it had a point beyond "crowdsourcing works",
I missed it.

~~~
dredmorbius
True for far, far too many articles IMO.

------
speeder
Assuming you can do that (ie: forecast world events), anyone know how you can
use that for your own advantage, beside investing in stocks (even if not
necessarily with the stock exchange, OTC or classic VC/angel investing enters
this too) or commodities?

------
syncsynchalt
John Brunner's book "The Shockwave Rider" (1975), which was heavily inspired
by Toffler's "Future Shock", had this concept (called a Delphi pool) as a
major and successful component of the future society.

~~~
gwern
Delphi pools were invented by RAND, not Brunner, IIRC, have been studied over
the years (Armstrong's _Principles of Forecasting_ has some papers on them),
and work differently from prediction markets: Delphi pools collect a small
number of experts, asks for written analyses and breakdowns and qualitative
assessments (not necessarily the precise numerical probabilities of prediction
markets), feeds each expert's analysis to the others, asks them for a new
analysis incorporating the others, and repeats this for 2 or 3 rounds until a
rough consensus has been reached. This is not very similar to prediction
markets.

------
joosters
Doug Stanhope has a good rant about these opinion polls:

[http://www.youtube.com/watch?feature=player_detailpage&v=Ryc...](http://www.youtube.com/watch?feature=player_detailpage&v=RycwYRcm3Lc#t=84)

------
Justen
Are the underpinnings of this experiment in line with the idea behind the
HyperLogLog story[0] I read about the other day?

[0][http://antirez.com/news/75](http://antirez.com/news/75)

------
grn
See also [http://www.johndcook.com/blog/2014/03/21/independent-
decisio...](http://www.johndcook.com/blog/2014/03/21/independent-decision-
making/)

------
thewarrior
If you have 3000 people predicting the outcomes of events with say a
probablity of 0.3 after 6 trials there would be 2 people who got all of them
correct and many others who would have a very good record out of pure chance
alone.

Unless they publish the detailed numbers we can't be sure of whats going on
here.

If some person with access only to normal sources is able to perform so well
against people with access to lots of classified sources and information ,
what are we to conclude ?

Either they are just lucky or the people at the CIA aren't as competent as we
think they are.

If we are unwilling to accept any of these theories then we need to know what
makes these people good at it. I don't see how just plain simple analysis can
beat people at the CIA.

~~~
adsr
But the point here seems to be to use the group average, not to find outliers
with good prediction ability per se.

~~~
logfromblammo
That's exactly it, which is why I was so confused about the "team" of
"superpredictors". While you might learn something by comparing their methods
with everybody else, you still have to do the same amount of scrutiny on
everyone in the crowd to learn anything useful.

With enough data, you can apply an automatic adjustment to even the most
horrible predictors, to make their contribution to the final result more
useful. If, for instance, someone always predicts exactly 10% more favorable
results for Israel, independently of the facts, you can just reverse that for
that one person on every question mentioning Israel, to produce a bias-
adjusted result. That is tremenddously difficult, though, and not nearly as
useful as just increasing the size of the crowd.

Which is the whole point. With a big enough crowd, you don't need to do that
at all, because the stupid little biases unrelated to the facts tend to be
noise, not signal.

------
happyhappy
"In fact, Tetlock and his team have even engineered ways to significantly
improve the wisdom of the crowd"

Does anyone know what these methods might be?

~~~
morganherlocker
One example briefly mentioned was providing feedback to respondents to let
them know when they were getting things right or wrong.

~~~
joosters
And how would that work?

Take their example question - 'Will North Korea launch a new multi-stage
missile before May 1, 2014?' That's a yes/no question. But none of the
participants _knows_ the answer. They are just trying to forecast the future
based upon the balance of probabilities. So if NK does launch a missile, the
people who answered 'no' were wrong, but _the reasons upon which they came to
pick 'no' could still have been right_. So giving feedback to these people and
telling them that they were wrong does not magically aid/improve predictions.

Perhaps a simpler example could be used. 'Will the next roll of this dice
score 3?' Well, it's obviously more likely that some other result will happen.
But if the 3 does come up, you can't say that all those people who said 'no'
are worse at predictions...

~~~
jbattle
Even trickier - it sounds like the participants are giving probabilities. So
if you guessed some possibility was 42% likely and then it did in fact happen
- are you right or wrong? I don't think there is an answer to that question.

~~~
llamataboot
Simple math applied to a large enough set of predictions will let you know
whether or not your probabilities are accurate. It is the same as playing
poker - every hand you are essentially betting on probabilities multiple times
(okay right now I thing I have a 50% chance of winning this hand and the pot
is X and I have to put in Y to keep playing, I'm going to continue). In the
short term it is very hard to know whether or not you are putting money in
when you "should" because of variance. In the long term, the variance
disappears and you can see that, for example, you are making an estimated 2
bets/100 hands you play which means you are "more correct" than the people you
are playing against at estimating the odds. A little more complicated than
that of course because you are also using bets as weapons to drive better
hands out through bluffing and semi-bluffing, etc but the general point holds
that a large enough sample size of actual results versus compared results
means that your probabilities will hold or they won't

~~~
joosters
Works great for poker, where you play lots of hands. Works less great for the
situation in the article. Just how many North Korean missile launches do you
need a person to predict before you know they are lucky or not?

~~~
llamataboot
Doesn't matter, as long as they predict a large enough sample of events. Just
requires an assumption that they will be as accurate on every type of event,
which probably isn't true, but close enough for statistics

------
ronaldx
I did exactly this in high school:

I set up an experiment to find the people best at guessing Zener cards picked
at random.

I surveryed 100 people and the best 5 did notably better than the others. I
dubbed these people "clairvoyant". (Spoiler: on retesting, they weren't, but
then again I only followed this process once...).

Wisdom of the crowds will work just fine for anything modestly obvious, and it
will fail for any black swan events that intelligence actually needs to be
aware of - if the predictors even get asked about those.

------
SolarNet
This reminds me of the future predicted in Rainbows End by Vernor Vinge.

Where he explores the ability to spin up pools of analysts like a cloud
server.

------
lukasm
Same rules apply to other institutions e.g. investment banks and
recommendations. Everybody sticks close to the herd.

------
avaku
Was the project shut down? (the website is not working, sorry if I haven't
missed that in the article)

~~~
gwern
GJP is still running and just added some new IARPA questions today.
Registration for the next season may not be open yet, though.

------
yiedyie
Crowdsource prophet. This is not new, that happened before with the "peak oil
community".

------
canvia
The formatting of the questions and answers could contribute to a candidate's
success as well. Unlikely events could be phrased differently than likely
events with the intention of skewing the success rate to the positive.

I would hate for data from this organization to be used for any sort of real
decision making process. You might as well hire a psychic.

------
anigbrowl
Inaccurate headline. Citizenship is not required for participation.

------
matznerd
How does this forecasting market compare to something like Intrade?

~~~
LanceH
Intrade is a futures market where people with insider knowledge can buy into a
position. So if you know of a scandal for a political candidate that will
cause him to withdraw, you can go buy up all the other candidates and make
some money when he does. This will alter his price, becoming a predictor
before the news actually comes out.

Additionally, there is the "put your money where your mouth is" aspect of it.
While every candidate in the primaries, is announced as, "the next president
of the United States", you can see who the people really think will win based
on which futures they purchase.

The CIA effort is more like taking a whole lot of people, asking them to
predict something and seeing who is good at it. There appears to be some
manipulation of those guesses across the breadth of the crowd but it is not
mentioned. In contrast to futures markets, the players in the CIA effort have
no stake in the game.

~~~
llamataboot
Right, so one would expect something like Intrade or other prediction markets
to leverage both the statistical "wisdom of crowds" and insider information to
be even more accurate.

------
secfirstmd
I noticed this phenomenon alot when I used to work at Intrade.

------
easy_rider
I sometimes have good runs in blackjack.

------
blazespin
Where's the API?

~~~
allochthon
It might be possible to put something together using Amazon's Mechanical Turk.

------
GarvielLoken
Intuitors.

------
stefantalpalaru
False classified information might actually work against the analyst who has
access to it.

The same piece of info, if public, will have to stand on its own, without the
bias introduced by restricted access / multiple levels of filtering / the
attached opinions of people wearing suits.

~~~
ahi
Even true classified information can cause problems if overvalued. The
questions seem to be rather general predictions so publicly available
geopolitical context likely matters more than any individual bit of inside
information.

