
U.S. Intelligence Community Explores More Rigorous Ways to Forecast Events - petethomas
http://online.wsj.com/articles/u-s-intelligence-community-explores-more-rigorous-ways-to-forecast-events-1409937859
======
rrggrr
Except that we don't know how well this 'focus group' forecasting performs
relative to voluminous reporting by analysts at State, CIA, RAND,
Congressional Research, think tanks, etc. Often the difference between good
analysis and bad is simply how successful the analyst is at getting their
projections read by the 'right' people. Along the same lines, if policy makers
could stop shopping for analysis that confirms their biases, much progress
could finally be made in international and public policy.

~~~
jacques_chester
> _Except that we don 't know how well this 'focus group' forecasting performs
> relative to voluminous reporting by analysts at State, CIA, RAND,
> Congressional Research, think tanks, etc._

Actually, we do. Philip Tetlock, who is mentioned in the article, spent more
than a decade collecting tens of thousands of predictions from geopolitical
experts. He found that humans are terrible at making predictions about complex
phenomena; typically we are stomped by simple linear models or probablistic
random walks.

Some humans were slightly better at predicting outcomes, but compared to
simple models, they still sucked.

It didn't matter what other variables were involved. Education, access to
classified information, experience, seniority, nationality, profession ...
_none of them mattered_. A Russian politics expert is about as likely to pick
Putin's next move as a financial economist who reads the _New York Times_.

Tetlock's book, _Expert Political Judgement_ , is comprehensive. I reviewed it
here: [http://chester.id.au/2012/07/29/review-expert-political-
judg...](http://chester.id.au/2012/07/29/review-expert-political-judgement/)

~~~
rrggrr
You don't understand how DC works. On a given issue analysis and projections
are produced by numerous agencies. For example a Country desk at State may
deliver an accurate estimate, but another group at state may water it down, or
it may be stuffed by a competing estimate from CIA. Since its politically
safer to bury your head in the sand than it is to fight for an estimate, the
process is ripe for manipulation. All this concept does is pass the buck
without - apparently - any comparison to all the analysis produced.

Tl;dr - This solution speaks to the failings of the intel process and not the
people.

~~~
jacques_chester
Tetlock's work was based on individuals giving unfiltered predictions.

------
shawkinaw
Reminds me a bit of Asimov's psychohistory [0].

[0]
[https://en.wikipedia.org/wiki/Psychohistory_(fictional)](https://en.wikipedia.org/wiki/Psychohistory_\(fictional\))

------
empressplay
Simply keeping those who successfully predict events over time will do nothing
but end with one incredibly lucky person, who will then proceed to get their
future predictions wrong most of the time, or at least as much as the group
you started with.

~~~
jacques_chester
I think the competitions are structured to show the difference between
distinction and calibration. Submitting outlier predictions constantly
eventually gives you the former but not the latter.

The easiest prediction in the short term is "Yesterday's Weather". Most of the
time, most things don't change much.

------
Teapot
It becomes more complicated as they then _act_ on those predictions. Thus
mucking up said predictions. And by doing so requires new predictions
(expensive guesses).

(I just watched DaysOfFuturePast)

------
dminor
Hah, I worked on this the first time around 10 years ago when it was called
FutureMAP. Didn't exactly end well.

------
Bud
I suggest that they hire Hari Seldon.

~~~
vdaniuk
We've still a long way from Gaia, but internet + biological computing may take
us to there faster than expected.

------
grizzles
I guess they haven't did their homework on how to detect precrime.

All they need to do is hire some Asberger-y women, put them in vats of liquid,
and give them some psychedelics. Ask them to predict the future.

How hard is that? You might argue that my approach is unscientific, but I
would argue that is at least as scientific as their forecasting approach.

