
Intelligent intelligence – Just how good are government analysts? - kawera
http://www.economist.com/news/science-and-technology/21607765-just-how-good-are-government-analysts-intelligent-intelligence
======
AndrewKemendo
For what it's worth, there is no method that goes untested for forecasting.
The CIA's Sherman Kent school is a treasure trove of analytical methods,
techniques and processes that have been studied and implemented over the
decades. A lot of the research has been put online for free [1].

As the article states, nobody outside of the business will ever hear about
99.9% of successful intelligence work done, and 90% of the people inside won't
hear about it either. So unless you are directly involved in analysis,
colletions or operations, it's impossible to get a good feel for efficacy.

[1][https://www.cia.gov/library/kent-center-occasional-
papers](https://www.cia.gov/library/kent-center-occasional-papers)

~~~
hueving
Let me try out my media spin skills on that. ;)

"Due to a shocking lack of transparency and accountability, we can't even tell
if CIA analysts are doing their job. It would appear that up to 90% of
analysts collect a paycheck without producing any measurable output."

~~~
AndrewKemendo
Ha, while tongue in cheek, that is something that is heard a lot _within_
agencies.

So much so that over the years significant layoffs happen and programs get cut
when higher ups in agencies don't have a proper understanding of some of their
organization's capabilities.

------
bcroesch
For anyone who is interested in this topic, Dr. Tetlock just released a new
book about it ([http://www.amazon.com/Superforecasting-The-Art-Science-
Predi...](http://www.amazon.com/Superforecasting-The-Art-Science-
Prediction/dp/0804136696)).

We're also hosting a public forecasting tournament for him and his team that
focuses on geo-political forecasting:
[https://www.gjopen.com/](https://www.gjopen.com/)

~~~
MathsOX
I'm about half-way through the book. Very easy, enjoyable read. Left wondering
why some of these Amazon reviews try to make it sound as if it's quite
academically/technically rigorous.

------
presidentender
The more famous an analyst is, the more likely he is to be confident in his
abilities, and the less likely to second-guess himself, to analyze his
analysis. Those analysts who must defend their work to superiors do a better
job.

The same thing seems to happen in other lines of work - when an engineer is
accountable to code review, he might do better. When an author is subject to
the will of an editor, his work is ultimately achieved faster, and with better
quality; witness George RR Martin's speed on the first three books in A Song
of Ice and Fire versus the glacial pace of the last two.

~~~
johnloeber
Your last sentence has me doubting: I think that 'faster and better' is
potentially true in measurable fields with reasonably objective criteria for
success (writing code is one), but it strikes me as untrue in the arts,
especially regarding experimental work, which is often all about defying the
norms and expectations a reviewer is in place to enforce.

~~~
hibikir
If your argument was correct, we'd see authors make their best work later in
their career, when they can ignore editors and publishers, because their name
alone will carry their work.

I haven't done a big study in this, but it seems to me that early works are,
on average, better than works made after artists have become famous. If
anything, they become complacent. It's not just GRR Martin: Look at the Harry
Potter series, for instance. In general, the commmercial success makes things
worse.

~~~
eutectate
An alternative explanation is that famous artists struggle to replicate their
initial success because it often required a large amount of luck.

~~~
thaumasiotes
[https://en.wikipedia.org/wiki/Regression_toward_the_mean](https://en.wikipedia.org/wiki/Regression_toward_the_mean)

Authors often do get noticeably worse after becoming famous, though. Take the
reasonably well-known example of Robert Jordan, who wrote a series of 5 great
books followed by 5 lackluster books. It's hard to say he just got lucky five
times in a row.

And you can look at the same question another way by reading self-published
(therefore, unedited) books -- I used to be quite open to reading those;
bitter experience has taught me not to bother.

------
MichaelGG
> they were underselling themselves, tending to err more than necessary on the
> side of uncertainty

That's not how it works. If you consistently misreport your calibration,
you're still miscalibrated. Consumers of the intelligence would (should)
notice the miscalibration and correct for it, regardless of direction.

------
lnlyplnt
Very Interesting, though I still wonder if intelligence agencies can out
perform betting markets (at least on questions with enough interest to be able
to generate a liquid price).

~~~
bcroesch
There was an article in NPR a while back about the Good Judgment Project
(which was run by Dr. Tetlock - mentioned in the article). In some cases,
betting/prediction markets (& other similar tools) were beating intelligence
analysts -
[http://www.npr.org/sections/parallels/2014/04/02/297839429/-...](http://www.npr.org/sections/parallels/2014/04/02/297839429/-so-
you-think-youre-smarter-than-a-cia-agent)

~~~
waterlesscloud
I participated in GJP for a couple seasons. Honestly, I didn't do that well.

But I was approaching it very differently than I would if it was my job. Since
the system that was set up gave high rewards to unpopular predictions, I just
gambled on the few most unpopular that had at least some shot at reversing. It
wasn't the smartest approach, but it was the most fun. If I was doing it for
real, obviously I'd go a different way.

The people who did best, at least from what I saw, tended to ride waves of
popularity on the more active questions, buying low and selling high.

What I did was like betting on a few biotech startups, what the best scorers
did was like riding waves of the market leading stocks.

In the end, I'm not that sure it had much to do with actual prediction of
events. Then again, neither did my approach. I guess I'm not sold on the
version of prediction markets they were using.

~~~
bcroesch
Kinda depends on which platform you were on (GJP used several). Some were
prediction markets while some were opinion pools, both of which are
scored/rewarded differently.

Our company (Cultivate Labs) recently acquired Inkling Markets (a very early
YC company that built prediction market software) and have been building a new
version of the PM platform, which will hopefully address some of the
risk/reward quirks.

If you're interested in this stuff, you might be interested in the two topical
PM sites we're launching:
-[https://sportscast.cultivateforecasts.com/](https://sportscast.cultivateforecasts.com/)
(obviously focused on sports)
-[https://alphacast.cultivateforecasts.com/](https://alphacast.cultivateforecasts.com/)
(officially launching later this week, focused on global finance, politics, &
tech).

~~~
waterlesscloud
Ah! I didn't know Inkling was a YC company. Interesting.

I'll definitely check out your new sites.

------
EliRivers
Adam Curtis' article on the competency of these people is entertaining and
contains some really quite disturbing information about the incompetency of
some professionals in this field.

[http://www.bbc.co.uk/blogs/adamcurtis/entries/3662a707-0af9-...](http://www.bbc.co.uk/blogs/adamcurtis/entries/3662a707-0af9-3149-963f-47bea720b460)

I don't always agree with Curtis, but his articles and documentaries are
always worth the time.

------
jessaustin
Intelligence "works" in the same sense that torture does: to further the
unstated private interest of whoever hired the intelligence and torture
agencies. If we're interested in the truth rather than The Truth, we can't
trust non-public sources that have repeatedly failed in public ways.

~~~
presidentender
I would encourage you to read the article. While I am inclined to agree with
your sentiment, the article presents a study that actually deals with the
concrete effectiveness of intelligence analysts.

~~~
jessaustin
Yeah I read the article; it wasn't very long. Either the authors of the study
are "insiders" with clearances, they have seen only redacted material, or
(most likely) both of those are true. All of those conditions ask me to
believe non-public sources. I won't do that.

[EDIT: I hesitate to join the ranks of those who complain about shitty-but-
common HN habits, but assuming for no reason that anyone didn't read TFA seems
nearly as common as it is shitty.]

