
What I Learned in 40 Years of Doing Intel Analysis for US Foreign Policymakers [pdf] - iuguy
https://www.cia.gov/library/center-for-the-study-of-intelligence/csi-publications/csi-studies/studies/vol.-55-no.-1/pdfs/CleanedPetersen-What%20I%20Learned-20Apr2011.pdf
======
lmkg
I work as a digital marketing analyst (think Google Analytics). While reading
this piece, I find myself nodding and agreeing with 90% of it. As it turns
out, the way that decision-makers incorporate data and expert opinion doesn't
change when whether they're in charge of $10k ad budgets, or ten-year
occupations of a foreign country.

Some observations of my own, that I can see echoes of in this paper:

Decision-makers have a mental model where analysts' job is to synthesize facts
and provide them, and it is the job of the decision-makers themselves to put
the facts together into a big picture and/or plan of action. This is generally
a poor set-up. Firstly, "facts" with any certainty are thin on the ground,
there's only inferences from small spots of data in between clouds of blind
spots. Secondly, the analysts have more experience with the trends that they
study, and often more training in how to incorporate them into a big picture.
Most managers will say "I want all the data," but if you give them what they
ask for, they will most often draw incorrect conclusions. You have to spoon-
feed them, even though neither party particular wants that.

People don't understand the limits of data, and they sure as heck don't
understand uncertainty. If you make 100 predictions, and are 99% sure about
all of them, then on average one will be untrue. And all you'll ever hear
about was that you were 99% sure and were wrong. In my job, there's actually a
consideration about not providing data that we don't totally trust (as opposed
to providing with caveats), because a single data point that's wrong erodes
trust in data in general. It sounds like that's not an option to the CIA: if
they're tasked with a question, they must provide an answer (which is a
problem itself).

We've found that if we our client to use our analysis, we actually need to
write it with _his_ audience in mind. We literally write it with the design
goal that the client be able to copy-paste our words or graphs into his own
deliverable. Because that is literally what will happen. The more effort it is
for the client to consume our work, the more likely they are to save
themselves the time and ignore it. And as bad as that is, it's also probably
the best way. If the client re-words things, he may misunderstand things.
Encouraging plagiarism cuts down on the game of telephone.

~~~
jonnathanson
_"...the client be able to copy-paste our words or graphs into his own
deliverable. Because that is literally what will happen."_

I've spent most of my career in marketing, and as far as I can tell, this
happens _everywhere_. Well, almost everywhere. The one exception I've
encountered was working at Apple, where people expected you to hammer out a
beautiful and convincing Keynote deck on your own. Everywhere else, people
more or less copypasted whatever decks, charts, graphs, and bullets their
agencies had sent them.

The problem with copypasting is that it _can_ lead to a game of telephone. A
sequential erosion in the quality of information. If you're not keeping track
of who's been lifting and plagiarizing and pasting which information from
which sources, at how many steps along the value chain, you end up with a deck
full of secondhand information and hearsay. The quality of the information (or
confidence in the information) fades a bit with each subsequent generation of
copypasting. Sort of like lossy data encoding. At larger organizations, as you
might imagine, there will be many "generations" of each presentation. This
makes the lossiness issue even worse.

The culprit here is always time. It's exactly as the author of this article
points out. People don't set out to make shitty decks, or to copy and paste
information, or to tl;dr their way through analysis. But that's just what
people have to do to survive. If you're spending (conservatively) 20-30 hours
a week in meetings, and many of those meetings leave you with deliverables on
tight deadlines, you're going to be eking out any and all efficiencies that
come your way.

I've never done any copypasting, and if I'm being honest with myself, it's
probably hurt me more than it's helped me.

------
jessaustin
To me this is an excellent argument for the disbandment of the CIA and related
agencies. This is obviously a thoughtful, knowledgeable, conscientious person.
He probably has colleagues who are the same. Yet, when organized as they
currently are, they can't think their way out of a paper bag. There must be a
better way to do this work.

~~~
mercurial
It may be more of a problem with the "service mentality" the document is
talking about. Bad things happen when what your customer actually wants is
"Find me a pretext to invade the Middle East and reshape it in a more US-
friendly form, because Western intervention in the area has always turned out
fine."

~~~
arca_vorago
Bingo. The problem is that in the intel community they used to have that salty
old graybeard that would tell general X and politician Y the cold hard truth
to power that they _need_ to hear, instead of what they want to hear. Post
9/11, there is evidence that it was mostly vp Cheney who physically intervened
in this process, personally pulling in analysts and grilling them...

The problem to me is that the same chilling effects we have seen develop in
the journalistic community have also grown in the intel community, and when
you have analysts who spend more time kissing ass than calling people out
bullshit like they should, we are only going to end up in worse geopolitical
situations.

NSA right now holds cards over CIA, keep that in mind as well.

~~~
flurpitude
_NSA right now holds cards over CIA, keep that in mind as well._

Can you elaborate a bit or suggest some reading on that point?

~~~
arca_vorago
Well it's more of an insight gained mostly from reading former-spooks books
and similar. The fact is that the resources, both monetary and physical, are
much larger than the company (CIA). It employs many more people as well, and a
lot of people don't realize that the agency (NSA) has paramilitary divisions
as well.

The main reason I say though is because NSA has the upstream under their
thumb. Pretty much anything the company does is going to be visible to the
agency but not the other way around, which creates an information power
disparity that is inevitable with the current organizational structures.

------
tjradcliffe
The point about "because" is very well-taken: _Every "may" and "likely to" and
"could" requires a "because" statement or its equivalent—the reason we believe
what we believe_

It is important to remember that any "may", "likely to" and "could" statement
can be replaced with a "might not" statement with precisely the same meaning,
unless you quantify the probabilities (difficult and often so imprecise as to
be pointless).

We see this all the time in tech journalism: "New technology might not lead to
breakthrough in flying cars!" has exactly the same meaning as "New technology
may lead to breakthrough in flying cars!" but for some reason we always see
the latter and never the former.

I recommend reading any article with unqualified "may" statements with them
flipped to "might not". It will rapidly become clear how much of this kind of
journalism is hopeful nonsense.

------
jqm
This sounds similar to being an IT guy in a big organization. They have 5
minutes, they think they know better (and occasionally they do), they might
listen to you and you have to beg them to do so.

Except the ending. That sounds like someone who has spent too much time around
politicians and become infected.

~~~
aero142
This is someone who moved up the ranks in the CIA to a very high level. They
are very adept politically.

------
flurpitude
_" Our ability to “raise the level of the debate” or to “help policy-makers
make the best decisions possible” or to “speak truth to power”—however one
defines the mission—rests on one thing and one thing only: our reputations for
analytic rigor, objectivity, and total integrity. Lose that and we lose
everything"_

It would have been interesting to see at least some acknowledgement of, and
response to, the fact that the CIA currently has acquired itself a reputation
for extracting false intelligence clumsily through relentless torture and
brutalization. How, despite all the very smart and experienced people working
for the organization, does that come to be the output? What goes wrong there?

------
stainednapkin
Great. This is how a deluded statist stooge thinks after 40 years of
indoctrination.

EDIT: Why was this even posted? Useless garbage.

~~~
nemo
Well, I found it equally fascinating and horrifying. You don't need to agree
with or admire an author to be able to learn things from their writing.

