
How likely is the NSA PRISM program to catch a terrorist?  - johndcook
http://bayesianbiologist.com/2013/06/06/how-likely-is-the-nsa-prism-program-to-catch-a-terrorist/
======
coldtea
> _How likely is the NSA PRISM program to catch a terrorist?_

A better question, which involves historical understanding about the issue
rather than going directly to the maths and taking the framing of the thing as
granted:

How likely is the NSA PRISM program even cares about catching terrorists?

Sure, they wouldn't say no if it did. It would even justify the program to the
eyes of the public.

But, thing is, similar programs have been going on for centuries in all
governments. They care only about two things: spying on third countries
(enemies, competitors, etc, things that effect commerce and military) and
spying on their own citizens (population control, etc). Those programs have
roots aeons before 9/11, and aeons before "terrorism" was any real concern of
the government.

~~~
jlgreco
The NSA isn't stupid. They know that screening for absurdly low incident
things is an exercise in futility. I draw the same conclusion as you because
of this: _they don 't care_.

...well, not about terrorists anyway. If you are interested in flagging people
that make up a much larger portion of the population then such systems are
very practical. A P(flag|pot head)=.99 and P(flag|sober)=.01 would make for a
distressingly effective system for example.

~~~
s_kilk
I don't think it's even really about flagging anyone. The objective is more
likely to be good old-fashioned careerism than anything else. These guys are
in the business of "screening for terrorists" (apparently), so they are
unlikely to want to see less terrorists in the world. That would put them out
of a job. No, much better to expand the definition of terrorist, and expand
surveilance so they have plenty of work to keep busy with.

------
mtgx
This shouldn't even be a question. Are we going to ask how likely is that
torture is going to work, too?

We shouldn't allow broad surveillance of everything you ever do or say online,
and we shouldn't accept torture as a means of interrogation either. This is
simply deciding about the kind of society we want to be.

Terrorists can be dealt with in other ways, too, and one of them would be
trying to avoid creating blowback and radicalizing future generations of
terrorists because of current actions.

~~~
jamesaguilar
Maybe this makes me a bad person, but I'd ask how effective torture would be.
If torturing, say, five people a year for one day each, prevented all deaths
from war and terrorism for the whole year? Make that deal any day.

To me the problem with torture isn't that it's "bad". It's that it doesn't
serve a useful purpose and serves many counter-purposes. If it wasn't so
clearly useless, I'd have to think a lot more about whether I'd support it.

~~~
jacquesm
> If torturing, say, five people a year for one day each, prevented all deaths
> from war and terrorism for the whole year? Make that deal any day.

Ok, we'll start with you.

This had better work!

~~~
jamesaguilar
I'm not really sure what you're getting at. I acknowledged efficacy in my
comment. Individual reluctance to be harmed has almost nothing to do with the
calculation of whether it's overall a better world with the policy or without.

~~~
jacquesm
It's only a better world from the perspective of those that are not being
tortured.

The fact that by massive voting imbalance those few being tortured would have
0 chance of stopping it does not begin to describe the injustices inherent in
your proposed scheme.

Ethics and principles are hard to reconcile with statistics and calculations,
by suggesting that there is a greater good for which you will discard your
principles in a heartbeat you are devaluing everything.

The question then becomes where do you draw the line?

A few individuals? A few tens of individuals? A few hundred? A few thousand?
As long as it is one person less than would be killed otherwise?

Best not to make that first step and stick to 'torture is bad, no matter what
the upside'.

~~~
jamesaguilar
I don't really believe in ethics or principles. Those are just words people
use to dress up their preferences about how the world should work.

Where do you draw the line therefore zero is a standard approach from the nays
in almost any policy debate, but it is easily defeated. I draw the line
somewhere. Not sure where. I'd have to figure it out if I ever heard evidence
that Convinced me torture is useful.

~~~
jlgreco
If torture were useful and ethics don't exist, then why would you draw the
line anywhere other than "one less victim"?

~~~
sokoloff
Because of a thing called the discount rate. For future or uncertain outcomes,
one rationally discounts the amount they are willing to pay right now, with
100% certainty, against the future and/or uncertain outcome.

There's a possibility (IMO, a certainty) that torture isn't 100% effective,
and there's some number greater than 100 of deaths that I'd need to avoid a
year from now to torture 100 people today. (some of those 100 might die anyway
before the terror event; we might thwart the terror event via some other
means; the terror threat may itself disappear [death of those terrorists or
other means])

It's just another case of "pay me $1MM today and I promise to pay you back
exactly $1MM in 20 years"...

------
tokenadult
An interesting application of Bayesian reasoning to the problem of screening
for low-base-rate phenomena. (And terrorist criminal activity is a lower-base-
rate phenomenon in the United States, so far, than prostate cancer or other
dangers that are screened for.) The mathematics, of course, is exquisitely
sensitive to exactly how sensitive and specific a screening program is. (By
the way, so far news reports are saying that the NSA program that is all over
the front page of Hacker News, deservedly so in my opinion, is not a screening
program but a data collection program, with analysis of the collected data
triggered only by other kinds of law enforcement evidence-gathering. We'll see
what further reporting says about that issue.)

~~~
suredo
There is only one reason for collecting this data, to analyze it... and the
biggest obstacle is the gathering of that data.

~~~
rdtsc
Well I guess technically there is this pesky thing getting in the way called
the Constitution. But they re-defined some terms, moved some things around,
decided that 'search' probably occurs when a human looks at the data (LOL,
founding fathers didn't know about databases ;-). So it is not just a
technical challenge, given their budgets they can solve the technical side it
seems.

Imagine also (and I've mentioned this before) there is no statute of
limitations on historical data in case of warrant/nsl, so if warrant is every
issued (and we know they are never denied) they can get _all_ your data from
years and years back. Emails you sent 10 years ago to secret lovers, jokes
about punching the president in the face when you were 13 years old, stuff
like that. Make no mistake they will find stuff to blackmail, jail, or scare
anyone into anything if they dig hard and long enough into everyone's past.

I suspect more of the people involved in this will decide to do what Manning
did. They see the waste, the lies, wars on terror, wars on drugs. If anything,
NSA and other XYZ agencies, love to hire patriotic people, and a fraction of a
percent of them maybe still are and will say "fuck it, everyone needs to know
about this".

------
dvt
Many of us with some (Bayesian) statistics/game-theory backgrounds are
familiar with these kinds of puzzles. There are many (many) such anecdotes
littered through math textbooks (HIV +/- testing comes to mind) as well as
some real-life scenarios (UC Berkeley hiring practices and women comes to mind
-- 1970s case).

HOWEVER, let me just say that two metrics in this little puzzle are simply
wrong (or at the very least unfair):

P(+ | bad guy) is simply a LOT larger than 0.99 if you want to play the game
correctly. The + comes from a POSITIVE outcome -- that is, a terrorist attack
is thwarted and someone is thrown in jail. Some noise in the system (i.e.
pizza orders) does NOT skew the positives down. After all, discerning between
pizza orders and terrorist activity is _part of the algorithmic process_ \--
the automated system may flag both cases, but the buck doesn't stop there (or
anywhere close).

P(+ | good guy), again, suffers from the same problems as P(+ | bad guy).
Unless there is evidence of 1% of the people being monitored being thrown in
jail for terrorism charges, that number is a lot (lot) less than 1%.

There have been plenty of bogus terrorism charges (see Guantanamo) and I think
there may be something here. But if we want to play this game correctly, we
need to be careful. To do this simulation, we need numbers that we will simply
never have access to: e.g. how many terrorist threats did PRISM avert?

~~~
vidarh
You are conflating the chance of being singled out of by PRISM with the final
outcome of the process involving its use, and the benefits vs. damage done.

He's not measuring, or even saying anything about, the latter.

The point he is making is that to get any benefits out of PRISM, no matter how
you value those benefits, you'll need to do a _massive_ amount of further
investigation to be able to identify an actual terrorist (unless you were to
take drastic action; e.g. imprisoning everyone that gets "reported" \- but the
high number of reports to bad guys also means that becomes "impossible")

------
bane
This is an incorrect question. Anti-terrorism is not the only thing the NSA is
interested in.

Better question: How likely is the NSA PRISM program to provide intelligence
useful to the NSA's mission?

~~~
flyinRyan
If you're going to pose that question, you better pose the other question: is
NSA's mission legal/moral?

~~~
bane
I wouldn't conflate legal with moral. Those are two separate questions.

~~~
flyinRyan
I was trying to save space. That was actually two separate questions.

------
gordaco
This is a very well known problem, but one that no politician on this planet
seems to understand.

Of course, as others have pointed out, the problem is not the likelihood of
getting a bad guy. The problem is that the means are unacceptable, no matter
the goal.

------
EliRivers
Is that what it's for? I thought it'd be a big retroactive database; identify
someone we want to know about now, and then look back through the last year's
records to see what she's been doing.

------
adventured
I wonder if the intense scrutiny these programs are coming under increases the
likelihood of catching the next attempted terrorist attack by using them. (if
you catch my cynical drift)

------
vsbuffalo
So? 1/10,100 isn't bad at all. Suppose you have many independent tests, each
creating watchlists. Suddenly you have the intersection of individuals hitting
each of these, and the number of "interesting" individuals narrows. This is
not a univariate problem; they're likely correlating everything to everything,
and have adaptive snooping methods based on how high you are ranked a risk.
Once you're high enough, field offices know about you and will keep a closer
eye. It's really not a needle in a haystack (and as a privacy nerd, it pains
me to say this, but I bet it is working).

~~~
mtgx
I think what his data shows is that _out of 10,000 terrorists_ , it might help
catch _one_.

Remember they've been doing this for more than a decade, and it still didn't
help them with the Boston bombers. They even had the guy in custody before.
Still didn't help. The abuse of surveillance power is not effective, even if
it was legal/constitutional.

~~~
dwild
Wait, media were talking about Prism before? You have no idea the effect they
had on terrorism. We can't tell with the information we have currently. We
don't even know which information is true. Maybe they saved ton of terrorist
attack, maybe they got multiple criminal or killers with that, but we won't
know because they never said it was with the help of Prism.

All we know is that this project cost 20 millions, they had access to
informations from a dozen companies, which is easily multiples hundred
terabytes, probably even petabytes. I doubt 20 millions is enough to do
anything useful with these informations. However can we even trust theses
slides?...

~~~
flyinRyan
Why on earth would you assume it's working? This is immoral and wrong, it's up
to the government to show that it's buying us something (but I'd be against it
even then). If they don't show that we should assume it hasn't stopped even
one crime.

------
flyinRyan
Well, it's been running for a while, how many _has it caught_? Oh, zero? Yea,
that's what I expected.

------
rangibaby
> How likely is the NSA PRISM program to catch a terrorist?

Chechen Pressure Cooker Bombers: 1 NSA: 0

------
craigyk
"So you're saying there's a chance..."

Haven't read it, but couldn't resist as it's an obvious conclusion.

