
The Surveillance Engine: How the NSA Built Its Own Secret Google - aburan28
https://firstlook.org/theintercept/article/2014/08/25/icreach-nsa-cia-secret-google-crisscross-proton
======
pdkl95
What I want to know is if this search tool is the backend for the "parallel
construction" application forms[1] from earlier this year?

[1]
[https://www.techdirt.com/articles/20140203/11143926078/paral...](https://www.techdirt.com/articles/20140203/11143926078/parallel-
construction-revealed-how-dea-is-trained-to-launder-classified-surveillance-
info.shtml)

~~~
atmosx
Possibly, among other things.

------
eyeareque
People always assumed they did this but now we have proof.

I thought it was amazing that the government spent so much time discussing the
call records being logged.. when they are doing so much worse. Maybe that's
how they keep people focusing on what the government wants to talk about? (aka
look over here, nevermind that thing over there...)

This find is way worse than call detail records..

~~~
knowaveragejoe
I think it's a combination of things. On the one hand, the phone records issue
is the most easily understood by the general public, and so it already had
traction there. It sort of fell into the government's lap that they could
focus all of their press' attention on that issue in particular.

I agree doubly so however - the phone records are near the very bottom of the
list when it comes to the severity of what Snowden's(and possibly other's)
revelations have brought to light.

------
DanielBMarkham
This has been released and is use by almost two dozen federal agencies and now
is the first time we're hearing about it?

When folks tell us crazy things, like the government is tracking every place
you go and your opinions through your cell phone and social networks, we're
supposed to say something like "That's extraordinary. With extraordinary
claims, we require extraordinary proof" Then, if they persist, we're supposed
to say something like "Such a program would require far too many people to
keep a secret. We couldn't even keep the atom bomb a secret. The government is
terrible at keeping secrets. Such a claim is just too far-fetched."

These are the traditional things taught to people who are supposed to be
clear-headed and rational. It's the way we engage crackpots without taking
them too seriously.

These responses seem to have failed us miserably in the current circumstances.
As it turns out, yes, that's what they were doing, and yes, it was
extraordinary and required lots of people to keep incredible secrets. But it
still happened.

These _things_ keep happening in the realm of automated surveillance, both by
the government and corporations (and worse, when corps do it and the govt
scoops it all up later) that would have been considered completely whacked
just ten years ago. The stuff of paranoid fantasies.

Our tools of rational inquiry have failed us.

~~~
cryoshon
For people paying attention to the evidence, the writing has been on the wall
for many years-- enough to build credible, defensible, strong suspicions, at
least. Of course, these people have been slandered in perpetuity by the
government, who stands to gain stability from the "rationalist" suspicions
seeming ridiculous. The confirmatory value of these reveals cannot be
understated: a subset of the people who have been much maligned as "conspiracy
theorists" were in fact reasonable in their assumptions and correct in their
hunches.

If these evidence-attenders and conspiracy theorists were maligned as being on
the lunatic fringe up until recently and were fully exonerated, what else
could they have been right about? This question has the government running
scared, scrambling to reassure the public that the now-redeemed former
crackpots are still off the rocker, and have nothing else that is relevant to
say about things which the public doesn't yet believe.

The public failed these conspiracy theorists because they accepted the
establishment friendly line that they were kooky-- the public accepted the
false associations and intentional counfoundment made between people with
legitimate but not mainstream ideas backed with evidence, and various cultish,
indefensible fantasy theories like aliens controlling the government, and the
moon landing being faked. Thankfully, the public received a gut shot when
Snowden went public with the truth. Maybe next time people will listen a bit
closer to voices that the government would love to suppress.

------
us0r
Somewhat OT but its amazing out of 3 "FTEs" 1 was for design ("GUI") and this
is what they came up with. [1] pg 22

[https://firstlook.org/theintercept/document/2014/08/25/shari...](https://firstlook.org/theintercept/document/2014/08/25/sharing-
communications-metadata-across-u-s-intelligence-community)

~~~
seiji
Also notice the entire screen shot. Netscape on Windows XP in, best we can
tell, sometime after March 2006.

Netscape. With an AIM launcher in the status bar. Accessing TOP
SECRET//COMINT//NOFORN resources.

~~~
Amezarak
It would be on a totally isolated network. Classified networks don't touch
unclassified networks.

~~~
seiji
Except by way of überadmin USB drives?

------
MalcolmDiggs
It's kind of remarkable that they kept the lid on this on long as they did.
With so many local/loosely-related agencies involved, I'm surprised _somebody_
didn't leak this much sooner.

~~~
Jimbotron
Yes, I find it VERY hard to believe that this went on and nobody mentioned it
at all. Not one disgruntled employee, not anyone mentioning it to a friend at
a bar? No way.

~~~
sentenza
Uhm. Just because you didn't believe it doesn't mean that it wasn't known. The
CCC was talking about this stuff years ago, some of it conjecture, some of it
information from old sources/anonymous sources and the like.

Now we have proof. So that's that.

Also, given the strong "disincentivization" for whistleblowing that is going
on at intelligence agencies, I am not at all surprised that it took a few
years before somebody went public with it.

Hell, it used to take decades before this kind of stuff became public. Just
this year we were made aware that the West German spy agencies opened _all_
cross-border inter-German mail. You know, the kind of thing that we in West
Germany were told only the East Germans would do. That was kept secret for
more than 20 years after it stopped being done, even though thousands of
people were involved.

~~~
TeMPOraL
Well, it is easy to dismiss people saying kind of stuff as 'crackpots' or
'tin-foil-hatters'.

I personally keep to a simple rule. If something is technologically
feasible[0], economically viable and someone has an incentive for it, it will
happen, period. This rule of thumb successfully filters out weather control
crackpottery while correctly identifying surveillance capabilities (and no,
it's not hindsight bias; the only thing I was surprised about Snowden
revelations was that it was underwhelming).

[0] - and keep in mind that pretty much anything is.

~~~
bad_user
In what way were the revelations underwhelming? For me overwhelming is their
ability to coerce companies and their willingness to do so, in spite of the
potential for a loss of trust in US companies. Sometimes such maneuvers can
have a big cost long term - I can imagine at least governments switching
software, hardware or services to more local providers. I mean, think about
backdoors - you may trust the NSA, but if backdoors exist, it is only a matter
of time before they get discovered by other parties. Countries now also have
the ultimate argument for the balkanization of the Internet. This can't be
good and all of this distinction between US citizens and foreigners doesn't
mean anything for foreigners.

~~~
TeMPOraL
> _In what way were the revelations underwhelming?_

The technological ones. I was expecting (and I still believe) that they're
tapping much better into the Internet that it is revealed, and that there are
many more hardware backdoors out there.

> _Countries now also have the ultimate argument for the balkanization of the
> Internet._

I agree. This is very bad. I didn't say I was underwhelmed with outcome
(though I am with the reaction of public and of Internet companies), just with
the revealed capabilities of the NSA.

------
curiousDog
Wonder how and where they recruit their top-talent. Pretty much every top
scorer in my school went to Facebook/Google/Msft/Amzn. The mediocre ones went
to Northrop Grumman/Raytheon/Rockwell and the like.

~~~
seiji
The Company doesn't exactly encourage recruits to advertise their new
position. You mostly see people just vanish for a while. Young recruitment is
largely done through direct faculty recommendations at schools, so it's more
of a "don't find us, we'll find you" proposition.

~~~
hiddencost
Yup. I had a friend who spent some time there. Easily the smartest kid in my
major. Hired due to a direct referral from faculty. It gets scary, once you
see how many math professors have security clearances.

~~~
eric_h
> It gets scary, once you see how many math professors have security
> clearances.

Makes sense, though. The people who are the best at math are going to be the
people who are the best at grokking and expanding upon the existing
understanding of the math involved in cryptography.

------
niels_olson
Is it just me, or is it tragicomical that I can't access firstlook from a
government network?

~~~
Tangokat
According to the Intercept, the U.S Military has banned employees from
visiting First Look because there could be classified information on the site.
[1] [2] (extra source since you can't see the real one..)

Maybe that is what you are experiencing?

[1] [https://firstlook.org/theintercept/2014/08/20/u-s-
military-b...](https://firstlook.org/theintercept/2014/08/20/u-s-military-
bans-the-intercept/)

[2] [https://thebulletwire.com/u-s-military-bans-glenn-
greenwalds...](https://thebulletwire.com/u-s-military-bans-glenn-greenwalds-
intercept/)

~~~
atmosx
That's similar to the wikileaks ban. I don't understand this policy: If
something is accessible virtually from everywhere, what do you gain by not
allowing your employees to access the info?!

The only thing I can think of, is internal issues at political level (who said
what lies to whom to achieve more funding about his work, etc.) which even
that won't take long before the interested party eventually finds out.

~~~
dublinben
It perpetuates the mythology of classification and information control. If you
have clearance, you can lose it by accessing classified information
improperly. That would probably mean losing your job. If you _don 't_ have
clearance, you should be considering all the information you can get your
hands on. _Not_ doing so, would be poorly doing your job.

~~~
atmosx
nicely explained, thanks :-)

------
eli
I guess it's always cool to learn about top secret stuff that spies do, but I
don't get why the specifics of how their search engine works is significant.
Am I just missing something?

~~~
pdkl95
This, as with most of Greenwald's work on this subject, has been a slow effort
to educate people about a very technical (for the average person) topic.
Building on previous reports that showed the type of activity and the threat
it can cause, this installment is about establishing the _pattern_ of
illegality.

If the NSA could justify the time and expense to make a search tool, there
must be a lot of searching needs... which aren't legal for the NSA to send to
the domestic-oriented agencies.

Also, more explicitly-stated details about how powerful "metadata" can be.
Obvious to anybody who understands INNER JOIN, but this is still new for most
people.

//Of course, I'm sure all this ignoring of warrants and the 4th Amendment, to
quote a DOJ lawyer in the recent EFF mess regarding Jewel v NSA, is all for
the "national security of this _company_ ". (emphasis mine)

~~~
motters
I think that's the key insight. It may not be legal for NSA to look at records
of US citizens, but build a search engine and there are other departments for
whom it is legal to do that. NSA just becomes the facilitator or router of the
data.

~~~
eli
Is that even what's being described? I don't see any reason why it would be
bad to let the DEA search an NSA database of intelligence about foreign
suspects.

The problem is (as it's always been) that they aren't just criminal suspects
and they aren't all foreign. But that was the problem whether or not the NSA
built a "Google."

------
DigitalSea
Imagine if hackers were to somehow to find out where this tool is being hosted
and then brute force their way in? Presumably such a tool is heavily secured
and off the grid, but if it is somehow accessible from the Internet and an
attacker were to find out where, it's only a matter of time now that the cat
is out of the bag. Could you imagine hackers having access to troves of
metadata and information like that? It's a scary thought.

I'm surprised such a thing took so long to be revealed. If you've got as much
data as the NSA has, wouldn't you want a Google like search engine to be able
to search through it? It makes so much sense which is why I am surprised some
people are surprised about this.

~~~
ChrisAntaki
Furthermore, think of how profitable surveillance on certain individuals could
be. Could a military contractor not have a thriving side business funneling
out intelligence on innocent American targets? Could multiple people not be
hired just to clear tracks and make justifiable excuses for accessing the
records?

If Snowden, as one person, accomplished what he did with (the majority of
Americans would say) good intentions, imagine what a team of people, who were
just as smart if not smarter, could do if they didn't have morals guiding
their actions. Imagine how well a team could cover their tracks, dot all the
I's, and cross all the T's, compared to just one guy.

This is all common sense, but it bears repeating.

~~~
smokeyj
I'd just play the market. Find dirt on CEOs, short their stock and pass the
dirt to the media.

------
wyck
Brought to you by meta data commodities and pattern recognition limited -
don't be evil. just be data.

------
fit2rule
I believe its really telling that, rather than take the tactic of fully
promoting open society, the inclination is towards more and more secrecy.
Like, dire, utter secrecy. Kill someone-style, secrecy.

Imagine the other end of the scale - where in fact every detail about
everyones lives is wide open and available for everyone and anyone to access.
Willingly. Freely. A new order of celebrity: total telepathy.

Do you think we'd be dealing with terrorism, then? Would there be the
idealist, killing souls, for a little private time?

------
srj
Maybe there was an earlier release I missed but this report looks to be the
first to show the extent of the call record data being collected. SIM, IMEI,
Lat/Lng and more. Logged for every American, every day, all since 2007. And
now shared outside of the military context it was gathered and authorized
under to civilian agencies.

It's possible the program has been reigned in since these slides were authored
but that seems unlikely.

------
BorisMelnik
I am just not surprised about this at all. They have more information than I
could ever imagine, wouldn't any rational person assume there is a search
engine to index, sort, parse, and return results?

seems like a lot until you consider how many indexed pages Google has:
[http://i.imgur.com/EqIJAoL.jpg](http://i.imgur.com/EqIJAoL.jpg)

why not throw in grains of sand or atoms in the universe?

~~~
Perseids
Nice comparison, though I hate it when graphics look like they were drawn to
scale, but actually aren't. (The Google indexed pages bar would have to be
more than 10 times larger.)

More on topic: Whether you can compare the index sizes or not mostly depends
on their signal to noise ration. Given the number of Google search request per
month and assuming and average of 5 viewed results, you can put a lower bound
of 99% of pages that are never seen by anyone. As the search results should
greatly overlap the actual number of relevant sites is probably orders of
magnitude less than 1%.

------
cryptolect
To me, this explains why the five eyes nations are pushing for (meta)data
retention legislation. It's a condition of participating in the scheme.

------
nowarninglabel
I don't think it remained "under the radar" like some are commenting. There's
mention of it in Snowden's disclosures and if you search around, lots of
government recruiting related info, e.g.,
[http://www.socnet.com/archive/index.php/t-108034.html](http://www.socnet.com/archive/index.php/t-108034.html)

------
Twirrim
I think I'm more surprised that this was a surprise to people.

Of _course_ they built a search engine. Wouldn't you? Don't you have similar
at your workplace? We use them all the time. Think about web interfaces built
on top of ElasticSearch, for example. Is that not a 'search engine'?

~~~
dredmorbius
It's not the building a search engine part.

It's the opening it up to LEO. Taking homeland security-justified warrantless
searches and turning those over to _criminal_ investigations is far beyond
unconstitutional.

~~~
mpyne
> Taking homeland security-justified warrantless searches and turning those
> over to criminal investigations is far beyond unconstitutional.

Except, that it's not.

It's all about _how_ the government comes into possession of information that
makes it the fruit (or not) of the "poisonous tree".

Once the government legally learns of some bit of information, there's no
requirement _by itself_ that they have to "pretend not to know" something is
going on. That's why a _third party_ voluntarily divulging information to the
government doesn't cause that evidence to get thrown out of court: There may
be a cause for civil or even criminal action against that third party, but
that doesn't invalidate the evidence.

Now, Congress has passed statute laws limiting information sharing between
intelligence agencies and law enforcement precisely because of the threat of
having this all-seeing eye subvert democratic government, but the limits were
never complete exclusions, even before 9/11\. And the reason statute laws were
needed is because there was no barrier to this activity from the judicial side
alone.

Parallel construction came in not because evidence was illegally gathered, but
because giving the (fully legal!) chain of custody for evidence derived from
intelligence sources would have quite naturally have "burned" that source or
method, so the NSA would require law enforcement agencies using that data to
use alternate (again though, _legal_ ) means to make a case in court, in order
to protect their intelligence source. It's the old apocryphal dilemma about
whether to let Coventry be bombed to protect ULTRA, applied to the post-9/11
world.

------
Dolimiter
Is anyone else puzzled that stories about the NSA are accepted as gospel fact
when posted on Hacker News?

The Reality Distortion Field appears, and people believe, because they want to
believe.

It depresses me, the lack of intelligent discourse.

Most tech people I meet _actually believe_ that the NSA records and stores all
telephone calls. It's depressingly stupid, but I have given up arguing, logic
and sense are not welcome when the NSA is the topic.

~~~
glitchdout
They do record and store (foreign and domestic) phone calls. Your comment
proves you're simply an ignorant or a shill.

Proof:

* NSA Collects ‘Word for Word’ Every Domestic Communication, Says Former Analyst [http://www.pbs.org/newshour/bb/government_programs-july-dec1...](http://www.pbs.org/newshour/bb/government_programs-july-dec13-whistleblowers_08-01/)

* NSA collecting phone records of millions of Verizon customers daily [http://www.theguardian.com/world/2013/jun/06/nsa-phone-recor...](http://www.theguardian.com/world/2013/jun/06/nsa-phone-records-verizon-court-order)

* NSA surveillance program reaches ‘into the past’ to retrieve, replay phone calls [http://www.washingtonpost.com/world/national-security/nsa-su...](http://www.washingtonpost.com/world/national-security/nsa-surveillance-program-reaches-into-the-past-to-retrieve-replay-phone-calls/2014/03/18/226d2646-ade9-11e3-a49e-76adc9210f19_story.html)

* NSA spying flap extends to contents of U.S. phone calls [http://www.cnet.com/news/nsa-spying-flap-extends-to-contents...](http://www.cnet.com/news/nsa-spying-flap-extends-to-contents-of-u-s-phone-calls/)

~~~
Dolimiter
I'm a "shill"? Really? This is the level of debate?

Also, none of your links in any way prove that the NSA stores all phone calls.
None of them. 1 is hearsay from someone, another talks about metadata, and
another says that the NSA can record any phone call (yes we know that).
Nowhere is there any proof that all calls are being recorded.

This is what I am talking about. It doesn't seem possible to have a
conversation about the NSA on Hacker News that is based on logic, reasoning or
sense.

~~~
drunkcatsdgaf
"MYSTIC is a voice interception program used by the National Security Agency.
The program recorded every phone call made within a non-specified country for
thirty days."

[https://en.wikipedia.org/wiki/MYSTIC_(surveillance_program)](https://en.wikipedia.org/wiki/MYSTIC_\(surveillance_program\))

~~~
mpyne
Again, this is moving the goalposts.

The original claim was "the NSA records all phone calls".

This is saying, "the NSA records all phone calls, _in a certain, non-domestic
country, for a 30-day window_ ", which is nowhere near the same.

You would think that simply accusing the NSA of what it's actually doing what
be good enough, but that's only rarely the case here at HN.

