
The NSA’s Hidden Spy Hubs in Eight U.S. Cities - BerislavLopac
https://theintercept.com/2018/06/25/att-internet-nsa-spy-hubs/
======
darksaints
Wow, just made a crazy realization. A while back I was downtown with my 5yo
kid, and we were by the bus stop right next to this building (You can actually
see the bus stop in the picture). The bus was late, and my kid was hyper, and
he tried going through the revolving doors. They were locked, no big deal.
After a while, he tried going through the doors again. At this point, _three
security guards with ballistic vests_ come busting out of the door _with their
hands on their holsters_ , and chewed me out for letting him play near the
door.

I know my way around the security industry. These weren't normal security
guards that get paid to watch cctv and call the real cops. They don't give
those guys guns or ballistic vests. To me it was completely bizarre that a
telecom building would have that sort of security. Now it all makes sense. I
actually wouldn't be surprised if they were actually military in disguise.

~~~
rconti
The one in the picture? Is that NY? I don't recognize the skyline.

~~~
saosebastiao
Ctrl-F Jovelle Tamayo

It's on the corner of third and lenora.

~~~
rconti
Crazy. I used to work on 2nd and Lenora and I still can't say for sure I've
noticed this building.

------
stephengillie
The Seattle location - 1122 3rd Ave - is an interesting spot. According to
Google Maps[0], next door is the FBI Seattle Division office. On street view,
the building itself has a street-level office with signs for both AT&T and
CenturyLink.

3rd Ave itself is slightly notorious - hosting the surface entrances for the
Seattle bus tunnel, and a large number of inter-city routes, the area has a
reputation for crime and the occasional death by shooting. It was a bit
worrying to commute through, as a bus rider, and seems like an odd location
for an ISP street-level office.

[0] [https://goo.gl/maps/8oxjkCL8JFm](https://goo.gl/maps/8oxjkCL8JFm)

~~~
closeparen
It seems odd to have an ISP office in the middle of downtown?

~~~
walrus01
Every major north american city has a legacy telco central office in downtown,
in a very central location, from the days of pulse-dial and then DTMF dial
analog phones. Always owned by whatever corporate entity the Bell System and
then ILEC eventually became.

Seattle, for example, also has the Elliot CO in Belltown: [http://www.co-
buildings.com/wa/206/](http://www.co-buildings.com/wa/206/)

~~~
autocracy
Yeah, you have to remember that in pulse dial days a central office station
had a reach of roughly 3 miles. The longer you go, the more you're paying for
cable, repeaters, or just losing quality. 90 volt AC for ringing has a limited
range!

~~~
walrus01
People who've never seen it in person find it amazing that things like 1200
pair phone cables exist... Here's a photo of just a 100-pair:

[https://imgur.com/gallery/f1EGp](https://imgur.com/gallery/f1EGp)

on this page is a photo of a 1200-pair:
[http://cityinfrastructure.com/single.php?d=RuralOutsidePlant...](http://cityinfrastructure.com/single.php?d=RuralOutsidePlant&t=Rural%20Outside%20Plant)

~~~
foobar1962
How do they colour-code the wires to identify them?

"Ah there's your problem: you've patched the Fire Engine Red pair, it's meant
to be the Ferrari Red. Rookie error."

~~~
tmm
> _How do they colour-code the wires to identify them?_

It’s actually pretty simple. There are only 10 colors: blue, orange, green,
brown, slate, white, red, black, yellow, and violet. They’re grouped in
“binders” (using colored strings). You’re likely familiar with the first four
pairs from network cables (which omit the white/slate pair). After cylcling
through blue through slate paired with white through violet (25 pairs), the
wires are bundled with binders starting with blue/white string. That gets you
to 625 pairs (the first picture posted above is 600 or 625 pairs). After that,
the binder groups are bound in a similar fashion (typically if you’re going
beyond 625, the slate/violet binder is omitted to get a nice round 600 in the
first group).

Wikipedia has a good article:
[https://en.m.wikipedia.org/wiki/25-pair_color_code](https://en.m.wikipedia.org/wiki/25-pair_color_code)

100-pair cable is only about 3/4” diameter. I have a 24-line 1A2 telephone
that uses 75 pairs just to connect to the phone switch and two 100-pair cables
feeding a telephone display case in my living room.

It takes me about a half hour to punch down 100 pairs on a 66-block. Old
school telecom guys could probably do it in under 10 minutes.

~~~
scoggs
What about if you are color blind? Change professions?

~~~
walrus01
Yes. Same problem for fiber fusion splicers. Worked for an ISP that hired a
color blind person as a field tech, he didn't last long.

------
kodablah
One wonders if a CDN might advertise, "we promise once your data enters our
network at our edge locations outside of the US, it does not traverse any AT&T
networks while reaching your server inside the US". Same with cloud companies'
private networks across regions.

~~~
lev99
If your communication is encrypted it shouldn't matter if it passes AT&T
networks.

Either

A. Popular and well known encryption algorithms are not broken by the NSA, and
your communication is private.

B. Popular and well known encryption algorithms are broken by the NSA, but the
fact that it's broken is top secret and the state will not do any actions that
revel the secret. Your communications are not safe, and while what you
communicate might make you the target of an investigation (if you're an
appealing enough target), the communications will not be directly used against
you in court.

EDIT: There is a third option, that your communication is being stored until
the encryption algorithm is broken or computation reaches a point where brute
force is possible (quantum computers). Long term storage of encrypted
communication is only economically feasible for a small subset of all
encrypted communication, so it's only a concern for targeted individuals where
the communication will be relevant to the state decades from now.

~~~
kodablah
> If your communication is encrypted it shouldn't matter if it passes AT&T
> networks.

That assumes metadata is irrelevant. The destination, time of day, and volume
of the traffic all have value separately and especially so when together. The
destination can be masked if you control both sides and AT&T is a go between,
but timing issues are subject to analysis unless you are a large enough player
to give safety in numbers or you push noise across your pipes.

~~~
mirimir
It's not that hard to anonymize metadata.

Just decide how thoroughly it must be done, and do what it takes. Plus a
safety factor.

~~~
throwaway2048
It is extremely hard to anonymize metadata, thats part of what makes its
capture and analysis so insidious.

Please tell me how i anonymize the metadata of where my cellphone is located,
which the telco harvests from towers its connected to.

~~~
Spooky23
Put it in a bag, and don’t use it.

------
peterwwillis
This is focused on AT&T, but there are other major datacenters in the US that
are also major transit peers for foreign nations, with entire floors dedicated
to government equipment. AT&T is a convenient one-stop shop, but they will
probably need to use other service providers as well.

~~~
nickpsecurity
Prior leaks showed they were paying multiple telecoms huge amounts of money to
do exactly that.

[https://www.forbes.com/sites/robertlenzner/2013/09/23/attver...](https://www.forbes.com/sites/robertlenzner/2013/09/23/attverizonsprint-
are-paid-cash-by-nsa-for-your-private-communications/#19e05c4743cb)

------
pandasun
For those who can't access theintercept for a variety of reasons:
[https://archive.fo/c0vWe](https://archive.fo/c0vWe)

------
souterrain
Is 811 10th (NYCMNY54) even interesting from an Internet perspective any
longer? Yes, the building is part of AT&T’s network (AS7018), but as far as
I’m aware, no peering occurs here at all. I’d look for equivalent shady racks
or rooms at 60 Hudson, 111 8th and 25 Broadway.

Maybe the comfortable relationship between NSA and at&t are more the driver
for the location, and the fiber taps are all backhauled here.

From a voice perspective, though, I’d think this was still a useful
surveillance point, given AT&T likely still tandems traffic here.

~~~
walrus01
Things are slowly moving out of 111 8th as Google bought the whole building
and is not renewing leases for certain tenants. Most traffic exchange happens
at places like 60 Hudson for Manhattan and then a whole bunch of newer,
modern, low and wide purpose built datacenters (mostly 1998 and later dotcom
1.0 boom and onwards era) in NJ across the river.

------
bonyt
Not the first time:

Room 641A:
[https://en.m.wikipedia.org/wiki/Room_641A](https://en.m.wikipedia.org/wiki/Room_641A)

~~~
XalvinX
12 years later and nothing changed. The SCOTUS killed the lawsuit and Congress
let everyone involved off the hook.

Since there are rarely, if ever, any consequences for breaking the law or the
Constitution, why would they stop?

------
helios893
[https://www.calyxinstitute.org/projects/encrypted-
internet-p...](https://www.calyxinstitute.org/projects/encrypted-internet-
peering-project)

------
yuhong
My personal favorite is how the current debt based economy allow almost
infinite amounts of government debt to be created. In fact, one of the reasons
why we got off the gold standard decades ago is military spending.

~~~
pm90
No its not. And I fail to understand why this keeps coming up. The gold
standard tied fiscal and monetary policy to an arbitrary commodity: gold.
Whereas value exists beyond the commodity itself. The gold standard had all
sorts of unintended consequences and is not fit for a modern economy.

What is perhaps more worrying is common currency. The Euro has lead to a lot
more trouble than it was worth, Frankly speaking.

~~~
yuhong
I am talking about why we got off the gold standard though.

~~~
XalvinX
Yup, at least in part to fund our military operations in S.E. Asia (Viet Nam,
etc.)

------
Firerouge
I wonder what sort of performance impact this has on American internet speeds.

At the very least, AT&T might be limited from upgrading their equipment until
the NSA can revise their monitoring systems to keep up.

~~~
IAmGraydon
They’re using optical splitters, so the original signal passes though
untouched. Because there’s no man in the middle, bandwidth shouldn’t be
affected.

~~~
Firerouge
That makes sense, but at the same time, I wonder if ISPs are hesitant or
inhibited from upgrading key pieces of infrastructure equipment out of
obligation to maintain these copying capabilities.

------
bhouston
This building in downtown Ottawa looks similarly imposing:
[https://goo.gl/maps/6xEgv4KwJ4v](https://goo.gl/maps/6xEgv4KwJ4v)

~~~
walrus01
99% odds that is either a ILEC telco central office or a large electrical
transformer substation. If electrical grid, similar to the big part with no
windows at 970 Burrard in Vancouver. BC Hydro headquarters site with giant
substation that later became office and residential condos.

edit: I just moved down the street a bit and it says Bell Canada on the
building. It's a CO.

Another way you can tell for sure with a building like that, is that there
will be a locked panel or set of highly protected hose ports for generator
diesel fuel refill, from alley or street side tanker truck delivery.

------
secabeen
It's interesting to see this, and the reference to "one million emails",
considering the prevalence of opportunistic TLS on MTA connections. Gmail
reports 89% of their inbound and outbound flow is protected by TLS [1].
Wouldn't that eliminate the ability of the NSA to intercept those messages?

[1] [https://transparencyreport.google.com/safer-
email/overview](https://transparencyreport.google.com/safer-email/overview)

~~~
detaro
They might still collect them, on the chance that they'll be able to break the
encryption later

~~~
cryoshon
or, on the chance that they were able to undermine the establishment of the
protocol to make it more amenable being broken using their methods.

------
chiph
The article describes a lot of the buildings a fortress-like. I actually don't
have a problem with that - in the event of a catastrophe, I want my
telecommunications to keep working.

The rest of it - splitting data traffic and sending a copy to the NSA - I'd
like to see how any international traffic is being sent through a land-locked
city like Dallas, which should only have domestic traffic in it. Which the NSA
shouldn't be looking at.

~~~
ilikepi
> I'd like to see how any international traffic is being sent through a land-
> locked city like Dallas...

Well, Texas borders Mexico, and although DFW is on the other side of the
state, it is a very large metro area. If it's a hub for domestic traffic, it
seems logical for it to also act as an interchange for international traffic
with Central and South America. The hostnames I see in a traceroute to
telmex.com (a big telco headquartered in Mexico City) from my office in New
England on Verizon FiOS appears to support the idea that at least some traffic
routes through Dallas before it crosses the border.

EDIT: clarification

------
bonestamp2
It could just be part of regular routing, but I noticed for example that all
my facebook traffic is routed to an address with an LA prefix in LA. That
doesn't seem strange except that Facebook doesn't have an LA datacenter (at
least not that I could find any public record of, and they seem to publish the
location of other data centers).

~~~
puzzle
There's a difference between a data center and a POP. The former is very
large. The latter is usually a rack or more in someone else's facilities. When
you try to reach Google or Facebook, you rarely hit their data centers
directly. They're out in the boonies, where land and power are cheap. Rather,
you talk to their proxies at the edge (POPs, often in large cities). From
there, your data is either returned immediately (e.g. cached objects such as
popular videos or the company logo...) or forwarded over their fiber to the
core clusters in the data centers.

~~~
oldgeezr
Just curious, is there a good site/book/etc to learn how the modern internet
actually works? As a lowly programmer, I have a good understanding of network
communications, and some knowledge of things like routing protocols, but I'm
completely lost when it comes to understanding how the modern internet
actually functions. Thanks!

~~~
puzzle
I'd normally recommend books like Google's SRE one, but at least in this case
it glosses over the detail of where GFEs tend to live:

[https://landing.google.com/sre/book/chapters/production-
envi...](https://landing.google.com/sre/book/chapters/production-
environment.html#fig_production-environment_life-of-a-request)

It used to be the case that they were mostly in POPs, but I think that with
Maglev
([https://research.google.com/pubs/pub44824.html](https://research.google.com/pubs/pub44824.html))
they can live in core clusters, too. Other Google sources go into more detail,
e.g.

[https://medium.com/@duhroach/profiling-gcps-load-
balancers-9...](https://medium.com/@duhroach/profiling-gcps-load-
balancers-94c552f06736)

[https://www.slideshare.net/MichelleHolley1/google-cloud-
netw...](https://www.slideshare.net/MichelleHolley1/google-cloud-networking-
deep-dive)

Back to your question, I'm not sure there is one good place to look up these
things, but presentations/papers by companies like Google and Facebook are
probably still your best bet. Stuff coming straight out of GCP teams will be a
little more enthusiastic in tone, but that's easy to tune out. :-)

Another good example is Facebook's Ben Maurer and his Fail at Scale talk,
which discusses a lot of details that are necessary for modern internet
services, such as queuing, session/application-layer congestion control,
canarying, advanced monitoring, etc.
[https://queue.acm.org/detail.cfm?id=2839461](https://queue.acm.org/detail.cfm?id=2839461)

------
jakenberg
There's an interesting 1960's era art installation by Anthony Heinsbergen on a
side wall of the LA facility that seems oddly related to this...
[https://i1.wp.com/upinthevalley.org/wp-
content/uploads/2015/...](https://i1.wp.com/upinthevalley.org/wp-
content/uploads/2015/12/IMG_2471.jpg?w=1024)

~~~
always_good
I don't understand "oddly" here. The art piece seems rather literal rather
than some odd coincidental association.

------
foobarbazetc
Lol, I walk past the one on Folsom all the time. The most obvious spy building
ever.

~~~
akira2501
Well, yea. I'd be surprised[1] if it wasn't on the list.

[1]:
[https://en.wikipedia.org/wiki/Room_641A](https://en.wikipedia.org/wiki/Room_641A)

~~~
foobarbazetc
Heh, see I had no idea what Room 641 was.

------
XalvinX
Anyone remember this from 2006?
[https://en.wikipedia.org/wiki/Room_641A](https://en.wikipedia.org/wiki/Room_641A)

I guess they just expanded, but the basic facts here are not new.

"the capability to enable surveillance and analysis of internet content on a
massive scale, including both overseas and purely domestic traffic."

AND

Klein claims he was told that _similar black rooms are operated at other
facilities around the country._

------
dooglius
The scrolljacking on this article is absolutely insufferable.

~~~
dopamean
It happens a lot on The Intercept. It's really super annoying.

------
XalvinX
I feel so safe and protected. NSA should give out branded teddy bears to kids.
(edit: added>> Oh, wait, the apparently already have them!!
[http://scienceblogs.de/klausis-krypto-
kolumne/files/2016/06/...](http://scienceblogs.de/klausis-krypto-
kolumne/files/2016/06/NSA-Gift-Teddy-bar-590x404.png))

Awwww....ain't that sweet??

------
reaperducer
I'm struggling to find the new information here.

We've known for years through the New York Times and others that AT&T helps
the NSA.

We've known for close to a hundred years about AT&T Long Lines networks and
hubs for that network.

It's only basic logic to put the two together and know that the NSA uses
AT&T's hubs.

What's new here?

~~~
txcwpalpha
Did you even read the article? The new information is that these _specific_ 8
buildings are specifically noted within NSA documents as the 8 locations
within AT&T's network that the NSA utilizes.

I don't know what The Intercept expects anyone to do with that information,
but that is new information.

~~~
reaperducer
I admit, I didn't read the full article because it's only barely readable on
my device, but if my scroll bar is accurate, I did read about 70% of it.

But my point stands.

We've known that these specific buildings are the key hubs in the network for
close to a century. And that they're hardened against nuclear attack, etc...

Maybe I was too deep into the phreaking scene in the early days, but I thought
this was common knowledge in technology circles.

~~~
txcwpalpha
You're still missing the point.

> We've known that these specific buildings are the key hubs in the network
> for close to a century. And that they're hardened against nuclear attack,
> etc...

Yes, it's been known that these specific buildings were key to AT&T's
infrastructure. But any speculation that these specific buildings (as opposed
to other specific buildings) were also key to NSA projects was just an
assumption. The new information, which comes from released NSA memos and
documents, shows that these specific 8 buildings are key to the NSA, meaning
it's not just based on assumption anymore.

There's some other new information in there from the memos/documents, too. You
really should actually read the article before mounting your high horse and
spouting off nonsense criticism about it.

------
azinman2
I’d love to see the intercept publish the equivalent for China, Russia, etc.
It feels the press has very overindexed into the NSA.

~~~
pjc50
It's not a secret in the West, or even in China, that there's a "Great
Firewall" which surveils and regulates all internet access.

It's just America that pretends it doesn't have a secret police.

~~~
JumpCrisscross
> _a secret police_

The term "secret police refers to intelligence, security or police agencies
that engage in covert operations against a government's political opponents"
[1]. We have no evidence the NSA is "used to protect the political power of an
individual" or even political party. They're an intelligence agency, purely
and simply.

[1]
[https://en.wikipedia.org/wiki/Secret_police](https://en.wikipedia.org/wiki/Secret_police)

~~~
boomboomsubban
That spies on members of Congress on the orders of the President...
[https://theintercept.com/2015/12/30/spying-on-congress-
and-i...](https://theintercept.com/2015/12/30/spying-on-congress-and-israel-
nsa-cheerleaders-discover-value-of-privacy-only-when-their-own-is-violated/)

~~~
tehwebguy
One of the few groups that _should_ be spied on

~~~
boomboomsubban
No, their activities should largely be public. The NSA having privlaged
information on their actions is dangerous, they have no incentive to share
them with the public unless it benefits the NSA.

~~~
tehwebguy
> No, their activities should largely be public.

I can agree on that

~~~
boomboomsubban
As important as that is, even if they're completely corrupt you still don't
want a secret police watching them.

------
rsa4096
First, I must immediately question the informativeness of anyone who thinks
Washington DC is located in Maryland. Second, pointing out addresses of major
AT&T PoPs is useless. Do carriers (still) mirror traffic to intelligence
agencies? Most likely. However, this makes it sound like AT&T and the NSA have
dedicated entire buildings for this purpose, which is ridiculous. Complete
sensationalism.

~~~
ggggtez
This first thing you said isn't in the article at all, you just misread it.

The second thing also indicates a lack of reading comprehension, thus
downvoted.

~~~
rsa4096
Please, explain to me how I misread the map. It clearly labels Washington DC
as a city in the state of Maryland.

~~~
pwinnski
You've misread the map by assuming "Maryland" was a state label, when evidence
within the same map clearly indicates it is not. "Northern California" and
"Southern California" are not states. Instead, they are areas, and Washington
D.C. is in the area of Maryland.

~~~
rsa4096
This is ridiculous, but I'll play along. California absolutely is a state.
Further defining the location of a city in a state does not really explain how
Washington DC ends up being classified as part of Maryland. Washington DC is
not part of the state Maryland. There is also no area (I assume by "area" you
mean metro area or region) that is commonly defined as Maryland. Washington DC
is part of the DC metropolitan area or "DMV," however it is in no way part of
Maryland anymore than New York city is part of New Jersey.

I was simply pointing out a minor (albeit, comical) factual error that
immediately made me question the legitimacy of the rest of the article.

~~~
nickdurfe
Perhaps you shouldn't discount the entire content of the article based on what
could be perceived as a minor mistake. There's quite a bit of corroborating
material in regards to the core theme of the Intercept piece (AT&T/NSA
collaboration); for example, the engineer who's quoted in the Intercept
article has been speaking out about NSA surveillance for several years. He's
been referenced in similar articles in various publications going back to at
leat 2007. The San Francisco address in the article was Mark Klein's former
workplace and he ostensibly witnessed fiber splitting equipment being
installed for use by government agencies. So if we are to take him at his
word, then his account certainly lends credence to the Intercept article.

~~~
rsa4096
Absolutely. I actually feel like pointing out the minor mistake distracted
from my actual point. As others have pointed out in the past few hours, this
article seems to do little more than point out where the NSA is fiber tapping
on AT&T's network. This hardly seems like breaking news to me. The exposure of
PRISM many years ago put the activity in the public eye.

To me, it seems like this article is sensationalizing a practice most were
aware of already. I suppose it is mildly interesting to highlight a bunch of
locations where it may be happening, but certainly not breaking news.

