
Why Amazon's data centers are in spy country (2016) - kposehn
https://www.theatlantic.com/technology/archive/2016/01/amazon-web-services-data-center/423147/?utm_source=atlfb&amp;single_page=true
======
gtrubetskoy
There is no mystery here. The "spy country" link is more of a coincidence;
after all, the Internet was a DARPA project, and Northern Virginia is where a
lot of defense and government stuff is.

When DARPA Internet was turned over to the public, one of the largest
interconnect points was at 8100 Boone Blvd (Tysons Corner), known as MAE East
[1]. Once upon a time I had access to it, as did countless other people
running small ISPs around the area and renting rack space in that building.
But the building (it's actually three buildings, and the MAE gigabit switch
[2] was in the basement across the street) was not really suitable for any
kind of a data center, though above.net did manage to convert the whole level
of a parking garage into one, albeit with very low ceilings.

Pretty soon everyone figured out that by Dulles airport rent is way cheaper,
you can build from scratch, there is plenty of electricity and water and
connectivity is cheap, and AOL was out that way too.

That's pretty much the story. Amazon barely existed then. There used to be a
nice bookstore nearby called "Computer Literacy Bookshop", which eventually
went all online and later got acquired by Amazon or B&N, can't remember...

[1] [https://en.wikipedia.org/wiki/MAE-
East](https://en.wikipedia.org/wiki/MAE-East) [2]
[http://www.fairfaxunderground.com/forum/file.php?2,file=9012...](http://www.fairfaxunderground.com/forum/file.php?2,file=9012,filename=mae-
east-in-the-parking-garage.jpg)

~~~
walrus01
Pretty much the same reason why there are a ton of carrier exchange points in
certain geographic areas of the SF Bay area. Or why the Westin Building in
Seattle and One Wilshire in Los Angeles have become the defacto carrier
exchange points/peering locations. Large numbers of ISPs began accretion in
certain buildings in the mid to late 1990s and the process has continued
through its own inexorable inertia.

~~~
greglindahl
IIRC when the Internet actually formed there were 7 places that you needed to
be at in order to have settlement-free peering. Looking at various not-so-
organized Wikipedia articles it appears that I am not recalling the earliest
days. I think all of these still exist; MAE-WEST is still at 55 South Market
in San Jose, for example.

At one point blekko moved into a brand new datacenter in Santa Clara which had
only limited fiber connectivity. @ChuckMcM twisted their arm for a free
interconnect to MAE-WEST so that we could move over our existing inexpensive
transit and interconnect contracts. Everyone's in MAE-WEST.

~~~
jlgaddis
I was just a pre/early teen when most of this was happening but I can remember
reading about it and being amazed, after becoming interested after reading
_The Cuckoo's Egg_.

Today, I'm a network engineer at an ISP. Go figure.

> _MAE-WEST is still at 55 South Market in San Jose, for example._

Yep, and fiber runs from there to 611 Folsom and into Room 641A.

(Edit: It probably wasn't quite _that_ early in the Internet's life but it was
still very young.)

------
rdtsc
People like to say it is s mini Silicon Valley here. A lot tech companies have
some presence here Oracle, MS, Google, Verisign, AWS, etc. Obviously this is
also government contractor land. Then there there are those buildings without
any signs but with 3 layers of fences, and every other person seems to work
for the "State Department" (cough cough CIA).

But I think another reason companies like it here is because of tax structure.
I hear it is more advantageous for them than being in DC or Maryland.

Other fun thing, apparently if you dig around Tysons Corner you have a high
chance of randomly hitting 3-letter agencies' fiber. And then black SUVs come
and talk to you in a surprisingly short amount time:
[http://www.washingtonpost.com/wp-
dyn/content/article/2009/05...](http://www.washingtonpost.com/wp-
dyn/content/article/2009/05/30/AR2009053002114.html)

~~~
openasocket
> every other person seems to work for the "State Department"

I actually had a professor who specializes in algebraic number theory quit and
move to DC, saying only that he was going to "work in Washington." I guess you
can't really be subtle, there's not that many jobs out there in algebraic
number theory.

~~~
ksenzee
I used to live near Fort Meade, and all our math PhD and linguist neighbors
said they worked for the "Department of Defense."

~~~
cperciva
I've heard that the official advice given to NSA employees is "If asked, say
that you work for the DoD but aren't allowed to be more specific; do not lie
or try to evade the question, since that will tend to attract even more
attention."

~~~
jonwachob91
The DoD is the parent agency to the NSA, so it is the truth.

~~~
type0
Yeah, but that's a bit like saying "I'm from planet Earth" when someone asks
you where you're from

------
ryanmarsh
If you want to know where all the data centers are in N. Virginia just fly
into Dulles and look out the window. The approach into Dulles is usually long
and low from the west and the data centers are the low large buildings with
big generators and lots of chillers outside.

There's a fuck ton of them.

~~~
walrus01
Aerial photos and satellite photos are also an easy way to spot 1MW+ sized
diesel generators and heat exchangers. I have found a few wall street trading
firms' unadvertised disaster recovery sites that way, when combined with other
on the ground publicly available info.

------
oculusthrift
I'm actually quite suspicious about everything Amazon is doing out there. I
was recently contacted about a job at Amazon that they advertised would gain
me a government security clearance. The main problem being that I'd have to
move to Virginia.

~~~
dsl
AWS maintains an entire region named GovCloud explicitly to isolate sensitive
workloads and meet compliance requirements. The "security clearance" is
basically just a background check to make sure you aren't a spy.

~~~
rdtsc
Not necessarily. I am guessing GovCloud doesn't require a security clearance.
Just have to be a US Person. That means a citizen or have a green card. (At
least for ITAR purposes that is good enough).

If they ask for a security clearance or are willing to sponsor it, it probably
has to do with that $600M private AWS instance for the CIA.

But yes if you have a clearance here you can pretty much walk out of the
building if you don't like your job and walk into the next one and they'll
give you one. (That's the joke anyway).

~~~
whorleater
> That means a citizen or have a green card

No, anything remotely close to the US government requires a full citizenship.
I got slammed by that when I was in the middle of applications for scientific
software companies which did government contracting on the side.

~~~
rdtsc
Depends on what you do. Often managers don't even know so they just use
"citizen" as rule to be safe.

Also worked for the military and we had a variety of citizens, non-citizens,
green card holders and non-green card holders. We had to go through silly
gymnastics on who can talk about what and work on what depending on their
status.

------
foob_newman
I don't think it has much to do with "spy country". The area just happened to
be one of the first to build a co-location/interconnect site where different
providers could join their networks. Then, as the article states, "Networks
build atop networks" and it grew from there.

~~~
ckozlowski
This is a good book to read on it.
[https://www.amazon.com/dp/B00LG92LYM/ref=dp-kindle-
redirect?...](https://www.amazon.com/dp/B00LG92LYM/ref=dp-kindle-
redirect?_encoding=UTF8&btkr=1)

It's a bit of both. Network providers were encouraged to build in the area to
support government requirements lead by defense and intelligence. But that
same infrastructure supports commercial. When Equinix built their exchange in
the D.C. area to support those customers, it snowballed from there.

It also helps that the D.C. area has cheaper electricity than many parts of
the country. And a major transmission line serving the east coast runs west of
D.C. Fiber + cheap electricity adds up.

It's more than just Amazon that's out here. Azure and Rackspace have
datacenters out here, as well as a large smattering of colos and smaller
providers.

~~~
foob_newman
I have had that book on my Amazon wishlist since Jan 2016, so I guess it's
time to get it!

------
lostboys67
Its not surprising that anyone running data centres doesn't advertise when I
worked for BT our data centre at staples corner had no external branding.

This was for security as DC and Exchanges are CNI targets - ironically the
PIRA did blow up the building next door.

------
bane
One of the thing this piece misses in describing the weird banality of
Northern Virginia is comparing it to someplace else. I'll compare it to San
Francisco. For reference for people not used to the D.C. area, Northern
Virginia (NoVA) is _about_ the same size as the San Francisco Bay
Area...without the Bay. The major city it's associated with is Washington D.C.
instead of San Francisco (two cities of _roughly_ the same size) and the
regions have roughly comparable household incomes. However, the GDP of the
Oakland-San Jose-San Francisco metro is about twice that of the D.C. metro.

Northern Virginia doesn't really have any incorporated large cities to speak
of but it's just at the southern end of the Northeast Megalopolis mega-region
[1] so it's quite urban in many areas. Notable "cities" might be Tysons
Corner, Alexandria (a city without a county), Arlington (a county with a
city), Reston, maybe a couple others. Large parts of the area are preserved as
National Monuments, National Parks and Battlefields from the Civil War.
However, most of it is very very suburb.

Despite the image of being , being near the seat of government it also has an
amazingly diverse population. The boring suburbs are also punctuated with
restaurants from almost every country you can guess, religious groups from JW
to Jainism and even local theaters that will play Bollywood to Korean Horror.

And I don't mean like a few representative examples, I mean like a dozen
competing large supermarkets that deal in goods for Asian customers from a
dozen countries, a half dozen for Latin and Hispanic, African restaurants from
around the continent, a reasonable Vietnamese-town like area, two (almost
three if the third one can ever get established) Korea towns, a couple quasi
Chinese areas, a dozen or so up-scale dining and shopping areas. One could
easily eat a meal from a different country for every meal of the day for
months and not exhaust the region.

The diversity is so intense and the population so large the area can support
very niche kinds of services, for example, you can find several Indian owned
coffee shops that focus on coffee and Indian desserts, or you can go eat
Chinese Hotpot, then hop out for a few hours of conversation at a local hookah
bar and finish your evening at a Korean spa, then wake up at 2 in the morning
and go get all night Salvadorean food. The next day you stop at a Korean
bakery for breakfast, work your hangover off on Ethiopian food for lunch
before diving into your choice of Persian, Afghan, Lebanese, or Kazakh style
kabobs for dinner. It gets kind of ridiculous and knowing how to eat with
chopsticks, what to do with injera, roti and how to eat Pho, Sam Gyup Sal and
that real Tacos aren't crunchy is basically expected among locals. Ndole,
Silpancho or Pupusas for lunch and Thai, American Chinese, Szechuan, Beijing
or Taiwanese for Dinner is not untypical. Locals feel weird in less diverse
environments and don't understand why they can't eat from the world when they
travel.

You get all the benefits of suburbia, but very few of the downsides (nightlife
could be better). The weather is the worst form of all the seasons: hot, humid
summers; cold wet winters; frequently overcast; with a few days of nice spring
and a few days of nice autumn with colorful leaves. Because of the huge
numbers of migrants, nobody knows how to drive in the winter, skiing sucks and
Atlantic water sports are boring.

Two major airports, rail links, several ports nearby, a thriving tech industry
(mostly serving the government) and a huge stable employer (the government)
mean that the economy is remarkably stable in the face of massive economic
up/down turns. It's hard to catch a wave up, but it's also hard to find
yourself without a job, especially if you work in tech. You can drive for 30
miles in one direction on one road and see almost back-to-back-to-back tech
companies. Traffic is easily #1 or #2 in the country with no predictable
patterns or ways to estimate transit time, so out of towners expecting back-
to-back meetings all over the area are very sad.

1 -
[https://en.wikipedia.org/wiki/Northeast_megalopolis](https://en.wikipedia.org/wiki/Northeast_megalopolis)

~~~
ckozlowski
This is an amazing post.

To give a sort of +1, I'll state that one never knows just how well they have
it until they experience elsewhere. I'm a Northern VA native, but I lived in
Germany for 3 years (returned last year). Germany is lovely, but came nowhere
close to the cosmopolitan area I was used to. Everything bane described is
true. The limited options for food was the most noticeable. If we were lucky,
we could find some esoteric Asian spices for our paneer from the Feinkost
downtown, but you'd pay 10EUR for a tiny glass jar. I missed the Lotte or
H-Mart where I could grab a bag of the same stuff for $2.

And the diversity prepares you as well. People in Northern VA, have to, as a
matter of circumstance, interact with people who may not speak the same
language. (Some don't like this, I, don't mind.) Jobs are filled by people of
all types. The services economy is hugely diverse, and you can quickly see all
of the different facets to the economy and the issues people discuss in a few
square miles.

Fairfax did toy with the idea of incorporating. Tyson's I'd agree is probably
the closest we have to a city outside of D.C. Fairfax around Rt 50 and Rt 29
is pretty dense.

I visited S.F. once, and I'm envious of the atmosphere there, with the
exception of the insanely high prices. (my perceptions of what constitutes
reasonable rent is already distorted by my growing up here) That S.F. still
keeps that vibe in the face of such prices makes me wonder how much it will
last. But D.C. is gentrifying quick, and there's old sections of the city that
are under threat of losing some of its diversity.

~~~
bane
> I missed the Lotte or H-Mart where I could grab a bag of the same stuff for
> $2.

It's really hard to explain to people not from the area how embedded the
ethnic diversity of the food scene in NoVA is and it really forms itself in
the consciousness of the locals. It really is its own thing. For example, when
I was in Portland, the locals talked up the local foodie scene quite a bit,
and the food really was excellent. But the diversity was totally different,
e.g. thousands of variations of a few kinds of foods -- like microbreweries.
The local competition in that limited menu-space made for excellent eating,
but the non-American food selection wasn't there. It wasn't until I was in
NoVA with some Portlanders that they "got it", we literally ate food for lunch
from a different country every day of the week for weeks before we started
ranging too far.

I think the only place I've been to that is similar in "feel" is NYC, but you
really have to hunt down the food in the various historic ethnic enclaves.

Here's an example [1]. Nowhere particularly interesting NoVA, a couple small
neighboring strip malls like anywhere, except there's: 3 Mexican places, a
Peruvian place, 2xVietnamese places, a Szechuan place, Indian food, Thai, and
an Indian grocer -- all mixed in with the usual assortment of fast-food,
Starbucks, autobody shops and other typical things. This isn't even a
particularly "ethnic" part of NoVA.

Here's another right up the road [2]: Pakistani, Turkish, Indian next to a
Dunkin' Donuts and a Subway.

Here's one strip mall [3]: Salvadorean, Guatamalan, American-Chinese, Russian,
Afghan, Vietnamese, Middle Eastern grocer, and a Middle Eastern restaurant
mixed in with a Panera, Italian deli, Popeyes, Subway and McDonalds.

These three places are probably within 15 minutes of each other.

When I'm in Europe, it's great eating the local food for a few days. But then
I find I start craving the diversity again and it's simply never there. Even
very cosmopolitan cities like London aren't quite there.

1 -
[https://www.google.com/maps/place/@38.9009113,-77.4508584,19...](https://www.google.com/maps/place/@38.9009113,-77.4508584,19z/data=!4m5!3m4!1s0x89b6415619677219:0x4041f0fc4316ad83!8m2!3d38.8942786!4d-77.4310992)

2 -
[https://www.google.com/maps/@38.9775979,-77.4099158,19z](https://www.google.com/maps/@38.9775979,-77.4099158,19z)

3 -
[https://www.google.com/maps/place/Granada+Restaurant/@38.968...](https://www.google.com/maps/place/Granada+Restaurant/@38.9688033,-77.385335,17z/data=!4m5!3m4!1s0x0:0x7b6eb65d0fc0d80!8m2!3d38.9687061!4d-77.3744747)

------
TazeTSchnitzel
An incalculable, but very significant, portion of all human knowledge is
locked away in secret, nondescript buildings.

I've always wonder what would happen if us-east-1 was somehow destroyed. How
much would be lost?

