Hacker News new | past | comments | ask | show | jobs | submit login
Why Amazon's data centers are in spy country (2016) (theatlantic.com)
157 points by kposehn on March 1, 2017 | hide | past | favorite | 56 comments

There is no mystery here. The "spy country" link is more of a coincidence; after all, the Internet was a DARPA project, and Northern Virginia is where a lot of defense and government stuff is.

When DARPA Internet was turned over to the public, one of the largest interconnect points was at 8100 Boone Blvd (Tysons Corner), known as MAE East [1]. Once upon a time I had access to it, as did countless other people running small ISPs around the area and renting rack space in that building. But the building (it's actually three buildings, and the MAE gigabit switch [2] was in the basement across the street) was not really suitable for any kind of a data center, though above.net did manage to convert the whole level of a parking garage into one, albeit with very low ceilings.

Pretty soon everyone figured out that by Dulles airport rent is way cheaper, you can build from scratch, there is plenty of electricity and water and connectivity is cheap, and AOL was out that way too.

That's pretty much the story. Amazon barely existed then. There used to be a nice bookstore nearby called "Computer Literacy Bookshop", which eventually went all online and later got acquired by Amazon or B&N, can't remember...

[1] https://en.wikipedia.org/wiki/MAE-East [2] http://www.fairfaxunderground.com/forum/file.php?2,file=9012...

Pretty much the same reason why there are a ton of carrier exchange points in certain geographic areas of the SF Bay area. Or why the Westin Building in Seattle and One Wilshire in Los Angeles have become the defacto carrier exchange points/peering locations. Large numbers of ISPs began accretion in certain buildings in the mid to late 1990s and the process has continued through its own inexorable inertia.

IIRC when the Internet actually formed there were 7 places that you needed to be at in order to have settlement-free peering. Looking at various not-so-organized Wikipedia articles it appears that I am not recalling the earliest days. I think all of these still exist; MAE-WEST is still at 55 South Market in San Jose, for example.

At one point blekko moved into a brand new datacenter in Santa Clara which had only limited fiber connectivity. @ChuckMcM twisted their arm for a free interconnect to MAE-WEST so that we could move over our existing inexpensive transit and interconnect contracts. Everyone's in MAE-WEST.

I was just a pre/early teen when most of this was happening but I can remember reading about it and being amazed, after becoming interested after reading _The Cuckoo's Egg_.

Today, I'm a network engineer at an ISP. Go figure.

> MAE-WEST is still at 55 South Market in San Jose, for example.

Yep, and fiber runs from there to 611 Folsom and into Room 641A.

(Edit: It probably wasn't quite that early in the Internet's life but it was still very young.)

One of the reasons Bletchly Park was chosen in WW2 was it was close to some of the core trunk network

AOL's first campus was actually in Tyson's Corner as well, but on the other side.

I can't remember exactly, but it was either at the end of Westwood Center Dr or Spring Hill Rd. I don't remember when they started moving out to Ashburn/Dulles, but I know for sure they still had 3 brick buildings in Tyson's in 1997.

Clickable image link (hotlinking is apparently disabled): http://imgur.com/a/JXjtD

People like to say it is s mini Silicon Valley here. A lot tech companies have some presence here Oracle, MS, Google, Verisign, AWS, etc. Obviously this is also government contractor land. Then there there are those buildings without any signs but with 3 layers of fences, and every other person seems to work for the "State Department" (cough cough CIA).

But I think another reason companies like it here is because of tax structure. I hear it is more advantageous for them than being in DC or Maryland.

Other fun thing, apparently if you dig around Tysons Corner you have a high chance of randomly hitting 3-letter agencies' fiber. And then black SUVs come and talk to you in a surprisingly short amount time: http://www.washingtonpost.com/wp-dyn/content/article/2009/05...

> every other person seems to work for the "State Department"

I actually had a professor who specializes in algebraic number theory quit and move to DC, saying only that he was going to "work in Washington." I guess you can't really be subtle, there's not that many jobs out there in algebraic number theory.

Old joke from the my University's math department.

Q: How do you apply to work for the NSA

A: You pick up the phone, call your mother, and ask for an application.

I remember hearing a slight variation...

Q: How do you get a job application for the NSA?

A: Pick up the phone and ask the dial tone.

I used to live near Fort Meade, and all our math PhD and linguist neighbors said they worked for the "Department of Defense."

I've heard that the official advice given to NSA employees is "If asked, say that you work for the DoD but aren't allowed to be more specific; do not lie or try to evade the question, since that will tend to attract even more attention."

The DoD is the parent agency to the NSA, so it is the truth.

Yeah, but that's a bit like saying "I'm from planet Earth" when someone asks you where you're from

Fort Meade? Not everyone living near there worked at Fort Meade!

Fort Meade is east, across the street, the Baltimore Washington Parkway, from Laurel, MD, and my wife and I lived there.

I worked at the Johns Hopkins University Applied Physics Lab (JHU/APL) in the group that did the calculations for the Navy's version of the USAF's GPS, well before the GPS. But my work was in the fast Fourier transform, power spectral estimation, etc. of passive sonar signals. Mostly I was writing PL/I code for the 360/91, 60 ns cycle time, 8 byte wide, 16 way interleaved main memory.

So, not everyone in that area worked at Fort Meade!

Well, since the NSA is actually in MD, you can be a bit more vague than that if you want to... It's technically not "In Washington" so you're not lying.

If you want to know where all the data centers are in N. Virginia just fly into Dulles and look out the window. The approach into Dulles is usually long and low from the west and the data centers are the low large buildings with big generators and lots of chillers outside.

There's a fuck ton of them.

Aerial photos and satellite photos are also an easy way to spot 1MW+ sized diesel generators and heat exchangers. I have found a few wall street trading firms' unadvertised disaster recovery sites that way, when combined with other on the ground publicly available info.

I grew up in Dulles (aka Sterling/Ashburn). Another way to go on a tour of data centers is to ride on your bike all along the W&OD trail [0] between Arlington and Leesburg.

Lots memories skateboarding around those places, then getting chased out.

[0] https://www.google.com/maps/place/Washington+and+Old+Dominio...

I'm actually quite suspicious about everything Amazon is doing out there. I was recently contacted about a job at Amazon that they advertised would gain me a government security clearance. The main problem being that I'd have to move to Virginia.

AWS maintains an entire region named GovCloud explicitly to isolate sensitive workloads and meet compliance requirements. The "security clearance" is basically just a background check to make sure you aren't a spy.

Not necessarily. I am guessing GovCloud doesn't require a security clearance. Just have to be a US Person. That means a citizen or have a green card. (At least for ITAR purposes that is good enough).

If they ask for a security clearance or are willing to sponsor it, it probably has to do with that $600M private AWS instance for the CIA.

But yes if you have a clearance here you can pretty much walk out of the building if you don't like your job and walk into the next one and they'll give you one. (That's the joke anyway).

"You can only obtain AWS GovCloud (US) accounts if you are an individual or entity that qualifies as a U.S. Person under applicable regulations. "


"§ 120.15 U.S. person. U.S. person means a person (as defined in §120.14 of this part) who is a lawful permanent resident as defined by 8 U.S.C. 1101(a)(20) or who is a protected individual as defined by 8 U.S.C. 1324b(a)(3). It also means any corporation, business association, partnership, society, trust, or any other entity, organization or group that is incorporated to do business in the United States. It also includes any governmental (federal, state or local) entity. It does not include any foreign person as defined in §120.16 of this part. "


> That means a citizen or have a green card

No, anything remotely close to the US government requires a full citizenship. I got slammed by that when I was in the middle of applications for scientific software companies which did government contracting on the side.

Depends on what you do. Often managers don't even know so they just use "citizen" as rule to be safe.

Also worked for the military and we had a variety of citizens, non-citizens, green card holders and non-green card holders. We had to go through silly gymnastics on who can talk about what and work on what depending on their status.

Working on GovCloud requires ITAR which is citizen or "permanent resident" (green card). There are other positions that require a security clearance, but GovCloud does not. GovCloud is open to non-governmental customers who need to meet stricter requirements (it's not solely for government use).

> GovCloud is open to non-governmental customers

Some excellent naming there ;)

It's still pretty apt, the reason the non-government customers are in GovCloud is almost always because their customers are in the government. For example, if you want to sell a service to US government departments, you'll find it much easier to do so if you have a presence in GovCloud yourself.

You got that wrong. It depends if you need to be a citizen or not.

I've personally known permanent residents to work in government contracting.

For your case, if I had to guess, it's probably the physical building/facility they required you to work at had restrictions on unescorted non-citizens.

I just assumed if they wanted him to move to VA is was likely for the pending build out of GovCloud east.

Edit: I found a map with the C2S region on it, and it does appear to be in the VA/MD area.

Yeah but he wouldn't need to get clearance to work for GovCloud.

The AWS "Region" which the super secret stuff runs in isn't plain GovCloud. It's Amazon C2S. It's not talked about much (at all). From my understanding, most people who work on this cloud do not need a clearance but do need heavy background checks.

Some people operate directly with the intelligence agencies (and work on their campuses as part of their duties for AWS). These people often handle sensitive data and therefore need clearance. Amazon often hires individuals who are located near intelligence services to ease this.

No, security clearance relates to this contract that AWS has:https://www.theatlantic.com/technology/archive/2014/07/the-d...

The GovCloud AWS region is located in the Northwestern USA, so it seems more likely that if he was doing GovCloud specific work, they'd have hired him in Seattle.

Though I'm sure AWS does lots of work with US government agencies in us-east-1, so it's not surprising that a government clearance might be required for some of it.

GovCloud is indeed in the Pacific Northwest, according to the 2011 announcement from Amazon itself. Specifically, it's in Boardman, Oregon, on a road named after famous explorers. It was their first structure in the area, confirmed by aerial photography and contemporary reporting. Smaller modular backups were active on a different site in Boardman right next to the interstate, and outside Umatilla near the prison. These other sites eventually evolved into the rest of us-east-2.

As for us-east-1, it all began in Chantilly in an existing low-rise building you could mistake for rentable office space, then expanded to several different sites between Sterling, Ashburn, and the airport itself, and perhaps surprisingly, to another place where famous battles were fought. Finally, they have a site a bit further away, near where Disney was going to build a theme park. That's where they want to expand more, but first they have to convince the power company to get them a dedicated line, instead of being fed from the same place as the battlefield.

Some sources:




Product teams support their services in all regions, meaning if a team is based in Virginia it still supports all the other regions simultaneously.

Why is that suspicious? There is an entire website just for jobs that require clearance. www.clearancejobs.com

There is plenty of governmental work out there that mandates certain levels of clearance. Even big consulting firms like Booz Allen or Accenture will require it for certain hires.

The presence of huge datacenter facilities in NoVA near Dulles greatly predates Amazon AWS/S3 or any significant Amazon presence online, except for its online store. Huge amounts of construction occurred prior to 2002. The fact that there's tons of dark fiber and it's easily trenched in green field type applications there is why it has continued. It reached a certain tipping point that a dozen square km of NoVA became the defactor carrier interconnect point for anything on the US east coast (outside of the NYC/NJ area).

But there seems to be a recent surge in data center construction in the area, with something like 7 (?) new DCs being built right now.

Yeah, because if you buy circuits into the right places in the same neighborhood you can reach a huge chunk of the internet via settlement free peering. Plus IX costs of course. And IP transit is epically cheap in the Dulles area, sub $0.10/Mbps.

Yeah I live off of Loudoun parkway and there are three or four of them going up now and 4 on my street built in the last two years.

I don't think it has much to do with "spy country". The area just happened to be one of the first to build a co-location/interconnect site where different providers could join their networks. Then, as the article states, "Networks build atop networks" and it grew from there.

This is a good book to read on it. https://www.amazon.com/dp/B00LG92LYM/ref=dp-kindle-redirect?...

It's a bit of both. Network providers were encouraged to build in the area to support government requirements lead by defense and intelligence. But that same infrastructure supports commercial. When Equinix built their exchange in the D.C. area to support those customers, it snowballed from there.

It also helps that the D.C. area has cheaper electricity than many parts of the country. And a major transmission line serving the east coast runs west of D.C. Fiber + cheap electricity adds up.

It's more than just Amazon that's out here. Azure and Rackspace have datacenters out here, as well as a large smattering of colos and smaller providers.

I have had that book on my Amazon wishlist since Jan 2016, so I guess it's time to get it!

Its not surprising that anyone running data centres doesn't advertise when I worked for BT our data centre at staples corner had no external branding.

This was for security as DC and Exchanges are CNI targets - ironically the PIRA did blow up the building next door.

One of the thing this piece misses in describing the weird banality of Northern Virginia is comparing it to someplace else. I'll compare it to San Francisco. For reference for people not used to the D.C. area, Northern Virginia (NoVA) is about the same size as the San Francisco Bay Area...without the Bay. The major city it's associated with is Washington D.C. instead of San Francisco (two cities of roughly the same size) and the regions have roughly comparable household incomes. However, the GDP of the Oakland-San Jose-San Francisco metro is about twice that of the D.C. metro.

Northern Virginia doesn't really have any incorporated large cities to speak of but it's just at the southern end of the Northeast Megalopolis mega-region [1] so it's quite urban in many areas. Notable "cities" might be Tysons Corner, Alexandria (a city without a county), Arlington (a county with a city), Reston, maybe a couple others. Large parts of the area are preserved as National Monuments, National Parks and Battlefields from the Civil War. However, most of it is very very suburb.

Despite the image of being , being near the seat of government it also has an amazingly diverse population. The boring suburbs are also punctuated with restaurants from almost every country you can guess, religious groups from JW to Jainism and even local theaters that will play Bollywood to Korean Horror.

And I don't mean like a few representative examples, I mean like a dozen competing large supermarkets that deal in goods for Asian customers from a dozen countries, a half dozen for Latin and Hispanic, African restaurants from around the continent, a reasonable Vietnamese-town like area, two (almost three if the third one can ever get established) Korea towns, a couple quasi Chinese areas, a dozen or so up-scale dining and shopping areas. One could easily eat a meal from a different country for every meal of the day for months and not exhaust the region.

The diversity is so intense and the population so large the area can support very niche kinds of services, for example, you can find several Indian owned coffee shops that focus on coffee and Indian desserts, or you can go eat Chinese Hotpot, then hop out for a few hours of conversation at a local hookah bar and finish your evening at a Korean spa, then wake up at 2 in the morning and go get all night Salvadorean food. The next day you stop at a Korean bakery for breakfast, work your hangover off on Ethiopian food for lunch before diving into your choice of Persian, Afghan, Lebanese, or Kazakh style kabobs for dinner. It gets kind of ridiculous and knowing how to eat with chopsticks, what to do with injera, roti and how to eat Pho, Sam Gyup Sal and that real Tacos aren't crunchy is basically expected among locals. Ndole, Silpancho or Pupusas for lunch and Thai, American Chinese, Szechuan, Beijing or Taiwanese for Dinner is not untypical. Locals feel weird in less diverse environments and don't understand why they can't eat from the world when they travel.

You get all the benefits of suburbia, but very few of the downsides (nightlife could be better). The weather is the worst form of all the seasons: hot, humid summers; cold wet winters; frequently overcast; with a few days of nice spring and a few days of nice autumn with colorful leaves. Because of the huge numbers of migrants, nobody knows how to drive in the winter, skiing sucks and Atlantic water sports are boring.

Two major airports, rail links, several ports nearby, a thriving tech industry (mostly serving the government) and a huge stable employer (the government) mean that the economy is remarkably stable in the face of massive economic up/down turns. It's hard to catch a wave up, but it's also hard to find yourself without a job, especially if you work in tech. You can drive for 30 miles in one direction on one road and see almost back-to-back-to-back tech companies. Traffic is easily #1 or #2 in the country with no predictable patterns or ways to estimate transit time, so out of towners expecting back-to-back meetings all over the area are very sad.

1 - https://en.wikipedia.org/wiki/Northeast_megalopolis

This is an amazing post.

To give a sort of +1, I'll state that one never knows just how well they have it until they experience elsewhere. I'm a Northern VA native, but I lived in Germany for 3 years (returned last year). Germany is lovely, but came nowhere close to the cosmopolitan area I was used to. Everything bane described is true. The limited options for food was the most noticeable. If we were lucky, we could find some esoteric Asian spices for our paneer from the Feinkost downtown, but you'd pay 10EUR for a tiny glass jar. I missed the Lotte or H-Mart where I could grab a bag of the same stuff for $2.

And the diversity prepares you as well. People in Northern VA, have to, as a matter of circumstance, interact with people who may not speak the same language. (Some don't like this, I, don't mind.) Jobs are filled by people of all types. The services economy is hugely diverse, and you can quickly see all of the different facets to the economy and the issues people discuss in a few square miles.

Fairfax did toy with the idea of incorporating. Tyson's I'd agree is probably the closest we have to a city outside of D.C. Fairfax around Rt 50 and Rt 29 is pretty dense.

I visited S.F. once, and I'm envious of the atmosphere there, with the exception of the insanely high prices. (my perceptions of what constitutes reasonable rent is already distorted by my growing up here) That S.F. still keeps that vibe in the face of such prices makes me wonder how much it will last. But D.C. is gentrifying quick, and there's old sections of the city that are under threat of losing some of its diversity.

> I missed the Lotte or H-Mart where I could grab a bag of the same stuff for $2.

It's really hard to explain to people not from the area how embedded the ethnic diversity of the food scene in NoVA is and it really forms itself in the consciousness of the locals. It really is its own thing. For example, when I was in Portland, the locals talked up the local foodie scene quite a bit, and the food really was excellent. But the diversity was totally different, e.g. thousands of variations of a few kinds of foods -- like microbreweries. The local competition in that limited menu-space made for excellent eating, but the non-American food selection wasn't there. It wasn't until I was in NoVA with some Portlanders that they "got it", we literally ate food for lunch from a different country every day of the week for weeks before we started ranging too far.

I think the only place I've been to that is similar in "feel" is NYC, but you really have to hunt down the food in the various historic ethnic enclaves.

Here's an example [1]. Nowhere particularly interesting NoVA, a couple small neighboring strip malls like anywhere, except there's: 3 Mexican places, a Peruvian place, 2xVietnamese places, a Szechuan place, Indian food, Thai, and an Indian grocer -- all mixed in with the usual assortment of fast-food, Starbucks, autobody shops and other typical things. This isn't even a particularly "ethnic" part of NoVA.

Here's another right up the road [2]: Pakistani, Turkish, Indian next to a Dunkin' Donuts and a Subway.

Here's one strip mall [3]: Salvadorean, Guatamalan, American-Chinese, Russian, Afghan, Vietnamese, Middle Eastern grocer, and a Middle Eastern restaurant mixed in with a Panera, Italian deli, Popeyes, Subway and McDonalds.

These three places are probably within 15 minutes of each other.

When I'm in Europe, it's great eating the local food for a few days. But then I find I start craving the diversity again and it's simply never there. Even very cosmopolitan cities like London aren't quite there.

1 - https://www.google.com/maps/place/@38.9009113,-77.4508584,19...

2 - https://www.google.com/maps/@38.9775979,-77.4099158,19z

3 - https://www.google.com/maps/place/Granada+Restaurant/@38.968...

I live in "down town" Reston, calling it a "city" is a stretch. To be honest, this diverse metropolitan environment that you are describing sounds familiar but it does not describe what is outside of my window. Is it possible that you lived closer to DC and are just assuming everything is the same all the way out to IAD?

Not to be terse, but you should probably get out of the town center sometime: here's a list of ethnic food within 10 minutes of RTC:

Bartaco (Latin) Barcelona Reston (Spanish) Our Mom Eugenia (Greek) Neyla Mediterranean Bistro (Mediterranean) Crisp and Juicy (Peruvian) Bento House (Japanese) Ariake (Japanese) Cafesano (Mediterranean) Singh Thai (Thai) Mon Ami Gabi (French) Enatye (Ethiopian) Lakeside Asia Cafe (Chinese) Cafe Montmartre (French/Vietnamese) Charcoal Kabob (Afghan) Tipicos Gloria (Salvadorian) Nomad Grill (Mediterranean) Balaji (Indian) Pupusas Express (Latin) Taste of the World (Indian/That/Filipino) Euro Bistro (German) Pho 75 (Vietnamese) Granada (Middle Eastern) Thai Luang (Thai) Russia House (Russian) Busara (Thai) Saigon Pho (Vietnamese) Turcuisine (Turkish)

and so on....add 10 more minutes and you literally triple the number of places....add in regional Chinese (Cantonese, Szechuan), end up in Sterling and you open up the Middle Eastern places, head South East into Annadale and you're in the 3rd largest Koreatown in the U.S...or go to Centreville and you're in the 4th largest.

I didn't even put the more quasi ethnic places like Italian or Bagel shops.

I guess my concept of a thriving urban environment is more expansive than "ethnic food within 10 minutes [driving]." For starters, how about a museum? We can skip talking about anything resembling an artistic/creative neighborhood because of the real estate prices.

I think it is a stretch to say Bagels are "quasi ethnic" but more importantly the mealy doughnut shaped loaves of bread around here are not "bagels."

Well, urban doesn't necessarily mean "Manhattan". But I get your point. Bizarrely, I'd argue that D.C. has probably less diversity than the 'burbs in many ways. For example, there's not a single Korean restaurant in the entire city. Heck, AFAIK I've never even seen an old fashioned deli in D.C.

NoVA also suffers from the worst habits of mid-20th century civic planning, designed heavily around cars. All of the urban areas in NoVA are urban automobile areas. It wasn't until recently that dense, urban, mixed-use classic city-type areas were even being built outside of D.C. and Arlington.

>how about a museum? http://www.restonmuseum.org/

okay okay, it's about Reston's history, but Reston is smack in the middle of tons of historic sites, many of which have small museums. If you want the big museum experience there won't be one until you get to Baltimore because the entire museum "scene" (such as it is) is swallowed entirely up by the Smithsonian, the National Galleries and the smaller museums all along and near the mall. I'd probably challenge the notion that urban cores necessarily have lots of museums though. Even Manhattan doesn't have as many as D.C.

If you want art, you're more likely to get into the local gallery scene. NoVA has quite a few fine art galleries.

Tyler Cowen (yes, the economist) even runs a blog where he talks about various hole-in-the-wall Asian restaurants in the DC area (particularly NoVa where I suspect he lives).


Yeah I just got back from staying with the inlaws for a couple of months and working remote and I was dying to have real food again. You could literally eat food from a different country every day for a month here without trying too hard.

An incalculable, but very significant, portion of all human knowledge is locked away in secret, nondescript buildings.

I've always wonder what would happen if us-east-1 was somehow destroyed. How much would be lost?

Applications are open for YC Summer 2021

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact