
Obama Campaign AWS Infrastructure In One Diagram - sethbannon
http://awsofa.info
======
minimax
Harper Reed and Scott VanDenPlas did a talk / Q&A at the Chicago AWS meetup
group a few months ago. My hazy recollection of the salient points of the
talk:

1) This represents a heterogenous set of services where the engineers in
charge of each service were more-or-less free to use whatever technology stack
they were most comfortable with.

2) Use puppet for provisioning everything. (There is some subtlety Scott went
into about being able to bring up a bunch of servers quickly which I have
since forgotten).

3) Everything shuts down the day after the election. Not much time spent
future-proofing the code when it has a well known expiration date.

4) Testing testing testing. System had serious problems and 2008 and they
didn't want to repeat in 2012.

5) Engineers and sysadmins had a lot of latitude to make changes in order to
respond to issues as they developed. It sounded like this is essentially
Harper Reed's management style (he seems like a cool guy to work for).

~~~
ROFISH
Kinda odd saying that it has an expiration date. Aren't the same systems
required for the next election?

~~~
zaptheimpaler
I was at a talk by Rayid Ghani, the chief scientist for the Obama campaign. He
said that they shut down everything after the campaign simply because they
can't afford to keep on all the staff full-time for the 4 years, but they
don't throw away the code.

~~~
charlieok
With the billions of dollars pumped into election campaigns, it's odd that a
political party wouldn't keep at least a small team on the code to continually
test, update, refactor, and generally improve the quality.

Fred Brooks, nine women / one month, etc.

~~~
ZachPruckowski
The problem is that political parties are fueled by donations, and it's hard
to get people to donate for something like that. Not to mention you'd be
competing with local/state/House/Senate elections with actual candidates for
that fundraising money. I've donated politically in the past and even having
only given small amounts, I still get emails daily and calls monthly asking me
to donate to about half a dozen groups (many of which I've never donated to).
Heck, wounded firefighters and cancer-striken kids have a low conversion rate
for their fundraising - "money for technical systems to help a candidate to be
named later" is even worse.

~~~
saraid216
This will seem a bit OT, but it is relevant:
[http://www.ted.com/talks/dan_pallotta_the_way_we_think_about...](http://www.ted.com/talks/dan_pallotta_the_way_we_think_about_charity_is_dead_wrong.html)

I'm bothered that we as a society have trouble recognizing the point and
purpose of overhead.

------
jared314
Previous Discussion: <https://news.ycombinator.com/item?id=5542368>

The diagram follows the style of the AWS Reference Architectures[0]. I have
yet to find a published set of diagram components that compare well to that
style.

Edit: Juan Domenech[1] has a small custom made component pallet in png form.
And, it looks like the original diagrams are custom Adobe Illustrator
drawings[2].

[0] <http://aws.amazon.com/architecture/>

[1] [http://blog.domenech.org/2012/06/aws-diagrams-
palette-v10.ht...](http://blog.domenech.org/2012/06/aws-diagrams-
palette-v10.html)

[2] <http://blog.domenech.org/2012/05/aws-diagrams.html>

~~~
edwinnathaniel
Kind of wish GAE to have some sort of whitepaper, reference architecture, best
practices/patterns as well.

Oh well, I guess GAE is for toy project _ahem_.

------
jwilliams
Wish this had a legend. Obviously the box sizes and positioning mean
something.

What I can see makes me a bit curious.

\- Frontend Web boxes are "big", where the (some) of the backend App boxes are
small. I'd have though it would be the reverse -- frontend horizontally scaled
first, backend vertically scaled first.

\- Sometimes they use Zones ABC, sometimes AB. This is cool, but a lot of the
"front" infrastructure is AB (including www), so not sure what advantage
having ABC on backend pieces are. These are obviously super-critical for some
special reason? I guess they also might be pieces that are-not/only-partially
replicated to the secondary site.

\- The failover US-West site is using Asgard along with Puppet, but the
primary US-East one isn't. I guess this is for managing failure scenarios?

\- Prodigious use of a lot of AWS services. Including internal ELBs all over
the place. SQL Server and PostgreSQL sneak in a few places.

\- Couldn't find the CDN!

\- Looks like staging and testing are a complete replica (hard to tell, the
resolution isn't quite there). They're big though. This is fine, but raises
the question why the secondary site is just a part replica? If you provision
the lot to stage, you'd figure you'd run the secondary the same way?

\- The Data Warehouse runs on the secondary site, but it only accessed from
the primary. Interesting. Wonder why they just didn't put it on the primary?

~~~
pfg
I remember reading somewhere that they were using Akamai as their CDN provider
(at least for their donation page).

~~~
bonertruck
Ask Scott why they ended up going with Akamai and how they, "reached out" to
the campaign. Funny story.

------
mratzloff
The most surprising thing about this is that test and staging are 1:1
duplicates of production. At the companies I've worked at dealing in scale,
that has _never_ been the case.

"Oh, you run 10 servers and 2 load balancers across these 2 sub-systems?
Here's 2 servers and a toy whistle for staging."

But then again, I suppose that's what you get when you can dump hundreds of
millions of dollars on a problem over a couple years.

~~~
peteretep
With Puppet, and with AWS, presumably it's possible to flip a switch, and 5
minutes later have a replica which is around for the 30m you want to test it
for, at 1/250th of the monthly cost

------
quellhorst
This would be much easier to read if the entire thing wasn't slanted.

------
parasubvert
Really jazzed by CloudOpt, which I didn't know about. I had a multi-region
architecture supporting AWS offshore development teams and HA between US-East
& US-West, and this would have been a godsend. We had Windows Servers as dev
workstations in Singapore and devs were based in India and Canada -- latency
was actually fine to the desktop, and we used Amazon's internet connection to
haul data back to US-East before hitting our data centre. This was
satisfactory for most connectivity (throughput was fine) but occasionally the
encapsulated routing would cause packet retry pains that traditional WAN
acceleration may have helped with a bunch.

But, overall, this is great work, and shows what's possible to those in the
enterprise that think that Amazon is still a bookseller, and clouds are
basically things in the sky you don't want on your golf days. This will only
help further the progression to way less wasted cost & time in IT. (Unless
you're a republican, I suppose. Or maybe especially if you're a republican,
you should be studying this for 2016 and considering an alternative to hiring
Very Serious IT Consultants.)

~~~
miles932
Cloudopt rocked, but it's being renamed Union Bay Networks or something.
<http://www.linkedin.com/company/union-bay-networks>

------
stevewilhelm
I would really be interested in know a) if I had a copy of the Puppet scripts
how long would it take get this server configuration up and running and b) how
much would the AWS costs be per month?

~~~
scottvdp
A) That's a bit nuanced and depends on the application you are referring to
(there are ~200 represented here), but if you had the scripts and the repos
and built from a vanilla AMI, a box would boot and configure within a few
minutes. We shortened the boot times greatly by using puppet to configure a
template node which we then used to create a generic versioned AMI per
autoscale environment and deployed with Asgard. If you had the AMIs, an EBS
backed instance would boot and serve traffic in anywhere between 15 and 30
seconds or so.

B) For costs, you can check the FEC filings... I cannot recall what they were
month to month, but it will be accurately reflected quarterly there, so you
can get a rough estimate. It was like a kabillion times more in October 2012
than it was in June 2011. The important part for us is that the costs scaled
with our demand.

~~~
csarva
Here are the numbers I pulled:

AMAZON $58,525.33 18-Oct-12

AMAZON WEB SERVICES $144,955.12 5-Nov-12

AMAZON WEB SERVICES $150,000.00 18-Dec-12 AMAZON WEB SERVICES $150,000.00
18-Dec-12 AMAZON WEB SERVICES $47,887.28 18-Dec-12 AMAZON WEB SERVICES $135.75
18-Dec-12

------
lifeformed
Man, I didn't realize big sites were this complex under the hood. That site
wasn't even complicated; if I recall correctly it was just some info pages and
payment. I don't know anything about backend stuff. All my projects are like
this:

[traffic] -> [hosted server] -> [Amazon CloudFront files]

~~~
thrownaway2424
I believe this includes all of their internal tools for campaigning and
organizing and suchlike, not just the public website.

~~~
jellicle
They're not internal; that's the point.

You want a citizen to be able to receive emails and give you money, then move
smoothly to calling other citizens through the website, being provided with a
walk sheet to go door-to-door in her own neighborhood, eventually leading a
small team of other volunteers.... all of these require interaction with the
database of other supporters. Ideally the campaign has over a hundred million
people all contacting each other and urging each other to vote a certain way.

------
trotsky
That just drives home how much simpler things have become with no SLA IaaS.
Instead of spending 20+ hours racking an east coast/west coast pair of small
cages or perhaps just 2 full racks each, you can save time and money by going
with simple and glitch proof scale out systems driven by one button
provisioning of hundreds of low performing vms. As an added bonus if your san
or ethernet collapses you can safely sleep through the pager - somebody else
will undoubtedly post the problem on their well monitored support forum.

Let's not forget the capEx advantage - It's not like the Obama organization
can afford heavy five or low six figure expenditures on infrastructure when
who even knows if they'll be able to raise a series A or will even be around
in a year.

~~~
bonertruck
The campaign isn't a tech startup. Its a people startup that uses tech to
facilitate the things that people do.

~~~
bonertruck
Perhaps that was a bit overly broad and not quite representative of what I
meant to convey. The problems faced by a campaign and the problems faced by a
tech start up are inclusive but not all encompassing. Every dollar you save on
your server infrastructure is a cell phone plan for a lonely field office in
Texas struggling to make a dent in a hopeless battle but they're there because
they care. Are you gonna tell them they can't have phones?

~~~
saraid216
> The problems faced by a campaign and the problems faced by a tech start up
> are inclusive but not all encompassing.

I thought this was pretty obvious.

> Every dollar you save on your server infrastructure is a cell phone plan for
> a lonely field office in Texas struggling to make a dent in a hopeless
> battle but they're there because they care.

I have no idea what you're trying to convey here.

~~~
bonertruck
There is an existing fundraising infrastructure, an existing polling
infrastructure, an existing media market infrastructure, all of these are
highly successful and competitive and get candidates elected. When you're
introducing sprawling new web based architectures into this mix you've got to
prove the efficacy of every minute decision and improvement. There is very
little trust.

~~~
saraid216
So... you're saying that a tech startup does not have to contend with existing
infrastructures? Do I have that correct?

------
dahumpty
Its interesting to note that the the boweb-production servers of "www" don't
have read-replicas (asynchronous replication) for read scalability. Anyone
know why? Typically, one would use read-replicas to alleviate read workloads
on the master.

~~~
ceejayoz
Most likely the www servers are serving almost entirely static, cached
content. Hell, they might not actually see any actual public traffic - they
might just serve as CDN origins.

~~~
scottvdp
This is 100% correct. We also generated static snapshots of the origin and
mirrored to S3. In the instance of something going awry with our primary
origin servers, we could fail back to S3 as an origin.

EDIT: I'll add that our origin offload for WWW hovered around 98%.

------
DigitalSea
Who would have thought the infrastructure behind Obama's site and whatever
else they're running was so big. I always assumed it would be running maybe
one or two instances, not require a whole freakin' diagram to explain.

~~~
volkadav
As pointed out down-thread, there's a lot more to this than a simple website
(voter analytics/GOTV stuff just for starters), and the potential user
base/traffic load is enormous (ramping up particularly as you near election
day).

Just as a data point in support of the kind of traffic that US presidential
elections can drive for election-related things, I happened to be working on
the web team for a major American newspaper circa the '08 election and we saw
something like 1.2Bn page views the month leading up the election. (Traffic
analytics wasn't my team, so that's a vague memory of being told that from
five years ago. Grain of salt, etc.) This is for simple news/op-ed, mind you,
and only one source of many for that information. I have no problems believing
that the main web presences of the two candidates would see even more traffic,
and they'd likely be more complicated under the hood than "hit homepage,
follow article link, read article."

------
georgecalm
It's interesting that apart from the DynamoDB and the SQL Warehouse
(Redshaft?) there's a pretty large LevelDB instance serving the
sites/services.

I also wonder if in reality there were only one or two instances in each zone
for each site behind the ELBs and the others were added to show how scalable
the infrastructure is; or if their number actually did correspond to the avg.
number of instances they ended up having to spin up.

~~~
skyebook
My guess is that they wouldn't have used Redshift as it only became available
for beta a few weeks before the election.

~~~
spudlyo
They used Vertica, which they didn't seem to be too pleased about. I think
they would have liked to use Redshift it was available.

------
gulfie
Where did this lovely map come from and how can I generate my own?

~~~
Ogre
I tried to figure out what software this was from, but I gave up. I did find
that Wikipedia has a nice list of network diagram software here
[http://en.wikipedia.org/wiki/Comparison_of_network_diagram_s...](http://en.wikipedia.org/wiki/Comparison_of_network_diagram_software)

I'd love to describe a network in (something like) graphviz dot notation and
get a diagram that looks like this out the other end. Usually I wind up just
using Visio and doing it by hand because work pays for it anyway, and it's
actually very nice.

~~~
miles932
It's good 'ol Inkscape :)

~~~
Ogre
Thanks for the answer! I was just playing with Inkscape the other day, I'll
make a point of learning it for real next time I need to put something like
this together.

------
WestCoastJustin
Here is a chat from AWS Invent BDT 207: Big Data and the US Presidential
Campaign [1]. Really interesting. They cover the diagram too.

[1] <http://www.youtube.com/watch?v=X1tJAT7ioEg>

~~~
miles932
That's me at the podium!

~~~
leozh
haha me too!

~~~
harper
me too

~~~
miles932
no, you're down front with the goofy glasses. answering most of the questions.
being smart. i'm the one futzing with the mouse and asking awkward questions
like a wank.

------
patrickg
Serious question: does the 3d mode add any information? I am distracted by it
but perhaps the size of the blocks have any attached meaning? Is there a 2d
variant available?

Otherwise: very interesting to see such a large infrastructure used for the
campaign. I wonder how it compares to an average campaign here in Germany. I'd
assume only a 10th of the servers (while spending the same amount of money :))
used here. I can't see anything special on the web before elections other than
some wordpress alike web servers.

------
pm24601
It will be interesting to see how Organizing for Action ( OFA ), the follow-on
group works out. So far they are raising money in decent chunks (
[http://www.latimes.com/news/politics/la-pn-organizing-for-
ac...](http://www.latimes.com/news/politics/la-pn-organizing-for-action-
fundraising-20130412,0,1428014.story) )

------
scribu
Did anyone else notice the blue narwhals in the Staging and Testing areas?
What's up with that?

Edit: D'oh: <http://en.wikipedia.org/wiki/Project_Narwhal> (disappointed it's
not an easter egg)

~~~
bonertruck
It is in a way. The department heads were very worried about the scrutiny and
tone of the press in covering project narwhal and actively discouraged its use
as a codename for the project.

------
TallboyOne
Mother of god...

------
Kiro
Is this complexity really necessary?

~~~
pm24601
Ask Mitt Romney that question

~~~
bonertruck
Yes and no. This diagram lays out how to do this kind of thing cheaply and
scale the systems up and down quickly. It allows the campaign to bring these
systems and services "in house" instead of contracting 3rd parties to manage
adding boxes at great expense at the last minute based on best guesses of
expected traffic. It allows you to respond to the spikes in traffic that are
expected during a campaign and during promotions that would otherwise bring
your site down or have you paying for unused bandwidth.

------
daenz
Is this not rendering for anyone else? Latest ubuntu + chrome. All I get is
black.

~~~
madelfio
Same for me, along with several other sites.

Ubuntu 12.10 + chrome 26.0.1410.63

------
white_devil
They misspelled "Wall Street".

------
sc00ter
Your tax dollars at works.

~~~
cmelbye
Please tell me that you weren't educated in the United States. I can't
understand how else someone could misunderstand such a fundamental concept of
our election process.

~~~
sc00ter
Ah, yes, because somehow wasting all that money on an election campaign is
okay so long as it is voluntary taxation. Then I remember that this is the
same group of intellectuals that voted up "Star Trek-like computer with voice
interface" (a 15 line mashup that is neither a computer, nor Star Trek-like)
to the top of HN. God bless America.

~~~
Samuel_Michon
Also the same group that created Y Combinator, Hacker News, and the Internet.
If you don't like it, you can see yourself out.

~~~
twic
We created the Internet?

~~~
Samuel_Michon
Al Gore is American :)

