
Inside Team Romney’s whale of an IT meltdown - shawndumas
http://arstechnica.com/information-technology/2012/11/inside-team-romneys-whale-of-an-it-meltdown/
======
smacktoward
_Orca had been conceived by two men—Romney's Director of Voter Contact Dan
Centinello and the campaign's Political Director Rich Beeson..._

 _To build Orca, the Romney campaign turned to Microsoft and an unnamed
application consulting firm._

This sounds like a _lot_ of failed IT projects in large corporations --
dreamed up by upper management types who don't know anything about tech, then
farmed out to consultants and tech vendors for the actual implementation. The
consultants and vendors then either proceed to wander in the weeds expensively
due to lack of direction, or actively seize on the client's ignorance and take
them for a ride.

The problem is that the people nominally in charge of the project, the upper
management types, don't have the experience or expertise to _know_ they're
getting taken for a ride until it's too late. So there's no way to hold the
contractors accountable, or to get the train back on the rails once it's
jumped off.

~~~
aaronbrethorst
And yet, despite this disaster,

    
    
        Mitt Romney’s campaign handed out more than $200,000
        in bonuses last month to senior staffers, according
        to new disclosure records filed Thursday.
    
        Richard Beeson, Romney’s national political director,
        received a $37,500 payment on Aug. 31 in addition to his
        salary, according to records filed with the Federal
        Election Commission.
    

[http://www.washingtonpost.com/blogs/election-2012/wp/2012/09...](http://www.washingtonpost.com/blogs/election-2012/wp/2012/09/20/romney-
campaign-gives-bonuses-to-top-staff/)

When I first heard about this, I remarked to a friend "what the hell are they
doing giving out bonuses? The bonus should be that your candidate gets
elected!"

~~~
lifeguard
Wow, I thought the bonuses he gave out at the convention for winning the
nomination made sense. I wonder what last month's were for, "winning" the
debates?

~~~
antidoh
Maybe to keep the rodents inside the vessel until the sailing date.

------
PakG1
When I worked for the Winter Olympics in Vancouver, it was pounded into our
heads all the time that there was no second chance. Everything we did would
get one chance only.

For that reason, we had two Technical Rehearsals. The purpose was to test both
our systems and our procedures, and identify how they could be refined for the
real thing. It was someone's job to actually design a huge number of expected
and unexpected issues so that tech operations teams at each venue could be
tested as to their response. Issues ranged from broken printers to network
meltdowns to broadcasting failures, etc. We had two Technical Rehearsals for
hopefully obvious reasons. And before the Technical Rehearsals, we also had
the test events (i.e. world class but lower profile sporting competitions held
at the Olympic venues to see whether we fell short in putting up a world class
competition event).

We also had Disaster Rehearsals to test our disaster recovery systems and
procedures, though I wasn't personally involved in those. I did get a string
of text messages when it was happening though. "Primary datacentre flooding,
starting emergency processes to switch to secondary datacentre now." etc.
Never mind that nobody could ever believe that the primary datacentre could
get flooded in a million years. The attitude was that any disaster was
possible, and we had to prepare for the impossible.

If you have one shot only, and that shot is important, you need to have a
large process committed to making sure that one shot goes well. And that
process needs to be managed by someone experienced in that type of thing.

~~~
unreal37
Ironic that Mitt Romney ran the Salt Lake City Olympic committee in light of
this.

Also, the Ars article seemed to imply they wanted to keep how it worked a
secret, so big testing like this would have tipped Obama what they were doing.

------
meepmorp
Zero training? A single web server? No end users even touching the system till
it's time for mission critical use? No frigging redirect of http->https? And
it looks like, as has happened more than once in my professional life, people
outside of the management structure saw possible risks and were blown off by
those in charge.

Even if you supported Romney, this has to give you some pause about the man's
management skills and who he hired to run things.

~~~
acslater00
It would give me more pause if Mitt Romney were involved personally in this
sort of a project.

~~~
jlgreco
If there is _anything_ a president's job _isn't_ , it is to micromanage.

~~~
anigbrowl
That excuse covers just about anything.

There's a big difference between micromanaging and verifying that your
critical infrastructure is in place when you launch.

~~~
jlgreco
I was unclear. I am saying that since it is not the job of a president to
micromanage, it is legitimate to criticise him for fuckups he was not directly
involved with.

If his job were micromanaging, then _"Hey, he wasn't in the room"_ would
excuse him of responsibility, but that is not the case. Since his job would be
pretty much the greatest opposite of micromanaging I can conceive of, there
can be no excuse. The failure of his campaign is his failure.

That still seems a bit unclear to me actually. Not sure how to effectively
reword this.

~~~
chris_wot
Let me try:

The United States has the following federal executive departments:

* Agriculture

* Commerce

* Defense

* Education

* Energy

* Health and Human Services

* Homeland Security

* Housing and Urban Development

* Interior

* Justice

* Labor

* State Transportation

* Treasury

* Veterans Affairs

Now there is literally no way you can micromanage you way through this.
Nobody, and I mean _nobody_ can possibly single handedly do it.

However, you can delegate. Part of that is to choose honest brokers and
managers who know their area well enough to ensure that their departments work
well.

In this situation it is _your job_ to delegate to the right people. If you are
the President of the United States of America and you aren't able to do this
with your own campaign to get elected, you must take responsibilty for your
failure and accept a great deal of responsibilty for the fact you did not get
elected to high office.

Harsh, but being the POTA is not a job everyone can do. I know I couldn't.

~~~
crag
But he wasn't president. Romney was managing a campaign. And a function
critical to his campaign - you gonna tell me he wasn't involved? Or at least
briefed?

I mean, wasn't he expecting to see the results?

~~~
jlgreco
It seems this is a particularly difficult topic to talk about. We all seem to
agree with each other.

------
lkrubner
No one should allow a business or non-profit to face a situation like this.
Only bad managers put something so important into such difficult
circumstances. Why go live the day of the election? Are they insane?

Unless the contract (to build this application) had some kind of clause like
"This thing must work perfectly or all the money will be refunded" then this
is a great contract to get. Let us consider this cynically for a moment. I
wish I could land contracts like this one. Can you imagine building an app
that the client will only need for 1 day, and you still get paid even if the
thing does not work? The chance for fraud is huge. Who would even allow this?
The people pitching this must have been some very talented salespeople
(perhaps they were convincingly able to argue that they were such huge
Republican supporters that if they lost then that would be punishment enough,
as it would be so very sad for them to see Obama win?)

I've worked with a lot of startups and clients and the staff always needs
training and you never, ever know what all the bugs are until you've gotten
some feedback from the staff who are using it for real. To suggest that a
software project is done the day you first give it to the staff is idiotic --
that's simply the moment you move to real world testing.

Some developers might suggest that a really good suite of unit tests might
provide enough insurance, but that is not true -- I've worked with non-
technical staff who tend to confuse bad UI decisions with software bugs, and
if a UI decision is bad enough, I think it is fair to classify it as a bug. If
some_important_action is crucial to the staff, and none of the staff can find
the link to some_important_action, then the UI decision that hid it is
effectively as bad as a Fatal Exception, but no amount of unit tests can tell
you that the staff finds the UI incomprehensible.

~~~
incision
>I wish I could land contracts like this one. Can you imagine building an app
that the client will only need for 1 day, and you still get paid even if the
thing does not work? The chance for fraud is huge. Who would even allow this?
The people pitching this must have been some very talented salespeople...

Anecdotally, this is exactly how large consultancies, particularly those who
specialize in serving the public sector, operate.

1) Deploy an army of "talented" salespeople to establish favorable executive-
level relationships by exploiting every available loophole to provide trips,
gifts, dinners and sporting events. Once the top has been won over, everyone
below will fall in line.

2) Write contracts meeting the barest minimum interpretation of the
requirements. Design them to specifically exploit uninformed procurement
departments which are better equipped to source millions in copier paper or
bolts than complex software.

3) Farm out any actual deliverable to the cheapest sub available nationwide.
Ideally, you will have hidden travel and per diem for the talent in #2 leaving
you free to fly in folks from the cheapest parts of the country.

4) Upon encountering any serious issue once work is underway simply scapegoat
the current PM and shuffle in a new one.

5) Go to #1 employing the latest project as a reference that likely won't even
be properly checked. In the event that diligence is performed, feed them to
one of your friendly executives.

~~~
TheCondor
Was this a talented sales pitch or just your standard politics and an old boys
club connection? No mention of the consulting company.. If it was my campaign,
I'd hang them out to dry.

Seems the IT staff was out of their depth but there were some very serious and
very fundamental issues if the users were confused as it sounds.

I say it as a joke but a barbed one: this guy thinks he was fit to manage a
nation when he couldn't knock out a web app for 30000 users!?! Really?

~~~
javert
Fortunatey, it's not the president's job to "manage a nation."

As a side note, an almost do-nothing president would be the best kind. (And,
as a side note to the side note, that was what was intended when the office
was created.)

------
SkyMarshal
_> "We asked if our laptops needed to be WiFi capable," Dittuobo told Ars.
"Dan Centinello went into how the Garden had just finished expansion of its
wireless network and that yes, WiFi was required. __I was concerned about
hacking, jamming the signal,_ * etc...*

This brings up a whole nother can of worms - electoral cyber warfare. What if
the Dems, or some sympathizer group, say some local Anonymous, decided to
disrup the GOP's centralized GOTV operation?

They could point LOIC at the IP receiving all the real-time field reports, set
up hidden wireless jammers around Boston Garden, and any number of other hacks
and exploits to disrupt this centralized system.

I use the GOP and Anonymous as an example here, but it could work in any
direction. It's one thing for a campaign to lose due to its own incompetence
and negligence, but quite another to lose due to outside interference and
being the target of cyberwarefare.

~~~
gamble
Politico's article on ORCA today claims that one of the major problems was
that they deployed the system in their election night HQ just prior to the
election, and the surge of legitimate traffic on election day triggered their
DDOS defences. They'd never tested it in that location with realistic traffic
levels.

~~~
rhizome
Deployed just prior: poor project management; late delivery.

DDOS defense: crashing

realistic traffic levels: more than the 3 ppl on the coding team

------
runn1ng
How effective these things actually are in US?

Why am I asking... I am in from Czech Republic, where I can't imagine anyone
knocking door over door and asking people to vote. Actually, this behavior is
so connected here to Jehova's Witnesses (yeah) that it would have only the
negative effect.

Similarly, I _can't imagine_ any candidate seriously considering calling
random people and persuading them to vote. Again, this behaviour is so
connected to annoying salesperson (most often mobile operators) that it would
hurt the candidate.

And I don't think our democracy suffers because we don't have that in here.

~~~
fernly
Our land-line telephone received at least six calls each day for two months,
from political organizations representing candidates at the federal, state,
and local levels. We never answer that phone, only let the answering machine
take the call and return it in the very rare event it is someone "real". But
we did not get calls on election day because we live in a state (CA) that both
parties assumed (correctly) would give a majority to the President.

"Get out the vote" operations (GOTV) are real and sometimes effective in
states where the outcome is expected to be close. In this election there were
a small number of these "swing" states where both parties thought they had a
chance to win.

In such states, if the early indications are that the voting is close, the
party organization can redouble its efforts to contact known supporters
(contributors, or merely people who are registered to that party in the public
records, or people who have signed a petition or gotten on the party mailing
list somehow) and persuade them to vote.

(This is because in the USA, unlike some countries, it is not mandatory to
vote, and rarely do even half the eligible voters do so. So if you can
persuade a person who was not planning to vote, to do so, that is a net gain
of 1 vote.)

The purpose of this "Orca" system was to collate the reports from volunteers
watching the polling places and identify precincts where a few extra votes
might make a difference.

You can see how serious the effort was, in that the Republicans had mustered
37,000 volunteers willing to do this sort of thankless work: to stand around
polling places, politely asking people how they had voted and phone in the
counts; or, sit in an office and make telephone calls to people who do not
want to be called and politely urge them to go and vote. That is a large
effort, and apparently wasted by poor planning.

~~~
brazzy
There are very few countries where voting is mandatory, perhaps none. And
still The USA has lower voter turnout than most, probably more because of the
separate registration step than anything else.

But the biggest difference is the per-state first pas the post mechanism,
which is what creates the phenomenon of "swing states" where relatively few
voter decide everything, and almost any expense of effort by campaigns is
worthwhile.

~~~
waqf
Voting is mandatory in Australia:
[https://en.wikipedia.org/wiki/Electoral_system_of_Australia#...](https://en.wikipedia.org/wiki/Electoral_system_of_Australia#Compulsory_voting)

------
brudgers
From Slate back in July: an interesting background piece about the campaign's
approach to staffing for data analysis.

[http://www.slate.com/articles/news_and_politics/victory_lab/...](http://www.slate.com/articles/news_and_politics/victory_lab/2012/07/the_romney_campaign_s_data_strategy_they_re_outsourcing_.single.html)

What the debacle brings to mind is the consequences of putting "idea guys" in
charge of software development when there is a hard deadline.

 _"Orca had been conceived by two men—Romney's Director of Voter Contact Dan
Centinello and the campaign's Political Director Rich Beeson."_

I imagine them saying, "We shall have an app," while thinking, "How hard can
it be?" While the developers were rationalizing, "Well, a web app is
technically an app...and a lot easier to complete than an iPhone app, an
Android app, and the web app we will still need to build for the old people
with a Windows desktop."

~~~
eli
A Web App is actually the right call. It was for volunteers doing GOTV in the
field, not as much for people sitting at a desktop. And you would want to be
able to issue updates quickly and carefully control access -- things that are
much more difficult with an App Store app.

~~~
lifeguard
Using Windows was a mistake, that's for sure.

Good thing for Democrats they didn't know about the open source ushahidi
platform.

~~~
WrkInProgress
How did you determine using Windows was a mistake from the information
presented in the article ?

~~~
njharman
"Using Windows was a mistake" is a truism.

Also, from the article, project failed. Failure is regularly equated with
having made mistakes.

~~~
smsm42
In the same vein, one could claim use of humans to write code was a mistake.
After all, the project failed, this means they made a mistake, so obviously
they should have used trained hamsters instead. Do you see where your logic
failed you?

------
WrkInProgress
Some key points -

1.) 11 database servers. 1 app server 2.) Not redirecting http to https 3.)
not stress tested or apparently tested in any way 4.) users got their first
taste of the system the morning of the election.

~~~
mrdodge
Sounds like Microsoft was trying to sell a few SQL server licenses.

------
dblock
The assumption that software can work from the first try is proven wrong, time
and again. Software is something that needs to grow, mature and stabilize. The
Romney campaign had the thing that's usually hardest to get - the human beta
test resources. It should have been dry run for months.

------
lifeguard
The campaign could not manage launching a web app and they wanted to run the
entire USA !?!?!?!

~~~
atomical
Yes, people who make mistakes can still run the entire USA. It has been done
before.

~~~
lifeguard
This kind of false equivocation is dishonest and intellectually lazy.

~~~
Xcelerate
I disagree. His point is very valid.

~~~
lifeguard
His use of "mistakes" is a verbal fallacy called equivocation. Not all
mistakes are the same.

wikipedia: Equivocation ("to call by the same name") is classified as both a
formal and informal logical fallacy. It is the misleading use of a term with
more than one meaning or sense

------
pkulak
We're all lucky that an organization too inept to manage a campaign is
ultimately not allowed to manage the country.

------
nhangen
Coming from a launch that had its own share of issues, I can say that things
like these happen for a variety of reasons:

1\. Developers overestimate their capabilities 2\. Developers underestimate
timelines

3\. Client overestimates developer throughput 4\. Client underestimates man
hours required for any given feature.

5\. Someone expects someone else to do QA. 6\. Project manager assigns
everything an ASAP priority

And this is just to start.

------
Bud
Perhaps Republicans should have spent more time actually displaying the
business and organizational acumen and discipline that they claimed as an
advantage over the Obama team, and less time suppressing the black vote.

Snark postscriptum: Also, next time, GOP, you might wanna hire some IT guys
who make more than $12 an hour.

~~~
acdha
You don't even want to think how much more than $12/hour they paid. Big
consulting companies are usually $200+ for fresh grads whose best work happens
on PowerPoint slides

~~~
dscrd
Perhaps he meant the guys who did the actual work.

------
mmonoceros
Another misstep of the Romney camp seems relevant, since a lot of the
discussion so far is about project management.

In September they admitted to wholesale copying of a key Obama fundraising
page, with Zac Moffat blaming the issue on "junior staff confusion":

[http://www.buzzfeed.com/zekejmiller/romney-campaign-
appears-...](http://www.buzzfeed.com/zekejmiller/romney-campaign-appears-to-
copy-text-from-obama-we)

The copy was lifted in its entirely. The similarity in the design is more
subjective. (They were so similar that a lot of the discussion online
speculated about a shared vendor -- which isn't the case. Quick Donate was a
custom product internal to the Obama campaign.)

Romney campaign seems to have had serious problems in a lot of their internal
processes when it came to tech/web. (AMERCIA)

------
danso
In the past week, there's been discussion of how sophisticated the campaigns
are at managing and leveraging the vast volumes of data they collect. I've
argued that while they may have lots of data, and that they _think_ they're
being sophisticated with it, the results don't seem to indicate that they're
particularly clever (or, more importantly, clever enough to be devious with
it).

Having access to data is one thing...having the logistics and middle managers
who have good insights is the other. I think it's safe to say that no matter
how much money and resources these operations have, it doesn't mean anything
without campaign operatives who have a real IT/data background.

------
antidoh
It's too bad the Romney campaign was unable to tap the services of an
experienced businessman who is expert at management and delegation.

------
SkyMarshal
_> To build Orca, the Romney campaign turned to Microsoft and an unnamed
application consulting firm._

Lawsuit incoming in 3...2...1...

I can't imagine the billionaires who funded the campaign and the various Super
PACs are going to be pleased to find out Romney likely lost because of a
failed GOTV effort.

~~~
ams6110
It's a stretch to think that a functioning mobile app on election day would
have made the difference between Romney winning and losing. The election
wasn't really that close. And I say that as someone who voted for him.

~~~
SkyMarshal
It's not a stretch to think that a functioning GOP GOTV campaign could have
made the difference in 4 swing states that tipped the election to Obama (CO,
FL, OH, VA), where the vote differential was just under 400,000.

[http://fivethirtyeight.blogs.nytimes.com/2012/11/08/as-
natio...](http://fivethirtyeight.blogs.nytimes.com/2012/11/08/as-nation-and-
parties-change-republicans-are-at-an-electoral-college-
disadvantage/#more-37366)

[https://docs.google.com/spreadsheet/ccc?key=0Au5WjgbQnYrSdFd...](https://docs.google.com/spreadsheet/ccc?key=0Au5WjgbQnYrSdFdpN2lWTlhrV2dkSExncjZrTmZQZGc)

Romney had 37,000 field volunteers, plus phone bank. I don't know how many
were in each of the four tipping point states, but had each of them been able
to bring in between 10 and 20 voters who didn't vote (of the 2 million extra
who voted for McCain in 2008 but were absent this year), Romney could have
won.

Instead it sounds like due to the massive Orca failure, they all gave up and
went home.

(and fwiw, I say this as someone who voted for Obama. Not sour grapes here,
just assessing the IT project failure.)

~~~
gordonguthrie
The result of Davies fieldwork in UK elections showed that active local party
activities (including GOTV) can be worth up to a 2%-4% swing - this is from an
election where 2nd generation electoral computing systems were used.

In Scotland we have 3rd generation systems (one system for all elections
vertically, local government, Scottish Parliament, Westminster Parliament,
European Parliament, and horizontally, 2011, 2008, 2007 back to the 80s).

The US parties be roughly considered as having 2 and half generation systems,
so you might expect a difference at the top end of that.

I don't follow US elections that closely but you can work the numbers out.

------
jongalloway2
This is an absolutely perfect use case for cloud hosting like Azure - dev
local, test on a few small instances, scale up as needed on election day, then
shut it down. From the article it sounds like they paid for eleven servers,
which was significantly more expensive _and_ lower powered.

~~~
brown9-2
This wouldn't have helped their training problem however - volunteers confused
about how to find the site, not sure how to use it, etc.

Several dry runs and stress tests would have helped them alot.

------
csel
11 DB servers and 1 web app? Let's look at how the db was structured.
Wait...nevermind forget it.

------
b1daly
I think there is a connection between the sort of hand-waving, know nothing-
ism of Republican political messaging and this tech fiasco.

That kind of thing might work in the soft field of persuasion, but is pretty
useless in creating a complex technical system. (By know nothing-ism of the
Republicans, I'm referring to the birtherism, beligerent rhetoric about
complicated diplomatic issues, anti-science views about global warming, trying
to paint Obama as a socialist, in general the whole "Guns, God, & Gays" type
of messaging).

------
irisshoor
This is one area you don't want to hack a solution quickly, but want to use
something that's been tried and tested in the field. You can be on the
bleeding edge when you have time to iterate and fix things, not when you have
just one shot. When it comes to having just one shot no one does it better
than NASA - [http://www.theverge.com/2012/8/9/3232160/curiosity-
mastcam-2...](http://www.theverge.com/2012/8/9/3232160/curiosity-
mastcam-2-megapixel-camera-project-leader)

------
inpHilltr8r
Faith based development.

------
xoail
1\. They should have hired best UX engineers for the design. 2. Hosted the
solution in the cloud and called and made sure the host is aware of the
upcoming traffic. 3. Rolled out the application in test mode at least a week
before so that staff can do trail runs. 4. Worry less about hacking and made
it easy to login (just via an ID) ~ me being Captain Hindsight.

~~~
acslater00
I'm not so sure. Intentionally disruptive behavior by the opposition -- or
even some motivated kid who wanted to cause trouble for the campaign -- would
be one of my biggest concerns if I were designing this thing.

------
CletusTSJY
As a conservative and a software engineer, it's heartening that they at least
had a plan and tried something. The fact that it was poorly implemented and
unstable doesn't surprise me. Most last minute projects end up that way.

------
djt
i find it interesting that they named it Orca as Obamas system was called
Narwhal.

for those of you that dont know, Narwhal is an animal, but its also a internet
meme. Obviously a very nerdy reference by Obamas team.

The fact that the Romney team had to name their software as a killer for a
Narwhal goes to show that they wanted to win at any cost, unfortunately when
that happens you sometimes lose sight of the goal, which in this case is to
have a working system.

~~~
jacalata
I find it interesting that they made this exact point in the article we are
discussing here, and you don't appear to have read it.

~~~
djt
they didnt mention that it was a animal made famous as a meme

