

An inside look behind Romney's loss: An epic failure of its Orca big-data app - CrankyBear
http://blogs.computerworld.com/governmentindustries/21310/inside-look-behind-romneys-loss-epic-failure-its-orca-big-data-app

======
nlh
Stripping out the politics here, some good lessons for all of us:

1\. Test your damn product before it goes live.

2\. User feedback, user feedback, user feedback. Things like the confusion of
the use of "app" (vs "website") and the lack of http->https routing scream of
things that a developer would think are "obvious" but even the most basic user
testing would reveal early on.

3\. Stealth mode can be trouble. This may be a controversial opinion but I've
learned to believe more and more that "Surprise! We're here!" approach does
more harm than good. There are limited circumstances where this is not the
case (situations where knowing the existence of a project can damage that
project) but I emphasize limited.

~~~
wpietri
Yes! Long experience has taught me that I can either look dumb in front of a
small, friendly audience of early testers or a big audience of people with
inflated launch-day expectations.

Not looking dumb is not an option. It is a big fucking world, and it will
never fit inside the 3 pounds of meat that people somehow expect me to think
with. Especially when some of that meat is dedicated to things like the ego
and false pride that tell me that my big launch will be magically perfect
because I am so smart.

~~~
lukasb
"Not looking dumb is not an option."

I'm going to be quoting you forever, thanks.

------
jd
Are we really to believe this is plausible?

1: Romney's big data app failed on election day which led to ineffective use
of their volunteers.

2: Romney's campaign performed exactly as predicted by the likes of Nate
Silver.

Therefore: the failure of their big-data app was inconsequential. Otherwise we
have to assume that the hundreds of polls were consistently biased in the
direction of Obama and that the failure of the Orca big data app exactly
compensates for this failure of the polls so that the outcome of the election
is nonetheless exactly as predicted!

Absolutely 100% ridiculous.

~~~
josephmosby
Two things to consider before immediately going to the "yay Nate Silver"
default:

1\. GOP had a consistent polling advantage on voter engagement, even going as
far back as August. The Romney campaign placed a lot of stock on this. (see
here: <http://www.politico.com/news/stories/0812/79665.html>) The polls might
break for Obama, but the Romney campaign was banking on something else...

2\. Polls attempt to account for the simple error that people are more likely
to say that they'll vote than will actually vote. (see here:
[http://www.gallup.com/poll/111268/how-gallups-likely-
voter-m...](http://www.gallup.com/poll/111268/how-gallups-likely-voter-models-
work.aspx)) The Romney campaign was predicated on the idea that GOP voters
were likely to get out and vote, while Obama voters would say they were going
to vote but then drop out on Election Day. That is a polling blind side that
Nate Silver couldn't account for.

The Romney campaign didn't accurately test their engagement assertion. They
relied on journalist polling, but engagement didn't turn into votes. The Obama
campaign did the opposite: they targeted voters who might be disengaged and
directly went after those with their GOTV efforts.

There is a HUGE debate that always goes on around election time with politicos
around who matters more. Is it more important to try to sway independents
(i.e., go after the middle), or is it more important to fire up your base?
Romney won independents but Obama got the base turnout, indicating that a GOTV
effort targeted at the base might have worked, had it been used effectively.
(see here: [http://abcnews.go.com/Politics/OTUS/exit-polls-obama-
loses-i...](http://abcnews.go.com/Politics/OTUS/exit-polls-obama-loses-
independents-swing-states/story?id=17656990#.UJ0_-nZ38Yw))

~~~
philwelch
The Occams Razor answer is that the "Obama supporters are stoned slackers,
they won't even turn out to vote" hypothesis was wrong, not that it was
counterbalanced by a failure within the Romney campaign.

~~~
bpick
Could be more relevant in Colorado and Washington. Just kidding.

------
danso
The OP borders on blogspam, read the more detailed account where it was
original posted:

<http://ace.mu.nu/archives/334783.php>

Among the more amusing errors: A checklist for poll-watchers left out a vital
item -- "poll watcher certificate" -- in a packet that went out to tens of
thousands of volunteers.

This should also be amusing to most web devs: > _Next, and this part I find
mind-boggingly absurd, the web address was located at
"<https://www.whateveritwas.com/orca>. Notice the "s" after http. This denotes
it's a secure connection, something that's used for e-commerce and web-based
email. So far, so good. The problem is that they didn't auto-forward the
regular "http" to "https" and as a result, many people got a blank page and
thought the system was down._

------
woodchuck64
"Project ORCA is a massive undertaking – the Republican Party’s newest,
unprecedented and most technologically advanced plan to win the 2012
presidential election."

I hope this puts the last nail in the coffin holding the remains of the hope
that Romney would have been a better business and technical strategist for
America than Obama.

~~~
viscanti
Because no good business/technical strategist has ever overseen a complex
software project that had a bug in it?

~~~
s_henry_paulson
The article doesn't really describe a "bug" so much as a complete lack of
planning, ignoring warning signs, brushing off questions and constructive
criticisms, and in general demonstrates the exact wrong way to implement any
system.

No, no good strategist would ever release a "big data" system without stress
testing it. You'd be hard pressed to find a job doing stuff like that.

------
flxmglrb
Romney lost because he was a terrible candidate, simple as that. Even the
Republican base hated him.

Right now the GOP leadership is busy flailing around trying to figure out why
they didn't win. The ones who have gotten closest to the truth are the ones
who have zeroed in on this year's lower-than-normal turnout of traditionally
Republican voters (as this article touches upon). However, the underlying
reason for that wasn't a failure of software or a lack of last-minute "get out
the vote" campaigning, it's because people just plain didn't like Romney.

Remember the primaries? For months the Republican primaries were a game of
"anybody but Romney". The primary voters at times even gave consideration to
total crackpots like Santorum and Cain. Think about it -- they sent the
message that they'd rather have some unelectable whackjob like Rick Santorum
than be stuck with Romney. That's how much they didn't like him. But when it
was all said and done, the Republicans finally gritted their teeth and
accepted Romney as the nominee, despite the fact that they really, _really_
didn't like the idea. Then on election day, many Republicans simply stayed
home rather than hold their nose and vote for the guy.

Even if this "Orca" thing had worked perfectly it's doubtful it would have
changed much. Republican voters' dislike of Romney just ran too deep.

~~~
ams6110
_unelectable whackjob_

Hard to say that, given how the so-called "electable" guy did.

My objection always was, the republicans nominated the guy who could not beat
the last guy who could not beat Obama (i.e. McCain).

~~~
flxmglrb
Santorum is "unelectable" in the same way as Sarah Palin. Since they are so
extreme, there will always be a small, dedicated swath of people who like
them, and they can even get enough local support to get elected to an office,
however once they hit the national stage they're toast.

Also, I completely agree it is ludicrous how the Republicans chose to nominate
someone less popular than McCain and are now acting surprised that he got
fewer votes. Gee, no one could have predicted that!

------
GabrielF00
FWIW the same thing happened to Obama in 2008, at least in the key swing state
of Virginia. I did some campaigning there for Obama the weekend before the
election and the system that managed their get-out-the-vote efforts had
crashed statewide. The precinct captains were handing volunteers old printouts
of where to knock on doors and then using a highlighter to mark your route on
a Google Maps printout. Given the limitations it was still an effective system
for organizing a lot of volunteers.

~~~
tedsuo
Yeah, strike lists are The Way It Has Been Done for over a century, it works
pretty well. What amazes me about this story is not that the software crashed,
but that they did not have precinct captains and strike lists at the ready to
step in. And that they were only rolling out their GOTV operation on the day
of the election, and not sooner? Pretty bizarre.

------
patrickgzill
The RNC establishment absolutely HATES Ron Paul's guts. Given the slap in the
face to all the RP volunteers, a large percentage of them voted for the
Libertarian candidate in protest, voted for Obama as a protest, or did not
even vote.

If you look at Ohio, Florida, and Virginia (I am going from memory here) - the
votes gotten by the libertarian candidate represented a large portion of votes
needed to swing the state to Romney. And those 3 states' EC votes would have
put Romney a lot closer, giving him 266 EC votes. Add in Colorado (numbers
below) and he would have won.

Florida: 29EC votes; difference was 61,000 votes, L. got 44,000 votes

Ohio: 18E votes: difference was 104,000 votes, L got 47,000

Virginia: 13EC votes: difference was 116,000, L and Constitution parties got
44,000 .

Colorado: 9EC votes: difference was 113,000, L and Constitution got 37,000 .

Clausewitz, in his treatise "On War" dedicates an entire chapter to "Moral
Forces" , what we would term "an organization's morale" .

Did crappy software contribute? Perhaps.

But wars and elections are not won by software, but by people.

That critical Libertarian slice or whatever you want to call it, of the
traditional Republican base, was demoralized over Ron Paul and that resulted
in an Obama victory.

Description of rule changes at RNC convention:
[http://articles.latimes.com/2012/aug/28/news/la-pn-ron-
paul-...](http://articles.latimes.com/2012/aug/28/news/la-pn-ron-paul-
supporters-walk-out-of-gop-convention-20120828)

~~~
qq66
So even making the incorrect assumption that 100% of the Gary Johnson votes
were coming from would-be Republicans, none of the outcomes would have
changed.

~~~
patrickgzill
That is not what I said... key sentence is "That critical Libertarian slice or
whatever you want to call it, of the traditional Republican base, was
demoralized over Ron Paul and that resulted in an Obama victory."

Meaning the libertarian section of the Republican base, not meaning the people
who necessarily voted for Gary Johnson.

------
tokenadult
From an Ask HN post a month ago:

<http://news.ycombinator.com/item?id=4583307>

> > Fairly often we see stories that link to a piece of useless blogspam
> rather than the original article. The guidelines say that these should never
> have been submitted. But are silent on what we should do if we notice the
> guideline being violated.

> > Is it helpful for us to flag those as we notice them so that editors can
> more easily find and kill them? Or would those flags just an annoyance for
> editors, and we should just ignore them in the belief that they will get
> deleted if they are a problem?

> Yes, flag them.

The story kindly submitted here was nothing but one blogger quoting some
earlier publications, with no new added analysis. There have been better
submissions to HN on the same underlying story today, from better sources.
This is one of the hot stories today.

------
jeremiep
This is pretty usual when government and software development are mixed
together. All of the bad developers I met at college are now working for the
government or are contracted by the government. Most hackers I know wouldn't
go anywhere close to such a contract.

My guess as to why it happens this way is that they check your scores rather
than your actual skills and anyone without a diploma is automatically
disqualified. Their culture is the opposite of hacker culture.

~~~
rprasad
* All of the bad developers I met at college are now working for the government or are contracted by the government. *

Interestingly, all of the government developers I know would run circles
around almost any startup programmer. These guys design and implement systems
that are designed to last _decades_ without even minor bugs. Frequently, they
manage to keep systems designed _decades_ ago from collapsing under their own
weight.

 _My guess as to why it happens this way is that they check your scores rather
than your actual skills and anyone without a diploma is automatically
disqualified. Their culture is the opposite of hacker culture._

Government hiring is credentials _and_ experience based. Don't be bitter that
you can't get a job because you lack the credentials or experience to make the
cut.

 _Most hackers I know wouldn't go anywhere close to such a contract._

A certain level of professionalism is expected for a real job. Nobody cares
what you wear, or where you work, or how you work when you're making yet
another Instagram clone. When you are designing and implementing critical
systems, being a professional matters. Most hackers simply don't have the
self-discipline to _get_ such a contract.

~~~
pnathan
> A certain level of professionalism is expected for a real job. Nobody cares
> what you wear, or where you work, or how you work when you're making yet
> another Instagram clone. When you are designing and implementing critical
> systems, being a professional matters. Most hackers simply don't have the
> self-discipline to get such a contract.

Disagree.

I work in critical system infrastructure. Dressing up nice to talk the
computer doesn't make the code better; aggressively making the code better
does. It's my opinion that the better dressed the guy writing code, the more
suspect his code is...

~~~
michaelt

      It's my opinion that the better dressed the guy 
      writing code, the more suspect his code is...
    

So do you think others make judgements about your ability to do a job based on
how you're dressed?

~~~
pnathan
Everyone is making judgements, are you wearing black, are you wearing a suit,
are you wearing fashionable clothing, etc.

My opinion/observation is that the more attention that is paid to your
clothing, the more your engineering/code quality suffers. It like the IQ test:
wearing suits impresses people who think highly of people wearing suits. Maybe
it's better to impress people who think highly of quality work.

edit: It's foolishness to dress finely to make people think you're awesome.
You're not more of a professional because you wear a tie and suit. You're just
a guy in funny-looking clothing who looks like important people. If you want
to be professional, do the job well, do it kindly, do it so it can be used for
its intended purpose for a long time, do it with respect for other people.
Dressing nice has so very little to do with professionalism.

------
adolph
It is pretty interesting that they didn't have the system up and running well
before early voting started--do campaigns not do get out the vote volunteer
work during early voting?

~~~
spamizbad
My hunch is that this project was started way too late. At the start of the
Republican primary in 2011, Romney's fundraising wasn't aggressive, and it's
unlikely his team prioritized a "Big Data" initiative from day 1. In the
ensuing primary slugfest, Romney probably needed to put a ton of resources
into advertising to win. He also had a very large campaign payroll: his people
were well compensated, but that didn't leave a lot money left-over for things
like Orca.

Based on this conjecture, Orca probably got started sometime in the Spring,
essentially giving the campaign 5-7 months to ship a bullet-proof GotV app....
alongside all the other stuff the team had to build (Tons of miscellaneous
internal tools, the public mobile apps, etc). A lack of time and resources
probably doomed it.

Moral of the story? Software is hard. Don't take it lightly.

~~~
btilly
Lack of resources was probably not a problem. The amount of soft money around
was tremendous.

But too much resources leads to the temptation to throw lots of bodies at it.
Lots and lots of bodies is simply not a good way to hit an aggressive
schedule.

~~~
ZachPruckowski
But most of that money couldn't be put towards a program like ORCA. The one
saving grace of Citizen's United is the limitation on PAC-campaign
coordination.

~~~
microtherion
Romney could have lent the money to his own campaign, couldn’t he?

------
jfc
I don't believe the app is the reason for the loss. It does, however, point to
a glaring weakness of the campaign: organization.

------
gfodor
An interesting takeaway from this article, if one believes this may have
helped lead to a Obama victory, is Romney's _campaign_ had a single point of
failure: that poor little web server.

One of the classic blunders!

------
programminggeek
I'm skeptical of blaming the tools so much as the people responsible for
buying them, building them, etc.

Also, no amount of data driven malarky changed the fact that a lot of people
weren't so excited about Romney or so upset about Obama that they changed the
current status quo.

President, house, and senate remain largely the same, so really, people
weren't convinced that a dramatically different direction was needed. At the
end of the day, Romney lost because he didn't convince enough people to
change. Some of that is money, some is tech, some is just he didn't do a good
enough job leading his campaign to victory.

~~~
wpietri
I agree about blaming the tools, but I think that last bit is false inference.
An awful lot of people were convinced that a dramatically different direction
was needed. It's just that we have roughly similar numbers of those people.

------
ck2
Why does everyone blame/praise the data?

It's the voters that made the difference.

We are talking 1% of people changed by campaigning, everyone has already made
up their minds. They spend a billion dollars on this nonsense and probably a
million "man hours" annoying people door to door. Just imagine what kind of
actual good that much money and effort could have done for schools, etc.

The population is just becoming more progressive and "values" from the 1950s
are less and less popular.

~~~
tedh
I'm a progressive but I don't know if I'm convinced that the populate is
moving to the left. I hope so but....

Romney's loss may just be that he just wasn't a candidate that excited the
right-wing enough for them to really get out and vote. Romney wasn't
conservative enough for them and he made a number of bone-headed blunders
along the campaign trail.

That said, I hope you're right and I'm wrong : )

~~~
ck2
The right would have voted for a lump of coal vs Obama after foaming at the
mouth for the past four years at Obama. Romney is kind of proof of that since
he was the pick of the primaries.

It's not a get out the vote problem, it's just that the old population is
slowly dying off and they can only brainwash their young to a certain level
when information flows so freely today.

I am disappointed there is only a 2.5% difference in the popular vote when
Bush was only 4 years ago but people have tiny memories and change happens
slowly. Values like anti-gay, anti-birthcontrol, etc. etc. etc. aren't going
to fly with most voters under 25 unless they are raised in a very conservative
environment and are never exposed to the rest of the world outside their home.

~~~
ams6110
_the old population is slowly dying off_

As if the young population won't age and replace them? Believe me, age,
responsibility, and having kids changes your outlook. When I was 25 and didn't
know anything, I was pretty liberal too. I voted for Jesse Jackson once. And I
supported Romney this year, even though he wasn't conservative enough for me,
he was better than the alternative.

~~~
orangecat
_As if the young population won't age and replace them?_

No, I don't think young people with openly gay friends and coworkers are going
to decide 10 years from now that they really should be shunned by society. And
if you're thinking at all rationally about marijuana, you'll recognize that
the greatest danger it represents to your kids is them being arrested over it.

The GOP is seemingly going out of their way to make sure that nobody under 30
even considers voting for them, except for the strongly religious. Which as a
fiscal conservative I find quite frustrating.

------
gojomo
There are more 1st-person details at the Breitbart account:

[http://www.breitbart.com/Big-Government/2012/11/08/Orca-
How-...](http://www.breitbart.com/Big-Government/2012/11/08/Orca-How-the-
Romney-Campaign-Suppressed-Its-Own-Vote)

I find this volunteer account notable:

 _I do not know if the system was totally broken, or if I just saw the worst
of it. But I wonder, because they told us all day that most volunteers were
submitting just fine, yet admitted at the end that all of Colorado had the
wrong PIN's. They also said the system projected every swing state as pink or
red._

Putting aside Hanlon's Razor ("never ascribe to malics what can be explained
by incompetence") for a moment: what _if_ some of these failures were due to
partisan/hacker tampering?

If you owned the server(s), you could scramble a state's PINs. You could break
redirects. You could bias the projections for overconfidence, or mark records
as 'voted' or 'followed-up' that hadn't been.

With a little phone-hacking, you could make sure attempts to call-in with
problems get no answers.

A single skilled amoral hacker could more easily suppress thousands of actual
votes this way, compared to voting-machine/voting-tabulation tampering. And
the details could disappear more easily into the secrecy and cover-your-ass
bullshitting that is common in a partisan, temporary campaign organization.

------
evandena
That wasn't much of a look inside.

------
Tsts
Romney never played the game to win, but to show there are even worse
alternatives than the black Bush.

~~~
jfc
What a vulgar sentiment! President Obama is hardly a black Bush, whatever your
thoughts about his policies. If you really believe they are the same--yet
utterly fail to articulate your rationale--there probably isn't much reason to
engage in a discussion with you.

