
‘We feel really terrible,’ says chief executive whose app roiled the Iowa caucus - AndrewBissell
https://www.latimes.com/business/story/2020-02-04/we-feel-really-terrible-says-chief-executive-whose-app-roiled-the-iowa-caucus
======
alpb
Seriously curious, there were only 1,765 precincts.

Couldn't they just create a Google Forms link share it to the field people
sending the results back? It would be only 1765 lines in a Google Sheets doc,
and would be practically free. (Not to mention more secure than a custom
implementation by semi-incapable development contractors like in this case.)

~~~
secabeen
They legitimately wanted more security than that, with two-factor and other
similar systems to prevent hostile actors inserting bad data.

~~~
zamadatix
Google probably does 2 factor better than $smallapp. I'd probably take more
than 1 form per report though (I think that falls under "similar systems"
stuff you mentioned).

~~~
wcfields
Although a little kludgey, it does cross of the boxes:

\- Only allow one entry per GSuite ID in Google Forms

\- Enforce yubikey 2FA on GSuite.

\- Issue each captain a Pixel 1 with Google MDM for remote wiping and app
deployment.

\- Turn off all other apps (Docs, Fi, Photos, etc...) in GSuite.

~~~
runawaybottle
Wouldn’t all of this be more elegant using Graphql?

~~~
ilaksh
Are you serious or making a joke? Not saying it can't be serious. But it might
be a joke. I can't tell anymore.

~~~
tmpz22
If that’s not a joke I’m quitting this industry and becoming a carpenter

------
war1025
Just saw this article from ProPublica [1] that basically implies they sent all
the data over http instead of https.

So much for "improved security"

[1] [https://www.propublica.org/article/the-iowa-caucuses-app-
had...](https://www.propublica.org/article/the-iowa-caucuses-app-had-another-
problem-it-could-have-been-hacked)

------
glofish
CEO claims the "The app was sound and good" but there was a "transmission bug"
in the code...

Other way to say it, if you don't count the mistakes - then the app is
excellent.

~~~
mcphage
“Other than that, how was the play, Mrs. Lincoln?”

------
lasky
I remember hearing on NPR how annoyed people were with the DNC’s lack of
transparency on this app, and their reasoning was “we don’t want to let the
Russian hackers know how we’re doing this”.

The more this saga continues the more the democrat gang seems like a troop of
out of touch hippie professors recruiting freshman to do their “technology”
work.

~~~
mercer
Or, few rungs up the ladder, it's a convenient way to get in Sanders' way
while claiming incompetence!

------
shadykiller
From the article here - [https://www.bostonherald.com/2020/02/04/companies-
behind-iow...](https://www.bostonherald.com/2020/02/04/companies-behind-iowa-
caucuses-app-failure-had-ties-to-hillary-clinton-pete-buttigieg-campaigns/)

Shadow Inc. — the tech company behind the app — launched in 2019 with backing
from political nonprofit ACRONYM, according to a statement from the nonprofit.

ACRONYM founder and CEO Tara McGowan, a Newport, R.I., native, said on Twitter
that Shadow is “an independent company ACRONYM invested in.

McGowan, who lists herself as a former journalist, married then-Hillary
Clinton campaign staffer and current Buttigeig strategist Michael Halle in
2015, according to a Providence Journal marriage announcement and Halle’s
Twitter account.

Not at all suspicious !

~~~
goatherders
It's not really. That's how most software projects happen. "We need XYZ
app/site/solution? Hm...I worked with these guys back in 2016 and they did
good work. I'll make an introduction...."

~~~
pasquinelli
But that first step, "we need xyz"... they didn't.

~~~
thedance
Also "they did good work" can't possibly be true, since Clinton 2016 was a bad
campaign, with bad data, that lost.

~~~
ncallaway
Your logic is there can be no individual person who did good work on the
campaign, if the overall campaign was bad?

~~~
thedance
We're talking about the person who _led_ the software and data efforts for
Clinton in 2016. Clinton's data was bad and misleading and stale, which
contributed to the candidate taking victory tours a week before losing four
states the campaign thought they would easily win. Yes, I think every high-
ranking official of that campaign should be unemployable in the same field.

~~~
ncallaway
Gotcha, fair enough

------
taurath
I really don't see a lot of accountability in any of his statements. It just
seems like "well this thing went wrong, somehow, and its nobodies fault and
hey nothing was lost". I find that to be absolutely abhorrant. Think about if
you're one of their junior developers - you'd probably be blaming yourself for
whatever bug that happened to get in. That the CEO wouldn't take any
responsibility is a complete lack of leadership.

To those devs, jr or otherwise - every step that was supposed to protect
against a failure in the app seems to have failed here, and you should not
feel bad. Mistakes happen and should be expected, but the damage that they are
allowed to wreck is the blame of poor management, not you.

~~~
alpb
Every single executive in this little software company used to work at
Clinton's 2016 campaign [1] so they have political connections.

Not to mention, political parties aren't transparently run in many cases like
we've seen in 2016 Democratic National Committee email leak. Most likely,
these people are feeding off of their political connections and harvesting
donation money from not just the party, but also the candidates [2].

This is also why I'm no longer donating to a political party directly, or a
candidate who would in turn would do the same. It seems like the only
independent candidate running under democratic party is keeping his campaign
money in-house.

[1] [https://www.crn.com/slide-shows/applications-os/who-is-
shado...](https://www.crn.com/slide-shows/applications-os/who-is-shadow-inc-
the-company-behind-the-iowa-caucus-voting-app-/2)

[2]
[https://twitter.com/BustinTrudeau/status/1224697566182498306](https://twitter.com/BustinTrudeau/status/1224697566182498306)

~~~
6nf
These developers also developed Buttigieg's campaign software and Buttigieg
donated to help develop the caucus app.

~~~
otachack
No COI to see here, folks.

Why isn't the DNC harder on crap like this?

~~~
commandar
>Why isn't the DNC harder on crap like this?

Well, for starters, because the DNC doesn't run the Iowa caucus.

~~~
munk-a
The DNC doesn't directly run the caucus, but the DNC (or maybe, more
precisely, the democratic party) chooses to accept the results of the Iowa
caucus and subsidizes (or entirely funds) the running of the caucus.

Additionally, the Iowa dem caucus (and primaries) aren't a governmental thing,
the political parties aren't government organizations - they're private
organizations that are just _all about the government and politics_ as such
the Iowa caucus is really weirdly in a grey zone where there is even less
necessary oversight than the administration of the general election in Iowa
will have - but sort of more oversight since the DNC could just reject the
results outright... Buuuut things like the voting rights act have forced some
regulations into primary operation - it's all quite confusing...

The TL;DR though is that the DNC doesn't technically run the Iowa caucus but
is more than capable of making the Iowa dem caucus committee feel a whole lot
of pain and hold them accountable.

------
munk-a
> Niemira also confirmed Acronym’s role as majority investor in Shadow but
> declined to name the company’s other investors. He also declined to name the
> members of Shadow’s board of directors.

Hey guys, you done messed up - no more lack of disclosure. Your company will
probably be dissolved due to this hilariously well publicized failure.

> The company’s mission is to help advance Democratic causes and candidates,
> and its employees were excited to have an important role in Iowa’s historic
> presidential caucus.

That sounds noble and all, but also pretty silly - how was your solution ever
going to help "advance Democratic causes and candidates" purely by serving as
a drop-box for everyone to upload voter tallies to. The employees at your
company can feel excited to be working to support the DNC - but it's really a
stretch to claim that your vote tallying solution was advancing Democratic
causes. It's like claiming, as a car manufacturer, that you're excited to help
advance the cause of addressing back problems because some of the people who
might buy your car end up driving to staples and buying a new office chair.

------
zomg
i love a smug, passive-aggressive apology. just OWN up to it, acknowledge the
implications of the issue you caused and provide an actual apology. unreal.

“I’m really disappointed that some of our technology created an issue that
made the caucus difficult,” said Gerard Niemira, chief executive of political
technology company Shadow Inc., in his first interview after the caucus. “We
feel really terrible about that.”

~~~
mxcrossb
> just OWN up to it

That’s how you end up getting sued.

~~~
ggm
Its not like they can avoid being sued by not doing this. It just changes the
cost point to be sued successfully, and actually denying impact and
consequence probably increases their burden. Owning up, being sued but showing
contrition has an upside.

------
iamaelephant
> The problem was caused by a bug in the code that transmits results data into
> the state party’s data warehouse.

The amount of data we're talking about here would fit comfortably in an Excel
spreadsheet. Why on Earth was there a data warehouse involved at all?

~~~
runawaybottle
From what I read on various articles linked in this thread and others, the
votes were being checked against the warehouse (who knows where the warehouse
is getting data, if that’s the one true source, why even have this app). It
failed to check against it properly (they were getting back an error). I guess
election night was the first time they tested this validation phase , as they
were not able to account for it and the app simply failed.

------
supernova87a
A typical non-apology. Making it sound like there were just minor subtle
errors that they could never have foreseen and were caught off-guard by.

They were deploying an app to a very inexperienced user group, via developer
certificate enterprise sandbox because they couldn't get the app done in time
for App Store review and approval. They didn't do the requisite amount of live
user testing and were sending out instructions up to 3pm on the day of voting.

They had a chance to evaluate whether they should take on this job and do it
right or not. They could've advised and warned the IDP that they were going to
get in trouble.

But they thought they could handle it based on their past inconsequential
apps. And they wanted the business and publicity of doing this job. They were
wrong, and now playing it off like it was some minor mistake.

It wasn't. Just a typical story of overly confident people getting in over
their heads, and now passing it off as no big deal when in fact it is a big
deal. Maybe if such behavior was fined or penalized there would be a lesson
learned...

------
ramblerman
First line in their job application:

"Shadow is a technology company dedicated to building power within the
progressive movement."

Can some things remain non political these days? What a weird sentiment.

Are we moving to a future where I need to check the political leanings of
software providers before they can agree to do business with me? I'm looking
for a professional to do a job, you can keep your politics at home...

~~~
skinnymuch
They aren’t progressive. So caring about the statement is pointless. North
Korea’s official name doesn’t mean shit.

------
jiveturkey
With all the furor around election tampering, why aren't there regulations
around vote counting hardware/software?

Is the CMM still a thing?

[https://en.wikipedia.org/wiki/Capability_Maturity_Model](https://en.wikipedia.org/wiki/Capability_Maturity_Model)

"The CMM was originally intended as a tool to evaluate the ability of
government contractors to perform a contracted software project."

For something as important as _elections_ , developers (organizations) should
be required to adhere to CMM 5 and use a very tight waterfall + TDD practices.
Further, the product and testing requirements should be public. Not this
shadow development crap.

~~~
Boxbot
the dnc and associated state organizations are not government organizations
and there are no rules or regulations on whatever processes they use to select
a candidate for the presidential election. they can literally do whatever the
hell they want.

for that matter there is no (federal level) law or regulation requiring a
public vote for the president at all. the constitution establishes the
electoral college but the methods by which electors are chosen and vote are
left to the state themselves.

------
cryptozeus
Basically someone didn’t do the load testing before prod release.

~~~
maxlybbert
That’s what I thought originally, but there really shouldn’t have been much
load (ignoring the load of downloading the app to begin with). There were
fewer than 2000 precincts, and each precinct had to upload around 100 bytes of
data. That shouldn’t have been hard.

One news story I read quoted an error message about a “missing protocol.” I
believe that the “coding error” was really a configuration error: somebody
forgot to include [https://](https://) in the web service URL. The app
recorded the data, but couldn’t upload it.

Load testing would have found this, but so would a single integration
test/smoke test.

~~~
runawaybottle
My thinking is they simply didn’t do any dry runs with possible simulated data
(bad data, malformed data, partial data), they didn’t consider all the ways
things can go wrong in real time. This is less about load testing but more
about exception handling/fallbacks.

~~~
maxlybbert
I definitely agree with you there.

------
Aperocky
> the app was sound and good, the problem was a transmission code ...

This guy doesn’t know what he’s talking about is he? Is this same person who
wondered why he could copy the google frontend and it would look literally the
same? What use is a browser with faulty internet ‘transmission code’

~~~
runawaybottle
Yeah, like what does that mean? Their post requests had bad data from the
frontend? How were they validating what they were sending?

------
craftinator
Most comments here seem to be about "what Shadow should have done", making it
obvious, given the distribution of HN users, that Shadow really should have
used a different systems approach. But I'd like to focus on a different
aspect: they used the systems approach that they did (given budget and
deadline), and the outcome was poor. Is it difficult, using their approach, to
accomplish what they accomplished? Seems like a single function call is the
likely culprit. What exactly caused this... A race condition, database
sharding? What's the cause of this issue, at a technical level?

------
quocble
Election software made by Shadow, inc. No joke

------
itqwertz
Does anyone have the source code or packages for the apps? I think the HN
community could have a field day with this!

~~~
akimball
Not open source ==> not auditable

~~~
itqwertz
Here's the React Native JS Code:
[https://raw.githubusercontent.com/jfbguy/iowa-caucus-app-
dec...](https://raw.githubusercontent.com/jfbguy/iowa-caucus-app-
decompiled/master/resources/assets/index.android.bundle)

We still need the index.android.bundle.map to faithfully deminify and audit.

~~~
itqwertz
Shadow team, if you're in the shadows, please come into the light and post
this file for us!

------
magwa101
What a den of incompetence, thanks DNC. Trying to rig Iowa and failed so badly
is laughable.

------
kraigie
But not terrible enough to issue a refund for the taxpayers.

------
quocble
Made by shadow, inc. No joke

------
thedance
This guy is quoted in Vice magazine throwing shade on Vertica, a database
written by a Turing Award recipient and used in many demanding roles. Then he
hired some Starbucks prep cooks (literally, check LinkedIn) with no education
and no industry experience and created a disaster. These events should replace
whatever is currently in the literature as an example of the consequences of
the Dunning-Kruger effect.

~~~
wheelerwj
I really don't think its okay to be criticizing the job history of people.
Everyone did something before they were a developer. You used to shit yourself
daily, but we don't bring it up during code review. So i don't see how a
previous position that lead towards being a developer is grounds for
derogatory commentary.

~~~
SideburnsOfDoom
It's Ok to have new / inexperienced devs on the team. In fact it's good for
the industry to bring them in.

The problems come about when there are _nothing but_ inexperienced devs on the
team.

------
mmcclure
If someone with mod powers sees this (or OP), the title has a typo in it. It's
intended to be "says CEO whose app roiled Iowa", not "rolled Iowa."

~~~
op00to
To “roll” is to mug or to rob someone. Seems fitting?

~~~
borkt
Agreed, I thought it was an intentional backhanded comment to the CEO.

~~~
AndrewBissell
A funny typo and one I wouldn't mind leaving too much, but I fixed it. :)

