

How Obama Raised $60 Million by Running a Simple Experiment - dsiroker
http://blog.optimizely.com/how-obama-raised-60-million-by-running-an-exp

======
maxklein
"That would have been a huge mistake since it turns out that all of the videos
did worse than all of the images."

It annoys me no end when websites have videos explaining their product and no
equivalent text. I hate watching videos about products. Always make the video
optional and secondary.

~~~
ryanwaggoner
Yes, always. Assuming your audience is maxklein. Otherwise, best to test.

~~~
Natsu
You're right that they should test, because their audience might be different.
But I admit to being the same way as him: when I hit an article that's nothing
but video, I skip it.

I can read a _lot_ faster than they can talk and I could have read a lot of
other things in the time it takes to watch a video. I also have a lot of
things I want to read and nowhere near enough time to read them all. In fact,
there was an article here on HN that linked to a bare video. I was going to
read the article, but when I saw it was a video, I closed the tab immediately.

So be sure to test and make sure, but remember that there might well be a lot
of others like me who would rather read than watch a video.

~~~
harisenbon
I'm similar. I'll watch about 2 minutes of a video, and see if the
presentation is engaging enough to warrant spending the time to watch the
video.

If I can download the video, I do that and watch it on my trainride home.

Else the window gets closed

------
guelo
The "Learn More" button and this whole page are spammy. I remember being
annoyed by this splash page, I just wanted to get to the main site. When users
saw the "Sign Up" or "Join Us" buttons they probably realized it was an email
phishing form and then searched for the hidden "Continue to Website" link. The
"Learn More" button otoh seems to be what you are looking for, an escape, but
when the button complained about the missing email some users probably went
ahead and entered it even though they didn't want to because it seemed like
the only way into the site.

~~~
kinofcain
Precisely. It doesn't take very many A/B tests on lead-gen forms to realize
you can get much better conversion rates if you're willing to trick your users
into clicking on stuff. "Learn More" isn't a better title for that button,
it's a deliberately misleading title for that button. You can have a button
labelled "Get a free puppy" and a button labelled "Spam me every 24 hours and
sell my e-mail address to the highest bidder". Guess which one will convert
better; Guess which one is honest.

------
run4yourlives
The best thing about this article is that it is a giant ad. Informative,
educational and useful, but an ad nonetheless.

Wonderful marketing and great article as well.

~~~
gallerytungsten
Indeed. I thought it was particularly clever how he apparently used the
campaign data, but ran it through his product to get the "experiment results"
screen shots.

~~~
natrius
I assumed that was Google Website Optimizer or whatever they actually used
back in 2008. If not, they should make that clearer.

------
waterside81
Kind of a side note, a complete noob when it comes to A/B, I tried out
Optimizely after reading this and I've gotta say, this thing is put together
well. I think I created 6 variations for our home page in about 2 minutes,
saved it, and started watching results flow in. Amazing execution.

~~~
WillyF
I completely agree. I just started using Optimizely today, and the ability to
make a small change instantly across all pages on your site is extremely
valuable.

------
ddlatham
He seems to assume that the contribution and volunteer rate of the added
marginal people (those who signed up after seeing the best performing media /
button, but would not have signed up with the default media / button) is as
good as the rest of the people (those who would have signed up regardless).

~~~
kenjackson
He doesn't mention it, but that's easy to determine. Simply see what the
average contribution of people who contributed during the A/B testing was per
image/button combination. If there not all about the same then there may be
some effect (negative or positive) that the added marginal people brought. If
they're about the same, I think its a decent assumption to believe that it
will be the same going forward.

And of course, you can simply validate it by seeing if it is the same going
forward.

~~~
dsiroker
_Simply see what the average contribution of people who contributed during the
A/B testing was per image/button combination_

We didn't measure the average contribution for each variation for this
particular experiment but in future splash page tests we measured eventual $
per pageview for each variation and it was very closely correlated to % signup
rate.

------
mmaunder
$400/month for 200K visits? The pricing seems to be based on how much they'll
contribute to your bottom line rather than what the market will bear, what
competitors are charging, whether they should focus on growth rather than
early profit, etc. This might be a great opportunity for someone to clone the
business and charge 25% of their prices, which will still be $250/month for 1
million visits.

~~~
patio11
You think the opportunity to be the cheap, crappy competitor with an audience
of technically inclined poor people is attractive? I think that is terrible
positioning. If you charged $50, that would still be a whole lot of ramen, and
you'd lose your customers to the first still cheaper, still crappier
alternative or to free options. (There exist compelling free options for A/B
testing if one is technically inclined.)

$400 is not a lot of money when you probably demonstrate increases in revenue.
(It is a fraction of BCCs gains from A/B testing. Nationwide political
campaigns, Fortune 500 companies, and real estate firms all have slightly more
money to spend than BCC does.)

------
lkrubner
The experiment showed that video was a poor choice for Obama. Images worked
better for him. However, I do not think anyone should generalize from this
example. Obama was supported by specific demographics. The poor and working
classes favored Obama to a disproportionate degree. These are the people most
likely to still be on dial-up Internet connections. There are the people with
the least access to broad band. Therefore, these are the people for whom video
would be a poor choice.

~~~
lkrubner
I wonder why this was downvoted? What part of this do people disagree with? I
made 3 main points:

1.) Obama had disproportionate support from the poor and working classes. This
is easy to verify, there were many news organizations that covered this. Here
is one example:

<http://geocommons.com/overlays/6095>

2.) The poor and working classes have less access to broadband Internet. Is
anyone seriously going to dispute this?

3.) Therefore video was a poor choice for Obama, since many of his supporters
lacked broadband Internet access.

My friends, instead of downvoting, why don't you explain what you are
thinking? Does someone disagree with my thesis? I feel like I'm stating
something fairly obvious here.

~~~
anthuswilliams
I didn't downvote you, but I do think you're mistaken.

1) Your source cites the November 2008 exit polls. The campaigning mentioned
in the article was being done through 2007, when Hillary Clinton still had her
hat in the ring. Now, it's hard to use campaign donations as a gauge of
support for presidential candidates (particularly among people who are too
poor to even afford broadband). However it's easy to point out that Hillary
Clinton's platform was more beneficial to the working class, whereas Barack
Obama's vague Yes-We-Can appealed more to broadband-connected hipsters like
me. As you can see from About's (pro-Clinton) US Liberals blog[1], half of the
Obama campaign chest going into 2008 came from individual donors, whose email
addresses were harvested on websites just like the one described in the OP.

2) I'll also dispute the claim that the poor have less access to broadband,
having grown up so debilitatingly poor myself. In America, internet access is
cheap, and even for the poor, the value of a fast, reliable internet
connection far outweighs the cost. Those who are remain on dial-up connections
are residents of low-density rural areas where broadband is unavailable. These
people are a) unlikely to have been Obama supporters, and b) unlikely to have
gone around donating campaign money on the Internet.

3) Barack Obama's was the most Internet-connected campaign in US history. If
dial-up users were as large a chunk of his support base as you say they were,
he would have spent a lot more time going door to door in Des Moines, and a
lot less time on YouTube.

[1] I wanted to link to the New York Times article, but it's probably more
informative all around if I just link to a blog which quote-mines it. At least
About.com doesn't make you log in.
[http://usliberals.about.com/b/2007/07/19/hard-truths-
about-d...](http://usliberals.about.com/b/2007/07/19/hard-truths-about-
democratic-2008-fundraising-results.htm)

~~~
lkrubner
This is not a subject where anecdotal information reveals much, because the
USA is diverse and few of us understand what all of the different population
segments face, in terms of information access, or Internet access. According
to Wikipedia the digital divide still exists, and income plays a role:

<http://en.wikipedia.org/wiki/Digital_divide>

While it admits to several definitions of the divide, it mentions broadband
access as one aspect of the problem.

My original point remains: Obama had disproportionate support from those
population segments that were likely to have limited broadband access,
therefore video would be a poor choice for him.

The larger point is that you can not generalize from this one example,
mentioned in the originally linked article. Video was a poor choice for Obama,
but might be a good choice for a new web startup that is pitched to a
demographic where broadband access can be assumed.

~~~
anthuswilliams
Of course the digital divide is affected by income, but you're missing key
pieces of the story.

1) The digital divide between classes is negligible compared to the divide
between rural and urban dwellers, particularly wrt broadband access[1]. Also,
n.b. class based Internet analytics is prone to error, since the poorest
segment of America lives in remote rural areas where broadband access is
unavailable to begin with.

2) As I pointed out above, the OP was in reference to 2007, when the
Democratic candidacy was still up in the air. Further, Barack Obama drew
support (particularly individual campaign donations) from young Internet-savvy
people, whereas the real blue-chip donations went to Hillary Clinton (and to a
lesser extent, Chris Dodd). Your data is from the exit polls in 2008, after
the Democrats had (more or less) formed into a united front.

This makes your larger point untrue[2]. The data from the Obama campaign
indicates that these particular videos were less effective than the images,
and to assume this is due to bandwidth complaints is specious at best.

[1][http://pewinternet.org/Media-Mentions/2006/Rural-areas-
laggi...](http://pewinternet.org/Media-Mentions/2006/Rural-areas-lagging-in-
broadband-Internet.aspx)

[2] Well, it is true that you can't generalize from this one example, and that
video may still be a good choice for web startups, but not for the reasons you
suggest.

------
Semiapies
_"Sending email to people who signed up on our splash page and asking them to
volunteer typically converted 10% of them into volunteers."_

Is it wrong for me to be a little startled by that conversion rate? I wouldn't
have expected anything that high.

~~~
kgermino
_Is it wrong for me to be a little startled by that conversion rate? I
wouldn't have expected anything that high._

It depends on how much of a commitment they're asking from volunteers. If
you've already signed up on the website it's not much more to volunteer to
pass out fliers at your office. If they want you to work a benefit or
something than ya I'd agree it's a little high.

Keep in mind too that these are people who have already actively signed on to
support the campaign.

~~~
Semiapies
Good points, though the best-testing design was the one that didn't phrase
signing up as a commitment, but as an opportunity to "Learn More", hence my
surprise.

~~~
kgermino
That's a good point and when I was reading the article I actually predicted
that the "Learn More" button would win because it was the least committal but
I never drew the connection between that and volunteer signups.

A few theories:

1) Maybe the feeling of not committing by pressing the button made people more
comfortable signing up to be a volunteer, figuring that it probably wouldn't
be too much of a commitment

2) Maybe the content on the Learn More page was just so good that the only
thing that really mattered was getting more people onto that page and they
would be comfortable

3) Perhaps the type of person more likely to click after seeing the picture of
the family (Stay at home moms perhaps) was more likely to have time/desire to
volunteer.

As I think about it, assuming that one of these ideas is right which is
probably not the case, I would guess that it's #3. I don't know the
demographics of the people who volunteered but around me it seems like it
would be the moms who go to the site, and they would probably be more likely
to be skewed toward clicking with the family and non-committal button, while
also being more likely to have the time to volunteer.

Disclaimer: This is pure, assumption based, uninformed speculation of the
worst kind.

~~~
d0mine
As I understand it there is no "Learn More" page just a button: you give your
e-mail, zip code and press the button.

After that they send you the e-mail with the suggestion to volunteer. So 10%
look impressive.

~~~
dsiroker
_As I understand it there is no "Learn More" page just a button: you give your
e-mail, zip code and press the button._

You are correct. All of the variations would redirect to the home page
regardless of what the button said.

 _So 10% look impressive._

Keep in mind that this is the percentage of people who volunteered at some
point during the campaign after signing up on our splash page-- not the
percentage of people who immediately decided to volunteer after signing up.

------
Shamiq
Glad they were able to actually write a short case-study on such a high worth
(I'm assuming) case. That's real transparency.

~~~
sudont
<http://www.designing-obama.com/> => I snagged the PDF when it was still free.
_EDIT: Still free. See responses._

VSA Partners on the Logo:

Part One: <http://www.youtube.com/watch?v=etEP1Bhgui0>

Part Two: <http://www.youtube.com/watch?v=ukIMW833EPE>

There's a ton of info out there on Obama's campaign, it's really one of the
best branding and marketing success stories.

~~~
bjonathan
The PDF is still free: <http://digital.designing-obama.com/>

You can also read "Barack Obama Social Media Toolkit by Edelman" :
[http://www.scribd.com/full/10807015?access_key=key-
nc75wovez...](http://www.scribd.com/full/10807015?access_key=key-
nc75wovez6hgum15t6)

------
jonathanjaeger
For any of those interested in seeing the Mixergy interview with Dan Siroker
(founder of Optimizely), check out this link: <http://mixergy.com/dan-siroker-
optimizely-interview/>

