

Was Nate Silver the Most Accurate 2012 Election Pundit? - rms
http://appliedrationality.org/2012/11/09/was-nate-silver-the-most-accurate-2012-election-pundit/

======
esurc
Thanks. I've been trying to find a good way to understand the accuracy of
Silver's predictions.

What's a fair benchmark? This article offers up a "coin flip" for each state,
computing that such a coin flip would have a Brier score of 0.25. (The Brier
score is a mean-squared error between outcome (1 or 0) and the percent
certainty of the prediction in that outcome. If a coin flip is the model, each
state's result of 1 or 0 would be in error by 0.5. The mean squared error
would be 1/51 * 0.25 * 51 = 0.25.)

But... that seems like too generous a benchmark. Take the simple model:
"assume 100% likelihood that state X will vote for the same party as it did in
2008." That guarantees that deeply red or blue states will vote the same way,
so it takes the non-battlegrounds out of the equation.

With this model, there would only have been 2/51 errors. This simple lazy
model achieves a Brier score of 0.039, beating Intrade and the poll average
computed in this article quite badly.

After working through this, I'm still impressed by Silver and the other quant
predictions. But I'm more concerned about media that rely too much on
reporting a single polls result as "news" rather than as part of a larger
tapestry.

Then again, it's the maligned media polls that are the raw input to Silver and
the other models. Unless the media keeps funding the polls, the quality of
these more advanced models will suffer.

~~~
gwern
Thanks for the suggestion. I've added your benchmark suggestion (along with a
bunch of other fixes and new data like a 2008 RMSE benchmark along the lines
of your Brier suggestion) into the R document:
[https://docs.google.com/document/d/1Rnmx8UZAe25YdxkVQbIVwBI0...](https://docs.google.com/document/d/1Rnmx8UZAe25YdxkVQbIVwBI0M-e6VARrjb0KdgMEVhk/edit)

(I don't know when the new numbers will go live on the blog; Luke handles
that.)

------
backprojection
Note how NPR is one of the most right-biased in this result. It's pretty
evident from years of listening that the NPR staff generally are progressives,
and would left. So I think this result exemplifies how genuinely 'fair and
balanced' NPR really is.

~~~
FaceKicker
Throughout all the criticism from pundits of Nate Silver et al. as a
liberal/commie/whatever for predicting that Obama would win, I couldn't help
but wonder if there was even any merit to the idea that an Obama supporter
would _want_ to say that Obama is going to win the election. I would think
predicting that Romney has a 10% chance of winning (making it tough but a
distinct possibility he would win) as Nate Silver did would lead to some of
the BEST possible voter turnout for Romney (making it the worst possible
prediction strategically, if you want Obama elected). On the other hand, a
prediction that Romney will win in a landslide would make many Romney
supporters stay home.

Is there actually evidence that higher poll numbers in favor of X lead to
higher voter turnout from supporters of X? It seems like everyone takes that
for granted but I've never seen any evidence that it's true.

~~~
showerst
It's worth adding that Nate is on the record as saying he'd have voted for
either Romney or Gary Johnson, if he would have voted.

<http://www.mentalfloss.com/blogs/archives/150042>

~~~
pronoiac
Okay, I found the reference someone cited as "Nate Silver openly rooting for
Obama". It's from _March 2008,_ before he joined the NYTimes:

<http://www.fivethirtyeight.com/2008_02_24_archive.html>

------
bo1024
I would like to see them add in David Rothschild at Yahoo[1], who's an expert
in scoring rules and prediction markets and whose February (!) predictions
were almost exactly on the money.

[1] <http://news.yahoo.com/blogs/signal/>

~~~
PaulMest
He works at Microsoft[1]. I have been following his predictions on
PredictWise.com for the last four months.

[1] <http://www.linkedin.com/pub/david-rothschild/12/651/681>

~~~
dreeves
Just to clarify, he recently moved from Yahoo Research to Microsoft Research
when the former imploded. Same with David Pennock and others. The Yahoo folks
in New York mostly switched to MSR (founding MSR-NYC) and the folks in
California mostly switched to Google.

(I used to be part of that research group but I left at the end of 2010 to
start Beeminder.)

------
nhebb
This compares top lines. I think a comparison of turnout model accuracy would
be more informative. Most of the models that erred predicted that the 2012
turn out would lean less Democratic than the 2008 turn out model, based on the
2010 mid-term turnouts and a (mis)perceived dampening of enthusiasm among
Democrats and increased enthusiasm among Republicans. Based on exit polling,
there was a drop off of 7 million white voters, and I don't think anyone who
predicted that.

~~~
ams6110
The reasons for the low turnout will be interesting to hear. Nationally,
nearly 3 million fewer votes for Romney in '12 than McCain in '08, and around
10 million fewer votes for Obama. Polls showed the election to be fairly
close, so that would not tend to explain either "giving up" or
"overconfidence" as a reason person didn't vote.

------
Osmium
Does anybody know more about YouGov's methodology? On the face of it, I'm
suspicious of their very low margin of error which seems substantially better
than any other poll out there, but you can't deny that their polling was
accurate.

Another thing that looks odd on that graph: the given polling numbers from
Washington Times/Politico/Monmouth/Newsmax/Gravis/Fox/CNN/ARG all look
_identical_ despite their differing margins of error (which suggests their
source data is different). What's going on there?

~~~
sethg
Here’s a good article from the YouGov site that describes their methodology:
[http://today.yougov.com/news/2012/10/23/obama-stays-ahead-
ju...](http://today.yougov.com/news/2012/10/23/obama-stays-ahead-just/)

~~~
Osmium
Thanks for the link. In short, it seems like only poll people they can confirm
are actually registered to vote based on the fact that:

"According to US census data, just 71% of eligible Americans are registered to
vote. In 2008, almost 90% of those who were registered did vote. So in any
poll, it is vital to know which respondents are on the register."

But there're a few more interesting subtleties in there too, so it's worth a
read.

------
swang
Article doesn't mention Sam Wang. His confidence level for Obama winning was
at 99%

<http://election.princeton.edu/>

~~~
peteretep
Oooh, I wonder what "he" might choose for a username on a social media site!

~~~
swang
I can guarantee, with a near 100% confidence rating, that I am not Sam Wang.
Merely coincidental that we have the same last name and first initial. I mean
there has to be at least 100 million people in the world with the surname Wang
so the odds that we are the same person are pretty slim!

~~~
crayola
"I mean there has to be at least 100 million people in the world with the
surname Wang so the odds that we are the same person are pretty slim!"

Except this seems to ignore that P(comment about Sam Wang | I am Sam Wang) is
much higher than P(comment about Sam Wang | I am not Sam Wang) :-)

------
interconnector
Actually the most accurate 2012 election pundit was Drew Linzer
(<http://votamatic.org/>). Provided Florida goes Obama's way, he correctly
predicted the electoral college - Obama 332, Romney 206.

~~~
thedufer
Didn't Nate Silver say that was the most likely split? There was a post about
it on HN for like the past 2 days.

Edit: The confusion is probably that Silver was reporting the average
electoral split as his prediction, when the mode is more important in what
you're talking about. His average was almost never a number that was actually
possible, since he was quoting them to the nearest 1/10th of a vote, so its
kind of unfair to punish him for not getting it exactly right.

~~~
interconnector
You're right, I didn't realize Nate Silver was using an average. Also after
reading the article, it looks like not just predictions for the 2012
presidential electoral vote, but other votes such as the Senate races are also
being compared.

------
brown9-2
It's worth noting that if you used the 2008 results as your 2012 prediction,
you would have gotten 49/51 on this scale.

~~~
gwern
Yeah, but if you simply predicted 2008, you'd get a mediocre Brier score on
your state victory predictions (because it would punish you for getting 49
compared to everyone who got 50 or 51), and the RMSE is even worse: the
margins were different and the electoral vote & popular vote _very_ different
from 2008.

------
SeanLuke
Slate Magazine found two other pundits which were as accurate as Silver.

[http://www.slate.com/articles/news_and_politics/politics/201...](http://www.slate.com/articles/news_and_politics/politics/2012/11/pundit_scorecard_checking_pundits_predictions_against_the_actual_results.html)

~~~
Sniffnoy
Josh Putnam is considered in the article, which looks at a finer level of
detail than just correctly predicting the states.

------
Osmium
Great article, but I disagree with the colouring on the first graph: if
reality was within the poll's margin of error, I don't think it should be
coloured, because that implies a bias that (probably) isn't actually there.

------
waterlesscloud
Isn't YouGov the XBox pollster? They're in the top group.

~~~
keypusher
Also interesting to their their margin of error is significantly lower than
any of the other polling organizations. From what I gather on Wikipedia, they
are internet-centric although I'm not sure if that includes XBox.

------
bwood
Is it possible to be more accurate than nailing all fifty states?

~~~
jfoutz
Well, maybe yes. One strategy for predicting the outcomes would be to paint a
quarter red on one side and blue on the other, and assigning the outcome of
that state to the outcome of the coin flip. That probably wouldn't be a very
effective strategy, but it's possible to still nail all 50 states (actually,
51 in this case). Silver's method is much much more accurate than the coin
toss method.

Silver takes all the polls, even the crappy ones, and includes them in his
calculation. If you're a bayesian you'd find this comforting because all of
the evidence is included in the belief. There might be some handwringing about
how important each poll really is. If you're not a bayesian, then you have
some other weird strategy that might or might not work.

So really, the predictions are nice, but what we're after is a system that
produces good predictions. It's not clear that Mr. Silver's is the best.
Perhaps there is some horrible flaw an evil agent could exploit that just
didn't get tickled this election. It's tough to say.

~~~
diminoten
> If you're a bayesian you'd find this comforting because all of the evidence
> is included in the belief.

Off-topic, but this is my beef with the "Bayesians" - it's _not_ all of the
evidence; it's completely absurd to believe that all evidence can ever be
accounted for, when considering _anything_.

~~~
gwern
"More of the evidence" seems like pretty much the same thing...

