
How Good Was 538? - robg
http://andrewsullivan.theatlantic.com/the_daily_dish/2008/11/how-good-was-53.html
======
llimllib
The race between 538 and intrade was basically too close to call. See:

[http://www.bayesianinvestor.com/blog/index.php/2008/11/06/in...](http://www.bayesianinvestor.com/blog/index.php/2008/11/06/intrade-
versus-fivethirtyeight/)

and

[http://www.portfolio.com/views/blogs/odd-
numbers/2008/11/05/...](http://www.portfolio.com/views/blogs/odd-
numbers/2008/11/05/and-the-winner-isprediction-markets?tid=true)

~~~
davidmathers
That portfolio.com link is being misleading by only showing the EV totals. If
we assume, as he does, that McCain will win MO that means the betting markets
got the right number because they got _two_ states wrong, whereas 538 got the
wrong number because Nate only got one state wrong. So which one was actually
more accurate?

I posted the predictions broken out by state here on HN the night before the
election: <http://news.ycombinator.com/item?id=353182>

~~~
llimllib
I tend to agree that 538 was better; I shouldn't have editorialized that the
race was too close to call. I really just meant to summarize the two articles
I'd come across that were on exactly this topic.

Thanks for collecting that data.

------
sown
Electoral-vote.com was pretty accurate, too.

<http://electoral-vote.com/evp2008/Pres/Maps/Nov03.html>

Chris Bowers wrote, "So much information is publicly available now that a few
nerds obsessed with poll numbers are much better sources for election
information than you will ever get from big media."
[http://www.openleft.com/showDiary.do;jsessionid=16AE5E72EE66...](http://www.openleft.com/showDiary.do;jsessionid=16AE5E72EE66F64BCDA37B4B70262523?diaryId=8971)

~~~
tallanvor
Yeah, Electoral-vote's results were pretty much the same as 538's. --And I
consider Tanenbaum's site to be better overall, because while it's not as
pretty, he's completely open about how he arrives at his results and makes all
the data he uses available.

~~~
sown
I also like that it is heavy on data.

Also, I get tempted by their master's program that I don't qualify for.

------
ivankirigin
Meta news.yc comment: when 538 called it for obama, the thread was killed:
<http://news.ycombinator.com/item?id=354128>

That was a mistake. People paying attention knew 538 was right, and
consistently better than the major news outlets.

Also, I feel the geek behind 538 deserved to be the top story, not some play-
it-safe news outlet.

~~~
hugh
I thought it was killed for being politics.

Unfortunately the later winner-of-the-election thread stayed up.

~~~
ivankirigin
Politics that matters should be on HN. Winning an election matters. He called
it with certainty.

------
Readmore
I fell in love with this site over the last few months. Not only were the
numbers almost dead-on but the daily updates were all well written and
insightful.

Great work!

------
kqr2
To be statistically meaningful, 538 will have to be this good for several more
elections.

~~~
randallsquared
I don't think so. They successfully predicted quite a few elections on
Tuesday. I think it's a statistics error to group them into a single datum
simply because they were all held on the same day.

~~~
khafra
Or an information-theoretic error: At the very least, we have one bit of
information per state, and one per Senate race. You could probably eke out
some more by comparing win percentages, although that's problematic without
knowledge of his algorithm.

~~~
llimllib
But you can't assume that these events are independent of each other, can you?

~~~
dangoldin
Can you assume that the next elections are independent of the current ones?

~~~
llimllib
Yes.

Which is not to say that this year's election outcomes don't influence future
events, but rather that the information flow from this year's elections will
be so insignificant compared to future information by the time the next
elections occur that we may accept them as independent variables.

(Certainly as _more_ independent than dependent, but I'd conjecture that the
effects of the current election are so small that they may be safely handwaved
away. Interesting arguments that this year's elections on any level are a
major determining factor in future elections are of course welcome.)

~~~
dangoldin
I'll take a stab at it but I'm gathering data now - I have a feeling that it's
pretty common for a president to get elected for a 2nd term - and repeat
offices in general.

~~~
llimllib
A person getting elected to repeat offices doesn't mean that their elections
are dependent on each other - it's easy to argue that a party usually allows
an incumbent to run, and each election is usually between only two people, so
a high re-election rate can be modeled without making elections dependent on
each other.

What you'd need to somehow show in order to claim that elections are not
independent is that the _data from the previous election itself_ influenced
the next election.

(right? I'm no statistician so if I'm being dumb somebody please correct me)

~~~
dangoldin
But if I am elected to an office and there is a 70% chance that if I run again
I'll be reelected doesn't that mean the latter is dependent on the former.

Anyways, I did some analysis of the Presidential elections data using
wikipedia and it turns out that there were 8 presidents who ran for reelection
and lost and there were 16 presidents who had more than 1 term in office.

I'll write this up in a blog post to show the data.

------
Oompa
The real question is: How good were the betting markets?

~~~
davidmathers
<http://news.ycombinator.com/item?id=353182>

Everyone got IN wrong. MO was almost 50/50, but Nate had it going to McCain
and the markets had it going to Obama. Right now, pre-recount, McCain won MO
by 0.2%.

On the morning of the election the markets flipped IN and gave it to Obama.

~~~
llimllib
> FiveThirtyEight got Indiana right on Oct 1

from
[http://www.bayesianinvestor.com/blog/index.php/2008/11/06/in...](http://www.bayesianinvestor.com/blog/index.php/2008/11/06/intrade-
versus-fivethirtyeight/)

(OTOH, intrade got NC when 538 didn't)

~~~
davidmathers
Why is Oct 1 significant? On the night before the election Nate had IN as a
64% McCain win. Also, Nate, intrade, and betfair all predicted that Obama
would win NC. See the link above for all the numerical details.

------
time_management
538 indeed kicked ass. They also got all of the Senate races right, except for
the Steven race in Alaska (way off, apparently, but who'd have expected a
felon to win?) and possibly Franken/Coleman in Minnesota, where they gave
Franken a 52% chance.

~~~
utnick
has he open sourced his model for election predictions?

I would be curious to see if it outperformed a naive averaging of the major
polls.

~~~
robg
No, and he probably won't. His baseball projections are based on a proprietary
algorithm (PECOTA).

They did seem to do better than Pollster which is a naive averaging.

