I built a model in a couple of hours on Sunday afternoon, which simply takes all the most recent polling data, takes an average, does a quick fudge to adjust for the number of polls, and then runs 10,000 simulations to get a probability for each state. The source is on Github:
Edit: full disclosure - with all data up to 6 Nov 20120 it predicts Colorado to be a toss-up, and I manually broke the tie in favour of Democrats, based on earlier models favouring them in that state.
The success, though, isn't his specific model. It's that Silver was able to market the idea that using statistical models is better than a table full of talking heads at predicting an outcome. And that a "dead heat" doesn't mean it's a coin flip. In hindsight, it seems almost ludicrous that it hasn't gained greater traction before.
He deserves a ton of credit for championing analytics into pop/political culture, but I'm sure we'll see many more models and a lot more statisticians during the next election, some of which will be better than Silver's models (if he's still doing them). I'm all for people asking to see the data instead of the media sound bites.
To keep this HN relevant: it was the marketing that drove his startup success. The product (the model) isn't quite perfect and an excellent substitute product was created by someone else in very short order (you). But he put together a great blog, grew out his brand, and eventually saw hockey stick growth.
A lot of forecasting problems have a structure like this one did:
- There is a lot of publically available data, and simple statistical models based on that data give pretty good predictions with very little effort.
- If you're an expert with a lot of time on your hands, you can put a lot of effort into improving the forecasts by 1-2%.
- In the end, it comes down to the effective business use of your model. Nate Silver had an excellent brand around his model.
In fact, I'd wager that he knows his model is more complex than it needs to be, but part of his brand is "rogue genius with a complex model that's far too difficult for mere mortals to understand".
Doing that would make your model more mathematically complicated, but it's no doubt one of the things that Nate Silver is doing.
But besides that, Silver isn't just trying to call the election the Monday before the election, he's trying to build a model that predicts the outcome as early as possible.
A lot of the stuff he's been throwing into it, like the 'economic factors' or 'convention bounce' adjustments are about keeping the model from going off into the weeds due to predictable fluctuations in public sentiment caused by ephemeral events.
If you compare the history on the right column of the site to the "now cast" tab, which excludes those adjustments, you'll notice that the former contains fewer and more modest fluctuations than the "now cast".
> Silver isn't just trying to call the election the Monday before the election, he's trying to build a model that predicts the outcome as early as possible.
That's true, but all the praise he's now getting is for correctly predicting the outcomes in every state - a prediction that he made less than 24 hours before the election took place (in fact, he changed his prediction for Florida from REP to DEM after I'd made my predictions, using up-to-the-minute polling data).
Not what he said on The Colbert Report. "It's not hard math. You take averages then you count to 270."
On the contrary, I think his brand is 'anyone who takes some time to study statistics & probability can work this out for themselves.' His key skill is demystifying the modeling and explaining that it's not the result of some secret sauce.
Billy Beane is more a someone who utilizes new types of stats in player evaluation, then the person who creates the stats, I'm not sure who the "user" of Nate Silver's metrics are.
It is interesting to note that much of Nate's work stems from his work in Baseball.
It's more than just marketing. It's largely about distribution.
A big part of Silver's success has not been his math, but rather his excellent writing. He offers easy to consume analysis of the polls and makes a boring (to the masses) subject interesting. He can be portrayed as a stats geek when presented to the public, but really he's a great communicator who also knows his data.
Pundits and the media gain from the narrative that it was on a knife edge because it keeps people coming back for more, but in reality Romney never really picked up, let alone sustained, the sort of position in the swing states that would have led to a win. It was never comfortable for Obama and Romney wasn't out of it but the idea it was neck and neck was fairly dramatically over stated.
Not undermining what Silver did but it seems to me that the same pundits who spoke out against him made him look more impressive than it was by claiming the race was closer than it was.
According to a couple of his blog posts, going just by the most recent polls (and their margins of error) Obama had a 100% chance of winning -- but in the past, polls have sometimes been systematically misleading.
As I understand it, the poll averaging works because of the central limit theorem. But with so few data points, maybe a dozen polls at most in each state, and often half that, why does it still seem to work? I thought you'd need a few dozen data points at least.
What am I missing?
In Ohio there were 44 polls in the month preceding the election, with a mean of 51.4% Obama, 48.6% Romney. If the confidence interval each poll is 4%, then the interval for a single poll is 4% / sqrt(44) = 0.6%, which is easily enough to make a confident forecast of an Obama victory in Ohio (my model was > 98% confident in its Ohio forecast).
In Alabama, the last poll was on Aug 16th, and it was 40% Obama, 60% Romney. Even with a margin of error of 4%, this state was clearly going to go to Romney (my model had 99.9% confidence in this forecast).
This pattern is repeated for almost every state - the states with few polls are not contested, and the hotly contested states are a focus for pollsters, so there are a lot of polls there. The only really difficult states were Florida, Colorado and North Carolina, where the candidates polled so similarly that even with 30-40 polls you didn't have sufficient sample size to make strong forecasts.
That's why Nate Silver said that Romney had about a 10% chance of winning. Statistically it was much lower, but there was a very real chance of systematic polling errors.
That's a bias issue - I was only addressing sampling variation. If you wnat your model to address bias as well, of course you can build that in, given some priors on what the distribution of the bias is likely to be. I didn't bother in my model (which is why I was forecasting 98-99% chance of an Obama victory).
We could discuss potential poll bias as well, but I thought that was a bit too much for this short comment.
I made a some money buying Obama on InTrade and selling on BetFair, at a time when Obama was at 65% on one and 75% on the other, but I didn't start doing it until Monday because it was a lot of faff to set up the InTrade account.
I haven't tried to get my money out yet. I'll let you know how that goes.
1. Even though any middle schooler with a calculator could have predicted the election 2 days ago, it seems that not many of them did - at least, if you judge based on the odds for a Democrat win you could get on InTrade and BetFair 2 days ago (65% and 75% respectively).
2. One neat thing about my model is that, in principle you can run it in "historical" mode and see what its predictions would have been at any given point in the past. So I can see what probability of a DEM win I would have assigned 2 months ago, and see if that's significantly different from the forecast from a more complex model.
But what people are saying is not "Nate Silver's predictions 1 month before are spot on". They are saying "his predictions ON THE DAY OF ELECTION were spot on".
Which is not that hard to do. Does not mean Nate Silver is a hack. Means people are not measuring his success appropriately.
House races seem to be far to data sparse to be reliable.