> the election were to happen today, what is the probability of each candidate winning?
He's explicitly not doing this.
Here's how I think about it. Silver is answering the question: "how much would the polls (as an aggregate) need to differ from the final result in order for Candidate X to win/lose, conditioned on some reasonable priors?"
Taleb is pointing out that the polls could be really wrong in all sorts of ways that are impossible to predict a priori.
The whole argument is sort of pointless from an intellectual/academic perspective. It's a war of public personalities more than anything else.
It's both the case that Silver designed a good piece of software that does what it's supposed to do and also the case that Taleb's skepticism is valid. But then, that sort of skepticism of statistical models is always valid, and yet we use these models to great effect in all sorts of settings.
If I am expected to make a better prediction of the election outcome by averaging your last 10 predictions than by taking your current prediction, then your current prediction is suboptimal.
But he provides no evidence that this is the case. The 2016 election was extremely volatile due to what actually happened during it, not the modelling. There were several bombshell press events all throughout the campaign that dramatically shifted polls.
That’s exactly Taleb’s point, both in this instance and in every book Taleb has written. Those bombshells should have been included in the model. Fat tail risks. If the modeling doesn’t include the potential for dramatic unknowns that can DO happen, then the model is no good. In the Clinton/Trump election, Trump is a showman and a name caller focused on ratings, Clinton a career politician with accusations of dirt, the model should expect “big things can still happen today, tomorrow, next week” and the absence if big events yesterday or last election holds little value relative to the probability of a big event happening in the next 24 hours.
Edit- I’m getting downvoted im guessing because I said Clinton has lots of dirt and called Trump obsessive, I’ve revised the comment to be less politically accusative. I’m not concerned with the politics, just interested in the obsession with these “predictions”
You’re not getting downvoted for saying mean things about the Clintons; don’t try and be a victim here. People disagree with your take on Taleb vs. Silver.
Yea that was my reaction. maybe, maybe not. I just decided to move along. I know it’s uncouth and against the rules to comment on downvotes on HN but, for the first time for me, I seem to have been downvoted several times here on HN in the last couple weeks. I can’t shake the feeling that I’m receiving downvotes as people want to suppress different views on certain issues. My comments here are substantive and thoughtful, I read this article, have discussed Nate’s projections at length with friends and have read all Taleb’s books except his latest. My initial comment may be wrong / worthy of rebuttal, but it’s not downvote wrong. (DanG- I won’t comment on downvotes again! Sorry!)
The 538 model does have fat fails, and also, adding even fatter fails doesn't address Taleb's fundamental criticism.
> Edit- I’m getting downvoted im guessing because I said Clinton has lots of dirt and called Trump obsessive, I’ve revised the comment to be less politically accusative. I’m not concerned with the politics, just interested in the obsession with these “predictions”
> If the modeling doesn’t include the potential for dramatic unknowns that can DO happen, then the model is no good.
Why do you think it didn't? It took a major news event breaking at the exact right time (too early, and people would realize it was meaningless, too late, and it wouldn't have time to get out). And even with that, Trump barely eked out a win. That seems unlikely-but-not-impossible, which seems to match with 538's estimations.
I disagree that it’s unlikely. Highly paid, brilliant people are pitted against each other with massive stakes. It’s very likely that something bizarre happens. And there are still thousands of other things that COULD happen that didn’t, like a candidate getting removed by assassination, car wreck, illness, fatigue, enemy attack on the State, pandemic, on and on, that will have a massive impact on the state of things. The model was misleading, therefore it didn’t take into account Trump teams’ plan and the voters’ action.
I really don’t know how to value/process information like “Clinton 90% most likely to win,” and “Clinton loses in hotly contested election.” How do you get from A to B. The outcome of that election is heavily into “butterfly effect” territory, Taleb says model should never have been so confident. I would agree. Was it bad input info into the model or just a bad model because it relies on inputs subject to bias? I don’t know but the output information is certainly less valuable than the attention it’s receiving. Seems largely academic, and worthwhile, but not worthy of broad attention outside of the quant circles. (Nate Silver 538 is a popular topic around me, deeply non-quant territory, self included)
> It’s very likely that something bizarre happens. And there are still thousands of other things that COULD happen that didn’t, like a candidate getting removed by assassination, car wreck, illness, fatigue, enemy attack on the State, pandemic, on and on, that will have a massive impact on the state of things.
Yes, there are lots of things that could happen, but they're all pretty unlikely. A huge impact times a low probability doesn't affect the outcome much.
> I really don’t know how to value/process information like “Clinton 90% most likely to win,” and “Clinton loses in hotly contested election.” How do you get from A to B.
Well, it helps to not start from "Clinton 90% most likely to win"—if I remember correctly (which I may not), the final odds for Clinton were in the 65%-70% range.
> Taleb says model should never have been so confident. I would agree.
538 was not especially confident.
> the output information is certainly less valuable than the attention it’s receiving. Seems largely academic, and worthwhile, but not worthy of broad attention outside of the quant circles.
Maybe, but it's popular because it's something that people want to know.
I was reading from the article: “Take a look at FiveThirtyEight’s forecast from the 2016 presidential election, where the probability of Clinton winning peaked at 90%“
> I disagree that it’s unlikely. Highly paid, brilliant people are pitted against each other with massive stakes.
By that logic, polls would be off every year in most races. But they're not. There are lots and lots of years where polls are highly predictive, including 2008, 2010, 2012, and 2018.
> And there are still thousands of other things that COULD happen that didn’t, like a candidate getting removed by assassination, car wreck, illness, fatigue, enemy attack on the State, pandemic, on and on, that will have a massive impact on the state of things.
It's unclear to me in which direction any of those things would push voters, to be honest.
If a guy in a MAGA hat assassinated Biden while he sat in Church, maybe one thing would happen. If a black bloc assassinated Trump while he walked down a suburban street then something else might happen. It's unclear to me that either of those scenarios is particularly likely, and it's also unclear to me which of those two scenarios is more likely than the other. Even 0.01% seems high for either? And they seem equally likely? So I guess add fat tails to both sides of the distribution. Which is what 538 does.
At work, in an area much more boring and less high-stakes than election modeling, we do our best to actively track these sorts of "out-of-distribution scenarios" and have a "Conservative Human Oversight Mode" the model gets pushed into whenever something crazy is happening. That mode does get activated! For us, getting rid of the model because it fails spectacularly every year or two would be economically idiotic. IDK what Taleb would do in our case, but I do know his hedge fund failed. WRT election models, I expect 538 would probably put a big warning banner on their forecast -- or even take it down -- if one of the candidates were assassinated. Which is sort of equivalent to "monitor for out-of-distribution and switch to human mode".
> I really don’t know how to value/process information like “Clinton 90% most likely to win,” and “Clinton loses in hotly contested election.” How do you get from A to B.
Silver's model gave Clinton a 70% chance, not a 90% chance.
On the night before the 2016 election, Silver predicted how you might get from Clinton's 70% lead to a Trump win [1]: larger than average polling error and undecideds breaking heavily for Trump. Which is exactly what happened.
(Read the headline of [1] again.)
Again, divorce yourself from the emotion of politics and personalities, and just treat it as another statistical forecast. It is what it is: not omniscient or genius, but a decent piece of software that does what it's supposed to do.
> I don’t know but the output information is certainly less valuable than the attention it’s receiving.
I tend to agree. I also think Taleb's rank skepticism of these models gets more attention that it deserves. Like I said in my original post, this whole contest is an intellectually boring fight between equally big personalities. It's entertainment for politics junkies.
> but not worthy of broad attention outside of the quant circles.
The one thing I appreciate about 538 is that they do pour a ton of resources into explaining -- in lay terms -- how their model works and what their model does and doesn't account for. I'm not aware of any other mass-consumed statistical model whose authors have put so much effort into explaining for those willing to listen. Maybe weather and climate models. I appreciate this because it provides me a touch-point when explaining work stuff to non-technical stake holders who happen to listen to the 538 podcast.
Anyways, junkies will be junkies and Taleb/Silver are their dealers. Point twitter and news sites at 127.0.0.1 in your /etc/hosts and go buy a nice bottle of scotch. It's going to be a long week.
> If I am expected to make a better prediction of the election outcome by averaging your last 10 predictions than by taking your current prediction, then your current prediction is suboptimal.
That's not how poll averaging works. Or, at least, that's not how poll averaging works in the 538 model.
That's not what the OP was saying. The argument is if somebody can take Silver's model's predictions over time and produce a better next estimate than it, then Silver's model is incoherent about its own beliefs.
I came here to say exactly the same thing. Silver and 538 are explicit about not doing this. Their models do include the uncertainty of future events, and do adjust closer to 50/50 given the same polls the further from the election it is. That's why Biden's odds have been ticking upwards slightly over the past couple weeks despite polls remaining pretty stable.
So in theory, the forecast should be exactly what it appears: the percentage of the time the candidate would be expected to win given the situation they are in at the time of the forecast. So if on Oct. 14 it gives Biden a 72% chance, it means that a candidate in Biden's position, with his polling numbers, and the economic conditions and everything else that is the case on Oct. 14, would go on to win from there 72% of the time.
I agree that it appears the volatility of the 2016 forecast was larger than you would expect given a correct model behaving that way. I expect that uncertainty is better expressed in the 2020 model. However it's certainly conceivable for such a model to swing correctly. IE, there's nothing impossible about a candidate having a 90% chance of winning on X date, then having circumstances change such that they only have a 50% chance at Y date. (In fact, if you ignore the discontinuity on election day, at least 10% of the time their odds would have to pass through 50% at some point, as they would go on to lose. And of course there would also be scenarios when it would dip to that level or below and yet still win.)
I think Taleb’s point is a bit more damning — implying the software does not do what it says it would.
If all sorts of things can happen, where you predictions change widely, Taleb’s argument is that you can’t use a number like 90%. You have to include the error, which ends up saying “50%” for both before the election.
This doesn’t mean that all prediction suffers from this effect, and you have to be skeptical everywhere. I don’t think that’s a fair characterization. Taleb has won by mispriced options. His main statement is that the true probability of this particular event is much closer to 50%
> If all sorts of things can happen, where you predictions change widely, Taleb’s argument is that you can’t use a number like 90%. You have to include the error, which ends up saying “50%” for both before the election.
Taleb is, simply, wrong.
The difference from 100%/0% is the uncertainty. If new information confirms rather than contradicts prior information, that uncertainty goes down, if it contradicts it, the uncertainty goes up. You expect most of the time the general trend over time to be declining uncertainty, but when there is a period of continuing new information at odds with the prior information, you get a period of increasing uncertainty.
So if you had a very large number of Presidential election cycles with the same model, you'd expect most of them to generally trend toward greater certainty over time, but you'd expect a few of them to have extended periods of declining certainty.
Does Silver's Presidential model work that way? It's hard to tell. One, because there aren't a lot of cycles to look at, and because they aren't the exact same model, is it's not quite what you'd want to look at it to assess that. Well, 2012 and 2020 have had very much the general shape of growing certainty you'd expect, while 2016 didn't.
That's what confidence intervals are for. You cannot condense both your expectation value and uncertainty in a single percentage (it is not correct to say that the probability of an outcome is 50% if 80% of likely outcomes result in one candidate winning). Otherwise you'll end up with every 5-week weather forecast saying there's a 50% chance of rain.
> You cannot condense both your expectation value and uncertainty in a single percentage
Yes, you can.
The predicted outcome is a particular electoral vote total. The uncertainty around that predicted value is what gets you from that to a probability that the outcome will be at least 270. That's a single percentage that results from the combination of the predicted result and the uncertainty around it.
We're not in agreement here -- the solution to uncertainty in statistics is to model the uncertainty and treat them as confidence intervals, not throw your hands up and give everything a coin-flip probability. The probability isn't "realistically much closer to 50%" -- that's not how uncertainties in statistics work. It is possible that (if the uncertainty in the model was smaller) that the "real" probability is 50% but there's no way of knowing if that is the case.
That is precisely how probabilities work. If you have a simulation that gives an 80% chance of one outcome, but you only have 75% confidence that the assumptions that the simulation is based on, your actual prediction should be closer to 50%. This is not "throwing your hands up", it is simply correctly assessing the possibility of model error.
There is no such thing as giving a probability and also giving a confidence interval, for an event with a binary outcome. Everything that your prediction can say about the outcome can be said with a single number between 0 and 1. To give an answer like "80% plus or minus 15%" just means you haven't finished calculating.
Right, but I think Silver is saying that averaging high quality polls produces a high level of certainty.
The only reason Trump has a 10% chance and not a 0% chance is that there's a 10% chance the polls are off by far more than they were in 2016 because Biden has consistently polled ahead of Trump in states totalling 270+ EV, plus he's polling at historically high levels nationally for a challenger and facing an incumbent with historically low approval ratings.
I think Taleb is saying "there's so much uncertainty it should be a 50/50 race" and Silver is saying "there's uncertainty, but it's equally plausible in either direction, so let's go off of polling data and historical examples which looks like a 90/10 race".
In the end they both should just bet their own predictions and let the winner emerge over time.
It’s impossible to be so certain of polls because certain demographics may change how forthcoming they are with pollsters over time.
Systemic bias is totally possible and can affect just one side, in a huge variety of ways, changing over time.
This is just one example of a fat tail event you can’t predict but which you can expect, and given Nate has only called a couple elections, there’s no way he can have 90% confidence a model of the polls will be accurate.
Models are backtested against the historical record where hundreds or thousands of elections (federal, state and local) can be used as training data.
For example, a weighted average of polls has Biden up ~5% in PA. In state-wide elections, I wouldn't be surprised if candidates with that sized lead in PA (and states with similar size and socioeconomic demographics) end up winning about 85% of the time, which is what the 538 model has Biden's chances at in PA. That 15% uncertainty accounts for the bias you describe. If polling was perfect and there was no bias, a ~5% polling lead (and exceeding the margin of error) would be a 100% guaranteed victory.
It would be great to see Taleb's forecast of federal and state elections so he has "skin in the game" as he likes to say. Assume you get points based on your % confidence, so if you predict there's a 55% chance of a particular outcome and you're right you get 55pts, and if you're wrong you lose 55pts. If Taleb pegs every race at 50/50 or 55/45 because you "can't predict elections", I doubt his score will be higher than Silver's. You'll end up with silly predictions like 40% chance of Trump winning Washington DC or 30% chance of Biden winning Wyoming, when in reality each have less than 1% chance of doing so.
When you make reasonable, weighted predictions based on state polling, you see that Biden has leads outside the margin of error in states totaling 270 EV. So, the 10% chance Silver gives Trump is saying, hey there's a chance that the polls could be extremely biased in all 7 swing states in Trump's favor. But, if Trump loses even 1 of those swing states, there are so many safe D electoral votes that Biden will win.
You totally ignored the point. Overfitting to previous results doesn’t account for the fact that the facts on the ground are changing. I mean this year alone we have a once in a century pandemic, can you fit that to any other election?
That's why you test predictions. Silver says 10% is enough uncertainty, Taleb says it's not. From the article: "Premise 1: If you give a probability, you must be willing to wager on it". Unless they're willing to publish and make a friendly bet on their predictions, it's just bloviating.
Every day/year/election is unique, but only to a certain magnitude. Some elections have had active wars, some health crises, some criminal scandals, some terrorist attacks, etc. In the end, the polls attempt to account for those uncertainties and historically that's been a much better predictor than a coin flip.
For example, Washington DC has never voted Republican since it won electoral votes in 1962. Polls have Biden winning 80-90% of the vote there. Even Taleb would agree there's a near 100% chance of Biden winning there, pandemic or not. Silver's methodology is just extrapolating that out to each state. When it gets to some tipping point states it's closer to 60/40 Biden or 70/30.
If Taleb wants to create his own forecast, it would be interesting to track it's performance over time. I have a hard time believing that model closer to a coin-flip is going to outperform weighted polling data and historical precedent over multiple predictions.
Right, but the original point was that Biden's lead was likely large enough in enough swing states to overcome a 2016-style polling bias, which looks like has occurred in some states. That's why it was a 10% chance for Trump, because he had to not just have a large polling error go his way, but also in enough swing states to total 270.
A landslide Biden victory was as probable as a narrow Biden victory. And both were more likely than a narrow Trump victory.
But that’s simply not true. Because Florida and Texas alone were so far off, 10% odds was definitely “way” wrong. Nate’s model predicted that if Florida went Trump the odds were at 33% or so. Texas moreso, etc. Since Florida went incredibly strongly Trump, we can say with certainty he was completely wrong on the 10% chance. That’s the point: he was way overconfident his past model would fit, it should have been closer to a slightly advantage, which would read something like 60/40 at best.
> Since Florida went incredibly strongly Trump, we can say with certainty he was completely wrong on the 10% chance.
The model generated 40k scenarios, and the scenario you're describing is one of them. The most extreme scenarios have Trump winning all swing states by a few %, which doesn't look likely. So what's playing out is not wildly outside the predictions by any means.
In other words, if a 10% chance happens, it doesn't mean the 10% prediction was wrong.
The only way to prove a prediction was wrong is to bet against it over time with your own predictions. As Silver has shown there's a huge appetite for election / sports predictions. Anyone able to beat him over time would have enormous income potential.
For example, if you thought Trump had at least a 40% chance to win, you'd have a great betting opportunity in the market that had Trump in the 25-33% range. You could have bought in then and sold when Trump's chance peaked at about 50-60%. If you arbitrage those mistakes you've identified in the market, overtime you could become very wealthy.
Until then, it's just pundits pontificating after the results are known without putting any money or reputation on the line beforehand, similar to a casual sports fan late in the 4th quarter: "of course the 49ers were going to blow their lead -- I just knew it!"
His probability of Florida being won by that much was basically 0. But it was actually almost the opposite, in the real world the probability of Biden winning was almost 0.
The polls were so far off systematically in one direction - it doesn’t mean he was right about the 10% chance, because that would mean there was equally likely a chance they were all off in the other direction, which is obviously just wrong. Polling missed huge groups of voters opinions - it was just wrong.
We have numbers now that show that Trump voters also aren’t forthcoming on their vote. Probably on purpose as an effect of last election and being spiteful towards pollsters in general. I actually had this as a strong prior, so my model predicted this would be close, but of course if you “just go by the polls” you’re essentially trusting a flawed system that is gameable. Anyone who trusts the polls is essentially a fool next time, they have been shown to be incorrect now twice in large amounts and I wouldn’t doubt that the “meme” the right has started to purposely deceive them continues even after Trump.
You are absolutely wrong on your 49ers analogy and the 10% covering this. Florida disproves it, as do the systemic nature of being off. If they weren’t off systematically, you’d have expected the polls to get Florida wrong one direction but other states wrong in a different direction. That would prove the model, but if every single state was multiple points off all in the same direction, you don’t get to call margin of error. Your bell curve was shifted one direction, it wasn’t a case of the curve being right but the dice rolled in the tail.
Yes, some states had big polling errors, while others like GA and AZ were reasonably accurate. But the model doesn't care if the polls are off in FL by 0.1% or 10% because it's a winner take all scenario. So while large polling errors are surprising, they have the same exact impact as a small polling error in a close race, which isn't surprising. But people like to cherry pick and say "how did you get X so wrong?" rather than "how did you get X/10 so right?" In fact, Trump might only end up winning 3 "upsets": FL, NC and ME-2, but they were all under 2% difference in the polls, so no one is shocked to see Trump win there. WI and MI would have been much bigger upsets, but that didn't end up happening and a 20k vote win is as good as a 2M vote win in the EC.
The 10% odds come from the path to 270. Here are the 9 states / districts that polled within 2% pre-election (considered toss ups): NE-2, AZ, FL, NC, ME-2, GA, OH, IA, TX. Even if Trump won all of those, he wouldn't get to 270. Using 50/50 odds for all 9 toss ups, Trump only gets a clean sweep 11% of the time. He would then still need another state like PA, WI or MI to get to 270. That's why Silver said that Biden's chances are so high, specifically because Biden could overcome a 2016-level polling error in every toss-up state and still win. In fact we saw 2016-level polling errors in some battleground states, but not all.
Another way to view it, is you have a 1/6 (16%) chance of rolling a 1 on a 6-sided die. Even if all 53 states and districts were projected at 84% confidence, and rolling a 1 meant "upset" and rolling 2-6 meant "polls were accurate", you'd end up with 8-9 states resulting in upsets! 84% confidence still means a lot of surprises. Their final FL prediction was only 69% confidence. Also, confidence intervals aren't linear, meaning that a 95% confidence anticipates half as many upsets as a 90% confidence interval. That +5% means -50% upsets, so 69 or 84% confidence is really not as sure a thing as it might sound.
So, yes it's worth re-examining why polling was very accurate in AZ & GA but very off in TX & FL (specifically Latino's in Miami). But a 10% chance was not crazy IMO given how easy it was for Biden to overcome even huge polling / vote disparities in multiple states and still win.
Again, those who disagree are free to get great odds in the betting markets.
Bringing in betting markets has nothing to do with the point, but you are right on one thing: if you had bet along with Nate’s odds in a simulation you’d lose money in a Monte Carlo simulator based on current results.
The end story is the polls weren’t a reliable measure of the vote. Their predictive ability was low, and even playing by the rules of Nate’s model you have to admit he was off by some 30% at minimum.
Nate’s model has the average PA poll at Biden +6 (!!) where the final result looks to be under 1 (and could go either way). It’s simply impossible to argue that was an accurate model.
> Nate’s model has the average PA poll at Biden +6 (!!) where the final result looks to be under 1 (and could go either way). It’s simply impossible to argue that was an accurate model.
538's PA indicator ended at Biden +4.7%. But the overall 538 model doesn't care if a candidate wins PA by +0.7% or +4.7%. The 538 model is designed to do 1 thing: predict the EC winner. It uses state-wide predictions as indicators, but you can't judge a model based on the performance of a single low-level indicator, you judge it by it's final prediction, which is Biden 90% / Trump 10% to win the EC.
But yes, the lower you go in the model the higher variance you'll see:
Level 1 -> EC Winner (538's focus -- could end up 100% correct)
Level 2 -> State Winners (low variance -- could end up 90-95% correct)
Level 3 -> State Polling Averages (moderate variance & could be 75-90% within margin of error)
Level 4 -> Individual Polls (high variance & could be 60-80% within margin of error)
If you click on any individual state (like PA: https://projects.fivethirtyeight.com/2020-election-forecast/...), you can see the state-wide vote projections. Biden was projected to earn between 49 and 55% of PA's vote with Trump expected to get between 45 and 50%. The final outcome looks certain to fall within those expected ranges. Again, the margin really doesn't matter, just that the idea of Trump getting above 50% and Biden getting below 50% seemed pretty difficult, hence Trump's low (but not impossible) 16% chance of carrying the state.
I like the way their "Winding Path to Victory" chart (https://projects.fivethirtyeight.com/2020-election-forecast) explains their model. It's basically saying: "Trump's path to 270 likely goes through PA, NE-2, AZ, FL, NC, ME-2, GA, OH, IA & TX, where he has a fighting chance of winning any of those, but a very low chance at winning all of them." On the flip side, "Biden's path to 270 goes through Wisconsin, Michigan, Nevada and Pennsylvania, where he's likely to win all of them". That seems like a pretty reasonable EC prediction to me.
If Silver was promoting the model as being able to accurately predict state vote percentages, I'd agree that it's underwhelming. But when you average all of that variability and uncertainty, you can get a pretty reasonable EC winner prediction IMO, at least better than others I've seen that had Biden closer to 95 or 98% to win the EC, or betting markets that had him as low as 60%.
If every single range it’s at at or below the lower bound for one candidate and at or just past the upper bound for the other, systematically, that’s the definition of a flawed model. You seem to breeze past every point I make, I suppose there’s no educating the unwilling so I’ll leave this thread. The bell curve was systemically off, the polls were systemically off. They actually missed the bounds in many races entirely. The betting markets were much more accurate.
> If every single range it’s at at or below the lower bound for one candidate and at or just past the upper bound for the other, systematically, that’s the definition of a flawed model.
Yes, I agree that individual low-level polls for 2 elections in a row have underrepresented Trump's actual support. But, 538's model addresses exactly your point: low-level indicators like state polling can potentially be systematically flawed and biased. By simulating those unreliable low-level indicators through 40k scenarios, the scenario where Biden systematically underperforms biased pre-election polls in key swing states but Biden still wins (as we're seeing) ends up being a very reasonable outcome and the large reason why Biden was favored overall. Trump needed to outperform the polls in 7 states to win, and that just didn't happen.
And the polls weren't off dramatically in every single race. Here are some of 538's last predictions in key states:
AZ: 50.7% Biden / 48.1% Trump
NE-2: 51% Biden / 47.8% Trump
GA: 50.1% Biden / 49.2% Trump
PA: 52% Biden / 47.3% Trump
NC: 50.5% Biden / 48.8% Trump
Each poll has a margin of error, usually +/-2.5% or more. So, 2-3% swings on Election day are completely normal. NC was off by a tiny margin and enough to flip the state to Trump, while the others were also off by tiny margins, but not enough to flip the state.
Sizable polling swings are not uncommon and historically have gone in both directions, but the model was more concerned about just how narrow Trump's path to 270 was, more so than a routine or even historical polling error.
> The betting markets were much more accurate.
Everyone who bet on Trump at a 30/40% chance to win is about to lose their bet. I definitely wouldn't have taken those odds for Trump to run the table in 7 straight must-win swing states all polling as a coin-flip and within the margin of error. In a completely random scenario Trump would have only had a 14% chance of pulling that off.
If you want a more data-heavy version go to the graphs, then click on the "electoral votes" tab. The shaded part is the 80% confidence interval of each prediction. This is the historical graph form of the "every outcome in our simulations" chart underneath the dot chart. It should be noted that the dot chart at the top of the page does kind of outline the confidence interval (giving a realistic example of what kinds of outcomes might happen) but is obviously geared towards lay-people who don't know what a confidence interval is.
> We simulate the election 40,000 times to see who wins most often. The sample of 100 outcomes below gives you a good idea of the range of scenarios our model thinks is possible.
I think it would be better if the confidence intervals were better signposted, but it's possible that only the electoral college win margins have solid confidence intervals (I'll admit I don't know enough statistics to know if you could trivially transform the confidence intervals of the electoral college votes to win percentage). The senate and house predictions explicitly show the confidence interval in their main graphs.
I think if you see 90% with 80% confidence interval, you can intuit that he isn't counting uncertainty in the tails.
Consider, Would you be comfortable making a bet with someone, on a 90/10 payoff? This, I think is Taleb's central point. Silver's algorithm is not counting the uncertainty correctly.
Well, he's right in the sense that there's close to a 50% chance of a swing in either direction (blunders, dirt, health issues, polling errors, etc).
However, when one candidate is up 8+% (even months out) against an incumbent with 4 years of net negative approval ratings, to call that a toss up is like saying the frontrunner is substantially more likely to blunder than the underdog.
As Silver points out however, a Trump win is just as likely as the largest Democratic landslide victory in modern history. The polls could move either way, but the only way they move enough for Trump to win is a larger than 2016 polling error in his favor in all 7 battleground states, which doesn't sound like a 50% chance at all.
To say Biden is up by 8 points is missing a lot of microstructure. In every state Trump needs except PA, he’s within about 2 points. In PA, the critical state, he’s down by about 4.5 points, but Biden’s lead has been diminishing rapidly in the past two weeks.
These polls also rely on a turnout model. There’s no way to calibrate these models for a pandemic election where some people may be more scared to vote than others.
I give Biden the edge but it’s nowhere near 90/10.
I don't know where you got your numbers from. I got them from Tanenbaums site https://www.electoral-vote.com/evp2020/Pres/Graphs/all.html and this shows that Trump would win all swing states and thus the election, if there was not the surprising swing in Texas from Rep to Dem lately, which broke Trump. You cannot really trust Tanenbaums summary as heavy Dem fanboy, which was wrong last time and is wrong this time, but the raw polling numbers are showing a clear picture. Early votes missed the Hunter Biden scandal, and Texas was champion in early votes.
Taleb made a career from stating the obvious and making it appear as a smart discovery. I've read some of his writings and have never found anything really new.
My favorite story when I hear someone dismiss an argument saying 'that is obvious':
Lazarsfeld was writing about “The American Soldier”, a recently published study of over 600,000 servicemen, conducted by the research branch of the war department during and immediately after the second world war. To make his point, Lazarsfeld listed six findings that he claimed were representative of the report. Take number two: '“Men from rural backgrounds were usually in better spirits during their Army life than soldiers from city backgrounds.”
“Aha,” says Lazarsfeld’s imagined reader, “that makes perfect sense. Rural men in the 1940s were accustomed to harsher living standards and more physical labour than city men, so naturally they had an easier time adjusting. Why did we need such a vast and expensive study to tell me what I already knew?” Why indeed.
But Lazarsfeld then reveals the truth: all six of the “findings” were in fact the exact opposite of what the study found. It was city men, not rural men, who were happier during their army life. Of course, had the reader been told the real answers in the first place, they could just as easily have reconciled them with other things they already thought they knew: “City men are more used to working in crowded conditions and in corporations, with chains of command, strict standards of clothing, etiquette, and so on. That’s obvious!” But this is exactly the point Lazarsfeld was making. When every answer and its opposite appears equally obvious then, as he put it, “something is wrong with the entire argument of ‘obviousness'”
Oh yes, I hate this kind of thing so much. People have so many pseudoscientific explanations and theories to support whatever random products and practices they like.
I call these “huh, that makes sense” explanations, since that’s often what people say after hearing them. You even used the magic three words above.
The related phenomenon in biology is called the “just so” fallacy; e.g. “X species evolved Y just so that they could do Z”. It’s a comforting story, but it will often lead you astray.
The same is true of pg, but his discoveries seem obvious only in hindsight.
(I’m objecting to “the discoveries are obvious” as a putdown. The simplest discoveries are some of the hardest. Though I doubt this matters for political pundit analysis.)
I have no horse in this race, but with the rarity of a unicorn, it could be what amounts to luck.
Out of billion people flipping a coin 30 times, a few of them may guess correctly all 30 times; but the books they write on coin flip guessing amount to meaningless drivel.
My point is that success or failure in such a complex, edge-case, luck-based world may have very little to do with knowledge.
PG, if you’re reading this, of course I mean no disrespect and I know little of the topic and nothing of this person at hand except that he seems to have run a failed hedge fund.
I don't know much about Taleb, but one problem I've always had with Silver is that he's very smug about this models but they often enough don't work out. Then when they don't work out, instead of reflecting on his models to improve them (at least publicly,) he just tells people they don't understand probability or his models. His sports models are -especially- bad; no better than a coin flip in some instances. I know there are a ton of Silver fanboys, but surely you can agree he has a pretty big ego, which always makes me take him with a grain of salt. When it comes to politics his former work with Obama's campaign and DailyKos also make me question if he can be truly unbiased in his predictions, but that's a separate issue.
Whose models are better? There have been years where Nate Silver's model has called every single one of the 50 states correct as far as which POTUS candidate they'd vote for. See: https://mashable.com/2012/11/07/nate-silver-wins/
He's not perfect (for instance, I think calling the 2016 model too certain early on is valid and I think some other people have basically equivalently-good models, but are less well known), but if confidence is smugness, then he has a right to be.
> Whose models are better? There have been years where Nate Silver's model has called every single one of the 50 states correct as far as which POTUS candidate they'd vote for.
Well, this certainly isn't how you'd go about evaluating model quality. Calling which candidate a state will vote for is not a good task; it's too discrete to be a good evaluator. For the same reason, no political model produces this output. You predict the vote share given to each candidate -- a nice continuous variable -- you don't predict who will win. Calling the state is a gimmick on top of that (since victory is a function of vote share, it's easy to degrade your model's output into victory predictions) which serves only to make the statistics worse.
Nate's probabilities are based on the electoral college, not the popular vote, so calling which way individual states goes is by definition necessary for a good evaluation.
Consider states like Maine who specifically allocate their electors by vote share rather than in a block to the winner.
Again, calling individual states is neither necessary for a good evaluation nor even helpful to it. It is a further processing step that runs on top of a finished prediction. If you have a vote share distribution, you can convert that into victor probabilities with basic algebra. You cannot operate in the reverse direction, and you cannot develop a good model that directly produces victor probabilities as its output.
To be perfectly honest, I think the Obama elections were a unique point in politics where models like Nate's worked a lot better. Realistically there weren't many up for grabs states, which means you could guess 8-12 states right and run the table. A lot has a evolved since then, especially in the social media landscape and the accelerated death of answering phones, which live polls rely so heavily on. Maybe Silver will hit this election out of the park, I don't know, but I would err on the side that he gets a number of states wrong.
It’s worth pointing out that Nate was mad at this result because it meant his implied uncertainty for each state was wrong. It meant his model was sandbagged to some degree.
What is your criteria for saying "his models don't work out"? If he says "candidate A has a 90% chance of winning", and then they don't win, that doesn't mean his model was wrong. In the same vain, if the 90% candidate won, it also doesn't mean his model was correct either.
You have to look at a lot of his predictions, and see if the percentages match up.
When you say his models don't work out, are you saying you have done that analysis and found that his percentages don't match the results? If so, I would love to see that analysis.
Put another way, if trump wins...again, do you take that as part of the expected outcomes or evidence your distribution is wrong?
If he said something was a 10% likelihood, and it happened three times in a row, would you still think the prior is that the event was actually 10%, or that the estimate of a 10% likelihood was off?
Pretty sure those are all based on sports or at least baseball. Question isn’t can silver predict baseball. We’ve know that since his time at baseball prospectus in 2007!
> he's very smug about this models but they often enough don't work out.
Taking all of the Presidential forecasts together, there's too few to really generalize about that. Taking the down allot results though, the odds have been pretty close to what he's forecast.
So I'm not sure what your basis is for saying they “often enough don't work out”.
> His sports models are -especially- bad
Maybe, but his political models are especially good.
> I know there are a ton of Silver fanboys, but surely you can agree he has a pretty big ego
He's never really come off that way to me, not that whether or not he has a big ego has any bearing on the quality of his models.
I frankly don't care if someone is smug or not; I want to know who builds the best models. If Silver's models are often wrong, who would you recommend we look into instead?
Silver might have the best models, but he isn't infallible, which a sizable number of his followers seem to believe...and I think that only feeds into his ego and is probably damaging to his models.
He recently tweeted that the only way Trump can win is through polling error or if Trump steals the election. Do you think that sounds like an unbiased person? His model can't be the reason why he was wrong, it had to be the input data! Imagine saying that to your boss.
His model is based off of point data. Ultimately what his model encodes is the polls+a degree of uncertainty of the polls are completely wrong. If you looked at just the polls, there was no way for trump to win, even assuming a polling error of the same variety as 2016, which appears to be about the situation we are in now.
So no, there's nothing wrong with his tweet. Otherwise, you should be able to explain how trump could win without a polling error. Which states that trump was polling behind by 5-7 points he'd win, and how he's win them without a polling error.
> one problem I've always had with Silver is that he's very smug about this models but they often enough don't work out.
What does it mean for a probabilistic prediction to "not work out"? If I tell you that the odds of flipping 2 heads in a row with a fair coin is only 25%, and it happens, did my model "not work out"?
> instead of reflecting on his models to improve them (at least publicly,) he just tells people they don't understand probability or his models.
I really don't intend this in any sort of rude way, but I think you might be interpreting his response as dismissive because it's accurate...
It's difficult to say a model "doesn't work out" based on a single event, especially when it puts a significant probability on the less-likely option. For example, 538's 2016 forecast gave Trump a roughly 1 in 3 chance of winning: that's _higher_ than the odds of flipping 2 heads in a row.
Measuring the quality of a model is more complicated: one way would be applying the model to the same outcome multiple times, but this isn't possible for single-event forecasts. Another way to see how calibrated your predictions are is to aggregate multiple predictions and see how often they line up with reality: an event predicted to occur with X% probability should occur X% of the time.
Lucky for us, 538 has done exactly this analysis[1]! Naturally, it's internal, so take it with a grain of salt, but it looks like their predictions are fairly well-calibrated.
> I believe Nate Silver is answering a subtly different question with his election forecasts. Each data point that Silver produces is answering the question: if the election were to happen today, what is the probability of each candidate winning?
As others have pointed out, this is definitely not what Nate's model was doing. In fact, he did have a version of the model that did exactly this, the so called "Now-Cast" and as expected it was even more volatile than the real model:
Right, the main model intends, in fact, to be predicting the probability of various outcomes when the election is actually held, that's why there is accounting for non-poll fundamentals (economy and other things) that are observed to influence poll changes over time that phase out as time runs out for them to move the polls.
I've recently come to believe that Taleb really doesn't understand probability. I know this sounds like some insane flippant remark, but I saw a video where he talked about the election forecast as a martingale bounded between 0 and 1 and then made some claims about variance with these bounds to back up his claim against Silver.
For his claim to make sense you would have to view probabilities as linear, which is not simply not how probabilities work. An event that has a 0.01 chance of happening is an order of magnitude more likely than one that has as 0.001 chance. However the different between and 0.5 and 0.501 is essentially negligible (hence probability is not linear because 0.001 does not mean the same everywhere).
I've generally given Taleb a pass as a bit of a crank that doesn't care about the philosophical interpretation of probability, but after seeing him misunderstand probability in such a major way I've started writing him off entirely.
edit: I should add I'm not particularly more convinced that
Silver really understands all that much either, just that this is all two very loud people that don't know much. Neither should be taken seriously. This isn't Bradly Efron arguing with Andrew Gelman.
There is a very clear flaw in your mathematical reasoning here, let's walk through it.
Consider your second probability comparison: the difference between a probability of 0.500 and 0.501. You have this correct, the difference is 0.001.
Now reconsider your first statement: that a probability of 0.010 is an order of magnitude more likely than a probability of 0.001. This is true, but is completely irrelevant because it is a completely different comparison. A more appropriate comparison would be to compare probabilities of 0.010 and 0.011. The difference of these is again 0.001.
You compared proverbial apples to oranges, and you cannot trust the conclusions you make from invalid comparisons. A difference in probability of 0.001 does mean the same thing no matter what the starting probability is. Probability is "linear" in this way.
Now that we know this to be true, I would encourage you to rethink your position on Nassim Taleb. He understands non-linearity quite well, in fact I would argue that his entire purpose on Earth (at least in his eyes) is to remind people about non-linearity as a concept. It comes across in all of his books and most of his public appearances.
On the matter of probability, I trust him very deeply. Whether or not I trust him is also inconsequential, because he has serious mathematical literacy and you can evaluate his propositions for yourself.
The difference in those probabilities is the same. In each case, the event with the greater probability is 1-in-a-thousand times more likely to occur than the lesser.
Let's walk through an example with these numbers. Each case has 2 probabilities and we'll assign them each to the possible outcomes of a coin flip.
Case 1:
The probability of heads is 0 and the probability of tails is 0.001. We flip the coin 1,000 times. The ideal outcome is that we see 0 heads and 1 tails.
Case 2:
The probability of heads is 0.499 and the probability of tails is 0.500. We flip the coin 1,000 times. The ideal outcome is that we see 499 heads and 500 tails.
(You'll notice that the outcomes don't add up to 1,000 in either case, and that's because the probabilities don't add up to 1. That means there is actually a third possible outcome, but that can be ignored for the purpose of this example.)
They each differ by a result of 1 result per 1,000 trials. This is because probability is simply the likelihood of an event occurring. There is no super-linear pattern going on here. It is no harder to get from 0.7 to 0.9 probability than it is to get from 0.2 to 0.4. It is just a measure of likelihood.
What is causing people to misunderstand this? I wonder if you're thinking about probability distributions vs. probability itself?
Adding and subtracting operations are natural in the log scale (think probability of multiple independent events), but not in the linear scale.
The interpretation of a linear difference of 0.001 in probability depends on the two numbers you're subtracting, which is not the case in log scale.
OP pointed out intuitively why adding and subtracting probabilities in the linear scale don't make sense (examples would be comparing 0.001 and 0 versus 0.500 and 0.499). It's the fold changes that matter when interpreting two different probabilities (e.g. making decisions under uncertainty), not the linear difference.
The proper interpretation of your Case 1 and Case 2 should be the ratio of heads and tails as you flip coins, not the difference in the absolute number of heads that show up in the two cases.
> The interpretation of a linear difference of 0.001 in probability depends on the two numbers you're subtracting
No it does not and I have shown why. A difference of 0.001 in probability means that the two events differ in their likelihood to occur by 1 in 1,000 trials.
> OP pointed out intuitively why adding and subtracting probabilities in the linear scale don't make sense (examples would be comparing 0.001 and 1 versus 0.500 and 0.499)
This is only intuitive if you have a misunderstanding of probability.
> The proper interpretation of your Case 1 and Case 2 should be the ratio of heads and tails as you flip coins
Clearly we have differences in what is intuitive and what is not.
I think OP is trying to say that people who interpret Case 1 and Case 2 in terms of absolute difference in number of heads rather than the ratio of heads to tails have questionable foundations in probability.
I probably won't be able to change your intuition, but let me try to show why thinking in terms of difference in log-odds is grounded on nice mathematical properties.
When thinking about how to compare two probabilities (i.e. a function that takes two numbers between 0 and 1, and outputting a real number), one thing that is nice to have is the function outputting opposite numbers for the complementary event.
A comparison function that fulfills this property (and other nice properties) is the difference in the log-odds.
For example, using this comparison for p1=0.002, p2=0.001 gives about 0.694. What's nice is that when p1=(1-0.002) and p2=(1-0.001) you get -0.694. So indeed this comparison function is giving expected results.
Using this comparison function to look at 0.500 and 0.499 gives 0.004, which is less than 0.694. This suggests that using this mathematical framework to compare probablities, (0.500, 0.499) is similar while (0.002, 0.001) is more different.
Indeed if you take the extreme case of 0.001 and 0 (I made a typo above) you see that the comparison function outputs infinity. This also makes sense when making decisions because when comparing an event that is impossible compared to an event that is merely improbable, the actual probability no longer matters for the latter event (think of your coin flipping case again with p1=0, and letting the other p2 be any number between 0 and 1).
Intuition is definitely subjective, and that is ok. I do respect different intuitions while trying to understand them. And the best thing for that is doing what you did, which is provide a more clear example.
Unfortunately, the example makes it more clear to me that probability differences are linear.
Your "difference of log-odds" function is a qualitative function that you've constructed to operate on probabilities. All you've done is take the log of numbers and compare them, and of course you're going to take on the properties of logarithms themselves: specifically that the difference between log(x2) and log(x1) decreases as x2 and x1 increase in value. That's just how logarithms work.
This says nothing about the relationship between probabilities.
You can't just apply any operation on any values and draw conclusions about it with understanding the interpretations of the values. Probabilities are likelihoods, not raw quantities. So the domain of likelihood has to be considered when analyzing probability values.
In the domain of likelihood, when probabilities are independent (i.e. flipping a coin once does not influence the next coin flip) you can interpret differences in probability as linear.
Based on what I've read of Taleb, if you think he doesn't understand some patently obvious facts about probability, it's going to take more than saying that you watched a video and didn't think it made sense for me to give your claim any weight.
Edit: as pointed out in the replies, I was wrong that the model was "explicitly" a now cast, and I would like future readers of my comment to know that without deleting things. I would still like to note that there was clearly a change in the 2020 model to increase variance as a function of time to the election when compared to the 2016 model. I think this is substantively the same.
Some of the top comments are making the same mistake, probably because they aren't aware of this twitter beef history (and I envy them), so I'm putting this in a separate comment.
These comments are all pointing out that Taleb's criticism is of something that Silver's models don't do:
> the election were to happen today, what is the probability of each candidate winning?
What people seem to be forgetting, is that Silver's 2016 model explicitly did this. Taleb made his criticisms starting in 2018 (even published an options pricing paper on the subject) then Silver changed his model for the 2020 cycle so that it no longer was an "if the election were today" model.
I can't say for sure whether or not Silver changed his model because of Taleb, buuuuut he deserves at least a little credit here. Yes, I know Taleb is a colossal asshole people love to hate, but he's usually right about these sorts of things.
This isn't true. He had both a "nowcast" (if the election were held today) and a prediction of the final result. The only change in that regard between 2016 and now was to stop publishing the nowcast.
Edit: Even after your edit I don't think the claim is true. The probabilities in this election didn't change much because the campaign season was comparatively uneventful and the polling didn't change much, not because of a fundamental change in the model. 2016 had tons of bombshell events which is what caused the chaos in the prediction model.
One of 538’s models did that in 2016, not all of them. They made it very clear that they did not recommend the “nowcast” because it tended to confuse people.
> What people seem to be forgetting, is that Silver's 2016 model explicitly did this.
Well, there was a sub model (the “nowcast”) in the 2016 (and, IIRC, earlier) forecasts that did this, but the headline model did not, it's always been a forecast of the election day results.
> I can't say for sure whether or not Silver changed his model because of Taleb, buuuuut he deserves at least a little credit here.
The change you are describing literally did not happen. The headline model was never an “if the election were held today model”, so it cannot have changed from that to something else. Giving Taleb credit for that change is as sensible as giving Trump credit for ending the COVID-19 pandemic.
I don’t understand why there’s so much attempted theory when we can just directly compare Nate’s model against a market, in this case, the presidential prediction market. What you found for 2016 was that the market both had about the same volatility over time as Nate’s model AND was more wrong about the end result, thus showing that the variances in the model were likely driven by actual uncertainty, not spurious over confidence.
If volume and stakes are low then sure, they're not too reliable.
If you don't think prediction markets aren't great predictors you should make bets against that market. If you're a better predictor than the market you should make good money. In practice most people can't outguess the market. Some experts can and do make good money.
The reason markets are a good predictor because as soon as an expert sees the market make a bad prediction, they'll make a bet against the market. This in turn will move the price towards the true price. Assuming there are enough experts out there with large enough pockets (which is true for popular bets) then the market will quickly reflect the experts' view of events which is by definition very hard to beat (you have to be more expert than the experts).
But the question that was being asked was whether there was an arbitrage opportunity available which, regardless of whether the market is manipulated or not, has an empirical answer, not merely a theoretical one.
So... Taleb is a like purist programmer (pick your favorite paradigm) and Silver is more like a pragmatist programmer that uses a "hybrid" language model?
There's values for both types in programmerdom, is there not the same for both in the world of statisticians?
> Taleb insists that Clinton never should’ve received a probability of winning of 90%. Even if polls were heavily in favor of Clinton at the time, he says Silver should’ve taken into account the uncertainty that polls would change over the next few months leading up to the election, or the possibility of major news breaking.
Silver's model fairly explicitly does address both the likely direction and the uncertainty in future poll movements, based on historical evidence. The 2016 model results were more volatile than you'd like to see, but it's at least plausible that's because 2016 was an unusual cycle out in the statistical tail in behavior. The 2020 forecast has behaved in a more conventional manner with mild noisy ups and downs early on, and then a smooth progression of greater certainty, the 2012 projection was similar. While this obviously is way to few to generalize from (but better than try to assess from the behavior of 2016 alone), it's consistent with the idea that the model usually behaves in the way you'd expect a forecasting model to behave, and 2016 was just a collision of unlikely events.
FTA, “ I believe Nate Silver is answering a subtly different question with his election forecasts. Each data point that Silver produces is answering the question: if the election were to happen today, what is the probability of each candidate winning?”
If that’s true, then Nate Silver just does poll analysis, not predictions. Predictions should be on the day of the election, not the day you’re reading their blog post.
The statement you quoted from the article (which is basically Taleb's criticism of Nate Silver) is not correct, the FiveThirtyEight prediction models are explicitly not answering the question "if the election was held today, what would happen". There are models like that on FiveThirtyEight, but the main question being answered is who is likely to win the actual election on the day it is held.
> There were models like that, they didn’t bring them back this year.
They weren't the headline forecast, but, yes, the NowCast existed (and still does, in fact, though it's not visible, just like the polls-only forecast model isn't visible any more; Silver references them periodically.)
But it's inaccurate to criticize the headline forecast based on the idea that it is trying to do what the NowCast does, and it is dishonest to criticize something other than the main forecast and say it is a criticism of the forecast, but not specify the alternative forecast you are actually criticizing.
* make predictions with a range
* or make predictions with a certainty coefficient (p value, etc.)
Predicting a single value without either is not a truly quantified "prediction."
It's weird now not getting that can lead to confusion. I agree with others in that those who don't communicate both values of information aren't being 100% clear.
This article is mischaracterizing Silver's model. The author writes:
"if the election were to happen today, what is the probability of each candidate winning?"
This is not at all what Silver's model does. Silver writes and makes clear that there is more variance in the model a few months out than a few days out. (In other words - Biden may be 90% now but four months ago if the polls were identical he would be at less than 90% since there would be more time for voter change and real world events.)
I honestly don't even get what Taleb is truly arguing here. He is using a lot of statistical arguments but ultimately Silver is making a model to predict an outcome. It is like a weather forecast.
I may say that four months from now the chance it rains in Los Angeles is 2%. However, the day before with more info I may be able to say it will rain 90% of the time. There isn't anything wrong with this, even though Taleb seems to suggest this wrong (from my read)
"if the election were to happen today, what is the probability of each candidate winning?"
This was a correct characterization of Nate Silver's model at the time Nassim Taleb made his critique. Since then, Silver has changed what the probabilities in his prediction model mean. This change was (briefly) discussed previously on HN [1].
For the record, I think Taleb was an asshole, but an asshole with an entirely valid point in this debate. And I think Silver reluctantly and quietly took it on board, and then went on to use bolt-on fudge factors so that the time-dependent predictions of his model roughly look like the time-dependent predictions generated by the rigorously defined approach (option price theory) that Taleb was calling for.
This was a correct description of one of 538’s models, the nowcast. It was never the one they recommended, and they did not bring it back later because it tended to confuse people. They actually made it very clear on their podcasts that the nowcast was a mistake.
Ahh, that's even worse for Nate Silver then, if the "polls-plus" and the "polls only" predictions were not assuming the election happened today (which it seems like they weren't, even in 2016 [1]), then Taleb's criticism is all the more accurate.
Whatever Nate Silver was doing to incorporate future uncertainty in 2016 was clearly inadequate for all the reasons that Taleb pointed out (you could reliably make money betting on Silver's prediction odds to tend toward their time-smoothed average, implying that their time-smoothed average was a better representation of the probability than the probability itself).
Taleb's criticism is inaccurate in any case; it would be a valid criticism of the model if, in a large number of election cycles, the 2016 shape was typical. Because what you'd expect a forecast model to do typically is to get gradually more certain over time, so if it consistently behaved the way the 2016 forecast did, that would be a problem.
But, you'd also expect, in a large number of cycles, to have some where you get patterns where you do start getting runa of information that contradict the impact of prior information, and where you do have the kind of reverses seen in the 2016 cycle.
And, while we don't have enough Presidential cycles to make firm descriptions of what a “typical” cycle would look like with Silver's model, we do know that both 2012 and 2020 look a lot more like you'd expect a typical cycle in a forecast model to look.
Taleb's criticism was that the chance it rains four months from now shouldn't reliably vary about a mean (say from 30% to 70% to 30% to 70% etc) in the four months leading up to that day. Because if it does, I can make risk-free money by betting it won't rain on days the payout for not-rain is high, and betting it will rain on days the payout for rain is high.
If a time-smoothed average of your predictions is a better prediction of the outcome than your current prediction, then your current prediction is wrong. The idea that someone can improve on your current prediction by averaging your last 10 predictions is a sign that your methodology for generating "current predictions" is deeply suboptimal.
Guessing at what Teleb would say: if it rained on 2% of days, and weather forecasts are only good a week out, you'd expect a reasonable model to have the probability sit around 2% until a week or so before the prediction date, then move up or down, maybe with some noise. If you saw the probability swinging from 20% down to 0.1% a month out you should be suspicious.
A 90% chance in the election a long way out is suspicious if we think that the chance effects are large and roughly even-handed. Regardless of assumptions about the distribution of chance events though, multiple large swings are themselves suspicious because they each by definition imply the observation of highly improbable events.
We've had Trump as President for four years, and it seems like most people have forgotten how incredibly unlikely that seemed in August or even October. Trump himself was laying the groundwork for launching a TV network until things changed quite dramatically at the end of October with Jim Comey and some other factors.
Taleb seems to be arguing that 90% confidence was crazy given what eventually happened, with the benefit of hindsight, but that does seem to ignore just how crazy the idea of a Trump presidency seemed before it happened.
In the scope of this thread - Silver v. Taleb - there's no need to mention the layperson's memory of their internal prediction. We know that 538 had a 30% chance of a Trump victory, which is the relevant piece of information - not our perception.
I disagree. Taleb's key assertion seems to be that Silver should never have given Clinton a 90% chance of winning with the level of uncertainty in that race. But there were times when Trump having only a 10% of winning seemed completely justified, and I think to say otherwise at this point seems like projecting backward with hindsight.
Taleb is saying that if your rain model four months out shows 2%, 8%, 3%, 9%, then it should have instead shown 5-6% that entire range instead. If your final prediction ends up at 90%, your path there should resemble a sort of random walk.
Put another way, the 538 model doesn’t add enough variance when we’re still months out. This shouldn’t be surprising, as Silver himself emphasizes that their primary metric is ‘last minute’ prediction accuracy.
I think what many people miss, and what Silver isn't likely to admit, is that 538 is entertainment. If their prediction were 50% split up until the last month of the campaign it would be boring and nobody would check out his site. Higher variance is desirable to him, and he can justify the whole thing in his post mortems by only measuring last minute accuracy.
> I honestly don't even get what Taleb is truly arguing here. He is using a lot of statistical arguments but ultimately Silver is making a model to predict an outcome. It is like a weather forecast.
"Making models to predict" is precisely the fluff that Taleb is against. There is no "if elections were held today", there are no do-overs, there is only one election. And you either win or lose (binary price [1]). So any prediction that doesn't incorporate the events between making the prediction until the election actually occurs, is worthless, or to be more precise: 50/50 (arbitrage free). The probability only goes up when you start counting the actual votes on Election Day.
If it's 50/50 up until election day, then it's pointless to try to predict. And maybe that's Taleb's point.
But is there value in a prediction that does try to predict using polling data and other information? If so, then regardless of whether or not 548 "does it right" by Taleb's definition, it's valuable, at least to some people.
The point of premise 2 is that an 80% probability in late June implies a very high level of confidence. Confidence that doesn't seem intuitively reasonable. I would say that few rational individuals would have taken a bet with 4 pays 1 odds that Biden would win on July 1. It would not be a very safe bet, because of the likelihood of unknown random events.
Therefore, what Nate's model was predicting in July probably wasn't the probability of Biden winning on November 3rd. That doesn't mean the model's not useful, but if it is trying to predict what you think it is, then it's overconfident.
As a non-huge fan of Taleb, and as actually a big fan of Silver, I think Taleb is right here.
The problems with Silver's results is that it is from a "seat-of-the-pants" model. It is trying to do some form of data mining with the data, but there is little underlying foundations to guarantee that it is even close to reality. Regular polls are more transparent because they (at least the well done) will tell you the methodology and the statistical limitations of the research. You can't get that from what Silver is doing.
Minor addendum, Silver altered his model to become more certain as the election drew near after 2016. In effect, it changed from a “what if the election were today?” Model to a “what will the outcome of the election on date ___ be?” Model just for this cycle.
I can’t say why for certain, but it definitely could be that Taleb’s 2018 criticisms played a part.
That's incorrect. At _least_ as far back as 2016 and I believe quite a bit farther back the model was not "what if the election were today?" There even used to be a separate side-model that was explicitly that, the "nowcast" I think it was called.
I suspect that 538 _never_ covered an election solely or even primarily with a "what if the election were today?" model, but my memory isn't that reliable back that far.
You are correct, thank you. I updated another comment of mine to reflect this.
Comparing them now though, Silver definitely changed the 2020 model to increase variance as a function of time to the election, which was Nassim's complaint in the first place.
> Minor addendum, Silver altered his model to become more certain as the election drew near after 2016. In effect, it changed from a “what if the election were today?” Model to a “what will the outcome of the election on date ___ be?” Model just for this cycle.
False. Silver's headline model has always been a “what will the outcome on election day” model. (Silver has a secondary “what of the election were held today model”, as well, which was published as an explicitly alternative model in previous cycles, with explicit notes about the difference in what it did from the main model, and is still run and occasionally referenced but not published.)
The disagreement between Taleb and Silver wheels on technical minutiae in statistics, but the discussions around them by lay-people seem to be about something else altogether.
The majority of 538's ad impressions come from people who start F5-mobbing the site during election time because they want to know before anyone else who's going to win. (A tiny few might be from people trying to figure out which races are worth donating towards.)
So that puts 538 into this awkward position where their audience is demanding something that 538 is unable to provide, but if they get too in-your-face in saying, "look, that's not how this works", then they stand to lose a lot of ad revenue during peak season.
That leads them to do very silly things, like bury this:
> A 10 percent chance of winning, which is what our forecast gives Trump, is roughly the same as the odds that it’s raining in downtown Los Angeles. And it does rain there.
...in the same page as this:
> We simulate the election 40,000 times to see who wins most often.
And that's some bullshit.
Silver and the rest of his staff over the last few months have spent a CVS receipt's worth of text on what went wrong with their predictions -- er, sorry, "models" -- in 2016, and why this year is different, and how uncertainty works, and why they're offering no guarantees really, "but please do keep coming back, and hey check out this cool new simulator thingy where you too can make guesses that are about as accurate as ours".
I almost really wouldn't care, except for one kind of big problem with 538: It is a political observer-effect in action.
Journos, hacks, and other newspeople trying to get one more page impression or soundbite before the end of the day keep referencing 538. When 538 says a candidate is doing well according to their models, it can and almost certainly is changing people's behavior. Likewise when 538 says a candidate is doing poorly. That's hugely problematic and, annoyingly, the more accurate 538 becomes, the more pronounced and dangerous this effect will be.
There's a small problem with your comment- 538 didn't predict the 2016 election wrong. They gave Trump a roughly 30% chance of winning. 30% happens. That's all. People act like 538 said "Clinton will win!" but that's absolutely not what they did.
538 gave Trump one of the highest chances of winning from a non-partisan outfit; 1/3 if I recall correctly. Far better than, say, the NYT that gave him 1% odds. I’m not sure how you can look at that and declare them to be hacks.
There is a lot of misunderstanding about now-casts and probability in this thread, but for me this discussion between two quite able mathematically minded poeple comes down to the exact same thing as what idiots argue with Nate Silver about. Nate Silver has a model that predicts the likely outcome of the election based on poll data, economic data, and a few others factors. Taleb saw Clinton's predicted victory peaked at 90% and says that's wrong - because the uncertainty is so high that you can't be 90% confident.
Even if Clinton were ahead at that point in the election there was a greater than 10% chance of something changing before election day. But this is just the idiots critique - You said it was 90% would happen but the 10% happened! Well, yes, that happens 10% of the time, and if you're making lots of predictions it happens often.
But let's examine the critique, was Clinton a 90% favourite to win? Well probably, I mean in order for Trump to win from that position not only did he need a decent polling error in his favour he also needed the almost unique situation of the FBI director announcing he was re-opening an investigation into Clinton. We can actually see from the polls the effect this had. That sounds like something, in combination with a polling miss, and in combination with a favourable distribution of Trump's vote by state combined to give Trump a win. It seems reasonable to me to fit that firmly under a 10% probability of happening. If that was in your list of likely scenarios at the point the model was at 90% then you are massively over-estimating how often shocks like that happen.
The 90% criticism of 538 only works if you can provide a reasonable discussion of why you think that what happened between the 90% and the result had more than a 10% chance of happening.
In fact I think 90% of Nassim's problem is that he says things like "When someone says are event and its opposite are extremely possible I infer either 1) 50/50 or 2) the predictor is shamelessly hedging, in other words, BS."
He's totally wrong. Nate says "Extremely Possible" because when he says X is 75% people hear "X is 100%", and if Nassim had actually paid as much attention to 538 as he should before making that criticism, then he would know that. Every single election cycle 538 have struggled against the fact that people don't understand probabilities intuitively.
Edward says that Nate's prediction should be interpreted as:
“If nothing else changes between now and the election, Joe Biden has a 85% chance of winning.” (Silver’s argument)
But the problem is this prediction is not testable as election only occurs once at the end. Thus, one should use Nate's early predictions as pure entertainment.
Nate's final predictions show that they are well-calibrated across all of his political predictions but it is hard to estimate how accurate his model predictions are just for presidential elections given he has predicted only a few so far (unless one assumes all elections have similar uncertainty, which is clearly not true).
Silver has such an army of fanboys that it’s impossible to even remotely criticize 538 without getting downvoted into oblivion. It’s a weird sort of fanatical statistical religious belief.
The fact of the matter is that election polling is not really accurate anymore. People are afraid to publicly admit their support of a certain candidate, even in a supposedly anonymous phone call. Considering that people have been doxxed online for this, I really don’t blame them for being skeptical and private.
That’s not even mentioning the social aspects of the media. Even if Trump were ahead in all the polls, do you think Silver and other journalist-statisticians would report it as such? I doubt it.
Pundits seems to consistently miss larger geopolitical trends, like Brexit, offshoring of jobs to east Asia, rise of right wing parties in Europe and India, and so on. These qualitative trends will end up being far more influential than a collection of polls.
I actually think both are wrong (Taleb little bit less so)
Because they are using empirical evidence (eg previous data) from 'experiments' that are effectively, unrelated to current situation.
In my personal view, of course,
Most previous elections in the last 30+ years, were between RINOs (Republican In Name Only) and Democrats.
Those are pretty much two fractions of the same bribe-taking, accountability-avoiding global Cartel.
This Election is between the representative of citizens of the Republic (Trump) and the Cartel's candidate.
This really did not happen from the time Bushes-then-Clintons took over both parties and turned them into co-existing fractions of one party.
- - -
Another analogy: we do not use same methods and design constraints when building anti-lock breaks control software, as we do when we build a daily revenue reporting system.
- - -
Frame of reference is different basically. And we do not know the tensor(s) that help us to move between the coordinate systems.
- - -
So we are not going to be able to use even 2016 polls (because then folks did not realize how Corrupt the RINOs+Dems are )
- - -
My prediction is that Trump wins, GOP takes House and retains Senate.
This will also be the largest percent wise vote for a GOP president, by Latino and African American voters.
- - -
Will check back in 9 days or so, to see if I was right !
Your argument doesn't hold water at all. Today's blurb in the forecast page - "First, the forecast is now totally polls-based; that is, any advantage our forecast gave Trump for the economy or for being an incumbent is no longer factored in. Second, the uncertainty around how much the polls will change between today and Election Day is also no longer an issue." - that means Nate Silver is explicitly saying his forecast is always trying to predict what is going to happen on November 3.
Heres an alternative hypothesis that involves no fancy math just anthropology - Taleb is right that the stats don't make sense, but Nate is also not an idiot - it's just in his financial best interests to make a website that can throw out confidence inducing pseudonumbers that will make people like us visit it every few hours for months at a time every two years. Perhaps he knows perfectly well the farce he's pulling (at least at a fundamental level though he might have convinced himself otherwise as academics always do) but just plays the game to make sure he's relevant. And thus, we are all the idiots.
“if I tell you an event has a 0% chance of occurring, I cannot change my mind and tell you tomorrow it now has a 50% chance of occurring. Otherwise I shouldn’t have told you it has a 0% chance in the first place”
I think this is the key to Taleb’s argument, and it’s pretty damning for Mr Siver.
If there’s high uncertainty, you can’t say whether someone’s probability to win is low. The uncertainty widens the probability.
For example,
Say I told you there’s a 2% chance it will rain Monday next year in New York
- This would _have_ to be a bogus prediction. So much could happen until then. I have _high_ uncertainty. There’s no way that uncertainty is accounted for with 2%
The prediction should have more data to it about variance, but it isn't necessarily wrong.
We have good data about rain, I am sure we could find out what percentage of the time it rains in New York on a given day during that time of the year. There might be high variance, but that doesn't mean it is wrong to say there is a 2% chance it rains on a particular Monday.
This is how insurance works, how sports betting works, how option pricing works. Just because something has high variance doesn't mean we can't give probabilities that work out over a large number of samples.
Imagine someone offered you this as a bet. They’ll pay you 3 dollars if it rains Monday next year. Otherwise you pay them 90 dollars.
Now, it’s pretty intuitively clear that at a year interval, there’s way too much uncertainty to take this bet. If you accounted for it, you would approach something like the average chance of raining on any particular day in November in New York.
You are right that with enough data, you would be able to price this correctly.
Taleb’s point is that Nate is not doing that, by demonstration: since Nate’s models oscillated so much, it demonstrates that he did not account for it.
First, your odds don't make sense. It wouldn't be 30x more likely to rain than not on a particular day. However, this is a detail and not that important.
Second, though, is that people make highly uncertain bets all the time, and the price for those bets DOES fluctuate wildly over time.
For example, you can currently place bets at a number of sports books for the winner of next year's NBA finals. There is a TON of uncertainty (we don't even know who is going to be on teams, free agency and the draft haven't happened yet... not to mention injuries, player improvement, etc)
That uncertainty is priced into the bet, and people will take both sides of that bet.
As the year progresses and things happen, that price will fluctuate as we get more information and we get closer to the finals. The favorite might change, and a 200 to 1 might end up being a 10 to 1 by the time the finals gets here.
It isn't that the odds were WRONG at the earlier time, they just had less information. It doesn't mean the odds makers didn't understand what the state of the world was, or that they didn't take into account the uncertainty of what could happen.
I am not quite following you... the "price" of a bet is synonymous with the odds (i.e. they are two different ways of expressing the same information). For example, the money line price of something that has a 90% chance of winning would be `-900`... meaning you have to bet $900 to win $100.
No one has odds that say the Warriors (or any team) has a 90% chance of winning the NBA title. In fact, the current favorite (the Lakers) are listed at +350 (meaning you will win $450 on a $100 bet).
I am not sure what you mean by the 90% chance thing...
> For example, the money line price of something that has a 90% chance of winning would be `-900`... meaning you have to bet $900 to win $100.
This is what Nate saying, when he says Biden has a 90% chance of winning.
You intuit correctly that it would be ridiculous for anyone to offer odds like that for the NBA title one year out.
Why do you think it would be ridiculous? Likely because you intuit that there's too much uncertainty for a chance to be 90% for any team to win the NBA title.
Taleb's reasoning is similar, but for the election. 90% Biden win implies way too much certainty.
> Say I told you there’s a 2% chance it will rain Monday next year in New York - This would _have_ to be a bogus prediction.
Yet betting odds exist, because we can predict things far into the future accounting for variance and risk by looking at history. If it's never rained on 2% of Monday's in NYC then a 2% prediction would make sense.
From the article:
> Premise 1: If you give a probability, you must be willing to wager on it
The best way to settle this is have Talib bet on Trump to win at 50% odds (or whatever he thinks is appropriate) every day up until the election and Silver to bet using his odds (currently around 10%). May the more accurate prediction prevail. Too late for this election, but there will be plenty more elections in the future.
Most betting markets are at 66%. That I believe is for who is sworn in in January.
Silver's forecast is slightly different. It's for who will win if all votes are counted and no courts intervene.
Given that Biden's lead in states exceeding 270 electoral votes is above the polling margin of error and even above an unexpected 2016 sized error, and that Trump would need a clean sweep of all 7 swing states, 90% isn't crazy considering Trump is an incumbent with a low 40% approval rating.
The extra 23% the betting markets are giving Trump are probably because of the courts and some 2016 "anything could happen" bias. The Supreme Court is conservative as are some of the attorney generals in the swing states. In 2000 that proved decisive and is probably an easier path to victory for Trump than winning a clean vote.
In 2016 Trump had a forecasted 30% chance to win. He lost the popular vote by 3M, but won 3 decisive swing states by a combined 80k votes. The betting markets assist to be giving him about the same 30% chance as 2016 even though the polling, approval rating, undecideds, etc. are all decidedly worse for him this time around.
Yeah, except both of them are modeling how people will vote, and people change their minds.
Saying that one should’ve held a constant chance of a candidate winning no matter what the polls do is frankly insane. This stance basically requires that you pretend that the outcome is determined and that voters opinions either don’t matter or don’t change. It’s utter nonsense, especially since we know that in 2016 undecided voters broke late for Trump. If your model doesn’t react to that, what are you even modeling?
Taleb’s point is not that your prediction should not change.
His point is that if it does change, and frequently at that, the purported 90% chance at t(n) is bogus. That prediction needed to account for the volatility
It's impossible to believe any forecast, nobody fessed up after 2016 nor did they explain why their models are wrong or attempt to update them. You're less than 24 hours away from the permanent death of the political polling and forecasting industry in the USA.
I don't believe this is accurate. NYT[0], HuffPo[1], Princeton Election Consortium[2], 538[3,4], and others did analyze their model's failures and update their models and methodologies in response.
Can you elaborate a bit on what you mean by, "nobody fessed up after 2016"?
From my understanding there wasn't the "big miss" that got called out in media, and to the extent that the rust belt/upper midwest had state level polling variance, it was determined to be due to the lack of accounting for education as part of polling.
I guess if it doesn't rain and the weatherperson says it should have rained, do we give up on even trying?
That's not true, there have been countless articles about why polling went wrong last time.
Also, polling wasn't that wrong. Due to sampling bias, several swing states were off by as much as 5-6 percentage points. But national polling bad Hillary winning by something like 4% in the popular vote.
Believing that occurrence of an event with a forecasted 30% chance (or 2% chance) of occurring invalidates a model is so fundamentally a misunderstanding of the topic.
My DM: "You're going to try to hit this guy with your bare hands, even though you have +0 strength?"
Me: "Does a 17 hit?"
My DM: "His armor class was 16. It was supposed to be impossible for you to hit him with a d20. I have just witnessed the permanent death of the dungeon mastering industry in Faerûn. I will now cope on Twitter, goodbye."
Well pre-2016 there was the assumption that the American electorate basically is capable of self-governing. This assumption has been invalidated so while the polling industry hasn’t gone away it’s now more like the sports forecasting industry, where people are speculating on an essentially theatrical performance, than something substantively driving good governance.
He's explicitly not doing this.
Here's how I think about it. Silver is answering the question: "how much would the polls (as an aggregate) need to differ from the final result in order for Candidate X to win/lose, conditioned on some reasonable priors?"
Taleb is pointing out that the polls could be really wrong in all sorts of ways that are impossible to predict a priori.
The whole argument is sort of pointless from an intellectual/academic perspective. It's a war of public personalities more than anything else.
It's both the case that Silver designed a good piece of software that does what it's supposed to do and also the case that Taleb's skepticism is valid. But then, that sort of skepticism of statistical models is always valid, and yet we use these models to great effect in all sorts of settings.