I disagree with the author on the idea that tail is too fat for isolated anomalies. There are most certainly events that can happen, which may lead to a red California or a blue Alabama.
Presidential assassination, war, video proof of something incredibly heinous (pedophilia?), etc. can absolutely lead to these outcomes. You don't even have to go that far back. Nixon and Reagan flipped states like no-one's business.
I do however agree, that 538's state-state correlation model seems weak.
California and Alabama would only flip during a wave, and that wave would consume any and all states. The fact that 538's model doesn't strongly show that pattern is a failing of it. But, it is not clear if a model that inaccurately models the unlikeliest of events (california flipping while Florida stays blue), does not necessarily mean that it is terrible predictor of it's primary target (Presidential likelihoods).
As a data scientist, I can totally understand Nate's hesitation. Do you impose strong priors on the model to reflect strong domain intuition or do build a model that best characterizes the data it is based on. In the presence of infinite data, you should abandon all domain based priors. For single digit data points, priors are essential. For any number of data in between, it is anyone's best guess.
I've always liked Enrico Fermi's attitude on this. When you're Enrico Fermi, you get to say things like "One data point gives you a curve. Two data points gives you the distribution about the curve."
The source is a long ago personal comment from my dad, who worked with and was personal friends with Fermi.
As to whether it's sarcasm, I'd describe it more as an inside joke with just a touch of the self-aware intellectual arrogance Physicists are famous for (see Lord Rutherford's "All science is either Physics or stamp collecting").
Trying to explain a joke is always dangerous so hopefully what follows won't be a mistake, but here goes:
The way I have always interpreted the first sentence is to hear Fermi saying that when you are doing an experiment, you should have a deep understanding of the family of curves or behaviors the system under test is expected to follow, including the full range of curves that would follow from interactions that are wildly different from what you might naively expect. If you really understand the Physics of the system (this is where it starts to blend the line from advice to a joke), the measurement of one single data point should be enough to tell you which of those possible curves describes the actual behavior of the system (the joke being both that it's funny to be that arrogant and that it's obvious to anyone you'd tell this joke to that mathematically you need at least two points to determine a line, so by saying one point gives you not just a line but a curve the speaker is purposely going over the top for fun). Moving into the second sentence ("two points gives you the distribution about the curve") takes the statement into full-on joke mode, with the comment shifting from an observation about knowing your Physics to an insider dig at the relationship between theoretical Physicists and experimental Physicists (Fermi being one of the greatest theoretical Physicists of all time). When he says that two points gives you a distribution about the curve, he's saying he as a theoretical Physicist understands the underlying Physics of the system better than the experimentalist understands the noise in their experimental hardware, or alternatively that the noise in the hardware is sufficiently uninteresting as to be irrelevant to him. The former view would simply be arrogance but leaving the second option open circles the joke back to include a bit of insider self deprecating humor in that he's purposely ignoring experimental error, a thing theoretical Physicists are famous for doing.
Awesome that there's the connection with your dad, I got the joke but it's also funny that some experimentalists have taken this literally AKA hand sketching the gravitational wave signature and looking for matches as done with LIGO.
It's bizarre there was no better analytical/computational way to come to what they were expecting.
Something could flip California and Alabama (example, Trump starts defending Roe v. Wade and in response Biden somehow manages to sound like he's opposing it). This would probably be some latent hidden variable, like whether the candidates are seen as socially conservative, which would effect all states (though California and Alabama would be the most impacted).
Meh. If you fit a model and don't explicitly constrain against "un-physical" results like negative correlations, you'll end up with them.
Constraining against them won't improve your models fit (usually by definition), and it doesn't always improve robustness (at least for situations near average)-- because they're acting to debias the model in ways that you otherwise don't have enough degrees of freedom to address.
A negative correlation here is also potentially historically supported, in the sense that sometimes DEM/GOP candidates are philosophically reversed in some way relevant to the state. As in, "The only way a GOP would get elected in X is if they had the DEM position on subject Y which would make them lose state Z, who cares as much about that subject as X but in the opposite direction."
Now-- it doesn't seem likely case in this election (e.g. Trump is not (currently) a massively pro-choice republican), so it probably shouldn't apply here-- but it's isn't hard for me to imagine how a negative correlation might show up out of the historical data.
> If you fit a model and don't explicitly constrain against "un-physical" results like negative correlations, you'll end up with them.
The Economist model does exactly that, and all of their correlations are positive.
I recommend reading their methodology, they know what they're doing (I wouldn't say the same about 538). Andrew Gelman has developed some of the Bayesian methods and software that people like Nate Silver use, he's the main author of what's considered a reference book on Bayesian statistics.
I think the question is if it matters to the predictive accuracy of the model. Just because it puts out results you can't envision actually happening on the margins doesn't mean they can't happen, or that they can't be valuable in presenting a holistic result.
It's clear that the models are tuned differently, but from Silver's replies in the PS's, it seems that he's ok with these artifacts being part of the model.
Yes, it increases the state-level and national uncertainty intervals (Andrew Gelman has talked about this several times on his blog), which improves Trump's odds.
Sure, but that's not necessarily wrong. Any decision in the model will change Trump's odds in one way or another. The question is if it makes it closer to the (unknowable) real odds.
Just because intuition says it should be longer odds for Trump doesn't mean that's right.
My statistics knowledge has withered away, but isn't this quibbling over overall approach? 538 seems do be doing a top-down approach while the Economist is more bottom-up. What is strange is that a person affiliated to the Economist is then asking why the 538 model's emergent properties aren't exhibiting more bottom-up characteristics .
> It didn't take very long to do the analysis. But it did then take another hour or so to write it up.
It's very interesting to see how long it takes people to do things. I am amazed that entire article took 1 hour to type up. I've spent entire afternoons trying to write shallower pieces of work.
I think a lot of people (myself included when I've felt the pressure to) lowball how long things like this take because a) it makes me look smart and b) people could judge of they knew who much time I actually wasted on it.
It looks like it was written as a single stream of conscience. While I couldn’t write that article, if I hit a flow state and was interested in the topic, it seems possible.
The trick is to think before you write. The same goes for programming. If you already know what you are going to write or build then you can reach very high apparent productivity, the time spent on thinking about it isn't accounted for.
Can someone help me understand what odds like this mean in the context of an election?
The model says that Trump has a 1 in 10 chance of winning. With a fair 10-sided die it makes sense that you have a 1 in 10 chance of any given side rolling face up. But what is the die that is being rolled in these election statistics? What is the "chance" element that is being predicted?
You're comparing two scenarios, one in which you know all the facts, and one in which you don't.
In the dice toss scenario, we know everything relevant. In the election scenario, we don't.
A model like this is attempting to say "these are the rules we think exist. Based on the rules, and assuming the data is off by some random distribution, here's what we think could happen".
What different forecasters disagree about is what the rules are. For example, the relevance of certain demographic characteristics and the potential variance between polling (conducted prior to the election) and actual election results.
There's a huge amount of assumptions, and forecasters disagree on those assumptions. We have very little historical data (polling is very recent) and even with complete historical data, future elections do not always conform to past elections.
I will veer this off into the dreaded political territory even though this is mostly a technical discussion.
The Democratic Party proved it was not as progressive as they thought as Sanders lost the primary. The reality is, the country as a whole is also not as liberal either, regardless of what these pollsters are asking people. You think the party is youthful, and ready for progressive ideas, but alas, the party wholly rejected an amazingly progressive candidate in Sanders. You think everyone’s super pissed at Coronavirus handling, and police brutality, healthcare, but alas, you find out people associate BLM protests with crime, and the virus with China, and socialism with unfair wealth redistribution. We can keep learning this the hard way I guess, this is America after all.
It’s important the technical discussions are happening this time around, because there was virtually none the last time. The post mortems for these forecasts being wrong again should be a death knell for accumulating bad data. I’m certain the models are good, but I’m not certain the data is.
Anyway, if you want my hot take, the conditional forecasting is to save their ass on election night from being embarrassingly wrong again. Imagine writing a giant if-statement that looked something like ‘and if(imWrong) changeMyAnswer’.
> Anyway, if you want my hot take, the conditional forecasting is to save their ass on election night from being embarrassingly wrong again.
Well Nate Silver wrote a full critically acclaimed book about why these types of forecast are more useful (and accurate) in reality because they account for uncertainty - he has been doing this for years, ever since he used to write similar algorithms to help bookies pick odds for sporting events, so I think your hot take isn’t based in any world of facts or knowledge on this.
Don’t trust a forecaster that says with certainty that a certain candidate will win, unless they have also bet their life’s earnings on it. Showing your statistical confidence level isn’t a bad thing.
I think it’s certainly more grounded in reality if you realize 538 is basically finished if they miss the mark again.
If you listen to what they say, they admit they were not able to measure for the no-colllege male demographic in 2016, or in other words, they couldn’t model identity politics. Why couldn’t they do that? I’m not sure, but they are certain they can this time around because they saw the 2016 data and now believe they have more complete data to not make the same mistake again.
They are looking at elections as if there are hundreds of millions of elections that happen every day and the data speaks for itself. No sorry, there’s very few elections to extrapolate the way they are doing it, and you really need to do sociopolitical analysis of things like a demographic identity bloc (no-college whites that feel some way about things) that really get you the accurate undercurrents that can sway an election.
Lastly, it doesn’t take a genius to sit there at 10pm on election night and go ‘well if Florida and Michigan went this way, then probably so will these other states in flux’. ‘Our forecast becomes more accurate as we get the actual poll closing numbers on election night’, ah I see, you’re all geniuses, I should have known.
> If you listen to what they say, they admit they were not able to measure for the no-colllege male demographic in 2016, or in other words, they couldn’t model identity politics. Why couldn’t they do that? I’m not sure,
You seem to have a fundamental misunderstanding of what FiveThirtyEight is trying to model, versus what pollsters are trying to model with the numbers they publish that FiveThirtyEight consumes. The kind of demographic weighting you're complaining about FiveThirtyEight being bad at is something the pollsters do, and is outside the scope of FiveThirtyEight's forecasting models.
> If you listen to what they say, they admit they were not able to measure for the no-colllege male demographic in 2016, or in other words, they couldn’t model identity politics. Why couldn’t they do that? I’m not sure, but they are certain they can this time around because they saw the 2016 data and now believe they have more complete data to not make the same mistake again.
I think you possibly misunderstand what 538 _do_ a bit. Their data is based on polling, so they can only work on what the pollsters do. Historically, pollsters didn't pay that much attention to education, beyond using income or class as a proxy for it; one middle-class white man was pretty much like another. This worked quite well historically, but no longer does (and it's not just a US phenomenon; it was also a contributor to polling problems for Brexit, notably).
In their current model, 538 assume a higher rate of uncertainty than last time round; also, some pollsters now model education. But really there's not that much they can do about stuff that pollsters don't ask about.
No, I don’t think so. If you build a model out of pollsters asking stupid questions, you deserve some blame.
I’ve got some basketball statistics to populate 538s model if their interested. Lebron did pretty good this season, hopefully they can correlate that with the black vote.
Their model is not transparent on any level, because if they
make it transparent, we’d easily be able to see why it’s ridiculous.
> I think it’s certainly more grounded in reality if you realize 538 is basically finished if they miss the mark again.
What does missing the mark mean though? In 2016 they proposed a c30% chance that Donald would win, and a 70% chance Hillary would win. Does that mean they were wrong? Not really, because that's how probabilistic forecasting works - and they stated their confidence interval - they were 70% confident that Hillary would win, but thought there was a 30% chance Donald would win.
> The Democratic Party proved it was not as progressive as they thought as Sanders lost the primary.
The FiveThirtyEight forecast for the Democratic primary [1] gave Biden the highest chance of winning for most of the process. He did have a steep drop in the month before Super Tuesday (followed by an equally steep rebound), but still, I wouldn't say the forecast was especially bad. That said, polling is always worse for primaries than general elections, since there are more candidates and fewer voters.
This sounds like a frequentist vs Bayesian statistics discussion, which involves (this is a simplification by me, a non-expert in the area) different definitions of probability. The frequentist view is along the lines of rolling a 10 side die hundreds of times, recording the results, and determining that each side comes up equally. The Bayesian view is that the probability measures our certainty about some event. For example, take the hypothetical point in time where all ballots have been cast, but have not been counted. One could use polling data, etc to model the odds that a particular candidate has won. However, the frequentist approach doesn’t really make sense here, as the ground truth already exists (all ballots cast), so rerunning the the event doesn’t make sense.
Once again, I’m not an expert, so I recommend looking for additional explanations, if you’re interested.
> what is the die that is being rolled in these election statistics?
A program which randomly generates outcomes for each state, based on probability distributions inferred from the polls, and calculates who wins the election given those outcomes. They run the program repeatedly and report the proportion of simulated wins as the probability of winning. https://en.wikipedia.org/wiki/Monte_Carlo_method
It means if you saw all of these facts in ten different events, you would not be surprised to see a one to nine split in results. Ish. As you are scaling up, if course.
So, think of it as saying these facts basically describe a ten sided die. With no other knowledge, the best you have is that you expect it to behave the same as any other ten sided die.
It's less about pure random chance, and more about our uncertainty. Compare it to a weather forecast that says there's a 10% chance of rain tomorrow. In the same way that weather forecasts get better over time (better atmospheric measurements, more sophisticated computer models), we could potentially do more to measure what the outcome of the election will be. And it might be theoretically possible (albeit highly unrealistic) to predict it with complete accuracy, given enough data. But we're not in that situation, hence uncertainty.
(There are a couple of caveats about election forecasting as opposed to weather forecasting. The first is the "October surprise," a sudden revelation that changes the election. This cycle, it was arguably Trump's covid diagnosis, although that tended if anything to push the results further in the direction they seemed to be going on their own, rather than upset any trend. The second is that, unlike with weather systems, measuring voter behavior (and widespread reporting on these measures) can change people's behavior. The effect of this is hotly contested, but one of the many explanations of Trump's victory in 2016 which hinged on turnout in a few key states is that those states were predicted wins for Clinton, so Clinton voters didn't bother voting. Despite occasional jokes to the contrary, it doesn't rain just to spite the weatherman.)
That's a good question and it's not clear. As someone mentioned, here "chance" includes both uncertainty (facts that we don't know), and randomness of nature (things that will happen in the future that cannot be deterministically deduced from the state of the world today). Depending on your philosophy these may overlap. Next, someone mentioned Bayesian vs frequentist.
The frequentist interpretation is roughly that if I go around making my best possible predictions, and we lump together all the things that I predict at 10%, about 1 in 10 of those things happen and the rest don't. But I wouldn't be able to be more specific about which ones in that group are more likely than others.
The Bayesian interpretation is that I can really view the world as flipping coins -- I don't care whether it's due to my lack of knowledge or "true" randomness -- and as far as I can tell, the coin flip involved here is 1 in 10.
We can also use a gambling interpretation. Here's one based on security of python's random module. Imagine the following three lotteries I offer you. In lottery A, you get $100 if Trump is elected. In lottery B, you get $100 if the following python code returns true on my laptop:
random.random() <= 0.09999
In lottery C, you get $100 if this code returns true:
random.random() <= 0.10001
If you would rather have lottery A than B, and you'd rather have C than A, then in some sense that you believe Trump has a 1 in 10 chance.
Now there's an interesting extra layer to all of this because it's a model predicting, not a person. In a short space, I would basically say that we've trained models to predict in ways that are not inconsistent with any of the interpretations above, when put into situations where that is testable. Then we use them in situations where it might not be, like this.
It pretty much means nothing. These sort of models produced wrong result again and again. For me the biggest question mark if that if we know that recent (last 5) elections were very close how can you predict somebody winning with 93% chance? Maybe I do not understand something here.
Agree. Polls are a bad metric to rely on. Only 2% of those asked respond. Impossible to get a statistical sample with that. People are bad at predicting their future behavior. They are dishonest, they don't know or just don't want to tell you. There's a huge class divide right now. The bigger the class divide the worse polls are historically. And we know they are wrong this time around. None of the early voting margins predicted have held close. Better predictors are Google trends, rally participation, and voter registrations.
Sure, these are all concerns. However, as long as they are not systematic errors for or against one candidate, they end up not mattering very much.
Andrew Gelman (the author of this post) has also done a bunch of work on how different parties supporters become more/less likely to respond to polls based on what the current results are, which has been incorporated into the newer forecasts.
It's typical to use "to" rather than "in" when discussing odds. So the odds of getting a 1 on a 10-sided dice are 9 to 1 against (odds are also typically specified with the larger number first, because of the overlap between mathematical odds and betting-shop odds). And the probability of it happening is 1 in 10.
(I suspect that counter-pedantry on these lines might be part of why your post is getting downvoted; I wasn't one of the downvoters fwiw.)
Nit: Have they counted for the possibility of a tie? US elections allow for a tie in the Electoral College (which then kicks off a supremely strange and legalistic process).
FiveThirtyEight has a page where you can choose winning states (condition on a certain outcome) and it will regenerate the prediction map, https://projects.fivethirtyeight.com/trump-biden-election-ma... This appears to be the what Andrew Gelman is also trying to do with their raw data.
At the bottom of the 538 page it says, " If you choose enough unlikely outcomes, we’ll eventually wind up with so few simulations remaining that we can’t produce accurate results. When that happens, we go back to our full set of simulations and run a series of regressions to see how your scenario might look if it turned up more often."
I interpret that as running a regression (linear?) and extrapolating it out to the tail where the conditioning is happening. This should eliminate the issue Andrew is seeing?
I'd argue negative correlation on conditionals distributions can be reasonable here.
In that particular WA-MS example, if Trump suddenly took more liberal positions and somehow won WA (e.g., announces he's pro abortion), he would in fact be more at risk of losing Mississippi. The idea that these two states are in play already is fringe and would require some major idealogical (or other third variable) shifts.
But then the correlations with the other 48 states are broken. In that insane scenario, Mississippi now votes for Biden (because I guess he’s suddenly come out as pro-life as well), but Alaska still goes for Trump.
The negative correlations don’t make sense. Maybe it’s a small problem and the model is solid overall, but... I don’t think you can justify that one effect.
I think you have to look at the joint distribution with Alaska included to draw any conclusions. Just looking at separate marginals will be uninformative.
To me, this mostly tracks with what 538 has said on record about how their model works and the design philosophy behind parts of it. To me, what Nate means when he says "directionally the right approach in terms of our model's takeaways" is that these sorts of wild and unintuitive outcomes are part of the point of the way the model is constructed.
Specifically, that when you get off into the weird situations like Trump winning Washington state, it's likely something incredibly weird has happened - something that likely has no historical precedent, so it may actually be a more sane thing to do to assume that now almost everything is backwards and Biden would win a bunch of states he shouldn't either.
To me, this points to a general willingness in the 538 model to just go "who knows" and build in some room for insane things to happen on the fringes. The Friday podcast episode about the 538 model specifically mentions that they have large/fat tails on their distribution that make it nearly impossible for someone to get over 95% chances of winning on a national level, and these sorts of wild results seem like the outcome of that. If you bake in an assumption that there's always a 5% chance of something crazy happening, that chance has to come from something in the data somewhere that reflects the ability of that to happen numerically, and thus will have numerical outcomes that seem impossible.
Yeah I think for Trump to win washington state, he'd have to do something to appeal to voters there in a way that would likely cause his red state base to abandon him.
The negative correlation makes sense when we think about how difficult it is for everyone in Washington to suddenly turn conservative and everyone in Mississippi to turn liberal. Much more likely is that the crazy thing is that the candidate or circumstances changed in some way.
It makes more sense if we ask...if a candidate wins NJ what is the chance they also won AK?
Every single one of these models will break down this year. We are living in an unprecedented time. I can't understand how we can model how many people will vote, when we don't even know how many people have moved out of cities this year. Half of my friends have left San Francisco - if as many people left Philadelphia, The Twin cities, Milwaukee or Pittsburgh, then that really effects the outcome.
That's a neat point. My gut reaction is that there probably hasn't been enough people migrating to make a big difference. As in, I doubt enough people left CA to make it cut Republican and I doubt enough people moved to SC to make it cut Democrat. But that's just a gut reaction - I have no clue really.
It's a neat possibility to think about though. If there were enough people who did that, it would really depend on the demographics moving and where they're going. It could swing the election either way. I wonder if anyone has found numbers on this and attempted to model it.
That's an interesting thought, but San Francisco aside, it seems like most people moving out of cities are just moving to the suburbs of those cities, which shouldn't affect the presidential election calculus?
I know people that have moved, but forgot to register. Its a fact of life that voter registration will take the back seat when managing more complicated things in life - like a move
That seems plausible. Still, I think the "exodus from cities" theory doesn't apply much to American cities outside of New York and San Francisco. Places like Philadelphia may have even seen a small uptick from NY emigration. But this is just a guess, it would be nice to see any statistics (I haven't found any).
Here in San Diego, for instance, we've generally had net inbound migration. It's pretty much the opposite of Philly, though, in terms of both coronavirus and violence. But it is an "alternate choice" backwater city, compared to SF or LA, kind of like the way Philly is to New York.
I wonder if that is true for people who were also likely to vote? I know registering to vote was one of the first things I do whenever I move, and I've also voted in every election since I've been allowed to.
> I can't understand how we can model how many people will vote, when we don't even know how many people have moved out of cities this year. Half of my friends have left San Francisco
Moving out doesn't stop you from voting. I didn't change my voter registration when I moved from San Francisco to China. Years later, back in California, I voted in San Francisco, where I was still registered, despite residing in Hayward.
For verification purposes, they asked me when I voted what my address was. I was allowed to vote despite not knowing my own apartment number.
I've made similar points on topics like this, and bar none, every single time, it is downvoted into oblivion. People seem to have a difficult time with forecasting data that goes against their preferred outcomes.
If you'd like to not be downvoted into oblivion you'd do well to provide an alternate theory. What's your prediction for the upcoming election? Why? What's your methodology? Or is your point that forecasting is futile? I can't tell. There's too much emotion directed at 538/etc. for me to suss out what your point is besides disliking 538/the media/etc, which just isn't particularly helpful for the discussion.
I wish it were that simple. HN just seems to have gone the way of reddit, where downvoting is disagreement. And of course, 3 minutes after I post this comment, it's downvoted.
I don't have a methodology, because I'm not a pollster with dozens of people at my disposal. I am just bemused and annoyed that things like 538 continue to be taken seriously when they continue to ignore sociological, historical, and cultural factors in favor of an overly-complex quantitative model.
Re: the upcoming election. I don't think we can be sure, yet. Certainly it will be close, and the Biden at 90% to win estimations make no sense to me. Biden is a much weaker candidate than Hillary and he continues to make blunders (i.e. I guarantee that his comments on fracking in the last debate just lost him Pennsylvania.) Trump seems to be finding a lot of allies in strange places, e.g. African-American celebrities. That may be an isolated incident, or it may signal some big unexpected changes.
At this point my estimation is Trump-Biden 55-45, for the simple reason that people tend to vote for economic issues and Trump has a better "perception" on this issue. "It's the economy, stupid." as James Carville put it.
> HN just seems to have gone the way of reddit, where downvoting is disagreement.
Alternatively it might be because you're objecting to the results of a well-documented statistical process, and then when being questioned saying things like "I don't have a methodology".
You'll notice that the comment you replied to (saying it was similar to your point) is not downvoted into oblivion.
> Biden is a much weaker candidate than Hillary and he continues to make blunders
That’s your view; however, he is polling much better than Clinton, which would indicate that voters don’t necessarily agree with you (or else just that peoples’ opinions of Trump are lower than last time round, or a combination. But really it hardly matters).
> (i.e. I guarantee that his comments on fracking in the last debate just lost him Pennsylvania.)
Looks like 20-50,000 people employed in fracking plus industries supported by it in Pennsylvania. And presumably most of those would be voting for Trump anyway; it’s not like Biden’s views on fracking were a total black box til now. So it only really matters if it’s very close anyway.
I agree that there are factors that so much of current reporting is missing. The black (and Latino) swing toward Trump, for example, is apparent in every single poll I've seen. As mentioned elsewhere in this discussion Nate Silver's discussed it, but little else other than the odd article like https://www.nytimes.com/2020/10/14/us/politics/trump-macho-a... (which basically attributes male Latino support for Trump to the same machismo that ruined their ancestors' countries). Same with the increase in black Trump support from 2016%'s 8%, even though it's pretty clear from history that no Democrat can win the White House without at least 85% of the black vote.
>At this point my estimation is Trump-Biden 55-45, for the simple reason that people tend to vote for economic issues and Trump has a better "perception" on this issue. "It's the economy, stupid." as James Carville put it.
My model is simpler. If Trump wins every other state he won in 2016, he only has to win one of MI, PA, WI, MN, or NH/NV. The first three he won in 2016 (and, as you say, Biden's views on fracking may very well cost him the state); MN Trump lost by 1.5%, so the state is only sightly behind the rest of the Midwest bar IL, and half of Minneapolis being torched this summer probably pushed the state over.
I'm not trying to get points, or keep points. I'm just trying to point out that the models are not going to work this year and its crazy to assume they will.
The models will work just fine. We are talking about large numbers here, and your small sample of anecdotes does not mean anything. No, there is no mass exodus. No, it is not going to change the results. Yes, the current polls are more accurate than just about any in history and we have an abundance of high-quality state level polls to back up the predictions.
If you want to know where the models are going to break down it is more likely that the state-level polling has over-corrected for the factors that caused them to miss the swing to Trump in 2016 and Biden's numbers are even better than what polls are saying (a prediction based on looking at polling at the congressional district level and then seeing how that differs from state-level polling -- the numbers are the district level are closer to national numbers than the lower statewide numbers for swing states.)
Nate Silver authored an adjustment to polls used in that model. Polls have more impact if they are more representative of statewide turnout among demographic things he chose like “black” and “low income.” This is why his predictions were so accurate for Obama’s 2008 and 2012 elections, and likely why they were so inaccurate in 2016.
Gelman’s own grad student is the only person to have academically published this approach, in a paper about polling Xbox Live users.
These guys sort of make a thing that is the same in many more ways than it is different. Why not just share the code is the biggest question?
Nate Silver said Trump had a 1 in 3 chance, which basically means one shouldn’t be surprised no matter the result. I’m not sure where this “all the polls were so far off in 2016!!” narrative comes from, but it’s wrong.
It comes from innumerate journalism, and an innumerate population. Next time someone laughs off being bad at math, you should point out that being unable to read is no laughing matter, and being unable to understand numbers shouldn't be either.
The only sensible way to predict probabilities that aren't extreme is to tell people how the model works and the figures it is currently spitting out. That's is the great thing about these kinds of blog posts, people are kicking the tyres, not just looking at the car.
Nobody predicting a one-off election with a rather special candidate would summarize a 33% chance as equivalent to having no chance.
> Next time someone laughs off being bad at math, you should point out that being unable to read is no laughing matter, and being unable to understand numbers shouldn't be either.
The narrative comes from the medias inaccurate and misleading coverage of the polls in 2016. Many news outlets all but declared Clinton president before the election.
But the media is not Nate Silver. He said Trump had about as much chance of winning as the Cubs had that year of wining the World Series, and obviously both happened.
That isn't the only thing that happened. Probably the 1 in 3 odds were too low given the data available, because the polling demographics were not adjusted for education. If you randomly sample 1000 people to represent several millions, you also collect demographic information to ensure that you properly weight the responses based on how skewed that demographic is in your sample compared to the total voting population. In 2016 they weren't correcting for education, which turned out to be a huge hidden variable. This is explained quite well by 538 themselves:
https://fivethirtyeight.com/videos/polling-101-what-happened...
And in particular any claim 538 was the site that was off the mark compared to other prediction sites is clearly based in a reality that is not shared with the rest of us. In the week before the election Nate and crew were posting articles specifically outlining the non-zero probability of a Trump win and if it happened how it was likely to happen.
Nate was the outlier in that respect. But it’s true that the polls aren’t weren’t all that inaccurate in 2016: a bunch of important swing states were within the margin of error and Trump won some important states by very small margins.
The mistake in 2016, IMO was a) the extrapolation that came from those polls and b) people paying way too much attention to national polls, which have very little connection to electoral outcomes, given the electoral college.
Also perhaps c) the larger public not “getting” statistics in the way they’ve been presented. The NYT had, if I recall, Clinton at 90% chance of winning. That still means that in one of every ten flips of a coin is a Trump win. But people read “90% chance” as “definite win”. I don’t actually know what anyone should or could do about that.
What Nate Silver got right in 2016 were the correlations in the Rust Belt, which were traditionally considered Democrat. 538's model predicted that losing one of those states would likely mean losing all of them for Clinton, because for example the polling errors were likely correlated. And indeed losing there is what cost her the election
Silver's 2012 book "The Signal and the Noise" discusses our inability to rationally process probability, pointing out that commercial weather forecasts (e.g. Accuweather) never list a probably of rain under 20-25%. A 5% chance of rain is a mathematical possibility but people "feel" like 5% = "will never happen" and get angry if it rains.
Nobel Prize winner Daniel Kahneman's life's work is about this, what he calls "System 1" and "System 2" of our brain, where System 1 is a fast responder that provides insta-feedback but is largely incapable of processing mathematical inputs. His 2011 book "Thinking Fast and Slow" summarizes his work well.
I'm not sure popular media can be trained to frame statistical probabilities in a way that doesn't provide people with the certainty they crave. But who knows?
I think it’s a confusion between the likelihood of winning, no matter by how many votes, and the predicted percentage of votes per candidate. The latter is more commonly presented to readers from polls. So it’s not too surprising if it gets mixed up with the former, which is used by Nate et al and uses also percent as the unit.
Say a national poll predicts 55% of votes for Clinton, 40% for Trump. Whereas 538 predicts 70% chance of winning for Clinton and 30% for Trump. It’s easy to confuse the two and think the second prediction is much better for Clinton when it might be much worse.
Like the coyote in El Viaje Misterioso de Nuestro Jomer, the cartoon fox tells you only just enough to light the path towards statistical enlightenment. You must walk it yourself.
> Polls have more impact if they are more representative of statewide turnout among demographic things he chose like “black” and “low income.” This is why his predictions were so accurate for Obama’s 2008 and 2012 elections
I find this argument strange, because black turnout was unusually high in 2008. That should have a negative impact on the accuracy of statistical adjustments, not a positive one.
I think he made an estimate for the increase in black turnout. If I were designing the model, and I believed turnout is the biggest factor (maybe inconclusive among political scientists), I would look at the circumstances where turnout changes based on candidate's demographics and validate it across statewide and congressional races.
However, we will never know, because they never published the code.
> I think he made an estimate for the increase in black turnout.
I think that kind of adjustment is usually the responsibility of the pollsters, with their likely voter models. I don't think FiveThirtyEight directly tries to also apply such an adjustment, because that would be at serious risk of overcorrecting.
Similarly, this year many pollsters have added level of education as a factor to their demographic weighting, to address a shortcoming in their 2016 performance. FiveThirtyEight consumes those poll numbers without adding their own layer of demographic adjustment.
I think polls being more representative of turnout amongst minorities could help indicate a potential black swan event for the election. If turnout does return to 2008 and 2012 election levels, polls featured in this fivethirtyeight article [1] indicate Trump is performing better amongst black and hispanic voters. Both demographics are seeing a 10-15% swing in support for Trump compared to 2016, which could theoretically cement swing states like Florida, Pennsylvania, and Michigan.
I don't think it's likely but if those polls are indicative of what's actually happening, we're talking about potentially a 2-4 million vote swing in Trump's favor. Here's a link to estimates of voter turnout in 2016 [2].
You're misreading or confused or something. The 538 piece points to a ~10% swing away from the (in the case of black voters) 82% that favored Hillary. This is very different from a swing all the way to a 10% preference for Trump. A bigger turnout by (these) minority voters, assuming they cast votes even vaguely in line with this polling, is more votes in Biden's column than Trump's. That's bad news for Trump.
I think you're misreading my statement, it's a 10% swing in the direction of Trump, not a 10% overall preference for Trump. From 82% favoring Hillary to 71% favoring Biden for black voters. That's a 10% change towards Trump's direction. If 16 million black voters participated in 2016 then that's around 13.1 million votes for Hillary and 2.9 million for Trump. If polling is correct this year and we see around similar turnout (not even an increase), then it'll be around 11.3 million votes for Biden and 4.7 million for Trump. So a 1.8 million vote swing in the black vote.
That's just a really rough calculation and doesn't account for the Hispanic vote either in that article.
Apologies, my above explanation is actually wrong. The article is actually referencing the margin of the candidate. So it was a 82 point margin between Clinton and Trump in 2016, which is still difficult to interpret because it doesn't mention if this includes 3rd party votes. But assuming 98% of voters voted Clinton or Trump, this would mean that 90% of black votes went to Clinton and 8% went to Trump in 2016.
This would then mean that if 98% vote Trump or Biden in 2020, we'll see something like 84% of the black vote for Biden and 13% of the black vote for Trump. A 5% overall change using 2016 voter participation numbers is still somewhere around 700,000 vote change in the black vote, which is certainly not insignificant. Adding in the change in the Hispanic vote (a margin change from 37 points in 2016 to 23 this year), this could certainly change swing state outcomes.
But unless I'm misunderstanding something, the change in approval is already factored into the current predictions. If there is a greater than expected turnout by a group of people voting - in aggregate - more for Biden than for Trump, then that favors Biden relative to the current predictions. So that's not going to be a reason for a surprise in the other direction.
There are plenty of reasons our predictions might not match reality, but they're not going to be wrong in that direction for that reason.
It wouldn’t be a black swan event, it would simply be a variable that got re-toggled on. As in, there was a large black turn out for Obama, and there wasn’t for Hilary. What if we turned that variable back on to true for Biden? That’s about all the rocket science involved.
> I’d think that if Trump were to win New Jersey or, even more so, California, that this would most likely happen only as part of a national landslide of the sort envisioned by Scott Adams or whatever.
That's a valid intuition to have but you can also clearly make the argument that if Trump wins California you're in such a weird scenario that using the traditional wisdom about correlation is dangerous. The point that 538 have tried repeatedly to make is that firstly: if you're conservative in your level of confidence you'll give a higher likelihood to outliers, and secondly: It's not particularly useful to focus on whether X has a 3% or 4% chance.
If Trump wins California, we aren't going to be talking about whether the chance was 3% or 0.3% we're going to be talking about that Nuclear explosion that wiped out 25million Californians.
For the same logic the reason that Trump winning Alaska given winning New Jersey is lower than given losing New Jersy is because your sample size is rubbish. The chance of Trump winning Alaska given losing New Jersey is an accurate number, the number of Trump winning Alaska given winning New Jersey is like saying "How likely is it Trump wins Alaska given the UK gains US statehood" it's like.... well... if that happens then we're so far outside of what the model thinks can happen then you should be that we're just gonna say it's 50:50 - because who the hell knows.
It's not like saying "Oh well if X swing state goes blue, Y will probably follow", the scenarios in this article are so bizarre that the model should rightly be very cautious and probably default to either refusing to give an answer or just default to 50:50 or the same probably ignoring that data. The implicit bias in this analysis seems to be that if NJ went Red that would be because Trump won by a big margin, but that's not a likely enough scenario to actually get numbers for, and is so unlikely that things like "The supreme court threw out all the ballots for inner city areas" start to become valid possibilities.
It is worth noting that they explicitly aren't accounting for any election tampering/throwing out ballots in the model, so that final hypothetical isn't something they're factoring in. Your "an explosion kills everyone in LA" is closer to the sort of things that the model is "considering" in so much as a pile of statistical code has any understanding of what could happen in the real world to cause the outcome it's putting forward.
This is the important point here IMHO. There are two errors that the tails need to deal with, voter shifts that are missed by polling and black swan events that completely upend the table. I think that the 528 model lumps a lot of the long tail into the second category, which then basically becomes a "let's throw out most of the rules and make wild guesses" territory. There is so little worthwhile information in those tails I am really surprised that this is the focus of the disagreement.
That's a pretty bold claim to make about essentially anybody in regards to the US presidential election. Not that I don't believe his account is even-handed and valuable, just that I'm curious what makes you say he has no dog in the fight.
I think the third post in that thread really nails the underlying issue here. We, as humans, know that it would be silly to think that some configurations could happen. We know there's no reason to expect Biden to win Alabama outside of him winning nearly every state.
A statistical model only has a vague idea of context/the real world. It looks at polls (and probably not really that many polls of Alabama or Mississippi or Alaska) and sees that, statistically, Biden should win 3% of the time or so.
It doesn't have a specific world set of events in mind that would cause that, it just knows that that's how the numbers go, and thus may lead to weird circumstances in the grander results because it has to make the world match the numbers in these small corners.
One thing to note: 538 uses a t-distribution (and calls out on regularly in their podcast that this yields much heavier tails than a normal distribution). Even 40,000 samples is not enough to characterize the tails.
So, it seems to me that the entire article is predicated on a faulty conjecture, namely that 538 uses a mixture of a normal distribution with an independent heavy-tailed one. (It's not explicitly stated what the author thinks the base model is, but I think "normal" is a reasonable guess.)
I'd be interested in seeing a reverse-engineering analysis of 538's choice of distribution parameters, and extrapolation from there to see if these pathologies still arise with (much) larger samples.
...
That said, ultimately, the choice of how fat to make the tails is a modeling decision, and how the models behave outside the regime of interest isn't as important as how they behave within the operating region. There are key ways we can evaluate goodness of fit once we have results (e.g. bias, MSE) which we can use to determine just how wrong the model was as a predictor, and chances are pretty good that we won't see, say, Trump winning NJ, so we won't actually be able to validate the tail correlation with the vote in PA. But we will be able to validate the correlation in margin between PA and NJ.
Maybe 538's tails are too fat, and every prediction in the 80-95% range ends up going as predicted. Or maybe they're not fat enough, and some races in the 99% bucket end up going the opposite way. Point is, we won't know for sure which models were the best predictors until we can verify the predictions.
(see: all models are wrong, etc. Newtonian mechanics work great as long as your objects are big and slow, for instance.)
It seems like the behavior between WA and MS could just be statistics saying that WA and MS always[1] vote for the opposite candidate, rather than considering a massive sudden change in the direction that one of them votes in. E.g. it's not reflecting who they vote, just who they most vehemently disagree with.
I'm not sure why that kind of interstate correlation should impact predictions?
<incoherent rambling :D>
IANAS but it feels like these correlations were added to compensate for the failure in 2016 to recognize that state A going one way implied that state B would also go that way. It "feels" like a more correct approach would be to compute some kind of error/weakness measure in a states polls by bringing in those of its geographical neighbors and incorporating the polling error of that entire block vs prior years. Or something.
The intuition I'm having difficulty conveying is that actual voting correlation is based on neighboring states only because you've got bubbles of ideology that aren't strictly cut along state lines. If strength of opinion in a bubble is going one way, then you'll see that mostly in the state at the center of the bubble, but the bubble still spreads into neighboring states, and a "stronger" bubble could push it geographically further into those neighbouring states, and/or could increase the bias in areas inside the bubble.
</rambling>
> I'm not sure why that kind of interstate correlation should impact predictions?
538 has low positive correlations between states on average, which actually has a big impact, it increases overall uncertainty (and therefore Trump's win probability). Why? If the states are not correlated, you usually end up with a few states going off the rails, like Trump winning Colorado without any nationwide swing.
No, what Gelman says is that he suspects that to compensate for the fat tails in 538's state distributions, they had to reduce the between-state correlations, to get a desired overall level of uncertainty.
This implies that correlations increase, rather than decrease, the overall level of uncertainty.
This is also easy to see from a basic probability perspective, using the concept of variance. For example, if you have two coin flips, with outcomes {-1, +1} chosen uniformly at random, then the sum has variance 2 if the flips are independent, but variance 4 if the flips are perfectly dependent.
Could this just be a result of low sampling by the author? If you reduce a sample to only include some tiny edge case, the resulting data points are going to be weird in random ways.
Somewhat interesting, however the guy lost me more and more the longer he argues. So, the various anomalies in the dataset are somewhat interesting, but having weird outliers in the margins is an entirely expected effect. Just because there are not many datapoints. So when you filter for something marginal like Trump winning New Jersey, then the statistical error increases and therefore it is entirely unsurprising that something weird happens. Thankfully, these systems are designed to work with probabilities, and these outliers are weighted down.
Additionally, getting worked up about a 3% chance of Biden winning Alabama. I mean, what does a 3% chance even mean for a one off event, compared to a 5% chance or a .3% chance? I know fully well, that it means I should bet $100 if I can get more than $3000 payout, but the trouble is that is only if we bet often enough. (Perhaps often enough on different things.) For a one off thing, the important part is, it is with a very high degree of certainty a loss of $100. So any claims that Bidens chances of winning are too high should be regarded with high suspicion.
Also, I listened eralier to Nate Silver's model talk [0], where he discusses quite a few problems with low quality polls in some states.
There are more than enough data points to determine the between-state error correlations, many of which seem to be very off.
> Additionally, getting worked up about a 3% chance
The weird between-state correlations actually have a large effect, they increase state and nationwide uncertainty and as a result Trump has a higher chance of winning.
It seems more likely that it's actually the other way. Nate Silver has specifically said he built the model to have relatively high uncertainty, especially with the volatility of this year, so this seems more like the outcome of intentional decisions to not let the model be overly confident.
I think it's an error in the model structure. If their goal was to artificially increase uncertainty, there are more reasonable ways to do that than adding weird between-state correlations (like the uncertainty index which is part of the model). WA and MS definitely should not have a large negative correlation.
As a US voter I am very frustrated With and tired of this obsession with election forecasting. Is it going to influence whether or not you go out and vote? If not, what is the point of it?
What value does something like fivethirtyeight add to our democracy, if any? Is this motivation the same as that of diving deep into baseball stats or Star Wars starship engineering, just like “nerding out” for its own sake?
Contrast the voter who never looks at any of these polls with one who keeps up with them daily. Is the latter voter better off in some way? Is this just about trying to read the tea leaves so you can strut and preen later about having been correct, should the dice roll be in your favor?
My concern is that these things are distracting and may actually dissuade some people from voting because they think they “don’t have to.”
Here’s an idea: everyone go vote for whoever you think the best candidate is regardless of what a stack of polls say.
Someone set me straight here, what is the point of all this stuff.
> What value does something like fivethirtyeight add to our democracy, if any?
One possible use of polls and election models is for helping people who want to donate to candidates determine which races are closest and where their money is most likely to have an impact.
> Here’s an idea: everyone go vote for whoever you think the best candidate is regardless of what a stack of polls say.
Because the US does not have ranked choice voting in most elections, polls are useful to determine which candidates are viable. If your preferred candidate is only polling at 5%, they are pretty unlikely to win, so you might want to vote instead for whichever of the leading candidates you find most agreeable.
Apart from what others have said, I think it also good to have independent polls to shine light on fraudulent elections.
For example, see what happened in Belarus recently where public polling is banned.
Well, I’m not particularly affected by this (I live in a state that hasn’t been considered purple for at least a decade and probably shouldn’t have been for the decade prior), but... most election systems have some form of strategic voting behavior—either built in (all manner of ranked voting), or based on understanding of the system and being underserved by it. In the US, that could look like leftish people in VA seeing that the state is reasonably safe to do its part to oust trump, and people of the same lean in NC not seeing the same level of safety. Or rightish people in Georgia feeling less safe bucking the Rs, and so on.
In this system, there are two ways to democratically influence politics by voting:
1. Vote your preferred candidate
2. Withhold your vote from the party more closely aligned with your views, in hopes of helping shift its coalition priorities
If you fall into the second category, accurate forecasting makes a strategic difference.
It's worth pointing out that the other thing 538 does besides political polling is analyzing the performance of sports teams. Americans are just drawn to the horse race. (See also: American Idol, which also involves voting.)
>Here’s an idea: everyone go vote for whoever you think the best candidate is regardless of what a stack of polls say.
On the one hand, I think I get your sentiment. On the other, I mean, we are all just solitary individuals floating through this life. Almost all of our important decisions are make at least in part (or more in some cases) dependent on the thoughts and actions of others. That's natural, right? You do otherwise in your life?
If the above is true, it makes total sense why one out of 7 billion plus people would want to understand the choices of others before making theirs.
Great question, and I share some of your concern, though I can imagine some positive framings in addition to what you wrote. For example, to use an analogy, what’s the point of trying to predict the weather, or trying to predict the stock market? There are lots of reasons including planning ahead for likely outcomes, the ability to protect against losses, and last but not least making money.
I can also imagine that the desire to talk about the potential outcomes is valuable as a social activity, and doesn’t necessarily need to meet a standard of influencing the vote, or adding to our democracy.
> My concern is that these things are distracting and may actually dissuade some people from voting because they think they “don’t have to.”
Of course if your concern is founded, this can go both ways... if the polls show the candidate you favor starting to lose, it could be a call to vote.
If polls are distracting and dissuade voters, then unfortunately election results might do exactly the same or worse. When a state has been solidly red or blue and not purple for 50 years in a row, people do (perhaps rightly so) jump to conclusions about the outcome in advance.
One question we could ask is whether, if voting were made mandatory, would election predictions go away? I’d speculate no.
Diving deep into baseball statistics has actually changed how the game is played. I understand your sentiment overall, I don't agree with your sentiment because there are valid use cases for predictive analysis in polotics, but regardless G whether I agree or disagree about political polling - using baseball analytics as an example is not appropriate.
The point is that people want to know what's going to happen before it happens. If I believed a Democrat was sure to win, I would want to allocate my stock portfolio different than if a Republican were to win.
Also, you can make money betting on the outcome itself. If the odds you get are underpriced relative to an accurate forecast, that's a great bet to take.
Furthermore, these forecasts influence where politicians put their focus. Let's say you're Hillary in '16 and you think Wisconsin is yours despite the forecast showing a narrow lead, maybe you should reconsider.
There's nothing scientific about any of this trash. It's a weird conflation of the degree of the incompetence of pollsters and the degree to which opinions can be changed within a span of time. And it's completely unfalsifiable. When Trump wins, the true believers will say "We gave him a 9.684% chance of winning! It's only your ignorance that makes you think we were wrong" and they go back to poring over their race tables.
It's an orgy of false precision.
edit: the entire debate is based on the weird assumption that if a prediction about a particular state is wrong, then pollsters must have systematically gotten middle-class Hispanic women over 40 wrong, therefore the odds of other states will change. It's all based in the reification of particular categories that are axiomatically significant for their profession.
> And it's completely unfalsifiable. When Trump wins, the true believers will say "We gave him a 9.684% chance of winning! It's only your ignorance that makes you think we were wrong" and they go back to poring over their race tables.
> It's an orgy of false precision.
The false precision is pretty obviously coming from you, not the FiveThirtyEight pages that never show more than two (or rarely three) significant figures, and emphasize in every other way they can that the numbers are approximate and uncertain. Have you seen the width of the 80% confidence intervals on their graphs?
As for falsification: all of their predictions are for testable outcomes. We'll always know soon enough who actually wins an election, and which states they won, and by what margin, and who turned out to vote. That's all public record. The only part of the post-hoc analysis that is non-trivial is figuring out how a candidate fared with specific demographic groups. It's imperfect, but between exit polling and precinct-level demographic information and election results, it certainly is possible to detect large pre-election polling errors resulting from inaccurate demographic weighting.
So, wait, you're offended by the election modellers making a prediction, and yet you yourself are making a prediction? What's yours based on? Time machine?
The negative correlation between NJ and AK is curious. What I'd like to see here (and in all of these forecast sites) is some confidence analysis. If you pick Trump in NJ, looking at the plots, you've selected a tiny fraction of the data to examine. Who cares if the predicted value is wonky; show me the confidence interval!
Treating this as a tea-leaf reading (that is, deliberately searching for meaning via free association, without investing it with a truth value) I'm reminded of the "own the libs" meme. I see folks on foxnews.com comments bragging about it; I see lefties complaining about it, but I suspect that it's overblown and not actually a driver behind people's decision-making. But that's what comes up for me when I see "NJ goes Trump" forcing "AK goes Biden".
I'm amused by the resulting thought experiment... if dems started airing "socialists for Trump" campaigns in otherwise safe GOP states, would it move the needle there? Even sillier: if you aired those ads in NJ, would it move the needle in AK?
Long Island is really red there. It’s really hard to say how a democratic stronghold like NYC and something literally a 45 min train ride next to it could vote so differently. Long Islanders are not separate from NYCers, they commute to and work in the city.
To your question, could experiments work in similar situations like this across the country for either side? I think so in the next 50 years as demographics shift (and I don’t think it’s as simple as urban liberals taking over, people do become more conservative as they get older). God knows the dynamic at work between NYC and Long Island in 2016, but it’s obvious things are in flux.
I’ll make a bold prediction here. If Long Island is that red again, yeah, you better believe the typical rust belt states are staying red.
I'm surprised that a lot of people almost worship the 538 model, when there is a better model with more competent people behind it. Economist is also more open about how their model works and it's open-source. After reading the methodology I was pretty impressed.
While numerically literate, I don't understand the details of the 538 or economist models. What I do know is that 538's model has had a great track record. It gave Trump one of the highest chances of winning in 2016. It did very well in prior elections. And both models are essentially predicting the same results: ~10% chance of Trump winning.
How is being slightly less wrong than everyone else "having a great track record"? Serious question. Because I find it hard to take any of them seriously after the debacle that was 2016.
They were generally correct with their prediction except in 3 states where nobody had been doing detailed polling because the pollsters didn't think it would matter.
It simply wouldn't have been possible for the models to be more accurate with the data they had. As they say, bad data in, bad data out.
That's why this time around the pollsters made sure to be more thorough in their polling.
> That's why this time around the pollsters made sure to be more thorough in their polling.
"This time is different."
I've heard that enough times to be highly skeptical. I'm also deeply skeptical of the notion that polling is even remotely correlated to actual results. Cultural and historical trends play a drastically higher role and are almost always left out.
> I'm also deeply skeptical of the notion that polling is even remotely correlated to actual results.
But it has been strongly correlated to the results in basically all elections so far in all democracies on the planet. Taking 2016 as an example there has been a very strong correlation between polling and the results. The national polling averages were only 3 points off from the actual result. If that's not correlated I don't know what is.
Polarization and the unacceptability of publicly saying "I voted for X" also didn't really exist prior to 2016. The fear of getting doxxed, combined with a record low level of trust in institutions and the media, leads to skepticism toward answering polls truthfully, IMO.
While polarization is bad now, it's nowhere close to historical extremes, and it's not even as bad as it was during the post-Vietnam era only a few decades ago.
As for fear of doxing: plenty of people openly supported and voted for George Wallace (a noted white Supremacist) back in the day, and even Roy Moore (accused pedophile) just 2 years ago. Proud Boys members openly pose for the cameras even as they espouse racist views, and QAnon members brag about being part of QAnon.
As a data point, I live in a conservative-leaning area of a purple/blue state. Along my normal driving routes I had seen quite a few, but over the past few weeks they have mostly been taken down (in every case, other republican candidate signs still stand). Scenario one, Trump supporters supported him all the way up until now, enough to donate to his re-election campaign to buy a yard sign, and in the final weeks of the election, decided that they'd had enough and taken down their signs. Scenario two, some random stranger decided that it's their right, nay, their responsibility, to moderate someone else's yard signs.
Not every Trump voter is a "Proud Boy" or noted white supremacist, maybe even some of your friends are who "just aren't interested in politics" because they know that if they were honest with you, you would flip out. Some understand that this attitude about yard signs is actually representative of an entire worldview, and opposing that might actually be the lesser of two evils.
Similar to how "not every Trump Voter is a >>Proud Boy<<" not every Biden supporter is a yard-sign stealing communist. Actually the extremists are in the minority in both groups. If you start going down that road and base your vote on how bad the worst people on the other side are, democracy is pretty much collapsing already.
Sure - I am above my quota for politics posts on HN but since this article is about problematic tail behavior I'll go ahead anyways. You are right that probably a very small minority of Trump opposers are actually stealing yard signs. However, a larger component are advocating for censorship of opinions and facts that disagree with their worldview, and many more will readily shut down any opposing discussion about immigration policy or international trade, supporting police, or even simply wearing a red hat, with strong and viral accusations of "racism" - signalling to their allies that this person should be punished, and many of those allies are happy to comply in whatever way their position in society allows them to. These attitudes and behaviors are pervasive and malignant; the yard signs are only a visible symptom of a larger problem.
Essentially I think that those narratives about one side "shutting down" the other side are not true for either party. In my opinion they are mainly the result of filter bubbles where you are constantly presented to all the outrageous things that the other side is doing (the above mentioned article is clearly an example) while actually people remain largely civil and are not tied to their political identity in such an extreme way.
The shy Tory factor[0] is likely to be even stronger this year than in 2016 when it comes to polls. After 4+ years of being harangued and called every name in the book by the vast majority of national culture (movies, music, TV, news, social media, news-entertainment), along with increasingly hostile projects such as https://donaldtrump.watch/ I imagine less of his more subdued supporters are going to be honest with pollsters.
But wouldn't that mostly be concentrated in places/areas where their votes aren't likely to matter? Having just driven through the US South in the last month, I can confidently tell you people are not in any way shy about their support for Trump. I saw more Trump signs and flags than I saw US flags.
It's also likely this works in both directions - if you support Biden, I bet you don't have a yard sign for it if you live in Mississippi.
By construction, the effect is strongest the more you're not in the majority, which also means your unspoken support is more likely to not matter on the actual outcomes of the election.
It's more relevant in swing states, which are by definition mixed. I.e. if you live in Pennsylvania, Michigan, Ohio, or Florida, you can't really be sure how your neighbors will react to a Biden/Trump sign.
If you're right, the support for Trump should be higher in polls where people talk through a phone menu or the internet, compared to live interviews.
Newsflash, they aren't, so your hypothesis must be they're too shy to admit their Trump support to a machine?
Everybody is just fighting the last war where Trump suddenly won for a couple of reasons. So, the Dems are scared they are missing something, and the Reps are going "Haha, who cares about modelling".
Will, the forecasts be perfect? Nope. But is the margin rather large, but not unsurmountable? Yes it is. Are the mistakes from last time repaired? Yes, they take care of uneducated whites. Is there evidence Trump has found a new source of voter support? I haven't seen it.
Anecdotal, but I have never met a Trump supporter that was shy about who they were voting for, this year or in 2016.
If anything, Trump supporters have been extremely vocal about who they were supporting, to the extent that they frequently violate social norms and try to takeover events and gatherings to make their political affiliations known, like this week with Among Us.
Saying that the ultimate outcome had a 1 in 4 chance is not wrong, slightly wrong, or less wrong. If the weatherman says there's a 1 in 4 chance of rain, and it rains, he wasn't wrong.
No it doesn’t, and this is a fundamental misunderstanding of how probabilistic forecasting works. If it rains 9 out of 10 times a weatherman says there is a 30% chance of rain, they are a bad weatherman, but they aren’t much worse than if it rained 0 out of 10 times they predicted a 30% chance of rain. A weatherman accurately assessing the probability of the weather forecast would see it rain around 3 out of 10 days they say there is a 30% chance of rain.
Only if they say that every day, and it rains every day. If the most likely outcome happened every time, then the model is likely wrong/under-confident. The prediction is never going to be 100% accurate until the event is happening/has happened. Up until then, there's always a chance you're wrong or something can change. Being wrong once isn't necessarily a sign that the whole system is messed up.
This is a fundamental misunderstanding of probability. Low probability events do happen, and it doesn't inherently mean the estimated probability was wrong.
Nowhere did I say it is. I simply think that, if one were following the right information, his win was not as unexpected as the coastal media presented it as being.
Personally I would have put it about 60-40 Hillary-Trump.
You started off by calling the 2016 prediction a debacle, but now you're saying you would have put the odds about 12 percentage points differently. That doesn't seem like a big enough disagreement to warrant the kind of vehement criticism you're throwing around.
I called it a debacle because 99.9% of media sources, pundits, politicians, political figures, or anyone else thought that Hillary had anything less than a guaranteed win. I'm simply suggesting it was actually always a close race, but that the media ignored this because it went against their ideological model / they weren't familiar with places like the Rust Belt.
You really need to be more clear about which of your criticisms are against FiveThirtyEight specifically vs against the media in general. Because now it's looking like you are complaining about the media in general and using that as justification for mistrusting FiveThirtyEight, in the context of a discussion specifically about FiveThirtyEight being a notable outlier from that general media trend.
It's still just unclear to me how FiveThirtyEight assigning Trump a ±30% chance of winning can be considered "accurate" or "good".
From my point of view, this is only because said people considered Trump winning so extremely unlikely that 538 getting it sort of right appears exceptional. In reality, they were still quite wrong, just slightly less so. Ergo I don't see much value in their model.
I'm seeing the same exact thing today with Biden at a 90% chance of winning.
At the end of the 2016 election, Donald Trump had a predicted 1/4 chance of winning the presidency. Does this seem like a massive debacle that Trump won under these conditions? Not to me.
That doesn't really answer my question. It only indicates that they were slightly less wrong than every other media source, not that they have a good model.
If I had a laptop that only worked 1/4th of the time, rather than 1/20th of the time, would that make it a reliable laptop? I don't think so.
If they were wrong, but “less wrong” than all others, you should pick their model (unless you have an oracle, because the alternative - flipping a coin or “relying on your intuition” is rarely better).
Also, it doesnt make sense to look at a single prediction to evaluate a model.
Out of all the predictions they have made (did you look at individual state predictions?), how many were correct (and how confident were they?) - how many were wrong (and how close to 50% were they?).
That is how you evaluate a model (aka cross entropy)
That isn't the criticism. The criticism is the appellation of it being "unlikely."
For example: anyone paying attention to the Rust Belt ±1980-2016 would have dramatically upped Trump's chances in Pennsylvania and Michigan. FiveThirtyEight had Hillary with 70%+ chance of winning both, which to me, shows a deep ignorance of actual cultural factors.
> anyone paying attention to the Rust Belt ±1980-2016 would have dramatically upped Trump's chances in Pennsylvania and Michigan.
There was a very decent chance that Clinton could have won in 2016 (if any factor had gone slightly better for her), and if that had happened, nobody would be saying this now. This is literal hindsight bias.
Aren't you doing the exact same thing that you're accusing me of?
My view is simple: the media completely, totally got 2016 wrong, mostly for sociological reasons. The people making the predictions simply had a huge blind spot. Brexit is another similar situation. The fact that Hillary almost won or Brexit almost didn't happen isn't really the point, because both things were never expected to be even remotely that close. Had the predictions been "Pennsylvania will be close", it would be relevant, but those weren't the predictions.
It seems like you are conflating probabilities with absolute certainties. If I had a 1:10 probability of winning the lottery, I would probably take it. If I had a 1:20 probability of getting injured if I leave my house today, I’d stay home. If I did get out but didn’t get injured doesn’t mean the model was wrong.
If there was sufficient data to assign a 0% or 100% probability to an event, that’s what a forecaster should do. If there isn’t sufficient data, then anyone who claims there is a sure thing is a charlatan.
If I tell you you're not likely to get two heads in a row, and you do, does that make me un-reliable?
It's unfortunate we can't just run the election again a few times, and actually find the rate at which Trump is elected given the polls.
And it's not empty signalling if 538 assigned Trump a higher chance of winning; they were pretty much the only ones saying he has a chance. That is why people think the models are useful.
If everyone was wrong, it is reasonable to believe a low probability event occurred. However, given the extent to which people predicted a Trump loss (say 1 in 20), which is significantly rarer, given that the event occurred it suggests the model that predicted a Trump win with the greatest probability to likely be a more accurate model.
For a post on a blog presumably focused on statistics the author seems oddly concerned with a model that predicts odd events as being "possible". The difference between "possible" and "impossible" is \eps << 1 - there's no real distinction to be made there in practical statistical terms.
Likewise asserting that California with a 3% chance of going Trump is absurd is an unreasonable degree of overconfidence. Assuming maximizing expected return, the author is implying that they would be willing to take a bet that Trump would lose California with odds >> 97::3, i.e. presumably they would take a bet where I bet $1 to every $99 they bet. To be critical of a model based on outcomes it predicts with tiny probability you need truly remarkably biased priors.
I don't know if I'm reading your comment wrong but I would happily take that bet- I think most people would. Want to make it? The election result of California isn't a matter of probability; it's an empirical matter that the number of people who will vote for Biden in California far exceeds the number that will vote for Trump.
To you and the other comment - as much as I dislike Taleb's rhetoric, this is precisely the sort of bet he has made a lot of money on. People round rare event probability down to zero, and if you bet against them enough with sufficiently extreme odds you'll eventually (and in expectation) hit a home run.
I would be more than happy to make this bet with anyone willing to take the other side - as in literally, find a modern middle-man system and I'm game.
But would Taleb specifically make a one time bet on one particular Black swan? Isn’t the idea the same as venture capitalism, where this idea only works if you do it with all possible black swans?
You’d have to be crazy to take this one specific bet, you can only realistically take all possible improbable bets.
Choosing to engage in these sorts of bets systematically exclusively and not individually only makes sense if you're not maximizing expected return. You also don't know if I engage in these sorts of bets systematically.
This is undoubtedly useful to know when playing around with different scenarios.
However, for my use of 538, I’m perfectly happy to ignore such scenarios (such as Trump taking New Jersey). I can call the election in his favour by myself in these scenarios without needing the model.
The thing about this election is that what if Trump himself is some sort of wildcard that can't really be properly forecasted in the polls?
Why is there so much fascination with polls to begin with? I understand that there are betting markets, but it seems sort of silly. If you had a 100% accurate poll, for instance, then what would be the purpose of the actual election?
> Why is there so much fascination with polls to begin with?
Polling is useful for candidates and gives them ideas on where to target outreach and spending.
For the rest of us, it gives us something to watch. With Presidential campaigns running for almost two years in advance of the actual voting weeks, there’s a huge gap between when the thing starts and when we see results. This way, people have something to fill the time. Even now that voting has started, we are still another 10 days until the voting is done and likely another 7 after that until we have a sufficient count to know who has been elected.
That’s a long time for a populace that’s worried, distracted, and interested, especially since so many of us live in states where—due to the mechanics of a broken election system—we can’t do much to influence the national outcome.
> The thing about this election is that what if Trump himself is some sort of wildcard that can't really be properly forecasted in the polls?
The only way that would work is if he made people more likely to lie about their voting intentions. Now, there may be something about Trump that makes polling methodologies less accurate (notably, many pollsters have started to take into account education, which turned out to be unexpectedly important last time round) but that points just to bad methodology, not inherent unpredictability.
Now that you bring that up, if we had a 100% accurate poll that would be really good for productivity wouldn't it? Perhaps it wouldn't give voters the same feeling of self-determination but it'd save a lot of resources in fundraising, going out to vote, counting votes
The sensation of self determination is the entire point of democracy though. We don't use democracy because we think masses of people are particularly wise; we use these systems because they feel more fair than the alternatives and that perception of fairness produces good results (peaceful power transitions.)
> I mean technically the election is a 100% accurate poll. You get data from every voter.
Actually an election is literally a poll in the sense of a sample, you are counting people at "polling stations" in order to gauge the public mood about who should be president.
When you see it that way, Nate Silver is predicting a sample of an unknown distribution, the "true" distribution of people's preferences.
The fact that you can only make one officially binding sample ("the voters") is a practicality, as is the fact that there's an electoral college that means votes have different values. The fact that turnout matters is another issue, a statistician might call it a sampling problem.
The problem is that 538 is correctly factoring the voting fuckery done at the state level which ruins the voting correlations. Gelman seems to be modelling fair elections - ha! Now the question, how did 538 come up with the correct model which takes into account vote manipulations at the state level? /s
I’ve decided to start my own election forecasting site that only ever gives 50/50 odds. Then I’ll just have to wait for the next 2016-style underdog win and my inevitable victory lap in the press as The Guy Who Called the Election.
Funny thing, can't submit this to r/politics, as they seem to have a tightly curated whitelist of allowed domains that must "Be notable, as defined by our domain notability guidelines. Notable domains will consist of news organizations, research organizations, political advocacy groups, governmental agencies / bodies, and political parties."
And apparently columbia.edu does not fulfill those criteria.
Don't bother, /r/politics is one of the most censored places on the internet. It's best avoided. Despite its description don't expect any actual adult discussion of politics there.
Only because people don't understand probability and statistics. 538 gave Trump a ~30% chance of winning. The fact the people seem to think anything less than 50 equals 0 is a problem with people's understanding of statistics, not the statistic itself.
The GP is saying 538 did better compared to other news outlets. You need to also provide a source for another news outlet giving Hillary less than 71.4% if you want to show that the GP is wrong.
The assertion you're responding to was not about Trump's odds in an absolute sense, but relative to other pollsters and forecasters. So pointing out the precise value FiveThirtyEight assigned to that outcome doesn't refute anything until you compare it to somebody else's number.
My biggest issue is when people say that it's a probabilistic model, and therefore it wasn't wrong in 2016 because 28% chance of winning is pretty high and you don't get probabilities. Well, guess what, this kind of model that provides a probabilistic estimate on a future event that cannot be repeated cannot be validated or falsified. It's basically junk science (if it has any aspirations of being scientific).
Basically what they did was bucket every prediction by odds. If they predicted 70/30, it went in that bucket. And they're "right" at about the rate of their predictions. In other words, for every 70/30 prediction they made, the people/teams with 70% chance to win, won about 70% of the time.
That shows that 538 in this case is pretty decent at calculating odds.
Yes, which is why it’s foolish to talk about the outcome of that single race. You are right that there will never be another 2016 election between Clinton and Trump. However, that is only one of hundreds of forecasts made by 538 across multiple election cycles, so we can see how often their probabilistic outcomes align with actual outcomes. The fact that their track record is fairly good across all of these is evidence suggesting that Trump’s win may be more likely to be the realization of a less likely outcome predicted by their model, rather than a fundamental problem of their model itself.
Like I told the other guy, I'm not defending his position, I'm stating it in different terms. And stating why pointing out the odds of a fair die isn't a good counter argument.
Grandposter doesn't believe the die is fair. That's a different argument than the guy I responded to made.
Sounds more like gambling than science; you're making a one in six bet that you can dupe me. More seriously, I would not accept your claim with evidence as weak as that. Roll the dice a few more times and then I'll credit your claim. Rolling the dice only once when you could just as easily roll them a dozen times is junk science.
Incidentally, this trick is something magicians sometimes do. Sometimes when a trick has gone wrong they'll make a wild guess. If they're right, the audience is impressed. If they're wrong, they'll brush it aside with some joke and the audience won't notice/mind much. This works for things card guessing tricks and puedo-psychic/cold reading stuff.
I suppose you meant that election forecasters don't claim election forecasting is a science, not 'no one.' I don't mean to quibble, this meaning was not at all clear in your original comment, nor did I expect that meaning from the context of the comment you responded to.
Presidential assassination, war, video proof of something incredibly heinous (pedophilia?), etc. can absolutely lead to these outcomes. You don't even have to go that far back. Nixon and Reagan flipped states like no-one's business.
I do however agree, that 538's state-state correlation model seems weak.
California and Alabama would only flip during a wave, and that wave would consume any and all states. The fact that 538's model doesn't strongly show that pattern is a failing of it. But, it is not clear if a model that inaccurately models the unlikeliest of events (california flipping while Florida stays blue), does not necessarily mean that it is terrible predictor of it's primary target (Presidential likelihoods).
As a data scientist, I can totally understand Nate's hesitation. Do you impose strong priors on the model to reflect strong domain intuition or do build a model that best characterizes the data it is based on. In the presence of infinite data, you should abandon all domain based priors. For single digit data points, priors are essential. For any number of data in between, it is anyone's best guess.