
How Good Are FiveThirtyEight Forecasts? - jacobedawson
https://projects.fivethirtyeight.com/checking-our-work/
======
empath75
All of the people complaining about how Five Thirty Eight is a fraud because
they "predicted" that Hillary Clinton would win should read this. They gave
Trump a significant chance of winning, more than almost any other poll
aggregator did. In fact, before the election, Nate made a post outlining all
the reasons that Trump could win and gave him a 36% chance of doing so. They
literally said that trump would win based on polls like that 36% of the time,
so it shouldn't have been that shocking that he did.

[https://fivethirtyeight.com/features/election-update-dont-
ig...](https://fivethirtyeight.com/features/election-update-dont-ignore-the-
polls-clinton-leads-but-its-a-close-race/)

You might say that a forecast that's so ambivalent isn't useful, but if
someone told you that there's a 36% chance of rain, you'd plan your beach trip
accordingly. I don't see why predicting a 36% chance of the underdog winning
the presidential race is any less informative.

~~~
keymone
can downvoters elaborate on why they disagree?

~~~
ghaff
It's probably a confusing the map with the territory sort of thing.
FiveThirtyEight predicted Clinton would _probably_ win and didn't. Many people
are (understandably) upset she didn't win and they're lashing out at
FiveThirtyEight for having given them what turned out to be false hope.

~~~
erentz
It might also be something a bit different:

There's a lot of people on the left who are angry with Clinton and the
Democrat establishment for running a very technical/data-driven campaign that
turned out to be very flawed and ran counter to common sense (even her husband
was telling her to campaign in areas she was ignoring due to this "data"). The
technically driven 538 sort of represents that attitude.

Nate Silver has also started to stray into political punditry which is turning
a lot of people off. People who liked him for his data are now having to hear
where he stands on issues.

~~~
ianai
For me, I’ll always monitor and follow his poll analysis. But something about
how he handled 2016 soured my view of him. It’s like he didn’t just “predict.”
He insisted on being correct in a way that was too overt.

Further, let’s not forget all of the dirty deeds orchestrated through
Facebook, online, and hacking. The election in three key states swung by tiny
amounts. It’s not correct to blame what otherwise worked for the perfect
storm. In fact, I don’t remember 538 sayin anything about fake news or
election hacking prior to the election. It was all polls of polls.

------
PaulHoule
This 'calibration' is a missing link for machine learning.

A prediction is _much_ more valuable if you know how good of a prediction it
is than if you don't.

~~~
paulgb
I'd caution about putting _too_ much stock in calibration, though. It's worth
noting that you can achieve perfect calibration just by giving each of n
candidates 1/n probability of winning.

~~~
robto
I don't think you can. If you give every candidate a 1/n probability in
winning, but the actual true probability is less than 1% (I'm looking at you,
Vermin Supreme[0]), then your calibration is going to be way off. Good
calibration is having your 65% chance of victory be right 65% of the time.

[0][https://en.wikipedia.org/wiki/Vermin_Supreme](https://en.wikipedia.org/wiki/Vermin_Supreme)

~~~
paulgb
Right, but say in 2016 you gave a 1/3 chance to Trump, 1/3 chance to Clinton,
and 1/3 chance to Vermin Supreme. Exactly one of them will win, so your
calibration for 1/3 will be perfect. If you do the same for elections with
different numbers candidates, you can have good calibration overall without
actually producing any information over the base rate.

~~~
robto
I disagree. Suppose you have an election with 3 candidates that runs every
year. If you predict candidate A win at 33% confidence, but that candidate
gets elected every year, your calibration will be way off. Calibration is
about aggregating, so you can't really cheat it. Your 33% prediction has to be
right 33% of the time - it's really hard to fake that.

Maybe I'm misunderstanding your position, though.

~~~
paulgb
That's right if you only predict the probability of candidate A, but if you
are predicting 33% for all three candidates each year, your 33% prediction
will consistently be right 1/3 of the time in aggregate.

------
joker3
This is good in the sense that having poorly calibrated forecasts is bad, but
it's not quite as strong as you might want it to be. Imagine that you're
tasked with predicting the next state of a traffic light that just cycles
through green, yellow and red over and over again. If at every step you
forecast that there's a 33% chance of each color, you have a very well-
calibrated forecaster, but it's clear that you can do better.

FiveThirtyEight probably actually is doing better, but I suspect that they're
not choosing to highlight those stronger metrics because they're complicated
to explain.

~~~
jsnell
But isn't that exactly what the last third of the article discusses?

------
petraeus
Can anyone explain how 538 got the last election so wrong?

Its quite obvious in hindsight their model is flawed, but Im not understanding
how is it flawed.

~~~
fourmajor
If you're referring to either the midterms or the 2016 presidential election,
they didn't get either particularly wrong.

------
cljs-js-eval
This gets to the core of Taleb's beef with FiveThirtyEight - what political
forecasts are calibrated? The ones that were forecasted 6 months out, or the
last forecast before the end of the election?

They may only be calibrated when enough of the facts are in to make predicting
easier. Still useful, but not calibrated in the way the general public might
think.

~~~
Symmetry
If you dive a bit more they look at each forcast at each time interval for
which it is made. So they include the 12 month out one, the 11 month out one,
etc when assessing calibration.

~~~
cljs-js-eval
Ah, that's very useful to know. I take my statement back then, this is better
calibration than I thought.

------
objektif
They obviously got 16 very wrong. However, what is even more concerning is
that Nate lately became a political hack, doing the ugly work for the
democratic establishment. Just look at his twitter you can clearly see it. So
I would take any "data science" he does with a grain of salt.

Edit: An example of his tweets:

"It's probably worth noting that while this group, Justice Democrats, calls
Biden "out-of-touch" with the "center of energy" in the Democratic Party, only
26 of the 79 candidates it endorsed last year won their primaries, and only 7
of those went on to win the general election."

Why didn’t he add the statistic that Biden already ran 2 times in the past and
won 0 times.

~~~
pwinnski
They did not, he has not, and one should not any more than one always should.

538's forecast was easily the most accurate for November 2016, by a decent
margin. The majority of critics of 538 are "sore winners" who delight in their
side having won despite being given a less than 50% chance, but as Silver
pointed out at the time: his chances were similar to those in Russian
Roulette, and most people consider that too risky to play.

~~~
objektif
If thats your understanding of making a prediction then he was accurate by
your standards. You clearly did not watch the election night predictions. NYT
live model was miles better than 538s.

[https://fivethirtyeight.com/features/how-i-acted-like-a-
pund...](https://fivethirtyeight.com/features/how-i-acted-like-a-pundit-and-
screwed-up-on-donald-trump/)

~~~
novia
Your link is from 2016 before the presidential election but after the
primaries. Nate is admitting that he underestimated Trump in the primaries and
talks about what happened. This does not seem particularly relevant to the
subject of election night predictions.

~~~
objektif
This is regarding his bias. If anything the link is even more relevant. He
repeated the similiar mistakes.

~~~
pwinnski
His main mistake as highlighted in that article is that they didn't use
statistics for the nomination process, but guesswork. For the election proper,
they did use statistics. So no, they did not repeat the mistake at all.

