
How Can We Tell Which Election Forecasts Are True? - nature24
https://www.quantamagazine.org/20161013-how-can-we-tell-which-forecasts-are-true/
======
basch
I find Silver's celebrity worship slightly irritating.

>FiveThirtyEight and the PEC both predicted the outcome of the 2012
presidential race with spectacular accuracy.

Markos Moulitsas and Drew Linzer (Daily Kos and Votamatic respectively) were
both more accurate than Silver
([https://www.dailykos.com/story/2012/11/14/1161465/-More-
accu...](https://www.dailykos.com/story/2012/11/14/1161465/-More-accurate-
than-Nate-Silver) [http://rationality.org/2012/11/09/was-nate-silver-the-
most-a...](http://rationality.org/2012/11/09/was-nate-silver-the-most-
accurate-2012-election-pundit/)) AND they teamed up this year
([http://elections.dailykos.com/app/elections/2016/office/pres...](http://elections.dailykos.com/app/elections/2016/office/president))
AND The Upshot is showing their forecast as the most confident in its
projections
([http://www.nytimes.com/interactive/2016/upshot/presidential-...](http://www.nytimes.com/interactive/2016/upshot/presidential-
polls-forecast.html#other-forecasts))

Why does Silver continue to be rewarded with praise of spectacular accuracy
while Markos Moulitsas and Drew Linzer don't get mentioned? Hell, Silver and
Wong are products OF Daily Kos. How can you write an entire article about
forecasting accuracy and not even link to their site? At least The Upshot
respects them enough to include them.

~~~
jahnu
538 was big before it even went to the NYT. It was a pretty fresh take on
things back then. It's not purely an academic exercise. Part of this is pure
entertainment and Silver is better at that. It's a complex product he's
selling and he's very good at it. I say hats off to him.

~~~
basch
sure, but this specific article was about which election forcasts are true.
Silver being the poster boy in pop culture is fine, but in this context he
shouldnt be the prime example.

------
skywhopper
I would ask instead: what does it even mean to be "right" about a statistical
probability of a future event? What does 54% chance mean vs an 84% chance when
you're talking about something that's six weeks away? If the event happens or
doesn't, that fact in itself does not tell you whether 54 or 84 was the right
number.

Each group creating predictions builds a mathematical model that simplifies
reality into a set of numbers that necessarily have large uncertainty and
error built in. The models are, by design, not intended to replicate reality.
They are intended to be a smaller, simplified version, retaining as many of
the relevant properties as possible, but with no illusions (one hopes) that
they actually represent reality. (If they did, you'd expect them to make exact
predictions, not percentage chances.)

Presumably these teams do their own math "right", so the only useful question
is "which model is _closer_ to reality?" But there's no way to know that,
except to test them an infinite number of times, and ooops can't do that
either...

~~~
platz
It is very simple. All one has to do is imagine the election being repeated an
infinity number of times under a stationary-non-changing distribution.

~~~
defen
I realize you're joking, but if the election were repeated an infinity number
of times, wouldn't the same candidate win each time?

~~~
platz
The distribution for the roll of a die doesn't change, but the individual
outcomes do.

It all boils down to how this random variable is defined.

~~~
defen
Right, but what mechanism would change the outcome of the election if it were
re-run? Are you suggesting that some percentage of people randomly choose
their candidate while they are in the voting booth?

~~~
platz
I know I do!

------
6stringmerc
Um, we can tell after the election is over? I mean, that's generally what a
forecast is tested by - the outcome versus the prediction. Every scenario is
different, so relying on one model is only good for one test I think. Not
saying building models is useless, I just think the analytical part is flawed
in general because it still relies on humans to respond and humans
are...difficult...

I know a guy who correctly predicted the Obama/Romney popular vote outcome
within 1% of each's actual take IIRC. As of now, he's got Clinton at 47% and
Trump 34%, based on nothing more than personal outlook, hunches, and mulling
over various polling data (he had 47% Clinton to 41% Trump just before the
Conventions). Point being it might be just as likely an irrational human can
point out human behavior (en masse, e.g. voting) on par with a statistical
model, given some particular circumstances.

~~~
matt4077
The problem with that it only tells us how good they were on election day.
It'd be great to somehow evaluate their earlier predictions, which is probably
only possible by looking at an aggregate of election.

(or maybe, since there are predictions and results for every state and every
senate race, it's actually possible to use those for an evaluation. Note that
it's not as easy as scoring a "won by trump" in favor of a forecaster who was
giving trump a 60% chance. You need a measure that expects your 60%
predictions to be wrong 40% of the time, like the Brier score)

------
chimeracoder
> Presidential election forecasts are historically successful and appear to be
> highly precise. Yet they’re often contradictory.

Eh, not really. Primarily elections (at all levels) are hard to predict for a
number of reasons, but the presidential general election is about as
straightforward as it gets.

Different forecasts vary with respect to the exact estimate and the degrees of
confidence, but all the scientific, polls based approaches tend to predict the
same range of outcomes. I wouldn't really call that "contradictory" for two
models to give the same basic prediction but with slight differences in the
errors and exact estimate.

------
maxxxxx
Why do people care so much about polling and forecasts? Why not spend that
effort on looking at issues, voting for the right people and after the
election see who won? This seems a total waste of time.

~~~
basch
the same reason fantasy football can be more fun than watching the games?

according to the forecasters the election is already over, who's model is best
is the only interesting undecided component left.

right now we have an election between a Blowhard Reality Star and a Crony
Crook, where most of the population would prefer neither.

so combining the realities of the choice already made (barring an insane
scandal) with an election between a douche and turd sandwich, statistical
models are about the only semi-honest thing we can turn to.

do you really think this election is about "the issues" and not who is better
at pandering? both of their campaigns are designed to tell people what they
want to hear, and figure out the rest later.

~~~
lintiness
you made me happy. people need to turn this shit off. i keep seeing a
commercial from somebody acutely interested in hillary getting elected: it
pretends trump makes fun of kids with spinal cancer. that pretty much sums up
the discourse right now, and it's not good for anybody or anything --
certainly not the direction of the nation.

~~~
basch
it's all a distraction to give the illusion of offering participation. people
can go home and feel "i contributed" having fulfilled their civic duty.
everyone paying any amount of attention knows its a rigged game ten ways to
sunday.

democracy in general is a feedback loop between the press and politicians with
the voting population as pawns/fodder. journalists and politicians have one
shared concern: staying relevant and needed.

------
losteverything
A question I recall asking a Director was why were there so many PhDs in our
marketing group.

They were all statisticians he said. They were brought in to support decisions
marketing made because as the Director said "people believe PhDs more than us.
They make our decisions believable."

Same for using JD Power. Totally result driven. They got a lot of money to
produce a result.

So to me anytime I see polls the only thing that matters is the money trail.

As far as elections go, we can see the accuracy in a month.

~~~
matt4077
I like how your argument flows from anti-intellectualism to conspiracies, only
to end up at fatalism.

Luckily, there are people motivated by such questions. And even it may not
make a difference to me except for it's entertainment value here, the impact
statistics have had on our quality of life is undeniable.

There was so much invention in statistics and operations research in WW2 that
you can make a credible case that it was the ally's decisive advantage. Since
then, medical research comes to mind as field where it has been useful to have
people who don't answer "dunno – why don't you wait a month and see" but "I
think this may be possible."

~~~
losteverything
Some things you can change the outcome by making decisions partially based on
statistics. Like who to choose for a hip replacement. Do you go to a hospital
that does only orthopedics or a general one?

I recall the event as a young marketer / techie because I always thought I
wanted a PhD. But when I saw how far they were away from business decisions
being only support I knew a PhD at this company was not for me.

There are people who earn a living with polls. This is a not-so-bad way to
spend ones time. There are worse.

------
ISL
By doing the experiment and letting the election happen.

Experiment is the arbiter of truth.

~~~
Scarblac
But say, a forecast says candidates A, B and C have 40%, 35% and 25% chance of
winning. Now the election happens, and one of them wins. Was the forecast
true?

~~~
basch
that is what confidence is for
([https://en.wikipedia.org/wiki/Brier_score](https://en.wikipedia.org/wiki/Brier_score))

"The reason for this is that pundits (the better ones, anyway) don’t just
predict election outcomes, they also tell you how confident they are in each
of their predictions. For example, Silver gave Obama a 50.3% chance of winning
in Florida. That’s pretty damn close to 50/50 or “even odds.” So if Romney had
won Florida, Silver would have been wrong, but only a little wrong. In
contrast, Silver’s forecast was 92% confident that Rick Berg would win a
Senate seat in North Dakota, but Berg lost. For that prediction, Silver was a
lot wrong. Still, predictions with 92% confidence should be wrong 8% of the
time (otherwise, that 92% confidence is underconfident), and Silver made a lot
of correct predictions. So how can we tell who did best? We need a method that
accounts not just for the predicted outcomes, but also for the confidence of
each prediction. There are many ways you can score predictions to reward
accuracy and punish arrogance, but most of them are gameable, for example by
overstating one’s true beliefs. The methods which aren’t cheatable — where you
score best if you are honest — are all called “proper scoring rules.” One of
the most common proper scoring rules is the Brier score. A Brier score is
simply a number between 0 and 1, and as with golf, a lower score is better."

[http://rationality.org/2012/11/09/was-nate-silver-the-
most-a...](http://rationality.org/2012/11/09/was-nate-silver-the-most-
accurate-2012-election-pundit/)

"Also note that Wang & Ferguson got a better Brier score than Silver despite
getting Florida (barely) wrong while Silver got Florida right. The Atlantic
Wire gave Wang & Ferguson only a “Silver Star” for this reason, but our more
detailed analysis shows that Wang & Ferguson probably should have gotten a
“Gold Star.”"

~~~
hsitz
"So if Romney had won Florida, Silver would have been wrong, but only a little
wrong."

No, Silver would not have been wrong at all, or at least we'll never know. He
doesn't predict a winner; he gives probabilities of victory. It would have
been perfectly consistent with Obama having a 50.3% chance of winning (or
63.5% or 32.8% or 99.9% or whatever probability you want to assign) and then
for Romney to have won. Even after the election was over, a Florida victory
for Romney would not have meant that Silver's pre-election probability
assignment of 50.3% to Obama was wrong. That's not how probabilities work.

Think of sporting events. A huge underdog with only, say, a 2% chance of
winning may beat the favorite. This does not mean that it had better than a 2%
chance of winning before the event. Depending on the reason for the underdog's
victory, it may not even mean that the underdog has better than a 2% chance of
winning if they play again. One in a million events happen, and it doesn't
mean that their chances of happening were greater than one in a million.

~~~
basch
read the last quoted sentence

~~~
hsitz
Not sure which sentence you're referring to. And I'm not saying there's no way
to tell which probability assignments are better. I'm just saying that the
mere fact that an event assigned a lower probability happens (e.g., if Romney
would have beaten Obama in Florida) does not mean the assignments were wrong.

This is consistent with Silver's own explanations of what he's doing, and (I'm
pretty sure ) he is careful to say he doesn't make "predictions". (Actually,
not sure, it may be merely that he often clarifies that a different result
than predicted does not make the probability assignment that prompted the
prediction "wrong". The prediction and the probability analysis it's based on
are different things.) I'm not saying that the Brier Score can't differentiate
between better and worse probability assignments, but I am saying it's
confusing the issue if it calls probability assignments "right" or "wrong".
They may be "better" or "worse" than each other, but not "right" or "wrong".
This may be using language in more technical manner, but it seems to me (and
to Silver) to be an important distinction.

~~~
basch
"Also note that Wang & Ferguson got a better Brier score than Silver despite
getting Florida (barely) wrong while Silver got Florida right. The Atlantic
Wire gave Wang & Ferguson only a “Silver Star” for this reason, but our more
detailed analysis shows that Wang & Ferguson probably should have gotten a
“Gold Star.”"

I think being concerned about this authors usage of the words "right and
wrong" misses the point. The media will say "you were wrong" if you were on
the wrong side of a 51/49% prediction, even if you shouldn't be penalized for
it. Wang & Ferguson got dinged by The Atlantic Wire for being wrong, when they
shouldnt have. The last quote in my post, supports your point. The person who
is least wrong is the person with the lowest aggregate Brier score. The author
was using right and wrong, because the articles they were referencing used
those terms. The quote was written with the colloquial language of what it
addressed.

~~~
hsitz
We're not too far off. Main thng in original quote that bothered me was "So if
Romney had won Florida, Silver would have been wrong, but ONLY A LITTLE
WRONG." Predictions of victory or loss are either right or wrong. There are
not degrees of rightness or wrongness with the prediction; it's either right
or wrong.

The probability analysis that the prediction is based on does admit of
degrees; one probability assignment may be better or worse than another. But
saying a prediction is "only a little wrong" implicitly refers to the
probability assignment it was based on; it confuses the concepts of
"prediction" and "probability". The probability assignment was not necessarily
"wrong" to assign a higher probability to the event that didn't occur. It may
have been a bad probability assignment, but discerning that would involve
considering factors other than the outcome of the event.

------
ythl
You can't. A forecast is just that - a prediction.

------
tomjen3
Based on past history, we can assume they have to give Trump a higher than
average of the current polls chance to win.

~~~
umanwizard
What do you mean?

~~~
lern_too_spel
Silver consistently underestimated Trump's chances in the primaries. He has
yet to explain what he did wrong then or how he has fixed it now. If you don't
want a Trump presidency, Silver's current projections shouldn't be very
comforting.

~~~
umanwizard
Right, but a lot of other poll aggregators who don't have that baggage are
also predicting a Clinton win, many with higher confidence than Nate Silver.

~~~
lern_too_spel
Did their polling analysis correctly predict Trump's nomination?

------
ctdonath
There are a number of emails leaked from various election polling companies
which indicate intent to spin the results and induce a particular outcome.

~~~
1337biz
That would explain why campaign intern polling is always such a sacred ground.

~~~
maxerickson
There are simpler reasons. Being able to direct the polling is useful (spend
resources on a given issue or not). Being able to spend more polling resources
on close races. Having guaranteed access to the internals of the poll. Etc.

------
cpr
Read to the end:

[http://www.zerohedge.com/news/2016-10-11/first-post-
debate-p...](http://www.zerohedge.com/news/2016-10-11/first-post-debate-poll-
gives-hillary-significant-lead-and-familiar-problem-emerges)

Edit: Correction noted.

Post debate NBC/WSJ poll was conducted by group specifically paid by the
Clinton campaign. Not exactly what you'd want for a major poll.

~~~
akhilcacharya
...ZeroHedge? Really?

~~~
cpr
Ad hominem.

Can you dispute the damning evidence?

~~~
r00fus
Just read the posts and comments there - lots of CT + right-leaning discussion
that seems curiously aligned with alt-right ideology.

ie, more heat, less light.

