
Why we should care about the Nate Silver vs. Nassim Taleb Twitter war - oska
https://towardsdatascience.com/why-you-should-care-about-the-nate-silver-vs-nassim-taleb-twitter-war-a581dce1f5fc
======
mturmon
This is an intellectually confused post. It is so unclear and muddled that
only a charitable reader would suppose the author understands the mathematical
and modeling issues here. I’m surprised the author published it in this form -
you don’t want to jump into a controversy like this with such an unclear
discussion.

As just one example, the whole digression on a “decision boundary” is
conceptually mistaken. Once you report a posterior probability, it’s up to the
user to establish a simple real number threshold for placing a bet (if we’re
talking about means to put skin in the game). The posterior you just learned
holds all information, your only action is to threshold it. That’s the Neyman-
Pearson lemma.

If you allow generalizations to interval-valued probability, which might be
sensible under these conditions, the situation gets more complicated. But the
writer of the post did not mention this.

Another place this came out is his initial discussion of aleatory vs.
epistemic uncertainty — this can be done clearly, but here we read:

> Aleatory uncertainty is concerned with the fundamental system (probability
> of rolling a six on a standard die). Epistemic uncertainty is concerned with
> the uncertainty of the system (how many sides does a die have? And what is
> the probability of rolling a six?).

This is just not helpful. He has said “probability of rolling a six” twice.

Source: do UQ in day job.

~~~
safgasCVS
Unlikely. Taleb seems to like how his argument has been summarised.

The crux of the issue taleb has with silver is showing probabilities without
making a decision/declaration is cowardly. When asked about making a
prediction he says one thing then follows it up with a “ but don’t be
surprised if it could be completely random.” And that’s the point - the fact
he covers his ass is the problem. Election forecasting is hard so don’t
pretend you know something you don’t if you won’t stake something on your
predictions. Oh so it could be anything and you don’t want to be held
accountable? Then don’t say a damn thing. Basically Silver never wants to be
held accountable for his models but always have uncertainty covering his ass
as a cop out. so I’m With taleb on this one. Either silver makes a claim and
sticks with it or don’t show probabilities and pretend he knows things

~~~
OscarCunningham
I disagree. Probabilities can be useful even if they're not 0% or 100%. For
example weather forecasts often give the probability of rain. My plans will be
different if the probability is 20% vs 80%. My plans don't tend to vary based
on who wins the election, but I imagine that that information is very useful
to some people.

And if your data is telling you to be 80% sure, I don't think you should fudge
it and claim 100%. Report the uncertainty that you actually have.

~~~
lftl
With the weather man, I could keep track of when it rains and doesn't. If it
rains significantly more or less than 80% of the time they say there's an 80%
chance of rain, I can say that their model is bad.

I can't think of a similar way to evaluate Silver's election forecasting
model. They very clearly aren't independent probabilities, and his model
changes significantly from cycle to cycle. Was his model good in 2012 when
every state went to where he predicted the likely probability was? Was it bad
when his model said Hillary had a 71.4% chance of winning?

~~~
matt4077
They do not only predict top-line presidential results, but every race, in
every state, for president, House, and Senate. Non-independence is accounted
for in the model, so they have no qualms about you judging them by the
calibration of their predictions, I. e. You want roughly 60% of their “60%”
forecasts to be right, and 40% to be wrong. If all of the races they predict
60/40 go to the more likely candidate, they themselves consider this a
failure: [https://fivethirtyeight.com/features/how-
fivethirtyeights-20...](https://fivethirtyeight.com/features/how-
fivethirtyeights-2018-midterm-forecasts-did/)

------
comex
The article seems like an oversimplification of the underlying dispute.

\- It’s not necessary or even useful to define an arbitrary “decision
boundary” unless you actually have to make a decision. From a Bayesian
perspective, a probability stands for itself: 100% means the event is certain
to occur, 50% means you have no idea, and numbers in between convey varying
levels of certainty. In reality, 538’s predictions are not true Bayesian
probabilities because they don’t take epistemic uncertainty into account, but
that’s a totally different issue.

\- “Wild fluctuations in a prediction” from new information are absolutely a
“normal part of forecasting” - _sometimes_. If I’m planning to flip two coins,
the probability of getting two heads is 25%; but once I flip the first coin,
the probability changes to either 50% (if I get heads) or 0% (if I get tails).
In the case of Comey reopening the investigation, even if the model had
included a probability of that happening, it would have been low and thus
wouldn’t affect the overall forecast much. But once that low probability
became a certainty, you would expect a sudden swing. The real question is
whether 538’s predictions are more swingy than they logically should be
(particularly earlier on), but again, that’s a different issue.

~~~
kingkawn
Seems like in addition to whether they're more swingy than they should be, are
they being presented in a way that is interpreted as more certain or
authoritative than they can possibly claim?

~~~
VintageCool
The 2016 election _was_ swingy and it _was_ uncertain.

In 2008 and 2012, the political media was myopically focused on the twists and
turns of the horse race and called it 50-50 dead even. Nate Silver's 538
modeled the election as basically stable with only small polling shifts, and
the only change in the last month was a steady decline in the remaining time
that McCain (or Romney) had to significantly shift the polls.

In 2016, the political media treated the election as if Hillary had a lock on
it, and entreated us not to get swept up in the daily swings in the polls.
Meanwhile, Nate Silver's model swung wildly with the polls, and Nate went on
the media to emphasize the uncertainty of the election. Both candidates were
disliked, and a large portion of the electorate was undecided right up until
the last week of the election.

I remember the following exchange in one interview:

Interviewer: If you say that Donald Trump has a 33% chance to win, what's the
chance that he actually wins? Nate Silver: One in three. I'm predicting a one
in three chance that Donald Trump wins in November.

As for the authoritativeness of their presentation, FiveThirtyEight has a
really beautiful forecast that presents their 2018 prediction as a probability
distribution:

[https://projects.fivethirtyeight.com/2018-midterm-
election-f...](https://projects.fivethirtyeight.com/2018-midterm-election-
forecast/house/)

~~~
coldtea
> _The 2016 election was swingy and it was uncertain._

In retrospect. In real life, only a few pundits called it from the start (even
explaining the mechanics, including Taleb and that Dilbert guy).

Most of the press and pundits was all certain about a specific outcome until
the very last minute.

~~~
VintageCool
The Real Clear Politics polling aggregator showed nearly a 10 point jump for
Hillary Clinton after "grab em by the pussy" came out. That lead then eroded
back to the mean over the course of the month.

That's what I mean by swingy. Nothing in 2008 or 2012 shifted the polls like
that. The polls in those elections stayed within a 2 point band the whole way.
That level of volatility was clear in the polls well in advance of the
election.

(And, 15% of the electorate was effectively undecided with 2 weeks to go,
which is much higher than it was in 2008 or 2012).

Such swings are also what Taleb and the author of this article are criticizing
Nate Silver for.

------
zimablue
To try and explain Taleb's argument as simply as possible: If someone tells
you today that an event is 100% certain, and tomorrow tells you that it is
impossible, then you intuitively will not trust their forecasting. If it's
certain, then it's certain not to be impossible tomorrow

Now if someone tells you something is a 90% chance today, and a 10% chance
tomorrow, again you don't trust their predictions.

There is a probability associated with "changes in probability". The
probability of going from 100 to 0 should intuitively be 0%. The probability
of going from 90 to 10 intuitively must be something like 20% (it's like 80 of
the 90s "didn't happen", with a probability of 20).

A key point is that you can add this up every day, if something goes from
90->10->90 then it's even more unlikely than going 90->10.

So if you extend that intuition, put some real maths behind it, you can tell
whether something is "likely to be a real probability" by the rate at which it
changes. And if you have lots and lots of repeated predictions, you can be
confident that something isn't a good probability (every timeseries has like a
<10% chance of doing exactly what it does, so their combined probability [of
being good probabilities] is like 0).

Now Nate Silver accepts this, but says that he's not actually putting a
probability on the event in the future, but some non-existent "probability on
that event in the future if the future was now". But that doesn't correspond
to anything useful or measurable (it's untestable!!), and most people will
assume it follows the normal meaning of probability, and it's honestly just
silly.

~~~
ubernostrum
I've always treated 538's forecasts as meaning nothing more or less than "We
have a model, we fed the current data into it and ran N simulations of the
event, and in X% of those simulations, this was the result". Which, as I
understand it, is literally how they're generated.

Any discussion of the accuracy of the model (which should be the only thing
that admits debate) has to be retrospective. Just saying "well the forecasts
changed" doesn't inherently discredit the model, especially because things
like elections turn out to be highly sensitive to tiny variations (even
something as simple as a rainy election day in a few key precincts can
completely flip an outcome).

~~~
huffmsa
Yes but that's not how the proles understand the blog. They take it as "x has
a y % chance of happening".

Which is Taleb's beef with it. The commoner thinks its Silver putting his
money on something, but when it doesn't happen he says "well I didn't say
_that_".

And it in the day and age of more robust machine learning models, that the 538
models are so volatile is a pretty weak excuse.

~~~
mercutio2
Humans are volatile. Politics even more so.

Our options aren’t “high confidence converging predictions, or 538”.

The options in the current era for understanding of the electorate’s mood are
“overly confident individual polls, poorly analyzed with completely inadequate
models”, vs “538-style epistemically humble models accompanied by discussions
of their confidence, which can be scored in aggregate after each election”.

I’ll take 538 any day.

~~~
huffmsa
But it's not about you. Just being on this forum puts you in the critical
thinking 1%.

538 fails in that most people think that the daily stats are Silver's betting
positions. He "predicted the 2008 election."

We understand the difference, but the crying campaign staffers last November
did not.

~~~
dragonwriter
> We understand the difference, but the crying campaign staffers last November
> did not.

Campaigns aren't relying on Silver’s model.

------
baddox
I appreciate the stuff about two types of certainties, but where are they
actually arguing anything remotely relevant like that? As far as I can tell,
based on what I see bubble up to my twitter feed and what tweets are displayed
in this post, the disagreement is based entirely on Talib’s claim that Silver
switched his prediction from 80% to 50%. This really has nothing to do with
actually understanding probability or prediction models. It’s just Talib
sharing some other pundit’s deliberate misinterpretation of a single phrase
deliberately taken out of context in an interview with Silver.

I don’t think their disagreement has much to do with science or mathematics. I
think the two guys have big online followings, each leaning heavily toward two
different political ideologies, and the two leaders just have to try to beat
each other up on a Twitter to decide which side is better.

------
mehwoot
_He should not have accepted the honor if he didn’t call a winner in any of
the states!_

And he didn't, as far as I recall. I remember him saying something to the
effect of "that isn't that impressive, even a simple method looking at polls
would have gotten nearly all the states correct". Don't confuse what other
people focus on for something he is boasting about.

------
tracer4201
It's not clear to me what the author is getting at. 538 is good at aggregating
polls and showing some probability. The author complains that's not enough
because a non zero probability of the less probable outcome gives 538 too much
cover when the less probable outcome occurs.

Isn't that a shared characteristic of any other model dealing with
probabilities?

Is the author trying to say Nate Silver gets too much spotlight considering he
doesn't guarantee outcomes?

I didn't sell in 2008 and kept buying. I have the same strategy today. Not
sure if that also makes me a guru if that's the bar (as the author applies it
to Taleb).

Disclaimer: not a mathematician

------
pizza
If 538 is trying to do aleatory probabilities, then it is kinda like a broken
clock - right sometimes, but as a tool for answers about future events of any
complexity (i.e. the real world, where actions have consequences, and errors
can be grave..), I'd avoid putting one's confidence in something that claims
it has a simple/'linear' relationship to the real future.

It is also a cop out to claim "everybody who thinks probabilities based on
initial poll responses are not representative of eventual votes" must clearly
not understand probability. Maybe I am straw-manning or just don't understand
the actual disagreement.

As a poll aggregator? I'm sure it's probably a fabulous tool for past,
historical, _observed_ data. The brand itself, probably even more valuable, as
a symbol with some sort of social authority in certain circles, I guess.

Anybody who reads my comment history will see I have obviously become a Taleb
stan, but his approach seems right to me: "show me the money". Which in his
case means mathematical papers of proofs primarily, actual money, second.

~~~
cm2012
Taleb is just mixing up the nowcast and the forecast, even after being
corrected many times.

~~~
zimablue
I think he would say that the "nowcast" is so useless and deceptive to be
malpractice, since everyone will interpret it as a forecast, and even if they
didn't it's not in any way useful. Which I agree with (unsurprising since I've
semi-invented it as an opinion for another person).

~~~
thousandautumns
It's only useless and deceptive if you willfully do absolutely nothing to
understand what it is, or how it differs from the other models 538 has, which
apparently is the case for Taleb.

~~~
zimablue
Or in other words, you're 99% of people who will ever hear the number

~~~
thousandautumns
It explains on the website what it is. And if you are going to rail against
something, you should understand what you are railing against.

------
potatofarmer45
Nate Silver built an aleatory model whereby low probability high effect events
(Comey reopening the investigation on eve of the election) are discounted
because if you add enough of these potential disruptive events the model will
be 50%-50% and that has no value to anybody. Who wants to read an article that
said "election forecast 50-50" every day.

If a 538 model says one outcome is 75% probable, it really should come with a
giant caveat saying "at current trends assuming nothing out of the ordinary
occurs". Taleb's beef is that if that is the case, the 75% does not mean in
reality this result is 75% likely to happen.

What is a really interesting takeaway is that if in an event where the outcome
you desire based appears unlikely, your job is introduce as much previously
undefined or discounted uncertainty. Ideally you engineer a black swan event
or at least do what you can to make it happen. This would be fun to model from
a game-theory approach instead.

~~~
waterhouse
> What is a really interesting takeaway is that if in an event where the
> outcome you desire based appears unlikely, your job is introduce as much
> previously undefined or discounted uncertainty. Ideally you engineer a black
> swan event or at least do what you can to make it happen. This would be fun
> to model from a game-theory approach instead.

In chess, if you are behind, John Nunn's two recommended strategies are "grim
defence" (if your opponent's advantage is not so large as to make it easy for
them to force a win) and "create confusion": create complicated tactical
situations and hope your opponent makes a mistake. The farther behind you are,
the more appealing "create confusion" gets by comparison.

------
c1ccccc1
I'm confused by the existence of this article. Comparing the accuracy of two
probabilistic predictors is pretty much a solved problem. (At least in
situations where the two predictors have both made predictions about a fairly
large number of events in the same set, S.) Just add up the log scores for
both predictors and see which does better. [1]

[1]
[https://en.wikipedia.org/wiki/Scoring_rule#Logarithmic_scori...](https://en.wikipedia.org/wiki/Scoring_rule#Logarithmic_scoring_rule)

Another thought: Although the author didn't lie about its meaning, I thought
the graph "Stated Probabilities Compared with Average Portions" was visually
misleading. At a casual glance, it appeared that 538 was all over the place
with its senate predictions. But if you look closer, you can see that many of
the red data points are at 0 or 0.5 or 1. This suggests that these data points
represent the outcome of just 1 or 2 elections, which isn't really enough to
say that 538 is doing particularly badly.

------
macawfish
I'm still waiting for somebody to talk about how tightly coupled Nate Silver's
forecasts are to the political processes he attempts to forecast.

What if Nate Silver were to run around on the football field before each play
whispering in the players' ears about the how likely they are to win,
according to his latest forecast. Imagine that they believed in him, or at
least felt some degree of superstition when he came around.

I just think there's something innately flawed about public forecasting of an
election. Mathematically and maybe even ethically. I'm still trying to figure
out how to articulate why I feel that way.

It could even be argued that polling itself could influence opinions by
introducing bias at a critical moment when someone is being asked to consider
who they are voting for. What if that's the moment when they make up their
mind?

P.S.
[https://www.frontiersin.org/articles/10.3389/fphy.2015.00077...](https://www.frontiersin.org/articles/10.3389/fphy.2015.00077/full)

~~~
dagw
_I just think there 's something innately flawed about public forecasting of
an election._

It used to be the case in France that you weren't allowed to publish polls two
weeks before an election for just this reason. They eventually scrapped this
with the argument that the all 'elites' had access to internal and private
polls anyway so all you where doing was denying the 'people' the same access
to information that the 'elites' had.

 _It could even be argued that polling itself could influence opinions by
introducing bias at a critical moment when someone is being asked to consider
who they are voting for._

This is a pretty well known effect, to the point that many political
strategists try to avoid publishing or drawing attention to polls showing
their candidate too far ahead in case it makes voters complacent and decide to
stay home. The ideal situation is to make people convinced the polls show the
candidates tied 50-50 going in to the election and that 'your' vote could
easily swing the whole election.

~~~
mijamo
This is slightly wrong. It was initially one week, not two. It was changed
because it was unconstitutional, as it limited freedom of expression too much,
not because it was useless or anything to do with elite vs people. And it
still exists, limited to the day of the election and the day before. Source:
[https://www.conseil-constitutionnel.fr/nouveaux-cahiers-
du-c...](https://www.conseil-constitutionnel.fr/nouveaux-cahiers-du-conseil-
constitutionnel/la-reglementation-des-sondages-et-l-election-presidentielle)

------
skolos
Author does informative critique of 538. I'm not sure at all that it has
anything to do with Silver vs Taleb twitter war though. It's more like premise
for expressing author's own thoughts.

However I totally dig what 538 does and appreciate their analysis. I
understand that there is a complex underlying model with many parameters taken
subjectively. The results of Monte Carlo runs of this model are very
interesting to me. The fact that they describe distribution of outcomes and
not just a single number is important property of a Monte Carlo model and I
would have an issue if that wouldn't provide that info. If I would like to
analyze complex event that has lots of moving parts and uncertainties I would
also build a model to see what kind of distribution I would get. The fact that
somebody published results of their model is quite useful.

The analogy would be European and American weather models - no one say that
their results are exact and everybody understand that uncertainty in result
comes from inherent uncertainty of initial state as well as shortcuts and
approximations each of the model takes. No one says that the results of these
models are useless because of that. And everybody finds it valuable if weather
forecast predicts rain tomorrow with 40% chance. So I don't see how political
prediction is only valuable (by the words of the author) if it comes without
probabilities attached to it.

------
lalaithion
> Practically what this means is that they do not predict a winner or looser
> but instead report a likelihood

Yes, this is good. It means that you can evaluate their accuracy to a greater
degree than % of times correct.

> Further complicating the issue, these predictions are reported as point
> estimates

They've learned from this, and now show their distribution of results very
clearly. However, there's nothing mathematically fraught about reporting a
prediction as a point estimate.

> The problem is that models are not perfect replicas of the real world and
> are, as a matter of fact, always wrong in some way.

...yes? And so what? This is a _fully general counterargument_ to all of
science.

> Predictions have two types of uncertainty; aleatory and epistemic.

Don't say things like this with 100% certainty when your source itself says
"the validity of this categorization is open to debate", and also when your
source is Wikipedia.

> However, as you can see, there is still a noticable variation of 2–5% of
> actual proportion to predictions. This is a signal of un-addressed epistemic
> uncertainty. It also means you cannot take one of these forecast
> probabilities at face value.

Models aren't perfect. Also, 538 is very careful to address their epistemic
uncertainty! They also try very hard to not change their models significantly
after they publish them, in order to avoid letting their personal biases
tinker with the results.

This post goes out of it's way a to avoid mentioning that there are pretty
well established ways of measuring the accuracy of predictions, instead using
things like "look, it's not quite a line" and "look, the line went up and down
before the election".

If you're interested in actually reading about this from a mathematical
viewpoint, read Taleb's paper, which has some actually interesting thoughts on
why 538's algorithms are too eager, or read Madeka's paper, which includes
comparisons of 538 to other people trying to make predictions, and finds that
actually, they were better than most other news sources.

[https://arxiv.org/pdf/1703.06351.pdf](https://arxiv.org/pdf/1703.06351.pdf)

[https://arxiv.org/pdf/1704.02664.pdf](https://arxiv.org/pdf/1704.02664.pdf)

------
jimnotgym
All of which increases the probability that I will ignore Taleb's tweets. I
follow people who are smart enough to disagree in a civil manner, and smart
enough to recognise that there may be things one doesn't know.

The way he talked to renowned classicist (and feminist, which seemed to be a
problem for Taleb) Dame Mary Beard in another spat was disgusting. Trying to
impose statistical certainty on ancient history ended up making him look
pretty dumb to everyone but him.

~~~
JabavuAdams
The problem is that the smart arrogant jerk may have the right answer
sometimes when it counts, while being a pain in the ass the rest of the time.
Better just to stay open to all feeds. He does come off as an arrogant dick,
wow.

~~~
alexandercrohde
I used to look at things this way, with a "just filter out the rudeness"
mindset. However in practical terms I think listening to somebody's rudeness
does have a really big cost (to estimate: how much would you need to be paid
to make it worth your time to listen to rudeness for an hour?)

------
mhb
Shouldn't a guy like Taleb be able to propose a bet with Silver that
demonstrates he (Taleb) is right?

~~~
starchild_3001
Best solution proposed here! Put your money where your mouth is Taleb and
Silver!

Btw, I read 3 of Taleb's books, 1 from Nate Silver. Both are pretty good.
Taleb is the more original, but also more often-objectionable.

~~~
dagw
_Best solution proposed here_

So what should the bet be? As far as I can tell they're not disagreeing on
anything that can easily be reduced to a bet

~~~
lixtra
Nate calls the chances like 80:20 Clinton:Trump. Taleb takes the side: 100,000
USD that Trump wins. If Trump wins, Nate pays 500,000 to Taleb. If Clinton
wins, than Taleb pays 100,000 USD to Nate.

Do this for each election.

~~~
dagw
But that doesn't seem to be the nature of their disagreement. Taleb's main
complaint seems to be about how the swings in the prediction probabilities far
out from the event indicates a fundamental flaw in how the underlying model
uses probability.

That being said it would be a fun little exercise to see who much money would
be made if they bet had $1 on every race 538 predicted in 2016 and 2018 using
the model odds.

~~~
mhb
It doesn't need to be a simple bet. It could incorporate the volatility which
Taleb might believe shouldn't be present in an unflawed model.

~~~
lixtra
Indeed after skimming through the paper[1] (I find it hard to grasp without a
solid background) it seems that Taleb wants to play this game every day before
the election and expects to come out ahead on election day when all the bets
are settled.

[1]
[https://arxiv.org/pdf/1703.06351.pdf](https://arxiv.org/pdf/1703.06351.pdf)

------
pmart123
I liked the first chapter of "Fooled by Randomness," but I stopped reading the
book when each chapter regurgitated the same points, each more nihilistic than
the last. Perhaps Taleb should concentrate on the accuracy of his own
predictions:

[https://www.businessinsider.com/taleb-every-single-human-
bei...](https://www.businessinsider.com/taleb-every-single-human-being-should-
short-treasuries-2010-2)

>>> "every single human being" should bet U.S. Treasury bonds will decline.

>>> Short the S&P vs Long Gold, in a 5 to 1 ratio. By gold Taleb means a
basket of precious metals including gold.

The second trade would have lost $125k for every $10k invested in gold and
$50k invested in SPY.

Did he miscalculate the risk of a black swan event...?

~~~
Animats
I'd be more impressed with Taleb if he published the results for his Emperica
fund for years other than 2008. Taleb's strategy was to buy options way out of
the money. In boring years, the fund lost money. It did really well in 2008.
Not enough numbers have been published to determine if his strategy was a net
win over a business cycle. The win seems to have coming from the fund's short
lifetime including 2008.

~~~
CosmicShadow
You only need to win once. One of his big things is about optionality and
anti-fragility. If you buy something that barely hurts you in terms of cost,
but can sky rocket massively if something big happens, you can capitalize on
it, and all other costs were meaningless. Even VCs try for that, although with
numbers that aren't nearly as good.

What do you mean by a business cycle? If his fund just goes until it makes it
big and then he stops, is that not a business cycle? Did he continue? I don't
know. It just seems weird to ask about a business cycle in the case of a
strategy that is very long played, waiting for unpredictable results. The only
thing that matters is can you play long enough to hit that result if that is
your game? It seems clear that it did win, or else he'd probably be focusing
more on it until he did? It's not like a regular business with actual output
or production that can be easily replicated and measured, it's just a play
till you win or lose deal and I'm sure there are others that did similar
things and lost, just like many more companies are dead than alive and we only
see the alive ones.

I understand wanting to see the numbers, but this is just one sample for a
strategy.

------
BurritoAlPastor
I'm infuriated that the author would allege that Taleb has a better way to do
this but just links to a highly technical paper instead of addressing it.

------
OscarCunningham
I think the difference in opinion is caused by Taleb modelling the election as
though all of the uncertainty is caused by the fact that people might change
their minds, whereas Silver is also including the uncertainty from polling
errors.

Silver's predictions fluctuate a lot as time passes. Taleb is assuming that
all of these fluctuations are caused by the electorate changing who they're
going to vote for. Then he says "if you knew people's opinions were as
volatile as this, you shouldn't have been so confident earlier!", or words to
that effect. If lots of people will change their minds, then early polling
won't be very informative about the final result.

But the fluctuations in Silver's predictions are actually caused by his
changing certainty about how accurate the polls are. Silver does take into
account that people might change their minds, but this isn't the cause of most
changes in his predictions. So people actually change their minds a lot less
often than Taleb's model of Silver's model suggests. This is why Silver can
make strong predictions early on.

------
backpropaganda
The difference between epistemic and aleatory uncertainty is not well
understood even in the machine learning world. Most applications of machine
learning to domains where uncertainty needs to be quantified use aleatory
uncertainty, when epistemic uncertainty is the right one. I wish we had better
ways to explain these concepts, and perhaps better terms too. Alternatives
like intrinsic/extrinsic or irreducible/reducible don't really work either,
because they make it seem like it's a dichotomy.

~~~
pizza
I haven't read any of his works but I have a suspicion that Judea Pearl writes
about causality in this niche with that inferential difference in mind.

------
lambdadmitry
I'm profoundly confused by the deep respect Taleb commands in the eyes of many
people here, apparently. The guy has a few good ideas that he repeats ad
nauseam, but he punches way over his own intellectual weight too often. A few
especially egregious examples are his deranged GMO hate "paper" which is
profoundly ignorant of the involved biology [1], his support of Syrian
government's slaughter of civilians [2][3], and his Putinversteher views ([4]
reads as a parody for anyone knowing anything about Putin and modern Russia).
Combined with his unlimited arrogance, I don't understand how people can
respect him intellectually.

[1]:
[http://www.fooledbyrandomness.com/pp2](http://www.fooledbyrandomness.com/pp2)

[2]: [https://medium.com/opacity/the-syrian-war-condensed-a-
more-r...](https://medium.com/opacity/the-syrian-war-condensed-a-more-
rigorous-way-to-look-at-the-conflict-f841404c3b1d)

[3]: [https://medium.com/@ameraidi/nassim-taleb-is-wrong-on-
syria-...](https://medium.com/@ameraidi/nassim-taleb-is-wrong-on-
syria-e1faa523824c)

[4]:
[https://twitter.com/nntaleb/status/773323163509850112](https://twitter.com/nntaleb/status/773323163509850112)

------
coryfklein
Re: the desire to choose a 50% (or other) decision threshold.

It doesn't have to be a binary choice. Knowing the distinction between 65% and
80% improves your betmaking capabilities by allowing you to hedge, in some
cases guaranteeing a profit [1] regardless of the odds and how they change.

[1] [https://help.smarkets.com/hc/en-
gb/articles/115001431011-How...](https://help.smarkets.com/hc/en-
gb/articles/115001431011-How-to-calculate-a-hedge-bet)

------
vertline3
Taleb's biggest thing is exposing holes in predictive models. Silver's biggest
thing is making and using predictive models. So it's only natural for these
two to quarrel.

------
mschuster91
Of course the US elections are not as accurately predictable as e.g. German
ones, or sports events. Not simply due to different pollster sources and their
data quality (which Nate Silver tries to account for).

The biggest and most unpredictable factor in US (and British) elections is
their winner-takes-all system, with the second layer of the Electoral College
in the US which does not account for popular votes either.

------
alexandercrohde
So does Taleb propose a better betting strategy? I'd like to see him put his
money where is mouth is, rather than just his reputation.

------
Cd00d
I would love to see the Nate Silver point by point response to this post. I
think it would be humorous, but also add some insights into his proprietary
black box.

I listen to the 538 podcast every episode, and I think I agree with cm2012
that this is a case of mixing up the "nowcast" with the actual forecast.

------
cperciva
The author doesn't seem to understand how the 538 model works. The 538 model
_does_ take epistemic uncertainty into account; it just doesn't call it that.
The model calls it "likelihood of polls moving by X% between now and election
day". In 2016 they offered a "now-cast" which didn't include that, and answer
the question "based on where the polls are right now, how likely is any given
outcome if the election were held today?" while the "classic" model answered
the question "based on where the polls are right now _and the fact that
unanticipated things will happen between now and the election_ , how likely is
any given outcome?". For 2018, they dropped the "now-cast", because people
didn't understand what it was saying.

True, the epistemic uncertainty is limited -- it's based on past elections,
because hey, 538 works with actual data. One could argue that rather than
saying "71.4% Clinton 28.6% Trump", a better model would have said "71.3%
Clinton 28.4% Trump 0.2% Election is cancelled due to nuclear war / natural
disaster / etc" but I don't think anyone sensible is interpreting the model as
excluding such extreme outcomes.

------
lavrov
I thought that the post was fairly unclear, but for me, it seems like the main
argument against 538 is that it makes unfalsifiable claims about the
probability of individual elections - picking a winner is a falsifiable claim,
but assigning a probability always allows Silver to claim something like “even
events with a 10% probability occur frequently” even if his model assigns a
high likelihood of victory to the loser of an election.

My favorite Nate Bronze takedown remains Carl Diggler[1].

[1]
[https://www.washingtonpost.com/amphtml/posteverything/wp/201...](https://www.washingtonpost.com/amphtml/posteverything/wp/2016/05/09/our-
fictional-pundit-predicted-more-correct-primary-results-than-nate-silver-did/)

------
empath75
Five thirty eight is pretty consistent with saying their forecasts more than a
few days out are unreliable. They’re fairly accurate with with their final
forecasts, though.

------
bshanks
You could use the Brier score to compare how good various forecasters are at
predicting something

------
tareqak
As a non-specialist, I like how the discussion boils down the picture at the
end: [https://cdn-images-1.medium.com/max/1600/1*D_tidaT-
fHMY3DRLg...](https://cdn-images-1.medium.com/max/1600/1*D_tidaT-
fHMY3DRLgjwekw.png) in that it is something I can understand.

~~~
cm2012
Nate Silver always calls the nowcast (blue line) for what it is - a forecast
for if the election was held that day. The forecast, made a few days before
election day, is the official forecast and the one that beats most other
estimates.

~~~
OscarCunningham
That blue line is a record of Silver's forecast, not the nowcast. You can
compare them here [https://projects.fivethirtyeight.com/2016-election-
forecast/](https://projects.fivethirtyeight.com/2016-election-forecast/)

------
akivabamberger
Why does this not just link to the Twitter war?

------
thousandautumns
I've been following this spat since it reignited several weeks ago, but also
since Taleb first attacked Nate on Twitter back before the 2016 elections, and
it has been the final nail in the coffin as to any supposed credibility Taleb
commands.

The quandary that FiveThirtyEight and Silver are in is that the general public
is stupendously bad with probabilities. Prior to the the 2016 election, Silver
was personally being attacked, in some cases by major media organizations like
the Huffington Post, and accused of tipping the scales towards Trump (for some
inexplicable reason), since their models were giving him approximately a 1 in
4 chance of winning, since everyone _knew_ it was a certainty Clinton would
win. Then, in the months following the election, the narrative somehow
switched, and the fact that Trump won despite only being given a 1 in 4 chance
by FiveThirtyEight meant that Silver was now a hack and the model was wrong.
This same criticism has been leveled by many right-wing media personalities,
who had been trotting out Silver's 2016 predictions leading up to the
election, as well as by the New York Times, whose own models were giving
Clinton nearly a 90% chance of victory.

So now, fast forward to this year's election, Silver is doing his best to
drill into people's minds that, yes, their model showed a Democratic win in
the house as the most likely outcome, but that absolutely doesn't not mean
their model says it is an absolute certainty, or that other outcomes would be
unusual. It isn't covering his ass, its educating the public on how
probabilistic statements work.

This is something Taleb _should_ know. But Taleb has proven himself to be a
hack and an intellectual parasite. He hasn't actually produced any ideas worth
discussing since "Fooled By Randomness" which was nearly 20 years ago.
Basically all of his works since then have been rehashing the same idea, or,
as with Skin in the Game, pseudo-intellectual ramblings containing little to
no interesting ideas, and what ideas do exist have been rehashed a million
times by others.

Since then, he has taken to making ad-hominem attacks and nonsensical
critiques on others in the public view, often on Twitter. I hate bringing up
Trump, as it happens far too often in discussions these days, but some of his
methods of "debate" are strikingly similar: strange monikers ("klueless
Nate"), schizophrenic jumps between topics/arguments, often indecipherable
language, and a ridiculous volume of output. All of this combines to mask the
fact that most of the time, he isn't _actually_ saying anything, or at least
anything sensible, but by shouting loud enough, long enough, and purposefully
making his point difficult to identify, it almost becomes impossible to argue
with him, and a non-insignificant number of people will assume he is right.

Interestingly, I've also found many of his supporters to be similar to some of
Trump's or other figures such as Musk. There is a cult of personality built up
there where they have established the person as a visionary first, and
therefore anything and everything they do or say is correct. Many of them
parrot back the same, shallow catchphrases of his, that rarely contain
anything insightful but are general enough that they can be thrown out in any
situation: critics or dissidents are "Intellectual yet Idiots" or don't have
"Skin in the Game". Note that the arguments of these critics or dissidents are
almost never addressed.

Unfortunately, the spat between Taleb and Silver (which actually started back
prior to the 2016 elections), is exhibit A in why Taleb is an intellectual
charlatan, and shouldn't be given the time of day. Silver isn't some perfect
person either, but in general he has been a reliable voice in the realms of
political prediction, fairly forthcoming on shortfalls, and his most damning
criticisms all seem to be built upon critical failures in the understanding of
probability, or a willful misrepresentation of his results.

------
gaius
No one should care about anything that happens on Twitter. It’s that simple.

------
einarvollset
[https://www.theguardian.com/books/2018/feb/25/skin-in-the-
ga...](https://www.theguardian.com/books/2018/feb/25/skin-in-the-game-by-
nassim-nicholas-taleb-digested-read)

------
013a
The whole 2016 situation feels like many people are borderlining on delusion,
Nate included. The Comey thing certainly helped Trump, but even _the day of
the election_ 538 was still forecasting a massive win for Hilary. How can the
probabilities be that wrong?

Because the polls were wrong. Or, at least, incomplete. It doesn't matter how
good your algorithms are if the data isn't there. And 2016 proved that 538,
and really most media outlets, don't have a grip on America.

In other words, the existence of a black swan event in that particular system
didn't really matter.

Silver is a glorified modern fortune teller, who seems to have deluded himself
into thinking that simply having data means you can prescribe meaningful
probabilities to the future. Assigning probabilities to a coin flip is easy.
Figuring out the most likely winner of a game of baseball is a little harder;
there's a whole movie about a guy who was basically doing that in 2002.
Politics is a completely different field. You can't possibly assemble all of
the data necessary to be remotely confident in the probability of outcomes.
And let's not forget unpredictable black swan events.

But the readers love it. And, especially during the elections, the media would
bring him out like a golden boy computer whiz, because he was saying the same
things they were, but he's really smart and has the data and magic algorithms
to back him up. And he was right that one time 8 years ago. He's about as
pointless as Sean Hannity, with the one exception that, at least recently,
Hannity was actually more correct about the future than Nate. It doesn't
matter if you are right or wrong, if the reasons are wrong.

~~~
dragonwriter
> but even the day of the election 538 was still forecasting a massive win for
> Hilary

It was predicting a large electoral win as the median scenario, but only a 70%
chance of Clinton winning at all, because discrepancy between actual outcomes
and polling were likely to be correlated across states, and many states were
very close. This is where Silver differed from the other forecasts that were
saying Clinton 90%+ based on the (historically false) idea that differences
between votes and polls were independent across states.

> You can't possibly assemble all of the data necessary to be remotely
> confident in the probability of outcomes.

You often can be more than remotely confident, but it's true that the national
result of the 2016 Presidential election wasn't one of those times. OTOH,
Silver’s forecast reflected that, since ~70% is extremely low confidence.

~~~
VintageCool
FiveThirtyEight also predicted a 10% chance that Clinton would win the popular
vote while Trump won the electoral college.

