
If You Say Something Is “Likely,” How Likely Do People Think It Is? - randomwalker
https://hbr.org/2018/07/if-you-say-something-is-likely-how-likely-do-people-think-it-is
======
crispyambulance
It depends on the context.

Hillary Clinton had a >70% of winning the US presidential election according
to the most responsible analyses (see 538:
[https://projects.fivethirtyeight.com/2016-election-
forecast/](https://projects.fivethirtyeight.com/2016-election-forecast/)).

Most folks took 70% to mean that she would certainly win and were bitterly
disappointed the morning after.

On the other hand no sane person would (willingly) play Russian roulette with
a 70% or even 5 of 6 chance of "winning".

In everyday life, one conflates probability with severity of outcome. This is
normal and is part of how people assess and act on risk in a qualitative way.

~~~
DINKDINK
>Hillary Clinton had a >70% of winning the US presidential election

Probabilities without confidence intervals[1] are by-and-large meaningless
(She has a 90% chance of of winning with a confidence interval of +11% -100%).
No amount of d3.js on 538's blog will change this.

[https://en.wikipedia.org/wiki/Confidence_interval](https://en.wikipedia.org/wiki/Confidence_interval)

~~~
OscarCunningham
From a Bayesian perspective, or from a betting one, it doesn't make sense to
put probabilities in a confidence interval. You might be uncertain about the
world, but you can be certain about how much uncertainty you have, since it's
a property of your own mind.

~~~
fragsworth
How do we quantify the difference between coin tosses, which we are _very
certain_ is 50% likely to end up heads, from political elections, where we
only have a few examples to go off of?

~~~
edbaskerville
In terms of predicting what's going to happen next, it really isn't that
different if your model's any good. Looking backwards, if the 2016 election
had been determined by two coin tosses both coming up heads, would we be
surprised? And yet 538's polling model indicated that the probability was even
better than two coins coming up heads.

They probably should have just written that right there: "71%, better odds
than two coins coming up heads."

But: in order to understand if the model is any good, you need to try to build
a good model, describe your model, and describe what you know about the
underlying processes. With words, visuals, and math. Which is pretty much what
538 does better than most.

------
roter
For what it is worth, the Intergovernmental Panel on Climate Change uses the
following definitions[0]:

virtually certain: 99-100%

extremely likely: 95-100%

very likely: 90-100%

likely: 66-100%

about as likely as not: 33-66%

more likely than not: >50-100%

more unlikely than likely: 0-<50%

unlikely: 0-33%

very unlikely: 0-10%

extremely unlikely: 0-5%

exceptionally unlikely: 0-1%

[0]: [https://ipcc.ch/pdf/assessment-
report/ar5/syr/AR5_SYR_FINAL_...](https://ipcc.ch/pdf/assessment-
report/ar5/syr/AR5_SYR_FINAL_SPM.pdf)

~~~
tinalumfoil
> virtually certain: 99-100%

Hopefully whoever wrote that never works in a data center, builds cars, sells
insurance, gambles, works on crypto, does scientific computing, or is ever
responsible for someone's life.

~~~
ehrman
If you're talking the realm of five 9s, etc, that's in reference to service
availability at some given point in time throughout the course of a year. If
you discuss the probability there will be an outage once during a year, the
answer is somewhere in the middle, around "more likely than not".

You could use this same probability around a pacemaker. The device is
virtually certain (99%) to function at a given point throughout the year, but
the probability that the device will not fail over the course of the year is
not 99%. If the pacemaker had a 99% chance of not failing once during the
course of a year, it would be virtually certain it would not fail during that
year.

------
twiss
Using numbers for probabilities is not a panacea either, when those numbers
are not the result of a calculation. One person might say "99% of the time" to
mean "almost always", and another person might say "9 out of 10 times" to mean
exactly the same thing.

The problem is that it's hard to estimate the probability of something that
you know happens "almost always" without doing the counting, which most people
don't do. And hence it's difficult to gain an intuition for what 99% means
exactly.

Furthermore, there is also the possibility that two people share the same
intuition about a given word/phrase, but then disagree about which _number_
describes that intuition. This could account for part of the results of this
paper.

~~~
c3534l
A good example of this is ratings on Amazon. You can give up to five stars,
but most people just give 1 or 5 depending on if they were satisfied or
dissatisfied. Additionally, people confuse precision for certainty and will
perceive an estimate of 89.1% as more accurate than 90%.

~~~
Izkata
Your examples are kinda funny, because that could easily be true. Now, if the
second one was 90.0% on the other hand...

------
dsr_
Related: RFC 2119,
[https://www.ietf.org/rfc/rfc2119.txt](https://www.ietf.org/rfc/rfc2119.txt)

which defines the key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL
NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL". Most
subsequent RFCs begin with a reference to it.

------
jcrites
This article, and some of the comments here, reminded me of another article
(and GDC talk) by Civilization game designer Sid Meier, about his experience
with players' perception of probability in games.

Sid's talk grapples with the issue "If the game says you have 3-to-1 odds to
win a battle, how often do players _actually_ expect to win?"

> When designing the combat system in Civilization: Revolution, Sid Meier
> found himself up against some interesting design problems. His players
> didn't understand math. In Civ Rev, the strength of units were displayed up
> front to players before battle to show the odds of victory. For example, an
> attacking unit might be rated at 1.5 with the defending unit at 0.5. This is
> a 3-to-1 situation.

> Unfortunately, the testers expected to win this battle every time despite
> there being a 25% chance of losing each time. Sid tweaked the math to make
> the player win more in this situation. Next, the reverse case was tested.
> The player had 1-to-3 odds. If they won, the math was functioning properly.
> They had a slim chance to win and they did.

> Sid identified a few cases of interest. When the player was presented with
> 3-to-1 or 4-to-1 odds, they expect to win. With 2-to-1 odds, the player will
> accept losing some of the time, but expect to win at 20-to-10, which is just
> a larger expression of 2-to-1. When the numbers get larger, the perceived
> advantage grows.

[http://www.shacknews.com/article/62807/sid-meier-and-rob-
par...](http://www.shacknews.com/article/62807/sid-meier-and-rob-pardo)

Here is a link to the actual GDC talk by Sid with the content about
probability: [https://youtu.be/bY7aRJE-oOY?t=18m22s](https://youtu.be/bY7aRJE-
oOY?t=18m22s)

(It's true. When playing a Civ game, it's not fun to have a 10-to-1 strength
advantage and lose the battle anyway.)

In later games, Sid removed probability from the game in favor of an outcome
that's predictable. Instead, each unit will be damaged in proportion to the
ratio of the units' strengths (or something like that).

~~~
throwaway37585
Maybe the key here is that strength ratio != odds, under most intuitive
definitions of strength. If someone is twice as strong as you, they're
arguably _much more_ than twice as likely to defeat you. If someone has twice
the _number_ of units as you, they're also more than twice as likely to defeat
you.

------
kashyapc
That impreciseness is one reason why William Zinsser in his book, _On Writing
Well_ , argues:

\- "Get rid of the adjective-by-habit."

\- "Don't hedge your prose with little timidities. Good writing is lean and
confident."

~~~
resu_nimda
_" Don't hedge your prose with little timidities. Good writing is lean and
confident."_

This is a separate issue, and one I would disagree with. In fact I feel like
this is a problematic attitude in our culture. Too many people overconfidently
asserting their subjective viewpoints. Real life is full of nuance, gray area,
second-guessing, and different perspectives; simple bold assertions usually
don't hold water.

David Foster Wallace, considered one of the greatest writers of recent times,
filled his writings with qualifications, side notes, devil's advocacy, and at
times insecure waffling. To me this is a more accurate reflection of the mind
and reality and should not be suppressed for a macho bravado façade.

~~~
kashyapc
True, I fully see your point of people imposing their subjective views over-
confidently. It is a real problem. :-(

But I don't think Zinsser there means to get rid of _all_ 'timidities'. The
subtext my mind's eye read there was: don't overdo it. And he goes on to
expand: "Every little qualifier whittles away some fraction of the reader's
trust. Readers want a writer who believes in himself and in what he's saying.
Don't diminish that trust."

For instance, I know smart people who often apologize for no reason, do
excessive hedging, or start a request with an apology. It gets _a bit_
annoying when I have to read or hear that everyday.

~~~
cat199
> It gets a bit annoying when I have to read or hear that everyday.

Or, this is an entirely subjective thing based on your own cultural
presuppositions, and you are actually simply rude/prideful.

The coin flips both ways - there is no 'norm' here other than
cultural/ethical/philosophical bias.

------
pacificpendant
When people say things like "serious possibility" they are not merely
intending to alter the perception of the likelihood but also attempting to
impress upon the listener the gravity of the event.

In cold war era America would people have felt any different knowing that a
Russian invasion was only 20% likely rather than 30%? For something so
/serious/ any non-zero probability is something that should be prepared for.

~~~
ScottBurson
Yes, I had the same thought. A "serious possibility" of global thermonuclear
war is 1% (maybe smaller). A "serious possibility" of a hangnail is 80%.

------
projectramo
Follow up study: when someone says something happens “90% of the time”, how
likely do they think it is?

------
will_brown
When have this type of imprecise (sometimes subjective) verbiage in our legal
standards:

1\. Beyond a reasonable doubt

2\. Perponderance of the evidence (“more likely than not”, which in theory is
>50%)

3\. Clear and convincing (which is somewhere in between the 2 above)

~~~
occamrazor
There is also a standard terminology for undesirable effects of medications:

\- Very common: affects more than 1 in 10 people – ie the risk is 10% or
higher

\- Common: affects between 1 in 100 and 1 in 10 people – ie risk is 1% to 10%

\- Uncommon: affects between 1 in 1,000 and 1 in 100 people – ie risk is 0.1%
to 1%

\- Rare: affects between 1 in 10,000 and 1 in 1,000 people – ie risk is 0.01%
to 0.1%

-Very rare: affects less than 1 in 10,000 people – ie risk is less than 0.01% (This includes isolated reports)

------
jmount
Another fun one: Fermi's definition of "remote possibility" as a chance of no
more than 1 in 10. (Scary when talking about the odds of initiating a chain
reaction, making a small atomic weapon, or even igniting the atmosphere in the
first atomic test.)

[https://books.google.com/books?id=2G2TlJOhGI8C&pg=PA280&lpg=...](https://books.google.com/books?id=2G2TlJOhGI8C&pg=PA280&lpg=PA280#v=onepage&q&f=false)

~~~
hueving
0 in 10 is still "no more than 1 in 10", so I don't see why that's scary.

~~~
fjsolwmv
0.9 in 10 is also no more than 1 in 10.

~~~
hueving
Right, but I'm suggesting that the odds of lighting the atmosphere on fire
were calculated to be much lower than 1 in 10, but there just wasn't a 'simple
English' category to distinguish them. So I wouldn't get scared until I saw
something that suggested it was anywhere near 1 in 10.

------
safgasCVS
"Lesson 1: Use probabilities instead of words to avoid misinterpretation."

Probabilities are meaningless unless it’s a repeatable experiment otherwise
its a ludic fallacy eg "There's a 70% chance of Hillary winning". This is an
un-provable statement. Either she wins and prediction was right, or she loses
and it counts as part of the 30%. This is Nate Silver's get-out-of-jail-free
card so even when he's wrong he comes off as being right.

i.e. this statement makes sense in a casino and nowhere else.

"Lesson 2: Use structured approaches to set probabilities."

Probabilities are meaningless by themselves. Path dependency matters a great
deal to actual humans but not to business professors. A strategy that works
well for the ensemble wont necessarily work well for the individual.

e.g. if you save for retirement for the average life expectancy then 50% of
the people would be screwed and 50% of the people would have saved too much.

i.e. The cost of being right/wrong is what matters and not the probability.

~~~
archgoon
> Probabilities are meaningless unless it’s a repeatable experiment otherwise
> its a ludic fallacy

Um... this goes against the entirety of the Bayesian approach to statistics. I
think you'd find a lot of very intelligent people who disagree strongly with
this statement.

The Bayesian approach takes probabilities as subjective confidences. You _can_
describe confidences as "well calibrated" if, when you look at their
historical guesses, if their 70% assessments are correct 70% of the time.

~~~
safgasCVS
". I think you'd find a lot of very intelligent people who disagree strongly
with this statement. "

I never claimed to be intelligent.

" You can describe confidences as "well calibrated" if, when you look at their
historical guesses, if their 70% assessments are correct 70% of the time."

\- Again, if you're trying to figure out if a coin is split 50/50 then yes,
but without being able to repeat the same experiment you're fooling yourself
and the whole aspect of bayesian thinking I think goes out the window. eg me
being right about unrelated topics doesn't mean I'm right/wrong about specific
topics.

------
anuj_nm
This makes me thankful for Amazon's document-writing culture. Ambiguous words
like very/could/should/few/large are avoided or at least qualified with a
range. This helps us avoid ambiguity, regardless of the amount of context a
reader has.

------
sanbor
In my experience, putting numbers to your gut feeling probabilities causes
people to not take them seriously. Example: "I think there is 30% chances that
I will get a promotion next month". If I say 30% people ask how I came up with
that number. If I say "not very likely" then it gets accepted as an educated
guess. Probably because we're used that those numbers must come from some
dataset. Or should I say "I think there is a chance of 40% that things are
like that because we're used that those numbers must come from some dataset".

~~~
a13n
30% is specific, as if it's the result of a calculation in a spreadsheet. But
with something like getting a promotion, how would you calculate that so
specifically? You can't. That's why people don't take it seriously.

"Not very likely" is a rough estimate, like "probably not", or "less than 30%
likely". It's a lot more believable that with the information you know, you're
able to predict with that certainty.

------
zaat
>When you use a word to describe the likelihood of a probabilistic outcome,
you have a lot of wiggle room to make yourself look good after the fact. If a
predicted event happens, one might declare: “I told you it would probably
happen.” If it doesn’t happen, the fallback might be: “I only said it would
probably happen.”

The same is true with numbers. If a predicted event happens, one might
declare: "I told you there 80 percent that will happen". If it dosen't happen,
the fallback might be: "I told you there are 20 percent it will not happen."

------
BrandoElFollito
I work in security and I am asked all the time about the likelihood of
horrible things which may happen.

I cannot, ever, give a percentage ("70% chance we will get hacked if this and
that"). I either say that I do it know and nobody will, or use "probably" or
"unlikely". These are wide, hand waving categories.

I do that not because I do not want to be wrong but just because I have only a
vague idea, classified into "maybe" and "maybe not"

------
GuB-42
It also depends on the individual. And as you get to know people better, you
tend to interpret their probabilities better.

Take the question "are you coming?". "Maybe" can mean "yes, except in case of
nuclear war" to "not unless you kidnap me", depending on the person.

------
Chronos309
I PERSONALLY decided if it seems like 20-25% probability, this is 'likely'
because it means I can accomplish it if I can try 4-5 times. If it is 1%, I'd
have to attempt 100 times to make a thing happen.

------
everdev
Aren't all adjectives relative and therefore arguable?

It's amazing how often they're used in debates or political speech (ex. good,
bad, strong, important, dangerous, etc.) when they clearly mean different
things to different people, but maybe that's the idea.

~~~
neom
Indeed, this is why the best communicators lean heavily on adverbs and pay
careful attention to what adjectives make sense for the audience.

------
aj7
I can summarize the article in a single sentence. “Something is likely” has a
huge variance.

------
cortic
Its very likely, that's its not as likely as people likely think it is.

------
any1
I usually just let the branch predictor figure it out on its own.

------
etqwzutewzu
likely

