
Pascal's Mugging (2009) [pdf] - srimukh
https://www.nickbostrom.com/papers/pascal.pdf
======
roenxi
This can be easily resolved by considering that the victim of the mugging has
finite resources. This is a usual remedy to problems were expected value alone
gives stupid results (such as Pascal's Mugging). Something similar happens in
lotteries where even if the expected value of buying a ticket is positive it
is _still_ not rational for an ordinary person to buy a ticket.

If I have $400 dollars I can't afford to take 1:1000000 risks that cost $200
each. I will go bankrupt with an enormous likelihood whatever the payoff.
There is a minimum cutoff involving cost/probability below which it does not
make sense to take up the opportunity.

There are links to similar theoretical ideas from the Pascale's Mugging wiki
page - although from the casinos perspective not the gambler's -
[https://en.wikipedia.org/wiki/St._Petersburg_paradox#Finite_...](https://en.wikipedia.org/wiki/St._Petersburg_paradox#Finite_St._Petersburg_lotteries)
and then
[https://en.wikipedia.org/wiki/Gambler%27s_ruin](https://en.wikipedia.org/wiki/Gambler%27s_ruin)
for example.

Most people will not take an 99% risk of going bankrupt in a game that will
consume all their resource reserves; expected value as a statistic does not
meaningfully capture the risk. Positive expectation, losing strategy.

~~~
petters
Yes, but it is still not trivial to formalize mathematically without running
into trouble: [https://www.lesswrong.com/posts/a5JAiTdytou3Jg749/pascal-
s-m...](https://www.lesswrong.com/posts/a5JAiTdytou3Jg749/pascal-s-mugging-
tiny-probabilities-of-vast-utilities)

~~~
roenxi
It is pretty simple to formalise - if there is a pool of people who routinely
accept positive-expected-value gambles where there is a P = 99.999999999%
chance of going broke then we expect everyone in that pool will be broke
unless the size of the pool is comparable to 1/(1-P). Tighten up the bounds on
'comparable' a bit and that is formalised.

The mistake is accepting uncritically that expected value is the best metric
to optimise. Nobody ever proved that expected value is a strategically
superior metric. In fact it would be quite hard to prove that since it is not
true. It leaves people vulnerable to making very stupid decisions as
illustrated in Pascals Mugging.

Optimium strategy involves at a minimum considering your available
opportunities and available resources. Opportunity alone is not enough.

~~~
ralfd
> Tighten up the bounds on 'comparable' a bit and that is formalised.

Is this like „draw the rest of the owl“?

~~~
garmaine
What owl?

~~~
OscarCunningham
reddit.com/r/restofthefuckingowl

~~~
PhasmaFelis
Linking to examples of the meme without linking the actual meme doesn't really
explain what "draw the rest of the owl" means. [https://i.kym-
cdn.com/photos/images/newsfeed/000/572/078/d6d...](https://i.kym-
cdn.com/photos/images/newsfeed/000/572/078/d6d.jpg)

------
samspenc
This is likely trending because John Carmack's recent post saying he was
moving to AI references Pascal's Mugging:
[https://www.facebook.com/100006735798590/posts/2547632585471...](https://www.facebook.com/100006735798590/posts/2547632585471243/)

His Facebook post was also discussed in detail on Hacker News here:
[https://news.ycombinator.com/item?id=21530860](https://news.ycombinator.com/item?id=21530860)

~~~
Erlich_Bachman
I really love how this is could be seen as a demonstration that HN (or any
other similar interest community, or probably the internet in general)
functions like a hive mind, a collective consciousness. I've read that post by
Carmack and I wondered exactly that "wonder what Pascal's Mugging is,
interesting".

Sometimes you go around and wonder various things, but don't look them up or
do them in the moment, and then sometime later your subconscious mind serves
you up with an answer, perhaps when you are more relaxed you just think up of
the answer, or it happens to come up in a certain context, the subconscious
lights it up there. It might be a word that you see randomly in a newspaper,
or you think of a person that was related when you met them etc.

And here the subconscious did the same wonderous thing, except it wasn't even
strictly my personal subconsciousness, it was the group subconscious that
found the information and presented it.

~~~
peterwwillis
But how useful is it? Pascal's Mugging was submitted to HN and discussed 8
years ago. If the collective consciousness keeps needing reminders of what it
once knew, this is probably still inferior to a single intelligent person who
reads a lot and remembers it all.

~~~
ncmncm
The population using HN takes in new members continuously, and the fraction
who read everything posted eight years ago is very small.

------
zenon
I hereby declare that I will expose everyone that gives in to Pascal's mugger
to a negative utility so great compared to whatever the mugger promises that
it is always best to keep the wallet. I cold be telling the truth. You're
welcome.

------
YeGoblynQueenne
I don't know the background to this but if I understand correctly, it kind of
pivots on the probability that the mugger is indeed an Operator from the
Seventh Dimension, that Pascal places at 1 in a quadrillion.

In that case, I have to wonder where this estimate comes from? I get that it's
just an arbitrary number and that any number would do, as long as it wasn't
zero, but that's exactly the point: why can't Pascal place the probability of
his mugger being an Operator from the Seventh Dimension at zero?

Is there any evidence at all to support the mugger's claim? Is there any
evidence at all that there is such a thing as a "Seventh Dimension" for which
the only thing we know is that its "Operators" have magickal, utility-
maximising powers?

And does the whole thing only work if we assume that the probability that
there is such a place and such people is more than 0?

~~~
Strilanc
If you set the probability at zero, you won't be convinced when they actually
_are_ an operator from the seventh dimension. That is to say, you run into the
opposite problem of being Pascal's Muggle [1].

1: [https://www.lesswrong.com/posts/Ap4KfkHyxjYPDiqh2/pascal-
s-m...](https://www.lesswrong.com/posts/Ap4KfkHyxjYPDiqh2/pascal-s-muggle-
infinitesimal-priors-and-strong-evidence)

> _A wind begins to blow about the alley, whipping the Mugger 's loose clothes
> about him as they shift from ill-fitting shirt and jeans into robes of
> infinite blackness, within whose depths tiny galaxies and stranger things
> seem to twinkle. In the sky above, a gap edged by blue fire opens with a
> horrendous tearing sound - you can hear people on the nearby street yelling
> in sudden shock and terror, implying that they can see it too - and displays
> the image of the Mugger himself, wearing the same robes that now adorn his
> body, seated before a keyboard and a monitor._

> _[...] "Unfortunately, you haven't offered me enough evidence," you
> explain._

~~~
YeGoblynQueenne
Thanks, that's an interesting read. But I don't think it addresses my
question: why shouldn't Pascal place the probability that the Mugger is an
Operator from the Seventh dimension to _zero_ (rather than an infinitissemally
small number)?

The point is that, at the time when the Mugger declares himself to be an
Operator from the Seventh Dimension who can offer large rewards etc, there is
no evidence to suggest he's saying the truth. No evidence at all. Accordingly,
the probability that he's telling the truth must be zero. Where does a non-
zero probability value come from?

Are you then saying that the probability of any reward should never be placed
to zero because that would not maximise rewards?

~~~
Dylan16807
Probability zero is the same as saying that it would take infinite evidence to
convince you. Even if someone provides amazingly convincing evidence, better
than you've ever seen, a flat 0 or 1 eats it.

> there is no evidence to suggest he's saying the truth. No evidence at all.
> Accordingly, the probability that he's telling the truth must be zero.

I don't think that logic works. What if the claim was "I have a five dollar
bill in my pocket"?

~~~
YeGoblynQueenne
>> Probability zero is the same as saying that it would take infinite evidence
to convince you.

That assumes I can't go back and change my earlier beliefs. But I don't see
why that's necessary. If I have no evidence that X is true at time t, I assing
a probability of 0 to it. If I acquire evidence that X is true at time t+1, I
throw out the 0 and assign a higher probability to X.

The world changes all the time. Why am I condemned to hold on to obviously
unsound beliefs for all eternity?

>> I don't think that logic works. What if the claim was "I have a five dollar
bill in my pocket"?

That depends. I've seen five dollar bills coming out of peoples' pockets
before (actually, I haven't because dollars are not common where I live but
Ok). I don't have to assing a zero probability to that. I have some evidence
that it's possible.

But I have no evidence that there even exists such a thing as a Seventh
Dimension etc.

~~~
Dylan16807
> If I acquire evidence that X is true at time t+1, I throw out the 0 and
> assign a higher probability to X.

> The world changes all the time. Why am I condemned to hold on to obviously
> unsound beliefs for all eternity?

Normally when you update a probability, how much you change it is based on the
strength of the evidence. If your probability of something is ultra-low, and
you see an event that's a million times more likely if that thing is true,
your new probability is roughly a million times higher. And for a probability
that's sufficiently close to 0 or 1, that pit is basically impossible to climb
out of.

Do you have an alternate method to suggest? What's the calculation you would
use? Note that "I'm seeing this with my own eyes" should only give you so much
change, because you might have accidentally taken a whole bunch of
hallucinogens.

> But I have no evidence that there even exists such a thing as a Seventh
> Dimension etc.

If you're setting a hard cutoff based on the silly Seventh Dimension stuff,
then you still fall for the version where I come to your house and sign a
document giving you a giant pile of money. That's how mortgages and business
deals work every day after all.

> How about the statement "Hillary Clinton is the President of the United
> States"? What probability should I assign to that? I know that the PotUS is
> Donald Trump. Does Cromwell's Rule mean that I have to believe that Hillary
> Clinton is the PotUS at least a little, because otherwise I will never be
> able to believe it if she ever gets elected president?

Not for that reason. But you have to factor in the chance that you got
confused, or your brain is failing to make new memories and it's actually
2022, or you just woke up from a really detailed dream about the wrong
president.

~~~
YeGoblynQueenne
>> Do you have an alternate method to suggest? What's the calculation you
would use? Note that "I'm seeing this with my own eyes" should only give you
so much change, because you might have accidentally taken a whole bunch of
hallucinogens.

I don't understand. How would it happen that I've accidentally taken a whole
bunch of hallucinogens? I never go near that kind of stuff.

>> Not for that reason. But you have to factor in the chance that you got
confused, or your brain is failing to make new memories and it's actually
2022, or you just woke up from a really detailed dream about the wrong
president.

I don't see how that would happen either. Why would my brain fail to make new
memories? Why are you saying that this might be the case?

I think this is just enhancing the deep unreality of what you are proposing.
If we need to assume that I'm in some kind of weird mental state that I have
no reason to be in for your whole proposition to make sense then I really
don't see the point of it, other than perhaps an interesting theoretical game.

~~~
Dylan16807
You can't come up with a one in a billion scenario that you would accidentally
take a hallucinogen?

You never ever have a dream that seems real for a few moments?

And failing to make new memories would be a specific but possible injury.

We're supposed to be working with very low probabilities here. That's the
whole point of the thought experiment. If you're going to round anything below
one-in-a-million to exactly zero then that's your prerogative, and it works in
everyday life, but it's objectively wrong; it would falsely reject the idea of
lightning strikes and winning the lottery.

> I think this is just enhancing the deep unreality of what you are proposing.

You didn't even reply to the part about removing all the silly stuff and
cutting it down to just "guy offers to sign a document for lots of money"...

~~~
YeGoblynQueenne
But I'm not hallucinating and I'm not dreaming either.

Also, I don't know why you're saying I'd round anything below one-in-a-million
to zero. I wouldn't. But would assign zero probability to a mugger being an
Operator from the Sevent Dimension because that's a patently absurd idea that
I see no good reason to grace even with the slightest degree of belief.

I mean, if you take what you are saying here at face value I actually have to
assume that there is a probability that there exists a Seventh Dimension with
magikcally powerful Operators inhabiting it. In real life, not just in the
context of Pascal's Wager. Because I can't assign zero probability to
anything.

That just doesn't make any sense at all.

>> You didn't even reply to the part about removing all the silly stuff and
cutting it down to just "guy offers to sign a document for lots of money"...

Apologies. I didn't understand what you meant with that and I didn't want to
clutter the comment space with more clarifying questions.

~~~
Dylan16807
> But I'm not hallucinating and I'm not dreaming either.

You've never been unsure if something actually happened for a moment? Because
even if you only spend a few moments like that per month, we can assign it a
probability.

> I mean, if you take what you are saying here at face value I actually have
> to assume that there is a probability that there exists a Seventh Dimension
> with magikcally powerful Operators inhabiting it. In real life, not just in
> the context of Pascal's Wager. Because I can't assign zero probability to
> anything.

You don't think there's _any_ chance that you fundamentally misunderstand the
universe and that there are powerful secrets being actively hidden from you?
It doesn't have to be real 'magic', just something too beyond your
understanding. I think there's _some_ chance of that. I'd say less than 1% and
more than 1 in a googolplex, to put some amusingly loose bounds on it. And
then you have to factor in the chance the guy picks you in particular to mess
with, but that's not an unreasonably large number.

> Apologies. I didn't understand what you meant with that and I didn't want to
> clutter the comment space with more clarifying questions.

You're objecting so specifically to the seventh dimension stuff, I thought it
would be simpler to cut all that out. The point of the thought experiment is
just a very likely but very positive act. And the way to have a productive
conversation is to respond to the strongest form of the argument. So in that
version, you can't just _declare_ that the person in front of you isn't a rich
guy screwing around and giving money to people that accept, because the
probability of that is clearly not zero.

~~~
YeGoblynQueenne
>> The point of the thought experiment is just a very likely but very positive
act.

(You mean very _un_likely eh?)

The reason I'm objecting specifically to the seventh dimension stuff is that
it's just something fanciful that someone came up with, so it's obviously fake
and I don't have to believe it even a little bit.

The probability of someone just handing out money (if I read you correctly
this time) is very low, but not zero, yes. But I'm contesting the claim that
I'm never allowed to assign 0 probability to anything, because then I'm at
risk of losing out. Sometimes, you don't risk being wrong by disbelieving
something.

Anyway I'm getting more and more confused by this conversation. I don't think
it's getting anywhere. Thanks for your patience- you have the floor.

------
skrebbel
Wow, I always thought of Pascal's Mugging as a satirical illustration of how
stupid it is to take enormous (or extremely small) numbers seriously. Turns
out people like John Carmack (see other comments in this thread) aren't
picking up on the satire. Or am I reading him wrong? I think he's smarter than
me, so what am I missing?

Given Bostrom's general love for mixing tiny probabilities with enormous
outcomes, what's the point of this article? It seems delightfully self-
critical. How can the conclusion be anything other than that we should _not_
be taking the AI/singularity crowd too seriously, as doing so would be akin to
voluntarily handig over a wallet to a mugger?

~~~
Erlich_Bachman
Sounds more like you are taking the argument too seriously and are trying to
read more into it than what it is.

It is just a philosophical story created to provide a certain line of
reasoning, as certain possible structure of an argument. People are free to
apply this argument however they want, it doesn't prove anything by itself, it
doesn't say anything about the world, it's up to the user of it. It does not
make any conclusions, it's just a story. Carmack used it to illustrate his own
beliefs (which are therefore: AI is possible and the payout for the AI is
exteremely high, even if probability for it during the next couple of years is
low).

Carmack did not mean that you should believe or not believe in AI or anything
else based on this argument. He just used it to illustrate what he himself
chose to do. He did not base it on this argument alone, he did not just hear
the argument and suddenly decide "now because of that I have to work on AI",
he based it on his experience and knowledge of the actual field. The mugging
argument is just a cute way of quickly explaining it.

> I think he's smarter than me, so what am I missing?

Wisdom and context.

------
praptak
Paradoxes such as this one point at some rarely discussed assumptions about
the utility function. The major one is that it exists. Or exists as something
more than a model which is only good for a range of payoffs.

~~~
adamchalmers
The point of paradoxes like this is to demonstrate that even in our incredibly
simple toy models of agents, we still run into issues. These paradoxes help
point out weaknesses of the models and act as new desiridata for better
models.

------
EvanWard97
Expected value != expected utility.

Just taking this seriously pretty much resolves all these problems.

It makes sense to take high cost/reward, low probability events seriously if
the _expected utility_ works out. Examples include reducing existential risk
substantially, even at cost to short-term utility (say, in the form of well-
being) to increase the probability that we can eventually figure out how to
optimally arrange matter and energy and trigger a utilitronium shockwave.

------
ivan_ah
In case someone is not familiar with _Pascal 's wager_, here is a table that
shows the expected values:
[https://en.wikipedia.org/wiki/Pascal%27s_wager#Analysis_with...](https://en.wikipedia.org/wiki/Pascal%27s_wager#Analysis_with_decision_theory)
(the appearance of infinity breaks the decision theory...)

~~~
MaxBarraclough
Pascal's Wager fails for the simpler reason that it ignores the possibility
that believing in god could send you to hell.

I see a similar problem here. Pascal should consider the possibility that the
mugger will use his (Pascal's) silly action as the basis for punishment, in
place of the promised reward. Slim probability, potentially very high cost.
The fact that this risk has gone unstated doesn't mean it isn't there.

~~~
adamchalmers
I agree. This problem is still useful though, because to analyse a lot of
counterfactual muggers, you must be able to analyse one mugger!

~~~
MaxBarraclough
I agree that both problems are worth analysing, even if they're both absurd.
Same goes for the Hangman's Paradox, a personal favourite. [0]

I gather that Pascal had never intended Pascal's Wager to be a watertight
argument, it was intended more as a plaything for showing how unlikely it was
anyone could make a watertight case for the existence of God.

[0]
[https://en.wikipedia.org/wiki/Hangman%27s_Paradox](https://en.wikipedia.org/wiki/Hangman%27s_Paradox)

------
dang
Related from 2011:
[https://news.ycombinator.com/item?id=2932157](https://news.ycombinator.com/item?id=2932157)

