
When Do Extraordinary Claims Give Extraordinary Evidence? (2007) [pdf] - gwern
http://mason.gmu.edu/~rhanson/extraord.pdf
======
nicholaides

        For example, you would assign a very low prior probability to your friend telling you that she met her 5’2” tall second cousin last Tuesday at   8:47am at 11 feet northwest of the smaller statue in a certain square. Nevertheless, after   she made such a claim to you, you would likely place a high    probability that the event happened as claimed. On topics   like this, most people are fairly reliable.
    

This is a bad example. "you would assign a very low prior probability to your
friend telling you __________". If it was a prediction, yes, that would be
extraordinary, but it's not.

Its the same as if someone told you they flipped a coin 100 times and got a
certain sequence of heads and tails. The probability of that happening is
1/2^100, but it's only extraordinary if you predict it in advance.

------
nickpsecurity
I've always found the "extraordinary claims require extraordinary evidence"
meme ridiculous. It defies reason and the scientific method. Extraordinary
claims simply need evidence they're true. That's it. Then, analysis and
experiments should be conceived to determine why they contradict the
"ordinary" model plus look for sources of error. If error or deceit are found,
then the extraordinary claim is rejected. If not, the "ordinary" model is
amended or rejected with data from new.

That simple. The need for extraordinary evidence seems to just promote
orthodoxy. Better to focus on data and methods.

~~~
paulddraper
In context, that expression refers to claims that contradict existing
evidence.

For example, if someone finds a way to create mass/energy, that would be
extraordinary. All existing evidence suggests mass/energy is conserved. Even
if no one has tried XYZ before, the evidence of XYZ creating energy has to be
weighed against the evidence of thousands of (other) processes _not_ doing
that. It's possible someone made a mistake, or the results are being
interpreted incorrectly.

While there is an ideal, the reality is that "evidence" is of varying types
and qualities, and that perfect evidence is not always feasible.

In the US, criminal convictions ("extraordinary" claims) require proof beyond
reasonable doubt. But civil convictions ("ordinary" claims) require only a
preponderance of the evidence.

~~~
andrewflnr
That may be the ideal usage, but I'm not sure I've ever seen it used that way.
If the phrase is about contradicting existing evidence, we should just say
that instead.

~~~
nickpsecurity
My point exactly. "Extraordinary evidence" just seems made up. I only require
more and/or better evidence to back up an extraordinary claim versus claims
contradicting it.

~~~
GFK_of_xmaspast
> more and/or better evidence

So, in other words, evidence that is beyond the ordinary. If only there were a
prefix that could be applied here.

~~~
nickpsecurity
No, ordinary evidence or models that fit the observed data better or make more
accurate predictions. I leave out the extraordinary part except for when
praising scientists for talented observations or methods.

~~~
paulddraper
Okay. Well, here extraordinary is _supposed_ to mean better/more then
ordinary.

But if it means talented or praiseworthy or something similar, then I whole-
heatedly agree.

