

Absence of evidence - henning
http://www.johndcook.com/blog/2011/02/22/absence-of-evidence/

======
baddox
The big point here, which the author doesn't mention, is that you can have
absence of evidence _without searching for evidence_ or you can have absence
of evidence _after rigorous searching_.

I believe that the former situation is the one covered by the saying "absence
of evidence is not evidence of absence." For example, I personally haven't
observed any evidence of gravitational time dilation, because I haven't
looked. I would be a fool to interpret this lack of evidence as evidence that
gravitational time dilation doesn't exist. The engineers of the Global
Positioning System have evidence.

However, after a reasonably rigorous search, one may indeed reasonably
conclude that something doesn't exist if no evidence of it has been found,
especially when evidence would be easily visible if the thing did exist. The
examples the author gave fall into this category: Most likely, given the
distribution of humans on Earth, there would be evidence of dodo birds if they
were extant.

Now, it's important to be careful when estimating the likelihood of stumbling
upon evidence of something. Hundreds of new species (most only subtly
different than known species) are discovered every year, and there are
undoubtedly thousands of oceanic creatures that have never been seen by
humans, much less captured and dissected. It would be foolish to deny that
there aren't hundreds or thousands of yet-undiscovered species simply because
as of now we have no evidence of them.

------
smanek
It's a pretty straightforward implication of bayes' law that the absence of
evidence is evidence of absence. In fact, we can quite easily quantify just
how strong the evidence it provides is.

[http://lesswrong.com/lw/ih/absence_of_evidence_is_evidence_o...](http://lesswrong.com/lw/ih/absence_of_evidence_is_evidence_of_absence/)

~~~
PaulHoule
But, with Bayesian methods you're feeding in prior information which can
itself be flawed. When it comes to some question like 'What is reliability of
the space shuttle?' the largest uncertainty in the calculation is that there
is some failure mode you haven't thought of. Despite decades of changes to the
space shuttle to improve safety, the official estimate of the failure
probability hasn't changed since 1975.

In conventional logic, we say that a reasoning method is sound if it never
comes to correct conclusions. One version of 'A of E' -> 'E of A' is the
"closed world assumption" which, in one variant, means that "if you haven't
been presented with X, then not(X) is true."

Now, we know that's not sound, not correct 100% of the time. However, when
doing 'commonsense reasoning', the CWA can sometimes be a highly effective
heuristic. (Note also that's there's no completely satisfactory way to deal
with logical negation in knoweldge based systems at this time, so we've got to
choose some poison, or otherwise deal with a system that "can't say no".

Part of that is that 'commonsense reasoning' is probabilistic. For instance,
an effective hold 'em player has to estimate the probability of improving his
hand at drawing, have a guess about what his opponent(s) may be holding, and
how the opponent may respond to his actions. From that he's able to estimate
odds which help him decide what to bet.

Poker's a pure example, but the same kind of probabilistic reasoning is
necessary for hunting, software project management, scientific research, and
many of things that people do.

------
_delirium
If you're going to take it as an absolute statement, sure. But the more usual
meaning in science, at least, is that the lack of evidence for something
doesn't _by itself_ constitute evidence against it (or at least, it's evidence
with weight arbitrarily close to zero). You need something more, such as
absence of evidence _and_ a reason to believe that finding such evidence, if
it existed, would have been at least somewhat probable.

(Which is of course why the example in this post is doing the calculation of
what observations we would've expected under each scenario.)

~~~
pjscott
Sometimes I think this would be a lot easier for everyone if we just hauled
out Bayes' theorem and had done with it. If a hypothesis (weighted with your
confidence in it) predicts that a set of observations would turn up more
evidence for it than you actually find, then that is evidence against that
hypothesis, and should cause you to lower your estimate of the probability
that the hypothesis is correct. And vice versa.

For example, suppose that I want to explain why my apples have been stolen
from my apartment. I live in a hypothetical universe where there are only
three possibilities, all of which have roughly the same likelihood, _a priori_
: spontaneous combustion, alien abduction, or theft by a troupe of dastardly
harlequins.

If the apples combusted, I would expect to see scorch marks, or perhaps soot
stains on the wall nearby. If they were abducted by aliens, I would not expect
to see anything, since everyone knows that alien abduction rays leave no
signs. Finally, if the vile harlequins made off with my fruit, I would expect
to see some signs of their passing -- face-paint on the doorknob, perhaps a
fool's cap hastily discarded in a corner.

Now, I inspect my apartment. I find no scorch marks, which is evidence against
spontaneous combustion. If my apples had burst into flame, it's technically
possible that they could have done so without leaving marks, but unlikely. I
find no evidence of alien abduction, which is exactly what I would expect to
find, so it does not change my estimate of that hypothesis' likelihood.
Finally, I find a scrap of ripped, brightly-colored cloth which appears to
have been torn from the outfit of an interloper hastily scurrying away through
my pointlessly-large ventilation ducts. My face grows grim as I raise my
estimate of the likelihood of my third hypothesis. Harlequins! Foul creatures
of the _Commedia dell'arte!_ My apartment has been defiled by their acrobatics
and shenanigans! And they stole my darn apples!

So, in the case of spontaneous combustion, absence of evidence was evidence of
absence. For alien fruit abduction, absence of evidence wasn't evidence of
anything. In the third case, there was evidence above what I would expect from
my prior beliefs, so that was evidence for the harlequin hypothesis.

~~~
_delirium
I buy that, but I guess to me that _is_ what the slogan is supposed to imply,
but I do agree it's not always taken that way. That is: merely not finding
evidence tells you nothing. However, not finding evidence, _when_ you have a
prediction of what evidence should be found, may be evidence against. So you
need two things there: 1) absence of evidence; and 2) a prediction of what
evidence there should've been. If all you have is #1 by itself, you have no
evidence of anything. Hence, absence of evidence (without more) isn't evidence
of absence.

Maybe I'm in the wrong debates, but most of the times when I see the slogan
applied, the person doesn't have the second part. For example, a common
elementary error is to conclude that variables are uncorrelated because you
calculated their correlation in your data set and it was not significant.
Failure to find a correlation in a very large population study that's large
enough that it should have found one, can be evidence of no correlation (or no
large correlation, at least), but you have to do a separate analysis for that.

------
tjmaxal
The turkey comment reminds me of black swan investing. Just because it hasn't
happened yet doesn't mean it won't in fact for certain things, it actually
increases the odds that it will happen the more time passes without it
occurring, like Volcanic eruptions.

------
Silhouette
Another slightly annoying saying is "The plural of anecdote is not data".

Any anecdote is a data point, if you assume that it is factually correct.

You may or may not be able to draw any significant conclusions from that data,
once you take into account sample size and bias, of course.

But unless you're going to "sample" the entire population of interest, pretty
much all statistical analysis in the real world is done based on multiple
anecdotes. We just try to minimise the bias and to choose a sample size large
enough to give results significant enough for our purposes.

~~~
Panoramix
I disagree. I interpret that phrase as stating that an anecdote is something
fundamentally different from a valid data point: the data point came from a
carefully designed experiment using properly tested equipment, control groups,
rigorous mathematical analysis, perhaps using double-blind, randomized trials
where it applies, etc... whereas the anecdote is a story you hear from
somebody, as per its definition:
<http://en.wikipedia.org/wiki/Anecdotal_evidence>

