
The Troubling Reality of Forensic “Science” - anon1385
https://firstlook.org/theintercept/2015/04/24/badforensics/
======
time_is_scary
I've had my own bad experience with forensic science.

I was a research intern at Oak Ridge National Lab. They put on several
educational events for the interns. One event was a guy from the group doing
forensic science to come in and talk about it. He talked about rate of decay
of corpses and things like that.

Then he brought out dowsing sticks. Said something about the bones of buried
bodies giving off magnetic fields that affect the sticks.

He then took a leg bone out of a trash bag, put it on the ground, and walked
passed it holding the sticks. The sticks moved when he did it, but of course
when anyone else in the room tried nothing happened.

Honestly, all the interns in the room were dead quiet. I think we were all
really confused, thinking that maybe it was some sort of joke or hazing. Or
maybe a test of some sort? It was really weird, but destroyed my regard for
forensic "science".

~~~
rjsw
He could have been a complete nut, or...

I have tried dowsing for water and found it very accurate.

We had a problem with the water supply to a house. There was a spring that fed
a collection tank through some pipes under agricultural land, we guessed that
a tractor had squashed the pipes somewhere along their length. A backhoe was
booked for the following day so it didn't really matter whether we got any
results or not but we decided to try dowsing to see if we could work out the
path of the pipes, just for a bit of fun more than expecting it to really
work.

We used some bent aluminium rods and walked about a bit in the field sticking
twigs into the ground wherever the rods moved. This worked for some family
members, not for others. The next day the backhoe dug out the old wooden pipes
and they were exactly underneath where we had placed the twigs on the surface
but several feet down.

~~~
EliRivers
Well then, can I interest your successful family members in a million dollars?
It's a genuine offer. One million dollars if they can do it. Dowsers have
tried and failed in the past, but perhaps your family really do have
supernatural powers.

~~~
rjsw
I'm more interested in finding out how we were "fooled".

Even if whatever you hold just acts as an amplifier for small changes in the
firmness of the ground underfoot it could still be useful.

~~~
sukilot
You were fooled by randomness and a poor analysis of prior probability.

------
formulaT
Coming from a statistics background, it seems like there is an easy fix for
many of these problems (trigger warning: p-values ahead).

For example, if you want to claim someone's fingerprints are a match for a
given fingerprint from a crime scene, you need to show that the probability of
a random person's fingerprints matching is small. One direct way to prove this
is to get the prints of N random people, plus the prints of the suspect, and
compare them to the print from the crime scene. If the suspect is a better
match than N(1-p) of the other fingerprints, then the match has a p-value of
p.

Even if matching is done by a human expert, a computer could be used to reduce
the set of potential matches before a human was used to select the best match
out of these candidates (of course, the expert would not know which one
belonged to the suspect).

I know all of this is obvious to someone with a thorough knowledge of
statistics, but I honestly think this is just lacking in the discipline.

~~~
hackuser
IIRC, part of the problem is that fingerprint comparison methods do not
produce consistent results. Different experts produce different results with
the same set of fingerprints.

Also, as the article says, fingerprints 'in the wild' are often incomplete,
smudged, etc.

EDIT: From the National Academy of Sciences report linked to from the article
(the ACE-V method is the standard fingerprint analysis):

 _the ACE-V method does not specify particular measurements or a standard test
protocol, and examiners must make subjective assessments throughout. In the
United States, the threshold for making a source identification is
deliberately kept subjective, so that the examiner can take into account both
the quantity and quality of comparable details. As a result, the outcome of a
friction ridge analysis is not necessarily repeatable from examiner to
examiner. In fact, recent research by Dror23 has shown that experienced
examiners do not necessarily agree with even their own past conclusions when
the examination is presented in a different context some time later._

Don't miss that last sentence.

~~~
formulaT
I don't see how any of those things prevent the methodology I described being
applied.

~~~
hackuser
For example, you say,

> If the suspect is a better match than N(1-p) of the other fingerprints, then
> the match has a p-value of p.

There may be no way to objectively determine if the suspect's prints are a
better match than others.

More generally, I lack your expertise but it seems to me that statistics
require data; we may not have reliable data. More colloquially, garbage in,
garbage out.

Anyway, I recommend the National Academy of Sciences report, which is authored
in part by their Committee on Applied and Theoretical Statistics.

------
hackuser
For those interested in these kinds of issues, I recommend the Reference
Manual on Scientific Evidence, written by the National Academies of Science
for federal judges.

[http://www.fjc.gov/public/pdf.nsf/lookup/SciMan3D01.pdf/$fil...](http://www.fjc.gov/public/pdf.nsf/lookup/SciMan3D01.pdf/$file/SciMan3D01.pdf)

Just the chapter "How Science Works" by David Goodstein, Professor of Physics
Emeritus (and former Vice Provost) at Cal Tech, is a revelation for anyone
outside the scientific community. Highly recommended.

~~~
jacquesm
Warning for those on mobile bandwidth, that's a 1000+ page PDF.

~~~
hackuser
Sorry. You can read it online, one page at a time:

[http://www.nap.edu/catalog/13163/reference-manual-on-
scienti...](http://www.nap.edu/catalog/13163/reference-manual-on-scientific-
evidence-third-edition)

There also used to be a place to download PDFs of individual chapters.

------
DanBC
There's a BBC radio programme talking about bad forensics. Only one episode so
far, but it talks about some of the same cases.

"Forensics in Crisis"
[http://www.bbc.co.uk/programmes/b05r3tf1](http://www.bbc.co.uk/programmes/b05r3tf1)

Edit this submission links to [http://www.washingtonpost.com/news/the-
watch/wp/2015/02/18/a...](http://www.washingtonpost.com/news/the-
watch/wp/2015/02/18/attack-of-the-bite-mark-matchers-2/) which is a better
read.

------
ChuckMcM
I wonder what became of the National Commission on Forensic Science[1] ?

[1] [http://www.popsci.com/science/article/2013-02/new-
commission...](http://www.popsci.com/science/article/2013-02/new-commission-
set-standards-troubled-forensic-sciences)

------
mathgenius
All of this points to the fact that it is just so hard to find out about
"reality". And when egos are involved triple that. Actually, you could
probably _define_ ego to be the denial of reality.

I'm still waiting for particle physicists to do double-blind experiments. (Not
going to happen.)

~~~
TheCoelacanth
I curious what a double-blind experiment would mean in the context of particle
physics. The particles don't know about the experiment?

