
Ontario wrongly separated families on results of Motherisk hair-testing program - okket
http://reason.com/blog/2018/03/05/motherisk-hair-testing-program-was-junk
======
mcherm
There is one simple technique that would allow nearly any forensic analysis
technique to be assessed. It would detect the difference between good science
(DNA) and junk science (forensic hair matches). It would work down to the
level of the individual laboratory (as in this example) or individual
laboratory researcher (Annie Dookhan). And it would only cost 10% extra.

That technique is blind testing with published results. 10% of all submissions
to the testing lab would be test samples -- a mix of a bunch of known negative
samples (from individuals completely unrelated to the crime in question) and a
few known positive samples (perhaps from someone already convicted with
overwhelming evidence). The lab would not be told which samples were tests and
which were "real", and the test samples would be evenly distributed (10% of
the submissions from each department; 10% of the "rush" submissions, etc.).
Finally, the results reported on these test samples would be made publicly
available.

Although this simple, effective, and cheap method would completely transform
forensic practice, it will not be implemented. The reason is incentives. The
parties involved in forensics are the police or prosecutors and the labs. Both
have an incentive to identify the guilty. The accused have a countervailing
interest (in correctly determining innocence), but they are not part of the
transaction, so their interests are not represented. In fact, the entire
system is incented to find people guilty, not to correctly assess people's
guilt. So most in the system are actually DISincented to discover and debunk
junk science.

~~~
darkerside
Something could trivially be done to the same effect. A private party, say an
investigative journalist, could submit a number of samples and report on the
results (perhaps that happened in this case and that's why we know about it?).
But to keep costs under control, instead of testing like this on every lab on
a regular basis, that private party could randomly select labs for testing.
While they're at it, they'd probably prefer to select labs with complaints,
rather than a random sample. And at that point, you have something resembling
the system we currently operate under, which has--to be fair--been completely
successful (if a bit later than ideal) in identifying this problem lab. But as
taxpayers, is it worth instituting a new regulated program over this
essentially free solution?

Besides, this seems like a case of closing the barn door after the horse has
escaped. Successfully identifying a problem lab--albeit tardily--is a signal
the system is working; not that it needs a complete overhaul. Rather consider
what completely unrelated problems would benefit from regulatory oversight at
a higher return on investment. Focus on those, not on a solved problem.

~~~
sbov
You're assuming there aren't any other problem labs that haven't or won't be
discovered. How expensive are these tests? How much of the police's budget is
it? Surely you know these things if you are talking return on investment,
because you must know these things to actually know what your return on
investment is.

25 years is a long time to be improperly ripping apart families. Is there a
point at which it becomes a problem? Death tardily solves all your problems
after only 80 years, so I assume somewhere between 25 and 80.

~~~
darkerside
I'm not saying it isn't a problem, only that we need to pick and choose where
our dollars go. Is this really a better investment than, say, increased
funding for public broadcast programming that might discover this kind of
thing in a less directed way? That way you don't end up with yet another
obsolete program hanging around costing money when this problem is no longer
relevant.

And I think the onus is on the suggester to provide data to prove ROI.

------
leereeves
Did anyone else think this article is saying that hair testing is "junk
science"?

That's _not_ what the report [1] actually says. The independent review
actually found specific flaws in the way the Motherisk laboratory was
performing these tests, such as failure to wash hair samples, non standard
hair lengths, misinterpretation of test results, failure to communicate the
limits of the tests, and the lack of proper forensic procedures such as
adequate records, oversight, and chain of custody.

1:
[https://www.attorneygeneral.jus.gov.on.ca/english/about/pubs...](https://www.attorneygeneral.jus.gov.on.ca/english/about/pubs/lang/)

~~~
clarkevans
To a lay person in a situation like this, how does the distinction you make
(between bad science and an incompetent application) imply an actual
difference? Parents got stripped of their children because of some
"scientific" process that the court relied upon.

~~~
arkades
Bad execution can and should be improved upon. Bad science (eg, matching
bullets to guns) needs to be thrown onto the trash heap of history.

The former can be debunked on a case by case basis. The latter needs to be
categorically banned from judicial proceedings.

~~~
koolba
Are you saying that ballistic fingerprinting is entirely bullshit?

~~~
bmelton
I don't know if that's what OP is saying, but it seems fair to say that it's
at least _enough_ bullshit that it should probably never be used as
evidentiary.

The states who pioneered its adoption (New York and Maryland) have both
abandoned it for being costly and ineffective. The FBI has abandoned
comparative bullet lead analysis for similar reasons[1]. The National Academy
of Sciences found the tests to be "full of holes", and that not enough study
was being done to ensure its reliability, which was a finding that the United
States President's Council of Advisors on Science and Technology under Obama
concurred with.

The general consensus now seems to be this[2]:

> An understanding of the metallurgical principles operative in the
> melting/casting process as well as the data acquired for this study,
> indicate that any forensic conclusions which associate unknown bullets with
> the “same source”, and/or “same box” should fail most or all Daubert
> criteria.

[1] [http://www.washingtonpost.com/wp-
dyn/content/article/2007/11...](http://www.washingtonpost.com/wp-
dyn/content/article/2007/11/17/AR2007111701681.html)

[2]
[https://www.sciencedirect.com/science/article/pii/S037907380...](https://www.sciencedirect.com/science/article/pii/S0379073802001184?via%3Dihub)

~~~
NutriSugar
>it seems fair to say that it's at least enough bullshit that it should
probably never be used as evidentiary.

Doesn't this also apply to eyewitness testimony, if not many other forms of
evidence?

~~~
boomboomsubban
Yes, ballistic fingerprinting was just brought up as an example.

------
simion314
IT is incredible how major decisions are taken because of this bad experts,
the science community needs to find a way to properly handle this "forensic
science" before someone invents a mind reader that will detect if you have bad
thoughts like pedophilia or similar and destroys more lives.

Probably you will need some system similar to physics where you can claim
something is true untill you have a big confidence in your experiments and the
experiments are replicated.

~~~
madaxe_again
It’s called the scientific method.

Unfortunately, the people passing judgment (judges, lawyers, juries) are
generally not scientists, and therefore are equally likely to believe the guy
with the paedo-dowser as the next guy.

Unreliable witnesses are as old as the justice system - “science” gives them a
veneer of respectability now, just as it has in the past.

I see two problems, neither of which have an easy solution:

1) scientific literacy is dire, and worsening. Education helps, but when anti-
science sentiment runs rife in popular culture and media, as it does now,
education is met with resistance. I have been reduced to a gibbering wreck
trying to explain to otherwise intelligent people that science isn’t “just an
opinion”, and that astrology and astronomy aren’t “just different, equally
valid opinions”.

2) the concept of the jury trial is broken. It used to be that you would be
tried by your peers - people who likely knew you, had a similar background to
you, could empathise and make judgments based on a clear understanding of your
world. Now, you’re tried by strangers who have usually been carefully selected
to be as hostile as possible without an outright admission of bias. I think
there’s a happy medium somewhere between the two, as there are obviously
biases in both cases.

~~~
loup-vaillant
> _I have been reduced to a gibbering wreck trying to explain to otherwise
> intelligent people that science isn’t “just an opinion”, and that astrology
> and astronomy aren’t “just different, equally valid opinions”._

A short sarcastic laughter would probably have worked better. If people sense
their status is lowering because they believe in this "equally valid opinions"
nonsense, they will have a strong incentive to update their belief.

Of course, many will be offended, in which case I recommend turning the
sarcastic laugh into a benevolent, condescending smile, with perhaps an appeal
to their intelligence to soften the blow. Ideally though, you should spin it
as if you were ridiculing a third party. This lets them avoid direct
confrontation if they so chose, and you still get to make your point.

In any case, they cannot be taught as long as they subscribe to epistemic
relativism. They first have to get rid of it, _somehow_. Doesn't matter if
it's for the wrong reasons. Only then can they be taught how to properly
change their mind.

~~~
imron
> offended, in which case I recommend turning the sarcastic laugh into a
> benevolent, condescending smile

Acting condescendingly will only serve to make people think you are a
condescending git.

~~~
loup-vaillant
Not if you have higher status to begin with. Not if you are their friends to
begin with. And if you merely have power over them, they will notice that
their beliefs will attract the contempt of people who have power over them. Of
course, this probably won't work on public web forums. Of course, this won't
work on your boss. Of course, this will not work well on strangers.

Then again, the best way to do it is to act as if you were ridiculing a third
party. If they get offended, try not to notice, and direct your condescension
at those other people you're pretending they're talking about. Give them a way
to save face.

Whatever the means, the idea is to stress that epistemic relativism is not
socially acceptable. Make them feel that thinking such thoughts has
consequences.

~~~
loup-vaillant
The downvote suggests I said something unacceptable. Maybe it was this:

> _Make them feel that thinking such thoughts has consequences._

I do not condone actually carrying out dire consequences, we've had enough
dictatorships to show us how this plays out. I'm talking about manipulating
people. A little. Because nothing more innocuous that I know of will do.

------
ZeroGravitas
_Just believe the parents_ seems like it may not always be the best policy
either. There's some really terrible parents out there. I don't know off hand
whether the numbers suggest more people are affeted by bad science and
government overreach, or bad parents and government indifference, but I feel
this blog is probably on the wrong side of the argument, even if there's areas
everyone should agree on (no junk science for example).

Though the blogger doesn't actually say believe the parents, they say believe
the mother (well, they say "don't automatically discount the mother" which in
context appears to be a weaselly way of saying the same thing) which is a bit
more specific. One of the cases seems to involve an accusation from a husband,
that the test backed up. So even if the lab was tossing coins to come up with
their results, we also have to assume a husband and father lied about his
child's mother's behaviour.

~~~
mseebach
Taking a child away from a family is one of the worst violations a government
can subject someone to, all the way up there with imprisonment. The argument
isn't a binary choice between trusting the mother or not, it's to raise the
bar for evidence significantly - which is what I take don't automatically
discount the mother" to mean.

This should be treated like a criminal matter, with the same standards of
evidence. If we're comfortable letting a criminal go free of prison because
guilt can't be proven beyond a reasonable doubt (and we should, for the
record), we should be comfortable letting a child stay with its parents until
the parents inability to care is established beyond a reasonable doubt. Or at
the very least, the parents in question should have a court appointed defence
lawyer.

~~~
ComputerGuru
I was with you until this:

 _If we 're comfortable letting a criminal go free of prison because guilt
can't be proven beyond a reasonable doubt (and we should, for the record), we
should be comfortable letting a child stay with its parents until the parents
inability to care is established beyond a reasonable doubt._

The difference is that in one scenario it’s about punishing something that
happened and is done and in the other it’s to prevent ongoing/future harm. If
you’ve seen some of the things parents are willing to do to their own kids,
you would see why it’s not that simple.

~~~
ramblerman
> The difference is that in one scenario it’s about punishing something that
> happened and is done and in the other it’s to prevent ongoing/future harm.

Letting a criminal go free (think serial-murderer, rapist) is just as valid an
argument for future harm.

I don't see why you treat these 2 cases differently.

~~~
dbajakkchs
The legal requirements to restraining an individual to prevent some future
violence with a known specific potential victim (eg. restraining order,
arrest) are lower than the “beyond a reasonable doubt” standard.

The rights of a child to be abused and the rights of a parent to not have
their child be taken away are both very important. It’s my subjective
impression that the “child abused” case is much more common than the “child
taken away where no abuse occurred” case. I see no reason that our laws should
be biased in favor of letting that case occur.

------
aunty_helen
>Motherisk disingenously argues its tests were meant for "clinical" use only

They would've known their tests were being used in a court of law. They
could've stopped this and they didn't. I hope they face consequences.

~~~
raverbashing
I'm not sure they would even need to have known (add a big fat disclaimer to
test results and be done with it)

------
microcolonel
The "family court" system in Ontario is provided by Social Justice Tribunals
Ontario, a system of kangaroo courts set up to handle all the icky things
Ontarians would rather not think about deserving a fair trial.

~~~
eumenides1
Kangaroo court? Based on what?

~~~
woolvalley
Family courts have different, less stringent standards than other parts of the
legal system, even less than the civil part, which is why people could call
them kangaroo courts.

For example, you could appeal a traffic ticket decision, but appealing a
family court decision could be fairly difficult to impossible task in some
jurisdictions.

"As you will learn from reading our interviews, in most states a "temporary"
decision made at a 10-minute hearing is generally unappealable and, from a
practical point of view, permanent. How reliable is the information that a
judge receives? Judge Duncan headlines one section "Lies Usually Go
Unpunished."

[http://www.realworlddivorce.com/Litigation](http://www.realworlddivorce.com/Litigation)

------
mseidl
[https://www.youtube.com/watch?v=ScmJvmzDcG0](https://www.youtube.com/watch?v=ScmJvmzDcG0)

John Oliver did a special about forensic science. The skit at the end is
hilarious.

------
qwerty456127
Even if true, the lone fact that a person had abused a substance (or not
abused but used - taking a substance without a prescription doesn't imply
abusing, there actually are some people capable of conscious and constructive
substance usage) means nothing and should never be used against the person in
any way. Separating families for just this alone is ridiculous. You still are
to prove that the person has developed actual addiction/habit, aggressive
behavior and really harms anyone but him/herself in any way (more than the
sanctions will harm them).

~~~
ransom1538
"You still are to prove that the person has developed actual addiction/habit,
aggressive behavior and really harms anyone but him/herself"

Nope.

If you get a minute, I would watch this video on child protective services:

[https://www.youtube.com/watch?v=9ETfgoSmTHI](https://www.youtube.com/watch?v=9ETfgoSmTHI)

~~~
qwerty456127
I don't mean it actually works this way (one actually has to prove the problem
is real, serious and actually affects the children to remove them), I mean it
should work this way (I've taken a look at the video, the first ⅓ of it)

------
eumenides1
Just as an FYI, Motherisk is a much larger program than just it's drug testing
lab (which is closed as of 2015). It's primary purpose is teratogen (study of
abnormalities of physiological development) information service providing up-
to-date information about the risk and safety of medications and other
exposures during pregnancy and breastfeeding.

Translation: Info distribution on the pharmacology of medications on kids.
I.e. is it safe to take Advil while breastfeeding? This kind of information
changes as the latest studies are done and motherisk has people take studies
and translate them to opinions mothers can use.

I've personally used their info and it's been very useful. Their drug testing
lab scandal is a travesty and should be condemned, but other offerings of
motherisk are scientifically sound. Did you know cough syrup for children is
ineffective and shouldn't be given? It's a new thing apparently.

Off-Topic: Why the hell motherisk decided to run a drug testing lab is beyond
me. It has nothing to do with its original MO.

------
sigstoat
i'm more involved in forensics than, uh, apparently the rest of you.

i suggest that when thinking about the various forensic "sciences", a good way
to think about their reliability is "was this concocted for purely legal
reasons, or is this a bunch of normal scientific/medical techniques bent to
forensic purposes?" that will quickly let you divide things up into "bullshit"
and "not bullshit".

bite marks, tool marks, firearm testing? all concocted to generate
convictions; bullshit. nobody uses any of those things for anything except
legal cases.

forensic toxicology, drug chemistry, dna? normal science bent to forensic
purposes; not bullshit. those were all developed elsewhere, by scientists
doing real science, and then later applied to legal cases.

can the folks doing the testing go bad? yes. the best and only way to address
that is for the defense in cases to bring in their own independent experts,
and do their own testing if enough sample remains.

at least in the US, the best way to make that happen is to fund public
defenders better, which solves all sorts of other problems, too.

------
ransom1538
There are also people rotting in US prisons for 'bite mark analysis'.

[https://theintercept.com/2016/09/07/white-house-report-
concl...](https://theintercept.com/2016/09/07/white-house-report-concludes-
that-bite-mark-analysis-is-junk-science/)

------
epx
Well, what to say about a country that forces men to pay child support to ex-
stepchildren?

------
jamesrgh01
There's also shaken baby syndrome that has come under scrutiny recently:

> “Our main finding is that there’s very low-quality scientific evidence for
> the claim,” says Niels Lynöe, a specialist in general medicine at the
> Karolinska Institute in Stockholm, and leader of the team, whose report was
> published last month. “You can’t use these studies to say that whenever you
> see these changes in the infant brain, the infant has been shaken – it’s not
> possible according to current knowledge.”

[https://www.newscientist.com/article/mg23230994-100-evidence...](https://www.newscientist.com/article/mg23230994-100-evidence-
of-shaken-baby-questioned-by-controversial-study/)

