
Court rules drug dog barely more accurate than a coin flip is good enough - pmiller2
https://www.washingtonpost.com/news/the-watch/wp/2015/08/04/federal-appeals-court-drug-dog-thats-barely-more-accurate-than-a-coin-flip-is-good-enough/
======
harryh
This article, and in particular the title, is horrible and shows no
understanding of the math in play. The facts do not indicate that the dog "is
barely more accurate than a coinflip" if 59% of the indications in question
turn out to actually have drugs.

Out of 1000 samples the dog is positively indicating for 930 of the (93% of
the time) of which 549 are true positives and 381 are false positives. Let's
be generous and say that all 70 remaining are true negatives.

If the dog was subbed out for a coin flip we would have a positive indicator
500 times of which 274 would be true positives and 226 would be false
positives. We would then have 275 false negatives and 225 true negatives.

So subbing in the dog for a coin catches 275 drug carriers at the expense of
an extra 155 false positives. We can argue about whether that is good enough
but comparing it to a coinflip makes no sense.

~~~
rayiner
The article's whole analysis is fucked up because it ignores the prior
probability.

> In U.S. v. Bentley, we see just how damaging the Harris decision really was.
> Lex, the drug dog that searched Bentley’s car, had a 93 percent alert rate.
> That is, when Lex was called to search a car, he alerted 93 percent of the
> time. He was basically a probable cause generator. His success rate was much
> lower, at 59 percent.

In other words, drugs were found in _55%_ of cars in which he was brought
in.[1] The base rate for drug possession in a car is almost certainly a tiny
fraction of that. So between the human judgment to bring in Lex, and Lex's
sniffing, the police are able to find drugs at probabilities that are
multiples of the base rate. In contrast, if you searched cars based on a coin
flip (i.e. randomly), you'd expect to find drugs exactly at the base rate.

[1] Being successful > 50% of the time is pretty much the definition of
probable cause.

~~~
CPLX
Not really. You have conflated a couple things here:

There is some aggregate base rate, which is the percent of a random sample of
motorists that are carrying drugs.

Then there is the rate of those who are actually carrying drugs within the a
group that the police have pulled over, and suspect are carrying drugs, and
have _also_ presumably asserted their rights and refused a consensual search
by the police and _also_ do not have some other attribute (such as smell,
items visible in plain view) that would provide probable cause.

You argue that "the base rate for drug possession in a car is almost certainly
a tiny fraction of that" but in fact the base rate within this much narrower
second group should be extremely high in comparison to the general population.
And the problem is that searching that group, regardless of what you think of
them, is illegal.

In order to make it legal you have to be able to show that you're going to do
something specific that will provide probable cause.

Showing that the dog is anything short of a clearly reliable and accurate
means of determining that a law is being violated means that the dog is not
providing probable cause. A dog that alerts to drugs more 90% of the time when
only 60% or so of the population in question has the drugs, has shown results
that aren't sufficiently correlated to the data it is supposed to discern.

When that is true it is just a tool for violating laws that prohibit searches.

~~~
rayiner
> Then there is the rate of those who are actually carrying drugs within the a
> group that the police have pulled over, and suspect are carrying drugs, and
> have also presumably asserted their rights and refused a consensual search
> by the police and also do not have some other attribute (such as smell,
> items visible in plain view) that would provide probable cause.

Yes, but that hurts your argument rather than helping it. The drug dog does
not, _by itself_ have to clear the probable cause hurdle. Rather, the totality
of the circumstances has to yield probable cause:
[https://en.wikipedia.org/wiki/Illinois_v._Gates](https://en.wikipedia.org/wiki/Illinois_v._Gates).

So say the base rate for carrying drugs is 10% in the population. The police,
based on their suspicions, pull over a population of people where 40% are
carrying illegal drugs. Then, the drug dog comes in, and he's barely better
than a coin flip--60% correct. Out of 100 people stopped, he'll correctly
identify 24 of the 40 carrying drugs, and incorrectly flag 24 of the 60 not
carrying drugs. But overall the process will be right 50% of the time, versus
only a 10% probability if you stopped drivers at random. That's probable
cause.

~~~
SilasX
But I thought PC couldn't count a officer's inarticulable hunch towards that
"totality of the circumstances".

For example, let's say that the dog is not sufficient probable cause. Consider
the cases:

1) drug dog indicates + suspect driving erratically

2) drug dog indicates + "I just have a feeling"

Am I correct that 1) might be PC but 2) could not be, no matter how accurate
that officer turns out to be in practice?

(In case 2, the dog's finding more drugs than the base rate would not validate
its accuracy, as the results are being pre-filtered to have a very different
distribution.)

~~~
rayiner
If they're finding drugs at several times the base rate, the officer's
suspicions probably aren't an "inarticulate hunch."

~~~
SilasX
The term I used was inarticul _able_ , and if you are unable to articulate
something (like to a judge), then that makes it inarticulable -- irrespective
of how otherwise grounded it was.

So rationally, the cop's "spidey sense" is indeed evidence. The question I
posed to you (and whoever's reading) was whether it counts as _legal_
evidence. And it's my understanding that, no matter how good your spidey sense
is, it does not become PC in a legal sense until it is articulable. As Thomas
Moore said in _A Man for All Seasons_ :

"The world must construe according to its wits; this court, according to the
law."

------
rdtsc
Dogs are a 100% proof way to always let the cops search your car. If the court
takes that way, they'll have to invent some other legal loophole.

Handler trains their dog to alert on cues such as saying "Good boy" or
"Find..." and so on. The dog understands what to do. Alerts and gets a treat
or patted on the back.

I am wondering what are statistics of "dog alerted, search performed, but no
drugs found", does anyone keep that? It would be interesting as it would
reveal the scheme pretty well. (They can of course say, well at some point in
the past the car had drugs in it...)

------
fiatmoney
The dog isn't even "as accurate as a coin flip". Given a 93% alert rate,
regardless of the subsequent true positive rate, information gain from an
alert is approximately zero.

Edit: "accurate as a coin flip" is a bad construction, but I wouldn't exactly
expect better from a newspaper. The information gain from a device that
activates randomly 50% of the time, 0% of the time, and 100% of the time are
all the same, which is to say zero.

~~~
gburt
This isn't being fair. I don't really think the post title is fair either.
That 93% isn't comparable to the 50% you're probably thinking for the coin
flip, or even the 0/100% mentioned in your edit.

\---

93% alert rate means:

93% = false positive + true positive

7% = true negative + false negative

\---

We're given "His success rate was much lower, at 59 percent." Assuming that
means that 59% are true positives, we know the false positive rate is 34%.

Without data on how many passengers are carrying drugs, we can't say anything
about the rest of that result. A plausible world where the drug dog may be
valuable is when he unquestionably rules out that 7% of the population that he
doesn't alert on, as in false negative = 0.

I believe the reason this number is so high (59%) is that the officer has
already used his suspicion to determine filter out most people. This makes the
value of the "ruling out" plausibility even more valuable: the drug dog
reduces searches compared to trusting the cop.

Ideally, they'd provide the receiver operating characteristics curve [0] or
the confusion matrix [1].

[0]
[https://en.wikipedia.org/wiki/Receiver_operating_characteris...](https://en.wikipedia.org/wiki/Receiver_operating_characteristic)
[1]
[https://en.wikipedia.org/wiki/Confusion_matrix](https://en.wikipedia.org/wiki/Confusion_matrix)

~~~
nickff
The real question is whether the dog is better than the officer at guessing
who has drugs. If the dog is much better than the officer, it would suggest
that a dog 'alert' should be given more weight than the officer's guess. If
the dog is no better than the officer, the dog is useless.

I say this because it is possible (and indeed likely) that the officer is also
better at guessing who has drugs than a coin toss.

Full Disclosure: I have never used or trafficked contraband, but I do not
support random searches or the use of drug dogs in public without a warrant,
regardless of their effectiveness.

~~~
LordKano
_If the dog is much better than the officer, it would suggest that a dog
'alert' should be given more weight than the officer's guess. If the dog is no
better than the officer, the dog is useless._

A very big problem is that drug dogs will alert when the handler gives it
subtle clues. This is done to get probable cause for a search when there is
nothing more than an officer's hunch or just a desire to hassle a particular
individual.

~~~
mejari
This is something I see claimed a lot. I'd be very interested in any
information you have about this. I find it very unlikely this kind of training
is part of the official training, so I'm imagining it's some kind of
unintentional conditioning? Or is it done on purpose by the officers?

~~~
lukeschlather
This is covered in the article, and it's quite straightforward. The handler
was giving the dog treats for alerting, rather than giving treats for finding
drugs.

The primary trouble I see is that there is no penalty for re-training dogs
like this.

~~~
mejari
The article only discusses that the handler gives the dog treats for alerting,
it doesn't go into the motivations about that, whether they're because the
officers aren't considering the Pavlovian conditioning involved and are
accidentally reinforcing such behavior or if it's intentional.

~~~
kd0amg
Does the motivation matter when we're talking about whether it establishes
sufficient justification for searching someone's car? After a clear flaw is
pointed out, a response of "We didn't _mean to_ ruin the statistical power of
this test" wouldn't do much to restore my confidence in its results.

~~~
mejari
I was specifically responding to this comment

>This is done to get probable cause for a search when there is nothing more
than an officer's hunch or just a desire to hassle a particular individual.

that claims that this is done purposefully. That's what I was wondering about,
not the overall efficacy of drug dogs.

------
mikestew
I am not a professional dog trainer, meaning I do not have a CPDT
certification nor do people pay me. However, I have extensive experience
training dogs at a local dog shelter (minimum two hours/week) in a specific
training atmosphere, and I have a fair amount of experience with "nose works"
which is somewhat related to "drug sniffing dogs" (nose works is where a drug
dog might start, for instance). In other words, if software didn't pay so
darned well I could go jump through the hoops and hang out my dog trainer
shingle. I also hang out with a lot of trainers who are certified and paid for
their work.

With my (albeit weak) credentials out of the way, my quasi-professional
opinion is that drug dogs are the polygraph test of the drug war. I've watched
bomb and drug dogs at work, and much like a polygraph, I think you can read
anything you want into the dog's actions. And unless you're 100% consistent
(and even pros aren't), the dog might mis-read the handler. There are retired
women who devote way too much time to the "nose works" sport with their
dog(s), and in competition even those dogs get it wrong a fair percentage of
the time.

For bomb sniffing and sniffing for bodies after a disaster, that's fine. Dog
says there's explosives, turns out Fido was wrong, no biggie. A few folks got
unnecessarily riled up, but no other harm done. Dog said it found a body, but
there's nothing but rubble? Meh, did some digging we didn't have to do, but
otherwise little harm done (though we could have been digging for a real
person).

Sniffing for drugs? False positives have a greater impact. And just because
the dog hit, and there were drugs, doesn't mean the dog was _right_.
Correlation and causation and all that, I suppose. Anyway, I'm explaining this
poorly, but I hope the gist gets across.

TLDR: I spend a lot of time around dogs trying to teach them things. They
might be more reliable than a polygraph, but I'm not willing to ruin people's
day or lives over the word of a dog.

------
jimrandomh
The article says:

> In 2013, the Supreme Court made things worse in Florida v. Harris. In that
> case, the court unanimously ruled that mere certification of a drug dog was
> enough to establish a presumption that a drug dog is reliable, regardless of
> the reputation of the certifying organization, regardless of whether that
> organization understands and appreciates the importance of training dogs to
> ignore their handlers’ suspicions, and regardless of the dog’s performance
> in the real world.

But the decision in Florida v. Harris says:

> A defendant, however, must have an opportunity to challenge such evidence of
> a dog's reliability, whether by cross-examining the testifying officer or by
> introducing his own fact or expert witnesses. The defendant, for example,
> may contest the adequacy of a certification or training program, perhaps
> asserting that its standards are too lax or its methods faulty.

...

> In short, a probable-cause hearing focusing on a dog's alert should proceed
> much like any other. The court should allow the parties to make their best
> case, consistent with the usual rules of criminal procedure. And the court
> should then evaluate the proffered evidence to decide what all the
> circumstances demonstrate. If the State has produced proof from controlled [
> __ _7] settings that a dog performs reliably in detecting drugs, and the
> defendant has not contested that showing, then the court should find
> probable cause. If, in contrast, the defendant has challenged the State 's
> case (by disputing the reliability of the dog overall or of a particular
> alert), then the court should weigh the competing evidence.

...

> Harris, as also noted above, declined to challenge in the trial court any
> aspect of Aldo's training. See supra, at 3. To be sure, Harris's briefs in
> this Court raise questions about that training's adequacy — for example,
> whether the programs simulated sufficiently diverse environments and whether
> they used enough blind testing (in which the handler does not know the
> location of drugs and so cannot cue the dog). See Brief for Respondent
> 57-58. Similarly, Harris here queries just how well Aldo performed in
> controlled testing. See id., at 58. But Harris never voiced those doubts in
> the trial court, and cannot do so for the first time here.

So what that supreme court case _actually* says is that the validity of a
drug-sniffing dog as probable cause for a search can be challenged in court,
but the challenge must be made in trial court rather than on appeal.

~~~
piker
The article says that certification establishes a _presumption_ of
reliability. That means that certification can shift the burden of proof to
the defendant. In that case, Bentley would have to prove to the jury that the
dog is unreliable. It's correct.

It seems Bentley's trial counsel erred by not presenting evidence that the dog
was unreliable, and hence did not meet Bentley's burden.

[Edit: clarity]

------
ericabiz
Not the first time, nor is this the worst offense perpetuated by drug dogs. A
town in New Mexico used one drug dog with an expired certification, "Leo", in
order to justify invasive anal probes on two men--neither of whom had drugs.
To add insult to injury, both men were later billed by the hospitals for this
non-consenting search.

Timothy Young, 2012: [http://krqe.com/2014/07/28/nm-driver-tells-story-of-
sheriffs...](http://krqe.com/2014/07/28/nm-driver-tells-story-of-sheriffs-
invasive-body-search/)

David Eckert, 2013:
[http://www.dailykos.com/story/2013/11/05/1253274/-Absolutely...](http://www.dailykos.com/story/2013/11/05/1253274/-Absolutely-
unimaginable-this-could-happen-in-America)

Both cases were eventually settled, leading to Young receiving a $900,000
settlement and Eckert receiving $1.6 million:
[http://articles.latimes.com/2014/jan/16/nation/la-na-nn-
new-...](http://articles.latimes.com/2014/jan/16/nation/la-na-nn-new-mexico-
police-search-20140116)

These two cases are an absolutely horrifying violation of human rights and I'm
really glad to see drug dogs being taken to task as part of what led to both
of these men being violated.

------
joesmo
This is hardly news. Until we put in real consequences for both police and
judges who intentionally misbehave or oftentimes fail at their job, none of
this will change.

Here's an idea: penalize the cop for every false positive. Docking pay seems
to be the best idea here. I guarantee drug dogs' accuracies will go up
nationwide within weeks (or however long it takes to retrain dogs and their
masters). Every 5 or 10 false positives, dock the cop extra pay or reprimand
him. Problem solved.

Here's another idea: hold judges accountable for police actions in the
courtroom. If the police lie on the stand, and it's shown that the judge had
"probable cause" the judge should be fired. The collusion problem between
police and judges, what the article is referring to tough maybe not
explicitly, will go away as well. Problem solved.

tl;dr: The only way to keep the police in check is to make them suffer real
consequences for fucking up. This will never happen in the US because the
citizens here have no idea about the atrocities the police are comitting and
the police prefer to keep it that way.

------
piker
To be clear, what this _appellate court_ and the _trial court_ were asked to
consider are different questions.

Trial courts find facts. For example, "this guy shot that guy". Mostly these
are considered by a jury, but in preliminary things such as Bentley's probable
cause hearing, the trial court judge can act as a fact-finder.

Appellate courts interpret laws. For example, strip searching a middle school
student for pain medication is an "unreasonble" search under the Fourth
Amendment.

Lex is demonstrably inaccurate. Bentley argued in _appellate court_ that Lex's
alert is insufficient to support probable cause under any circumstances--i.e.,
a "matter of law". Bentley did not argue in _trial court_ that Lex's alert was
unreliable in his situation--i.e., a "matter of fact".

The ruling simply reject's Bentley's argument, but preserves future
defendant's rights to make the latter argument.

------
daenz
AdaBoost is an algorithm for taking weak classifiers (performing only slightly
better than random chance) and boosting/weighing them such that together they
are a strong classifier.
[https://en.wikipedia.org/wiki/AdaBoost](https://en.wikipedia.org/wiki/AdaBoost)

To play the devil's advocate, if the dog alerting to drugs is only one of the
classifiers in play (others being the suspects look/act under the influence),
and if it's weighed correctly in an officer's mind, it is a key part of the
strong classifier.

~~~
shkkmo
But if you are going to use that reasoning, then you need to remove the dog
from the equation and only judge the officers on their actual success rate,
utilizing any method at their disposal. What should this rate be? 50%? Any
officer who falls below that rate should be forced to get approval from
another officer with a higher success rate? They can then choose to use dogs
to supplement certain officer's accuracy without giving them incentive to
train the dogs to be "probable cause generators".

Using a weak classifier to serve as the basis for getting around a
constitutional right is pretty messed up.

~~~
daenz
> Using a weak classifier to serve as the basis for getting around a
> constitutional right is pretty messed up.

I was playing the devil's advocate, in case you missed that part, and my point
was that a classifier being weak does not necessarily make it bad, as
evidenced by AdaBoost.

------
ryguytilidie
At that point why not just have officers approach a suspects car with a box
with a button and two lights, press the button and if the light turns green,
there is "reasonable suspicion"? Just have the light turn green everytime and
they can always search you and F the 4th Amendment.

------
mangeletti
[https://en.wikipedia.org/wiki/Base_rate_fallacy](https://en.wikipedia.org/wiki/Base_rate_fallacy)

------
deutronium
I wonder if we could use bees instead then, as there seemed to be some
interesting research on training them to detect drugs and presumably they
wouldn't be biased by human behaviour especially.

[https://en.wikipedia.org/wiki/Hymenoptera_training](https://en.wikipedia.org/wiki/Hymenoptera_training)

Alternatively are there any portable mass spectrometers now?

~~~
DanBC
Another thing Bees were being used to detect was explosive, but researchers
discovered that bees detected the impurities in explosives. If you gave the
bees pure samples of explosive they didn't do very well.

Professor Jackie Akhavan (Professor of Explosive Chemistry) mentioned this on
the BBC programme "The Life Scientific", where a scientist talks to other
scientists about their work. It's for a general audience so nothing
particularly technical, but still interesting.

~~~
deutronium
Cheers, I'll see if I can grab that.

~~~
DanBC
Sorry! I forget to include the link. Here's that specific episode:
[http://www.bbc.co.uk/programmes/b04jk360](http://www.bbc.co.uk/programmes/b04jk360)

> Jackie Akhavan, Professor of Explosive Chemistry, tells Jim al-Khalili all
> about the science of explosives. She explains exactly what explosives are
> and how to make them safer to handle.

> She started by working on how to make fireworks safer and has been involved
> in research with bees to see whether they can be used smell different types
> of explosives. Her current project involves testing the rocket fuel that
> will be used in Bloodhound, the British designed and built supersonic car
> that aims to reach a speed of 1,000mph.

> Her work involves finding out how to best detect explosives in airports and
> elsewhere, teaching security professionals how to differentiate between
> false alarms and the real thing. She also works on explosives used in
> warfare and discusses the ethical issues involved.

------
njharman
All this doesn't matter until the root cause is fixed. Police will find a way.
The root cause* is mentioned in the article, property seizure laws. As long as
there is monetary incentive Police will be __compelled __to violate the
constitution.

* The real root cause is the criminalization of recreational drugs. But ending Prohibition is basically a non-starter.

------
Animats
Hm. What would be the probable cause level required for a search of a
politician's records, phone calls, and bank accounts for evidence of bribery.
We could probably use big data to establish patterns which are indicative of
bribe-taking that would be demonstrably more reliable than 50%.

------
shkkmo
This headline is pretty terrible.

Keep in mind when looking at these statics that the number for this dog is not
based on a random sample. Therefor it is impossible to actually gage the dog's
accuracy from these numbers, since we don't know the ration of false:true
negatives, only the number of false:true positives. The ratios could be
identical, which would make the dog's alerting behavior random. There could
even be a higher ratio of false:true negatives, which would make the dog worse
than random.

The only meaningful number is the 93% ratio which basically means that the dog
is very close to being a 'probably cause generator' as mentioned in the
article.

------
javajosh
So drug dogs become a kind of police "search anything" power totem.

(This as opposed to the "kill anything" power totem that is the cops brain and
tongue via _I thought he had a gun_ or _I thought my life was in danger_ ; or
the "break into people's home and terrorize them" power totem of the knock-
less warrant; or the "track anyone, listen to anyone" power totem of the
Patriot Act, etc. at the federal level.)

~~~
merpnderp
Before the US revolutionary war they were called "General Warrants" and
allowed British soldiers to search and seize as they pleased.

------
nosuchthing
Merits and repercussions of drug prohibition aside, the fact that the dogs can
signal a false positive on command seems unsettling.

------
cle
Not only is this an important civil rights issue, but it's a safety issue too.
If the dog is used as a proxy for the bias/hunch of the officer, rather than
objectively, then adversaries can use that to their advantage, e.g. to divert
resources away from a real threat.

------
jacobheller
Here's the full text of the case for those interested:
[https://casetext.com/case/united-states-v-
bentley-13](https://casetext.com/case/united-states-v-bentley-13)

------
Zikes
Statistics should be a high school course.

------
doomsells
Upvoted because drugs are good.

