
Why facial recognition has led to false arrests - dnetesn
http://nautil.us/issue/89/the-dark-side/the-bias-in-the-machine
======
pbhjpbhj
>But the next steps, where investigators first confirm the match, then seek
more evidence for an arrest, were poorly done and Williams was brought in. He
had to spend 30 hours in jail and post a $1,000 bond before he was freed. //

Seems more like "why lack of rigour in policing has lead to false arrests".

Facial recognition is evidence, not proof. They clearly didn't view the video
and compare it to the person's photo if during post-arrest interview they
could immediately tell it was the wrong person?

Do they also just arrest people based on other 'witness' reports without any
effort towards corroboration?

~~~
humanistbot
This is what social psychologists call "automation bias," which has been
empirically observed in all kinds of situations from all kinds of people:
[https://en.wikipedia.org/wiki/Automation_bias](https://en.wikipedia.org/wiki/Automation_bias)

When we are told that a prediction is from an automated system, most people
are more likely to believe it and not subject it to the same skepticism and
critical thinking that we do to predictions from humans. Even if there is a
human-in-the-loop system that requires a human to confirm it, that's often
just laundering the machine's prediction through a human. When we are primed
by being told this is what The System predicted, we're more likely to override
our normal processes. This is especially the case when The System has worked
99% of the time --- which is still a lot of false positives.

Although this can also be the case with priming from predictions or judgments
by other humans. The Asch conformity experiments involved planting fake test
subjects in a group activity where subjects had to give answers to a very
obvious perception question, and the real test subject could hear everyone
else's answers. The fake subjects went first and all gave the blatantly wrong
answer. The percent of people who conform varies with how you run the
experiment, but typically a majority of the subjects end up giving the obvious
wrong answer that everyone else did:
[https://en.wikipedia.org/wiki/Asch_conformity_experiments](https://en.wikipedia.org/wiki/Asch_conformity_experiments)

~~~
pbhjpbhj
I can see it happening when checking takes a long time - but they're probably
man-days into an arrest at the point they're sitting in an interview room
post-arrest. Against, the LEO looking at their tablet screen prior to
arresting the guy, seeing he looks different and not arresting him?

Are the LEOs getting bonuses for raw number-of-arrests?

~~~
monocasa
It depends on the department, but raw arrest counts are absolutely part of
individual officer performance metrics, with low arrest count officers being
"pressured to find something" at the end of the month.

[https://www.nbcnewyork.com/news/local/nypd-officers-
arrest-q...](https://www.nbcnewyork.com/news/local/nypd-officers-arrest-quota-
exclusive-interview-pressure-numbers/748594/)

~~~
BelleOfTheBall
Which is absolutely a big issue with our policing system. Most of the time
they just arrest easy targets like sex workers or people who could be charged
with 'public misconduct' or something like that.

------
tyingq
_" These contain some criminal mugshots, but the bulk of the images comes from
non-criminal sources such as passports and state driver’s license
compilations; that is, the databases mostly expose ordinary, generally
innocent citizens to criminal investigation."_

Worth noting that a mugshot does not imply criminality. As the lead off in the
article shows, lots of innocent people get arrested and thus, have mugshots on
file.

~~~
boomboomsubban
Fairly banal crimes can also lead to mugshots. I had to have one taken over a
parking violation.

------
throwaway4747l
I mean, if your algorithm is accurate 99.9% of the time, and you're using it
24/7 in a country with millions of inhabitants, you should expect tens of
thousands of false positives _every day_. This is why some wet blankets have
been strangely skeptical of delegating all responsability to The Algorithm.

~~~
sevencolors
I love how you just handwave away the possibility of destroying someone's life
with a false positive. This is an area like automated driving where the error
rate needs to be 0 if it's going to be relied upon as truth.

~~~
throwaway4747l
I'm actually agreeing with you, perhaps the sarcastic tone didn't go through.

------
renewiltord
Most people are incompetent. The tax lawyer down the street, the pharmacist,
the dental assistant. It's just that most people aren't in a position to hurt
others because there are checks:

\- the recipient checks the pharmacist's work

\- the dentist checks his assistant's work

\- the state allows the tax lawyer some leeway

16% of Americans have an IQ less than 85. 50% have an IQ less than 100. If you
build a tool that requires a person to have average IQ, it will fail with half
of all users.

That doesn't mean it's a bad idea: after all, you only need to convince the
buyer, not the user.

------
grumio
What's the incentive to reduce false arrests? Are the arrestors' jobs at
stake? Is their wallet at stake?

Why put in the extra effort if there are no consequences for being wrong?

Most of the time it seems the worst-case scenario for the perpetrator of the
false arrest is being relegated to desk-duty or a paid suspension. If there's
a settlement it's the tax-payers that foot the bill. Until that changes,
there's basically no reason for them to make the effort to reduce false
arrests. This isn't a problem that will be solved with recognition algorithms
getting a few percentage points more accurate every year. It will require
actual reform.

------
NPMaxwell
This article appears in my HN feed with an article about throwing away React.
The React article explains that having code that prevents engineers from doing
whatever they want to is essential. If we can't trust engineers coding, why do
we trust police arresting? Restraints are needed.

------
oppositelock
The best algorithm that NIST tested had a 0.01% false positive rate and it was
uniform across races.

Consider an airport like Heathrow, which uses facial recognition widely.
220,000 people travel through heathrow per day. With a 0.01% false positive
rate, you are falsely flagging 22 people per day.

You can not rely on facial recognition to jail people.

------
supportlocal4h
Facial recognition seems a bit like shoe print id. A valuable clue, but not
enough to ruin somebody's life over. That is so obvious that anecdotes about
false arrest seem like obvious human incompetence.

~~~
raxxorrax
I don't think it is valuable. Especially with falling crime rates, developing
such systems is basically bullshit job in a bullshit industry.

Exaggerated, but I haven't really seen neither need nor success of any facreg
system.

Sure, security agencies might be able to supply some cases when it helped to
catch a criminal, but the side effects suggest it isn't even slightly worth
it.

~~~
tetromino_
Falling crime rates? For some categories of crime, sure, but e.g. _murder
rates_ this year have risen sharply in major US cities.

[https://nyti.ms/3gyoVqd](https://nyti.ms/3gyoVqd) \- "At least 336 people
have been murdered in Chicago through July 2, according to the Chicago Police
Department; because murders typically increase in the summer, the city is on
track to match the 778 deaths in 2016, its deadliest year since the mid-1990s.
<...> Chicago is not alone. Before the coronavirus hit, homicides were
escalating nationwide in early 2020, and although the lockdown brought a
pause, they began rising again as the stay-at-home measures were lifted. A
national study showed that homicide rates fell in 39 of 64 major cities during
April and began creeping up in May."

[https://www.nytimes.com/2020/08/04/nyregion/nyc-shootings-
co...](https://www.nytimes.com/2020/08/04/nyregion/nyc-shootings-
coronavirus.html) \- "On Sunday, another 19 people were shot in New York City,
one fatally. Through the first seven months of this year, shootings were up 72
percent over the same period last year and murders rose 30 percent, even as
reports of other violent crimes like rape, assault and robbery fell."

~~~
raxxorrax
I think these problems might have to do with a current conflict within the US
society that needs to be talked about and surveillance or the lack of it is
neither the reason nor the solution.

~~~
tetromino_
I don't understand why you think it's not a solution.

Witnesses refusing to talk to cops is a problem. Given the "current conflict",
witnesses may be even less willing to talk than usual. And camera surveillance
with a _good_ facial recognition system (i.e. one that has been trained and
certified to recognize faces of all ages, ethnicities, and phenotypes, not
just white college students) would certainly seem like a solution that could
help track down murderers.

~~~
bkor
UK has loads of camera surveillance, it didn't really help with the crime
rates. Seems better to avoid people getting into crimes. Meaning: reduce the
difference between super rich and very poor, ensure people have the ability to
have a good life (basically: afford to start a family), etc.

Anyway, that's what helped in NL.

------
bluesign
Problem here is for arrest they claim to be using not facial recognition but
witness identification. In theory it sounds like good.

But the problem is when you identify some similar face to suspect then show
this with some random faces, you are actually giving a multiple choice to
witness, which is biased for the match you have found.

------
cultus
I'm not trying to be snarky here, but can anybody point to an example of
machine learning used in law enforcement that hasn't lead to inequitable
outcomes?

~~~
hammock
Equally non snarky, what is something in law enforcement that has led to
equitable outcomes?

~~~
glenda
The more I look into the history and tactics of modern policing the more this
becomes apparent.

Police are willing to use broken machine learning technology because it “works
for them” - they don’t necessarily care about accuracy they just want someone
locked up.

~~~
vorpalhex
The police are fundamentally a low paid blue collar labor force with union
protections. They are not your college professor or the protagonist of a
murder mystery, but more like a plumber or construction worker.

The moment you understand that, it's not a mystery that they sort of blindly
accept tools that seem to work without deep introspection. They are not
equipped to be a data analytics shop, they are equipped to fill out paperwork
and sometimes get into tussles.

~~~
bkor
> The police are fundamentally a low paid blue collar labor force with union
> protections

Please keep in mind that this is not true worldwide. In various countries
you're only admitted after 1.5-2 years of police-specific education. You might
still not be a police officer if they seem you unfit.

Also, you don't seem to value plumbers or construction workers. A colleague
used to be a construction worker. It's very interesting to hear the knowledge
he has on that. Plus pretty damn handy for understanding how to improve your
house. Just because he did a lot of work with his hands doesn't mean he isn't
damn bright.

~~~
vorpalhex
At no point did I indicate I don't value blue collar workers such as plumbers
or construction workers, quite the opposite. At most I said the police were
low paid blue collar workers - and in general I do believe most blue collar
workers are underpaid. For what it's worth, I do also value our police forces,
even with their flaws. But a blue collar profession has a different set of
knowledge handling than a white collar one - it's not that one is better or
worse, but they are intrinsically different ways of knowing.

> Please keep in mind that this is not true worldwide

You're correct and my comment is US centric. What I find interesting is that
some US cities have seemingly moved farther from a more educated police force,
removing requirements around having college degrees and reducing trainings.

------
BurningFrog
Every method of finding criminals result in innocent people being arrested
from time to time. That's the nature of arrests.

Maybe there is something uniquely bad about facial recognition, but to show
that, compare it to other techniques, not to a perfection standard.

~~~
Jtsummers
There are several issues, though.

1\. As discussed in the article, due to training sets or methodology, these
systems are poor at distinguishing non-Caucasian faces. Or at least poor
relative to a human looking at the image and the person together.

2\. Police (though not just them) have an unfounded trust in the machines.

3\. Police take this unfounded trust and push through _arrests_ before they
realistically should (like in the opening story of the article).

4\. People arrested for felonies are fucked in the US. It can take years to
get your record expunged. Until then your name shows up in lots of searches,
you can lose licenses to conduct some kinds of business (or be unable to
obtain them). (Expunging can also be very expensive.)

You may not mind innocent people being caught up in the dragnet, but as a
society we probably should care a lot more than we do.

~~~
BurningFrog
Yeah, those issues may be big problems. Or they may not.

My point is that the way to know is to compare this tool with others and weigh
pros and cons.

------
kortex
> Williams’ case is a signal to stop the ad hoc adoption of facial recognition
> before an injustice occurs that cannot be undone.”

Arguably, Williams' case _is that injustice_. It should not have happened in
the first place (although anyone in the field could have told you it would be
inevitable).

Clearly the machine-human interface is insufficent. A big honking warning is
still insufficient to overcome automation bias (see other top level comment).

Perhaps the "lineup" functionality needs to be baked in. The system won't
divulge details until a(n) operator(s) selects hits from a pool.

------
Dominisi
I worked in a Law Enforcment office that used Facial Recognition. Every single
time we would give officers a potential match, it came with a disclaimer that
it was a POTENTIAL match. It was up to the detectives and officers to actually
do their job and find other pillars of information.

This isn't a failure of facial recognition, its a failure of the stupidest
people I went to high school with thinking its some sort of perfect system.

~~~
teachrdan
If the software vendor knows the end user will not use their product as
intended, at what point is the vendor complicit in the impacts of that misuse?

------
GuB-42
The real question is: overall did face recognition lead to a higher proportion
of false arrests, and even more importantly false convictions.

Clearly, facial recognition can make mistakes that lead to false arrests. Of
course, it is important to minimize that number, that alone doesn't tell us if
facial recognition is justified.

The thing is:

\- An arrest is not a conviction, it is important to know (if possible) how
much of these facial recognition false arrests led to false convictions. I
suspect not a lot, because judges and attorneys are still human.

\- How much false arrests did facial recognition prevent? It can happen if the
system correctly identifies the culprit before an innocent is arrested. It can
also be used to cross-check false/unreliable testimony.

Note: I am not talking about the privacy implication of facial recognition,
only its effect on criminal investigations.

------
jakobmartz3
Those police should have been required to make sure he was a match. Feel like
it was mostly due to racism and thats fckin horrible. Then they blame it on
the computer. like yeah but its also their fault for not double checking.

------
mytailorisrich
The issue highlighted in this article is that it seems that police deemed
facial recognition as reliable as DNA so that if the computer says it's that
person then they are already guilty.

This is a simple issue that could be easily solved by making sure that facial
recognition should be treated as a very good lead but not as a smoking gun, so
that the police approaches "suspects" more neutrally before deciding whether
to arrest them.

~~~
teddyh
So how do you get police to behave that way?

------
sdsvsdgggggg
As if humans have never misidentified people or arrested the wrong person?

I find these articles very silly.

Maybe the photo on the drivers license was not good enough, so only in the
real world could the police see that it was not the same person.

In any case, why not scan the database automatically (the evil, evil facial
recognition), and then double check by humans?

Even when humans police, I think it is always just about probabilities. Then
they follow up (ideally) and try to drive the probability of being correct
higher.

Obviously nobody should be tried automatically without any recourse.

~~~
Shared404
> why not scan the database automatically (the evil, evil facial recognition),
> and then double check by humans?

Because people tend to think "The computer said this was right, therefore it
is unequivocally the correct answer." This is obviously incorrect, but is how
many laypeople view it, even if instructed otherwise.

~~~
tehwebguy
In fact that's _exactly_ what happened. The police realized it was the wrong
guy and still brought him in, he still lose 30 hours and had to post bail.

I cant even imagine if two morons showed up to my house, saw that I was the
wrong person, and then arrested me in front of my family anyway.

~~~
sdsvsdgggggg
But then it can't possibly be the "people just believe computers" effect,
because the article explicitly states the police realized the computer was
wrong.

~~~
Shared404
Then in this case it's the same as phishing emails. People know it's wrong,
and yet still follow along anyways.

This is not necessarily a forgone conclusion, but occurs to often to dismiss.

~~~
sdsvsdgggggg
Nothing indicates that it is specific to algorithmic facial recognition.
Actual humans make mistakes identifying suspects all the time. At the very
least, the article should address that and compare the reactions, to prove
people are more likely to act despite knowing better when the id was made by a
computer.

~~~
Shared404
A reasonable point.

------
deepstack
On hacker news, can we please submit https link instead of http. An S goes a
long way in this hostile internet environment now days.

