
Amazon’s Face Recognition Falsely Matched 28 Members of Congress with Mugshots - okket
https://www.aclu.org/blog/privacy-technology/surveillance-technologies/amazons-face-recognition-falsely-matched-28
======
alphabettsy
Some commenters here mentioned they may have setup the test incorrectly and
that may be, but I think the problem this highlights most is that technology
used improperly, especially by law enforcement can have major ramifications
and consequences. Is the contractor that makes software for your local
department going to follow best practices and have the algorithm audited by
experts? Will they release the code? Those are real considerations for any
program that has the potential to help ruin someone’s life.

It also possibly highlights the fact that algorithms are not immune to bias
when they are designed by humans. Obviously I don’t think this bias is
intentional, but so much of it isn’t and happens anyway.

~~~
dfundako
I think a fun thought exercise is finding the fine line between tech and
guns/alcohol/cars. You cannot sue a gun/car/alcohol manufacturer if their
product is used to injure someone because it functioned as designed but was
used maliciously. How does that legal precedent work when extrapolated to tech
and something like facial recognition? If it worked exactly as designed and we
know it has a margin of error (or can be used improperly and have disastrous
results, like a car or gun), could Amazon or a tech administering it be liable
for someone falsely imprisoned?

~~~
guitarbill
It's still ethically questionable. In fact I'm struggling to come up with a
better example than facial recognition tech (except other mass surveillance).
Maybe cutting corners while developing driverless cars that results in the
death of a pedestrian.

Almost every engineering discipline has a code of ethics [0][1][2][3]. It's
time software "engineering" grew up and did the same.

I rarely see ethics mentioned on HN, and granted, people's view differ. But
it's weird we're not having that conversation at all.

[0] [https://www.raeng.org.uk/policy/engineering-
ethics/ethics](https://www.raeng.org.uk/policy/engineering-ethics/ethics)

[1]
[https://www.ieee.org/about/corporate/governance/p7-8.html](https://www.ieee.org/about/corporate/governance/p7-8.html)

[2] [https://www.nspe.org/resources/ethics/code-
ethics](https://www.nspe.org/resources/ethics/code-ethics)

[3] [http://www.asce.org/code-of-ethics/](http://www.asce.org/code-of-ethics/)

~~~
Kalium
You're right! This is a critically important conversation that we _absolutely
need to have_ within our profession. It's very often ignored and there's no
support system for people who take ethical stands.

So. Let's talk about ethics. I, personally, subscribe to the ACM code of
ethics.

I think Rekognition, as built and presented, falls fully within that strict
ethical code. It can be put to uses that are unethical, but that does not fall
upon the people who made it. Certainly, an engineer creating a system such as
the one the ACLU created would be acting unethically.

------
joemaller1
Things I want to know:

\- Show us the side-by-side images of the false-positives. Are the matches
plausible?

\- What is the demographic distribution of the mugshot database? If the data
is disproportionately biased, then that bias would be reflected in the false-
positives. A casual skimming of some mughot websites shows a potentially
significant racial bias.

~~~
sudhirj
Whether the matches are plausible is completely besides the point. This isn't
an evaluation of Amazon's recognition quality, it's a demonstration that the
system is fallible and notes on the consequences.

Same thing re data setup. Not the point again. Even if law enforcement hires
the world's best computer vision experts (fat chance, it'll go to cheapest
contractor) this is a note that there can be mistakes that get people killed.

~~~
petermcneeley
What are the consequences?? How is this any different than the suspect
sketches? This is a far worse situation and still quite minor
[https://www.cbc.ca/news/canada/british-columbia/bc-mother-
ch...](https://www.cbc.ca/news/canada/british-columbia/bc-mother-child-no-fly-
list-1.4390544)

~~~
sudhirj
What does the link have to do with sketches? It's about name similarity on the
no fly list.

Regarding sketches, they're actually interesting because they force a human to
exercise judgement, being ambiguous by nature. This is fine - any law
enforcement officer stopping someone who looks like a sketch is going to give
them the benefit of the doubt. On the other hand, if a computer tells an
officer a person ahead is an 89% match with a known killer, the conversation
will start with guns drawn, and that will greatly increase the likelyhood of
things going south.

You really don't want to be scanned by a trigger happy cop having a bad day.
God knows who you happen to look like from that angle.

~~~
zo1
" _[..]conversation will start with guns drawn, and that will greatly increase
the likelyhood of things going south._ "

That is where the two sides of this argument diverge. First: That things "will
start with guns drawn". And Second: That things are more likely to "go south"
for an innocent individual incorrectly blamed by this.

~~~
stevew20
Police have a poor history in the US of exercising trigger discipline. A fair
amount of this is due to overly aggressive training that is designed to keep
the officers as safe as possible, sacrificing safety of anyone perceived as a
threat (real or not), but another large portion is caused by the culture of
dominance and entitlement that is endemic to our law enforcement.

If police engage a target with a weapon already drawn, there is a long list of
things that person can do that will get them shot, and a very short (and often
unclear) list of actions that will keep them safe.

Source: 10 years PMO in the Marine Corps, 1 month SFPD (then quit because
their training was so shitty)

------
twothamendment
They have Montana Rep. Greg Gianforte in the list of false positives. Did they
call it a false positive because they are sure he wasn't included in the
25,000 mugs they loaded? His mug is out there.

[https://www.google.com/amp/s/amp.usatoday.com/amp/756343001](https://www.google.com/amp/s/amp.usatoday.com/amp/756343001)

------
cthalupa
[https://www.theverge.com/2018/7/26/17615634/amazon-
rekogniti...](https://www.theverge.com/2018/7/26/17615634/amazon-rekognition-
aclu-mug-shot-congress-facial-recognition)

>Reached by The Verge, an Amazon spokesperson attributed the results to poor
calibration. The ACLU’s tests were performed using Rekognition’s default
confidence threshold of 80 percent — but Amazon says it recommends at least a
95 percent threshold for law enforcement applications where a false ID might
have more significant consequences.

Presented without real comment on my part.

~~~
fluxsauce
> but Amazon says it recommends at least a 95 percent threshold for law
> enforcement applications where a false ID might have more significant
> consequences.

That's not in
[https://aws.amazon.com/rekognition/faqs/](https://aws.amazon.com/rekognition/faqs/)
\- while that threshold may minimize false positives, I'm curious if it was in
any documentation that the ACLU saw. Or, to the bigger point, is it in any
public documentation?

~~~
dfundako
I would be shocked if LE started using this tech, imprisoned a bunch of
people, and used the excuse "Well, we followed the FAQS." I would imagine they
would work very closely with Amazon and run a ton of calibration. For the ACLU
to say that we used the default settings and it doesn't work correctly is
disingenuous.

~~~
delecti
I would not be shocked at all if some LE somewhere used the default settings.
It seems absurd to assume otherwise. It's not as though LE has a great track
record.

[https://en.wikipedia.org/wiki/No_Fly_List#False_positives](https://en.wikipedia.org/wiki/No_Fly_List#False_positives)

------
AdmiralAsshat
I don't see why this is a bad thing.

Falsely arrest 28 members of Congress due to poor face recognition, and the
problem of facial recognition in law enforcement is resolved the next day.

~~~
dragontamer
Congress doesn't stick up for each other. As soon as those 28-Congressmen are
gone, they'll start passing legislation that those 28-people didn't want.

Congress is a battleground. Not a club or secret society. You send
representitives from your state to try and get a slice of the pie, and I send
my reps from my area to try and get a slice for me.

------
andrewguenther
> People of color were disproportionately falsely matched in our test.

They are trying to make this racially charged without giving enough
information to verify their claims. If you use a dataset of mugshots, that's
statistically going to have more data on people of color. If you have more
data on people of color, it is more likely to match people of color. Claiming
the algorithm is racist because your data is racist is inflammatory bullshit.

~~~
rrego
Pretty lousy argument. Firstly, the majority of inmates in the US are white
(58%). I'd assume mug shot stats are similar, or at the very least not mostly
black people.

Moreover, if a algorithm enforces bias to the detriment of inoccents, it's a
bad algorithm

~~~
andrewguenther
> Firstly, the majority of inmates in the US are white (58%). I'd assume mug
> shot stats are similar, or at the very least not mostly black people.

I actually did not know this. Thank you. I had conflated incarceration rate
with inmate population. If anyone is curious for a source, see here:
[https://www.bop.gov/about/statistics/statistics_inmate_race....](https://www.bop.gov/about/statistics/statistics_inmate_race.jsp)

> Moreover, if a algorithm enforces bias to the detriment of inoccents, it's a
> bad algorithm

I agree with this 100%. But they haven't provided enough information about
their dataset to make this conclusion. If they provided enough information for
someone to independently analyze the data and reproduce the experiment, I
would flip immediately.

------
mnx
I don't see any mention of them verifying those members of congress are not in
fact the same people as the mugshots (/s)

~~~
yjftsjthsd-h
I admit that was my first interpretation before I remembered the issues with
accuracy.

------
ggambetta
Maybe their face recognition tech can see the future?

~~~
sheeshkebab
Or past

------
reaperducer
Well, it's not like police are using Amazon's face recognition technology in
dragnets.

Oh, wait:
[https://www.usatoday.com/story/tech/talkingtech/2018/06/29/c...](https://www.usatoday.com/story/tech/talkingtech/2018/06/29/capital-
gazette-gunman-identified-using-facial-recognition-technology/744344002/)

------
stretchwithme
People have similar faces. Who knew?

The technology is not perfect. And should never be used as evidence of a
crime, just as an indication that two images might be of the same person and
to have humans look at them.

And then those humans will also make mistakes. People do look alike. And there
are twins.

I think a jury should require more evidence than just similar appearance. But
such a match is a strong indication of where to look.

~~~
see-dante
I don't think that's the point -- the problem isn't that "oh there are other
processes that [...]," it's that this wave of technology is misused, is bias,
and care must be taken when any of the signals we create.

This is especially important as automation enters the equation.

------
lbriner
The scary part is that people believe "science" because it is created by
people who they think are "clever" and thefore not likely to be wrong.

How many people have been falsely convicted because "DNA"? 1B-1 odds of a
match, "and sir, yet you claim you were not even in the area?".

There should be a way of recognising the parts of the science that are
basically correct and the parts that are either less reliable, open to bias or
could simply be broken due to incorrect process/mistake in the lab.

~~~
ceejayoz
This is a long, horrific read about exactly that, resulting in the execution
of an almost certainly innocent man.

[https://www.newyorker.com/magazine/2009/09/07/trial-by-
fire](https://www.newyorker.com/magazine/2009/09/07/trial-by-fire)

> Many arson investigators, it turned out, had only a high-school education.
> In most states, in order to be certified, investigators had to take a forty-
> hour course on fire investigation, and pass a written exam. Often, the bulk
> of an investigator’s training came on the job, learning from “old-timers” in
> the field, who passed down a body of wisdom about the telltale signs of
> arson, even though a study in 1977 warned that there was nothing in “the
> scientific literature to substantiate their validity.”

> In 1992, the National Fire Protection Association, which promotes fire
> prevention and safety, published its first scientifically based guidelines
> to arson investigation. Still, many arson investigators believed that what
> they did was more an art than a science—a blend of experience and intuition.
> In 1997, the International Association of Arson Investigators filed a legal
> brief arguing that arson sleuths should not be bound by a 1993 Supreme Court
> decision requiring experts who testified at trials to adhere to the
> scientific method. What arson sleuths did, the brief claimed, was “less
> scientific.” By 2000, after the courts had rejected such claims, arson
> investigators increasingly recognized the scientific method, but there
> remained great variance in the field, with many practitioners still relying
> on the unverified techniques that had been used for generations. “People
> investigated fire largely with a flat-earth approach,” Hurst told me. “It
> looks like arson—therefore, it’s arson.” He went on, “My view is you have to
> have a scientific basis. Otherwise, it’s no different than witch-hunting.”

------
ericcumbee
At the risk of being down voted....it recognizes 28 members of congress as
criminals....i’d Call that a good start.

------
drpgq
I work in the industry and obviously I can't say for certain, but I don't
think Amazon has anywhere near the most accurate face recognition software
available. If you want to see where the tech is in terms of performance, the
various NIST tests are highly informative.

------
ejlangev
Amazon really has to take this back to the drawing board. ~5% accuracy is
pretty terrible...

------
aarong11
Could this be down to the people training the algorithm being predominantly
middle class white people? It's a pretty well known phenomenon that people are
bad at recognising features of people of other races.

~~~
rcar
Racial minorities are gong to be greatly overrepresented in a collection of
mugshots, so it's almost certainly just the case that it's easier to make
mistakes in that direction.

If the algorithm is, say, 99.99% accurate on faces that are the same race,
gender, and roughly age, and there's 100 faces in there that match those,
it'll correctly say no match .9999^100 = 99% of the time. If there's 5000,
it's only going to find no match .9999^5000 = 60% of the time.

------
jamisteven
In other news, Amazons "Freudian Slip" Machine is 100% accurate.

------
squozzer
America is in too much danger to take any chances. The 28 positives should be
held indefinitely until _they_ can prove they're innocent.

Presumption of Guilt is the new normal, isn't it?

------
mc32
There are definitely concerns about false positives, but it has to be compared
to the current system. Are the results, effectiveness, better?

When the gov steers opinion, we call it manufactured consent, when public
advocacy organizations engage in sloppy methodology to further a cause, I
propose calling it manufactured outrage.

------
imnotlost
The algo missed 507 of those crooks!

------
exabrial
Are they sure this was a mistake? I wouldn't be terribly surprised to find out
if that many _had_ mugshots.

------
knorker
Sure. "Falsely" :-)

------
j_lane
maybe we should let the cops have this...

------
hanselot
What percentage of people in the openly available mugshot database were people
of colour, and why should that not be relevant?

~~~
bunderbunder
It's relevant, but not as an excuse to dismiss the problem.

One of the perennial problems with machine learning is that it has a tendency
to intensify pre-existing biases in the system. It's not just that the
algorithm tends to reflect biases in the source data, it's that that
reflection tends to encourage the people using the system, who generally have
some role in creating that source data in the first place, to become even more
biased.

~~~
hanselot
biasception. This can be the title of a new film where ai-powered echo
chambers produce progressively more polarized societies until civil war and
anarchy destroy all of humankind. Oh wait, this might just be reality already.

~~~
dang
We've banned this account for repeatedly breaking the site guidelines. If you
don't want to be banned, you're welcome to email hn@ycombinator.com and give
us reason to believe that you'll follow the rules in the future.

[https://news.ycombinator.com/newsguidelines.html](https://news.ycombinator.com/newsguidelines.html)

------
nolite
predictive analytics..?

~~~
rebuilder
So phrenology again? IIRC, that idea didn't exactly hold up to scrutiny.

~~~
onemoresoop
I think it was humor and made me laugh a bit.. phrenology cannot be taken
seriously

------
sadris
Internet used to be delivered through unreliable 28.8kbps modems. I didn't
think for a second that we should cease using the Internet.

Bone marrow transplants for cancer treatment used to have a 20% failure rate.
I didn't think for a second that we should cease using bone marrow
transplants.

And yet why does the ACLU think we should cease using a technology to deliver
potential location hits on wanted criminals because its not 100% perfect?

~~~
mrweasel
The problem is that it's only 95% perfect in this case (28 out of 534, being
identified as criminals). That a pretty big margin of error when the result is
possible arrest. The scope of its use is also in question, sure if you looking
a someone who kidnapped a child, 95% is good enough. If you use it for any
minor violation you risk harassing a large percentage of the population, who
then will need to prove that it wasn't them. Some will flat out deny being the
guilty party, and there's a 5% margin of error, so they may very well be
right. Then what will you do, drop the charge? In that case what's the point.
Or will you spend police time finding evidence that some random person stole
$10 worth of good at the super market?

The cost associated with being just 5% wrong is huge, and that's not including
the emotional damage done to falsely accused citizens, or the decrease level
of trust in law enforcement.

~~~
sadris
They weren't "identified as criminals." They were identified as _possibly
being criminals_. (And as other posters have shown, they were identified with
only 80% confidence as being a criminal.) And an officer can review the
evidence presented by the report and make a human determination to follow up
with a physical arrest.

~~~
SmellyGeekBoy
Perhaps more of an issue in a country where being arrested can involve being
murdered by the police.

------
aurizon
Only 28? Should be a complete 1:1, meaning they all seem crooked in one way or
another...

