
Defense of Amazon's Face Recognition Tool Undermined by Only Known Police Client - smacktoward
https://gizmodo.com/defense-of-amazons-face-recognition-tool-undermined-by-1832238149
======
phyzome
« Currently, the only known law enforcement client using Rekognition is the
Washington County Sheriff’s Office in Oregon. (Other clients may exist, but
Amazon has refused to disclose who they might be.) Worryingly, when asked by
Gizmodo if they adhere to Amazon’s guidelines where this strict confidence
threshold is concerned, the WCSO Public Information Officer (PIO) replied, “We
do not set nor do we utilize a confidence threshold.” »

That's worrisome indeed. Understanding confidence thresholds is _essential_ to
using software like this. Thinking they don't "utilize" one means they don't
understand the first thing about it.

~~~
yorwba
> Understanding confidence thresholds is essential to using software like
> this. Thinking they don't "utilize" one means they don't understand the
> first thing about it.

Thresholding is not the only way to make use of a classifier that ouputs a
confidence score. In particular, the slide shown in the article indicates that
they use the confidence score to sort search results in descending order. That
means that there is no fixed threshold and the PIO's statement is not
worrisome at all.

The system described is much less dangerous (in terms of overconfident
decisions) than one that only returns results matching a threshold, because it
means that the user also occasionally sees low-confidence results and learns
that they can't simply defer decisions to the machine.

~~~
b_tterc_p
Unless the cops aren’t exactly virtuous actors... which they frequently are
not. You can’t assume they care if the results are good.

It is another tool to justify police action with a low true positive rate.

~~~
pdkl95
> It is another tool to justify police action

"We were following the AI _leads_ " is the perfect excuse for parallel
construction[1] (evidence laundering).

If Rekognition supports limiting the search to the people in a specific town
(or other small group), it might be possible to force someone to be
_somewhere_ on the list of matches. That would give bad actors the perfect
tool for manufacturing "probable cause".

[1]
[https://en.wikipedia.org/wiki/Parallel_construction](https://en.wikipedia.org/wiki/Parallel_construction)

~~~
Spivak
If parallel construction is in your threat model then does the existence of a
slightly better face search database really matter?

~~~
craftinator
Damn, beat me to it

------
Finnucane
If all of your users are using your product wrong, the problem is not your
users.

------
anth_anm
I should do a version 2.0 that just has a colorimeter and a probably cause
light.

------
bsenftner
This article, and all the "fear of FR" articles in general do not describe how
FR is used. 'What is the process of an FR analysis' sure would be useful in
this discussion.

It's quite simple: a face image is submitted, and then a sorted by "match
score" result is available. This is the same gallery, now sorted in respect to
the submitted face image. That "confidence threshold" (also called
verification threshold, and a number of other similar terms) is the threshold
within the sorted results after which one can generally discard possible
matches between the submitted face and a face in the gallery because the
similarities between the submitted face and the candidate face have little
resemblance. Imagine a series of faces on a horizontal row: the left-most is
the highest match score, and moving right is the next highest in the sort, and
so on moving to the right. The left-most image looks the most like the
submitted face image, and as one moves right each image is a little bit less
similar.

This confidence threshold loses accuracy when a) the submitted image is a less
than a perfect image (often the case), b) the gallery is composed of images
that are less than perfect (often the case), and even c) the types of cameras
used to generate the images are radically different, with different dynamic
ranges and potentially poor video encoding parameters. As a result, a few
things are done: 1) multiple images (even if less than ideal quality) (and if
available) are added to the gallery for each person, and 2) the confidence
threshold is treated as a soft number, with sliders even to dynamically grow
and shrink the net cast by the analysis. Also notice this is interactive -
that implies a human operator. The entire FR process is a selection guidance
tool, not an authoritative selection tool. The FR operator is keenly aware of
the fact that they are sifting through a large number of low quality data
points. FR is a tool to help sift sand.

Disclaimer: I am a lead developer of a leading FR product. Not Amazon's.

Also, to speak about racial bias in FR: it is a stretch to call it "racial
bias", perhaps more of a "low facial dynamic range invisibility". Both people
with very dark skin and people with very pale skin have the same situation:
there is very low variation from the brightest non-highlight part of their
face to the darkest part of their face. This means there is very little
distinguishable information for FR to use, if it can even identify the portion
of an image that contains a face. Such individuals are simply hard to see with
FR, and when identified and tracked as a face, there is very little
information to use for differentiation. I don't know if I'd call that a
"racial bias". These people are almost FR invisible, which benefits them in
this situation, making them very hard to distinguished with FR analysis.

~~~
jacquesm
> I am a lead developer of a leading FR product.

Wish you'd spend your time on something that will move the needle in a more
positive direction. The long term applications of the tech that you are making
are downright scary.

~~~
bsenftner
Who do you think is correcting FR users misunderstandings and insuring the
poor practices you're reading about do not come true? I think my skills are
critical here.

~~~
hnaccount141
What makes you think that you (or anyone else for that matter) are capable of
preventing the abuse of FR? I appreciate that you and many others in the space
are working with the best intentions, it just seems like any potential
benefits of FR are dwarfed by its potential use as a tool of oppression.

~~~
bsenftner
What makes you think that you (or anyone else for that matter) are capable of
preventing the abuse of FR by walking away and declaring an easily created
technology simply taboo? It is going to be developed, like it or not. May as
well have intimate knowledge of it, help to educate others, and actually know
how to defeat it. Walking away you're just blind.

------
bradgnar
This article is kind of buzzy, doesn't really make sense, and never actually
states what amazon is defending their tool from.

~~~
salty_biscuits
A poor understanding of ROC curves. A concept that is surprisingly difficult
to explain to people with low maths abilities and will lead to misuse. Even
without the "confidence" not being a true probability. My anecdote is I made a
classifier to speed up the search through a list of images. It worked great
and saved lots of money. There was an auto accept (no review) for a confidence
greater than 95%. After a while they get back to me and say "we have run some
analysis and it is getting about 5% or the auto accept wrong, how can we make
it get them all correct?". Of course set the threshold to 100%. But then it
doesn't accept anything. Then give me a few years and more devs to develop the
classifier to make it more powerful. Can't you just tweak it to make it
better... And so it goes

