
81% of 'suspects' flagged by Met's police facial recognition technology innocent - howard941
https://news.sky.com/story/met-polices-facial-recognition-tech-has-81-error-rate-independent-report-says-11755941
======
whatshisface
Bayes' theorem strikes again. Even if the odds of a false positive are only 1
in 1000, if 1 in a million are criminals then out of 1000 flagged people only
1 might be a criminal.

~~~
jdietrich
...which makes an 81% false positive rate remarkably good. You've massively
whittled down your list of potential subjects and made the job of humans
immeasurably easier. An 81% false positive rate would be nightmarishly awful
if this system served as judge, jury and executioner, but it's incredibly
useful as a means of improving the signal-to-noise ratio of surveillance data.

~~~
e40
That would be true if wiggling out of the net of justice when you are innocent
were not so expensive. There are a lot of poor people who can't afford the
system you speak of.

~~~
jdietrich
The world is not America. _Your_ criminal justice system might be a Kafkaesque
nightmare, but ours is mostly OK.

~~~
throw20102010
Both the USA and UK provide legal aid for poor people even if the procedures
are different. If you cannot afford a solicitor, the government will help you
pay for one, but they probably won't be the best. In both countries rich
people get better legal representation than poor people.

Don't make the UK out to be some paradise (legal or otherwise), it struggles
with plenty of its own problems.

~~~
jdietrich
The UK doesn't have cash bail, so nobody is stuck in pre-trial detention
simply because they can't afford to buy their freedom. It doesn't have elected
prosecutors, judges or sheriffs. It incarcerates less than a quarter as many
people per capita. It doesn't have three-strike laws and uses mandatory
minimum sentences only for an extremely limited number of serious offences.

The UK is far from perfect, but most of the criticisms of the American
criminal justice system simply don't apply.

~~~
merpnderp
Aren’t people far more likely to be a victim of crime in the UK?

~~~
dazc
From petty crime, I suspect, yes. Why, because even when criminals are
convicted they spend very short periods of time in prison (if at all). And, on
release, they have much less chance of integrating into normal society than
previously and fall into a pattern of reoffending, going to prison,
reoffending, and so on.

In the UK, it seems, almost everyone has a tale about when they were burgled,
when they were mugged or when their car was broken into.

Is this so in other countries, I think not?

~~~
mantap
I have never been burgled, mugged or had my car broken into. The only crime
that I worry about on a regular basis is pickpocketing but this is similar to
other European countries.

------
rndgermandude
Berlin did experiment with facial recognition. They had 300 test subjects that
regularly used a particular train station. They claimed to have achieved a
65.8% detection rate, or up to 200 properly identified persons per day (so 1/3
were false negatives). However they also identified 0.67% wrongly, so given
90,000 people using that station a day, that means about 600 people were
wrongly accused by the system each day (false positives).

They declared the system, which wrongly suspected up to 600 people a day at a
single train station of being wanted criminals, but missed a lot of actually
wanted criminals too, a major success.

Mind you, they used high resolution photographs as the training input for the
system, something police does not necessarily have available of criminals and
suspects they try to track down.

[https://www.ccc.de/de/updates/2018/debakel-am-
suedkreuz](https://www.ccc.de/de/updates/2018/debakel-am-suedkreuz) (German)

~~~
DangerousPie
I don't understand the pessimism about this. In any real-world scenario this
type of system would be followed by human review of the photographs, followed
by actual investigation. Nobody is proposing arresting all 600 people straight
away.

Whittling 90,000 photos down to a set of 600 is a major accomplishment. Even
if 599 of those are false positives, that 600th person would probably never
have been caught, because it wouldn't have been feasible to review all 90,000
photos manually.

~~~
mLuby
>Even if 599 of those are false positives, that 600th person would probably
never have been caught.

Society needs to accept that some percentage of crimes will _invariably_
occur. Our question is "what maximum level of crime can we accept?" which then
determines the time and resources required to achieve that level. I have no
patience for this "zero-tolerance" rhetoric.

~~~
DangerousPie
I don't disagree with your point. It doesn't really have anything to do with
the question at hand though.

All I'm saying is that these tools are clearly doing a useful job and
pretending like false positives are an insurmountable problem is a bit silly.

~~~
rndgermandude
Are they tho? Are they doing a better job than say police randomly checking
people?

Let's actually do some crude math to try to estimate how many new cops we'd
need to hire in Germany just to check out those false positive hits. I mean,
what's the point of such a system if you do not check out the hits and make
sure, right?

Let's assume it takes police 15 mins to check out a potential hit in person
(finding the person, checking their identify and asking some questions), and
they work in teams of 2 to be able to secure each other.

The Deutsche Bahn alone says in 2018 they had 5.7 million passenger in Germany
per day traveling by rail[1]. Let's assume all the train stations get equipped
with with this tech and let's assume their 0.67% false positive rate holds.
That's 38,190 checks you need to do on innocent people, per day, or 1591.25
checks per hour.

A two person team can do 60 / 15 = 4 checks per hour. So you need to have 400
such teams at any given time. A shift is only 8h not 24h and people only work
5/7 days per week, so 400 * 2 * (24/8) * 7 / 5, so you need to hire at least
3,360 police officers (well, probably more like 5000, because we completely
disregarded a lot of variables, such as vacation days, etc) just to harass DB
rail passengers who were incorrectly classified as threats, nationwide.
Remember, those are all false positives, not one of those people is actually a
criminal the system is meant to be looking for.

IIRC an average police officer costs the state about 90K per year on average
in Germany. So you'll be spending 300+ million EUR per year on harassing
innocent people just because your tech is kinda shitty.

Now if those 3,360 police offers would do actual useful work instead and
prevented or solved some crimes...

Also, I don't want to be that person who happens to have a similar enough face
to some criminal so the face detection triggers on me all the time.

[1]
[https://www.deutschebahn.com/resource/blob/3992284/f9331633c...](https://www.deutschebahn.com/resource/blob/3992284/f9331633c8a19470629f9e3aa6d5fe8c/20190328_bpk_2019_daten_fakten_en-
data.pdf)

~~~
DangerousPie
Admittedly I don't have any data for this, but I think it's fair to assume
that this number of 600 would go down to 10-20 or so after a quick human
review of the data. So I think your estimate of 15 min per "hit" is wildly
overstating things.

And again I don't think you're asking the right question to begin with. There
is a whole separate argument to be had about whether we as a society actually
want these kinds of systems and whether we think they are worth the cost
(whether they require 300 or 3000 additional officers). But this system is
being developed under the assumption that people do think it's worth it, and
it does a pretty good job of making it a lot cheaper than it would otherwise
be. I wouldn't call that a failure.

As a side-note, I would imagine this would not actually be used routinely in
every single station, at least not in the beginning. You could only activate
it when needed to catch very high-profile criminal. For example after the
attack on the Berlin Christmas market the terrorist managed to travel all the
way to Italy (?) before being stopped. If the German police could have fed his
photo to this system that would have been extremely helpful, even if it would
have meant a couple hundred officers having to review photos for a day.

~~~
rndgermandude
>Admittedly I don't have any data for this, but I think it's fair to assume
that this number of 600 would go down to 10-20 or so after a quick human
review of the data.

Yes, you have no data for this. Sieving out 95% of hits just because a human
glanced at the results is a ridiculously optimistic estimate, even if you
manage to staff the screens with humans who are actually good with strange
faces; and most humans are not. Humans on average not being good with faces
(of strangers) is why witness statements are so unreliable and put so many
innocent people in prison (and we have data for that). And you want to sieve
out faces that the computer already thought were similar enough to warrant a
match, which won't make it any easier.

They're trying to sell a horrible system that does not work correctly to the
public. It doesn't matter if the public thinks it is worth it after they
massively misrepresented what it actually can do. (Oh, and the German public
isn't exactly known for been keen on surveillance either).

>I wouldn't call that a failure.

I would and I do.

>If the German police could have fed his photo to this system that would have
been extremely helpful, even if it would have meant a couple hundred officers
having to review photos for a day.

This is not backed by any data either. He could have easily been one of the
1/3 of people the system fails to detect. Or he could have further disguised
himself for the system not to trigger. Oh wait, he did[1], with a beanie and a
scarf and later with hood. And nobody took notice, because a lot of people
looked like that as it was cold outside. In the summer, you'd just get some
big ass sunglasses to cover enough of your face that the already poor system
would not have triggered.

And by the time the police actually had a preliminary ID and a preliminary
picture of a sufficient quality that would have allowed facial recognition, he
already had left Germany. And by the time they actually had fingerprint
evidence he even was in the truck, he had traveled 3 or 4 countries.

The likelihood that such a system would have helped finding Amri, even tho he
actually took trains and didn't just go into hiding close by or took a car, is
minuscule. Those "couple hundred officers" would most probably had more
success if they spent their time actually investigating Amri and his
acquaintances[2]. Also, let's not forget that by the time we would have used
the facial recognition system, he had already murdered the truck driver and
carried out his attack murdering another 11 people, injuring another 55.

So in the end, even your terrorist-strawman here is pretty bad at justifying
such a system.

[1]
[https://www.spiegel.de/politik/ausland/bild-1128683-1091474....](https://www.spiegel.de/politik/ausland/bild-1128683-1091474.html)
[https://www.spiegel.de/politik/ausland/bild-1128683-1089847....](https://www.spiegel.de/politik/ausland/bild-1128683-1089847.html)

[2] Well, maybe... Given the massive failure by law enforcement before the
attack, I am actually not too sure. Amri had been under surveillance for
months before the attack, and they even had a snitch close to him, who even
drove him from NRW to Berlin. He had previously been convicted in Tunesia (in
his absence because he had already fled to Italy) of robbery, and was further
investigated for theft and drugs, and in Italy he was convicted of arson and
battery, and spent more than three years in jail, where he started a lot of
fights and was already being tracked by Italian counter-terrorism because he
showed lots of signs of radicalization. He was under surveillance in Germany
by counter-terrorism as well, but they further found out that besides
violating the movement restrictions of his asylum, and using multiple
identities (they knew about 3 or 4 before the attack, but in the aftermath
they found he was using up to 14), and his links to radical Islamist groups,
he also paid his bill by dealing drugs. He also was known to the US agencies
and had been put on the US No Fly list. His asylum had been denied, which
prompted him to try to go to Switzerland with fake documents (to marry
somebody there and avoid deportation), but he was caught and jailed, but then
let go again because his replacement papers from Tunesia were expected to take
more than 3 months to arrive. Let go even tho he was known to be a flight
risk, a violent convicted criminal, a drug dealer and very chummy with
radicals, and even tho he could be jailed for more than 3 months for being a
deportee alone awaiting deportation.

This guy was a walking red flag, and German law enforcement had enough
evidence to put him into a prison cell for the drug dealing alone, and he had
a deportation order, but they decided not to act.

------
fencepost
The panopticon _is_ coming and its accuracy will improve. Facial recognition
by itself may not improve much, but when paired with other information it's
going to skyrocket in accuracy, and I suspect it'll happen in China first if
it hasn't already. That other information will include things like the
possible identities of other people in the same vicinity that have known
associations, locations pulled from recognized devices, other fuzzy
identifications along travel routes, gait analysis, matching facial
recognitions with clothing identification to improve tracking, etc. None of
the individual parts are that difficult, they just result in a massive amount
of fuzzy data to be mined and computers can be very good at that mining.

Another factor is that this may be challenging when starting from a facial
recognition point in a crowd and trying to identify a person, but what if
you're starting from a list of "the usual suspects" who've been thoroughly
photographed, had gait analysis done and had their social networks mapped?
What if you start from "do we see any sign that any of these 50k individuals
are in the crowd?"

~~~
heavenlyblue
What if you’re breaking the privacy laws doing so?

Marijuana is illegal, so human tracking should be illegal too.

~~~
stordoff
> Marijuana is illegal, so human tracking should be illegal too.

I'm not sure I really see the connection here.

~~~
heavenlyblue
Marijuana is outlawed due to social conservatism.

Privacy must be preserved for the same reason since we still haven’t yet lived
without privacy for long enough to see the consequences.

------
deogeo
Yet another article framing dystopian levels of surveillance as only bad if
they malfunction. As if the only reason we wouldn't want to give complete
control over our lives to authority, is because the authority might make
mistakes.

~~~
magicalist
> _Yet another article framing dystopian levels of surveillance as only bad if
> they malfunction_

Seriously, it's amazing how well framing a conversation works to divert it.

Where are all the comments about the stasi, securing databases, corrupt law
enforcement, and not collecting information in the first place? Instead we're
arguing over what percentage of "innocent" people can be hassled by the total
surveillance state for it to be acceptable.

------
kd5bjo
There aren’t enough details here to tell whether or not this system is truly
useful. If its false negative rate is good and other methods successfully
detect the false positives, like traditional detective work, using it as a
screening tool reduces the number of people that need to be investigated by a
factor of 200 or so^.

However, those are a lot of assumptions— there’s lots of ways this could
actually be as bad as it looks on the surface:

\- Is the output of this system being used as evidence in a courtroom instead
of a lead generator?

\- Is there some systematic bias in the false positives that result in a
particular class of innocent people being harassed by the police during their
investigations?

\- Does the system reliably include the actual suspect in its set of potential
suspects?

I’ll refrain from commenting on the social acceptability of this system, as I
have never been a part of UK society and thus lack a baseline against which to
compare it.

^My brain is refusing to do the arithmetic tonight for some reason.

------
AnimalMuppet
How does this compare to humans? If you have a human police officer watching a
crowd and singling out "suspects", what percentage of those are innocent?

~~~
namanyayg
No human police offer could process these many faces or remember suspect
databases, of course our current tech is going to be faster here.

More inaccurate, sure, but I'm thinking that with the amount of government
money involved it'll slowly improve. Or at least society will reach a stage
where we don't care about the incorrectly flagged anymore.

~~~
pjc50
The met used to claim they had "super recognizers" doing exactly that. I'm not
sure if it was true or just a lie to cover for the face recognition tech.

~~~
namanyayg
Amazing, I wonder how they test to qualify a super recognizer

------
darawk
Reducing the search space from everyone in a crowd to 5 people seems...pretty
great technology wise. I'm not sure what the problem is here.

I mean, I get that people don't like facial recognition for privacy reasons.
But that seems totally orthogonal to this report, which is criticizing its
efficacy, on very odd grounds.

~~~
mola
When you are on of a crowd, you don't get treated like a suspect. When you are
1 of five you get treated like a suspect. A criminal. That direct implications
on your well being. 4 people out of 5 will be Innocents that get treated like
criminals. I don't want to live in that world.

~~~
darawk
Ok...but that happens all the time. If you look like someone that just robbed
a liquor store and you're in the area, cops will stop you until they can clear
you as a suspect. This is how policing has always worked.

~~~
duxup
Is a dragnet where everyone in a given area is checked even all that common?

Seems unlikely.

But with technology we can suddenly do it 24/7, numerous places, and for all
sorts of crimes at the push of a button...the implications of that many people
as suspects, that volume of innocent people accused seems pretty bad.

------
makerofspoons
I have yet to work with a commercial facial recognition product that produced
a reasonable number of false positives or negatives when dealing with 'live'
data like that of a security feed of a crowd. Is there a facial recognition
model or tool out there that actually works in that use case? It seems like
the Met's police got sold some technological snake oil that wasn't close to
ready.

~~~
jdoliner
It really depends on if their goal with this was actually a tangible result or
just security theater. Snake oil's not a bad purchase if you're planning to
resell it.

------
kazinator
19% of suspects flagged by Met's police facial recognition technology are
_not_ innocent.

That's pretty darn useful!

~~~
xmichael999
Right, I made this point kinda of in another comment. It really depends on how
you want to frame the conversation. I agree with the general argument that
randomly stopping and questioning people because a computer hit sucks, but
knowing UK law enforcement they likely politely stopped the suspects and asked
to see ID. Once they provided ID they were free to go on about their business.
This technology is catching a ton of people that would otherwise not be
caught.

~~~
mantap
And what if you don't carry ID, as is normal in UK. Do they arrest you?

~~~
kazinator
Suppose your face is identified by a machine as looking like someone in a
criminal database who is wanted. Suppose that you in fact are a dead ringer
for that person in the database; even to a humans who double check this match,
you look exactly the same. You have no ID. What are they to do?

------
sigstoat
precision vs recall.

if it correctly finds anyone on the watch list, but they comprise only 19% of
the results returned, and the police force knows it, that strikes me as a
great screening tool. just need some people to sift through it.

i doubt it finds the folks on the watch list that reliably though.

------
gwern
Or in other words, 19% flagged were guilty.

How large does it have to be to be useful?

Bueller?

Bueller?

~~~
ggggtez
The problem is that innocent people could go to jail because they fall into
that 81% of incorrect matches. The justice system, at least in the USA, has a
presumption of innocence. The idea being that when there isn't enough
evidence, it would be better to let a criminal go free than send an innocent
person to jail.

I wouldn't be surprised if these systems are used _specifically_ in cases with
weak evidence. If you had phone records or other hard evidence, you wouldn't
need to rely on face matching. So the faulty system ends up used in the exact
cases where it can do the most damage to innocent people.

~~~
xmichael999
I assume - I don't know, but my guess is the 81% are simply quickly stopped
asked to prove their identification. Once they show their drivers license or
any other basic form of ID that proves they are not the suspect they are free
to go. I highly highly doubt they are detained, frisked, or even slight
questioned beyond being asked to proof who they are. To me, that seems
reasonable. In fact I've been randomly asked to show my ID in various
countries - cop simply stopped me and asked me to show my identification.

~~~
NeedMoreTea
There's _no_ ID requirement in the UK[1]. Driving licence forms a de facto ID
for those who have one, but you don't even need to carry it when driving, let
alone when walking! So potentially gets far more involved, and yet another
small chip is taken from those Peelian principles (Policing by consent by
citizens in uniform).

[1] There's several voluntary ID schemes for younger folks to prove they're
old enough to be in a bar, buy alcohol or cigarettes etc.

~~~
xmichael999
I was no aware the UK does not require that you carry ID. I'm not sure how I
feel about that law, in my experience most of the world requires you to carry
ID. I don't see it at all unreasonable that an adult be required to carry ID
at all times. I am Canadian by birth and Canada has the same policy but in
practice it is best to carry your ID. I've live around the world and
everywhere that I know of outside of europe and the english speaking world
requires you to carry some form of government issued ID.

~~~
NeedMoreTea
Ireland and Australia don't, or didn't, not sure about others.

The UK had them through the war and got rid in the 50s because they were
thought an infringement of civil liberties - _because police were asking to
see them as habit, not for reasons of suspicion_ \- funny how attitudes
change. Not having ID makes something like facial recognition stations set up
in the street seem an even larger overreach. We're _supposed_ to be able to go
about our business without interference unless suspected of something. OK, it
doesn't always work quite that way in practice, but the principle used to be
much clearer, and far better respected... I rather resent losing that.

It made for a bit of an oddity back where there were the troubles in Northern
Ireland and the heavily policed border between north and south. You could get
on to a ferry to Dublin from the UK with _nothing._ No ID, no passport, nowt.
I did that several times, I don't remember anything other than a ticket check.
Change some pounds into Irish pounds when you get there.

------
jammygit
This is a temporary state of things and people need to be less caught up with
the technical shortcomings. It’s the world where machine vision is fully
accurate that is the real problem, not only the one where the machines are
buggy

------
ggggtez
Put another way: they are saying that 81% of positive flags are actually false
positives. Yikes. That might be better than random chance, but in the court of
law that should hardly be considered evidence.

------
ineedasername
The article doesn't really go into enough depth on the repercussions here one
way or another:

-What is the level of impact on the individuals caught up in a false positive? A minor inconvenience? A major issue?

-It takes resources to deal with false positives, even 1 in a 1000 can be a lot of people. The question isn't how much it costs though, it's how much does it cost _without_ the automated facial match, and how much to the false positives cost, and _what is the difference between them_?

------
bb88
For every criminal found that means 4 innocent people are flagged. Hence the
80% rate.

But if there are 1 in 10,000 people which are criminals, then 5 total people
out of 10,000 will be flagged. So the chance of being flagged would be 5 in
10,000 or 1 in 2,000 which would be about a .05% false positive rate overall.

I think everyone forgets about in a situation like this is the false negative
rate -- or missing a flagged person. If they miss 90% of the people they flag,
then the system is pretty much worthless.

------
keiru
1% error would arguably be scarier for that 1%, just pointing it out. I hope
this tech evolves along some strong regulations on how they are used as
evidence.

------
I_am_tiberius
Also if this technology works perfectly I don't want to be checked everywhere.
It should be a human right to be excluded from facial recognition checks.

------
xmichael999
On the surface this sounds bad, real bad. However on the flip side it means
they found a number of criminals they would have never found otherwise.
Hopefully they can tighten the false positive ratio dramatically but this
technology for all its faults is finding real criminals that would otherwise
not have been found.

~~~
TuringNYC
Your having already labeled those found as “criminals” rather than “suspects”
speaks volumes to the problem with legitimizing such systems.

------
Animats
We need is a "you only get hassled once" system and policy. If you're stopped
by the cops for looking suspicious, but not arrested and convicted, you should
get a goodguy tag that prevents you from being stopped for a few years.

~~~
stordoff
That seems open to abuse - wander around London until you get pinged with a
false positive (vary your appearance - hat/glasses/beard - to make it more
likely), then go and commit your crime, knowing one of the major tools is
neutralised. Either it's good enough for general use, and we don't need this
(_maybe_ have an exception for people who have been hassled multiple times),
or it shouldn't be used.

~~~
jimclegg
Remeber the black guy with no record in Florida that was stopped 20-30 times a
week, making his life unlivsble (hard to keep a job).

His life is going to get a lot more interesting with technology like this,
since we know the false positives tend to have a specific race.

------
OrgNet
just keep doing it... one day it'll come down to 75%

------
OscarCunningham
This would be good if only we knew that they wouldn't be treated as though
they're more reliable than they actually are.

------
jacquesm
Proponents of this system will chalk this up as proof the system works because
19% of the suspects were not.

------
tracker1
So the machines think all <insert-identity-group> look alike.

------
LudwigNagasena
I think even 95% would be good enough, 81% is commendable.

------
onetimemanytime
so 19% were criminals :)

------
cedivad
Doesn't sound too bad to me. I mean, moral implications aside this is a fairly
successful technical achievement.

~~~
arkitaip
May I suggest that you up whatever success metrics you have because a 19%
success rate is abysmal. I swear we have no quality standards in our industry,
let alone strong ethics.

~~~
jdietrich
If you had a system that picked the winning lottery numbers 19% of the time,
you'd be a very wealthy man.

~~~
TuringNYC
If I had a system where 19% of my coffee refills were toxic I’d be a dead man.

My point is, in some cases false positives are ok, in other cases they are
not.

------
eesmith
_The Met prefers to measure accuracy by comparing successful and unsuccessful
matches with the total number of faces processed by the facial recognition
system. According to this metric, the error rate was just 0.1%._

Turn the machines off and the error rate will be 0%.

~~~
Traster
The Met prefers a metric where adding an arbitrarily large number of clown
faces to the database significantly increases their performance metric.

