That's the point? These systems invariably turn out to not be any sort of AI at all, but sales people ready to make a call any way their customer (police) wants it made.
Later it's presented as AI to a jury. It's basically the same scam as lots of forensics, junk science to get a desired result.
I generally agree, but wouldn't necessarily say a lot of forensics is junk science. The science is generally sound (procedures and theory). Usually it's the prosecutors that present the forensics in a way that misrepresents what the results actually mean. Or lab techs make mistakes or perform misconduct (How to Fix a Drug Scandal on Netflix is interesting).
Do you have some links for this? My understanding is the interpretation part can be BS, but stuff like finding and identifying potential ignition or accelerant locations and types should be more settled.
That doesn't exactly support your claims for most of forensics. If you read/skim the NAS report these articles are based on, you will see that much of the issue is not with the underlying science, but rather in the "interpretation" part. Yes, the odontology seems to be unsupported. But even the fire analysis that you list as an example is said to have a scientific basis like chemistry, but that the interpretation part has issues.
While my stance on forensics is not as negative as the parent comment, I don't see how you are drawing such a sharp distinction between the "science" and the "interpretation". They are not separate things.
When a scientist proposes a new test, they include measurement techniques as well as a method of interpretation. If a medical test is used in a way that is frequently prone to misinterpretation, it would be fair to call that entire test useless.
"If a medical test is used in a way that is frequently prone to misinterpretation, it would be fair to call that entire test useless."
And yet there are many examples of this with malpractice, unnecessary deaths, etc. Lyme tests are notorious for false negatives depending on the lab you use. It's also a widely held belief that you want tests done in the middle of the week instead of on a Friday or weekend because some techs just want to get out of there and are more prone to mistakes.
The reason I'm drawing a hard line is because of fact vs opinion. This same line gets drawn in court. They will ask experts if it's a fact or an opinion that they testified about (usually to attack a witness). Autopsies are full of facts, and then there are opinions on what those facts mean. If it's an opinion, it's possible to counter it with another professional opinion.
The only sort of opinion they should be sharing is included, excluded, or undetermined. Hair is not a "match", but it could include or exclude someone and maybe mathematical probabilities.
I disagree. The canonical example here are p-values. Completely valid and accepted interpretation which will forever be abused. In the medical realm, we see the same things with false-positive and false-negative rates causing massive confusion. An even more recent example is COVID vaccine efficacy. I would take the bet that 90% of the general population could not give the accurate definition of what the definition of efficacy is.
If you are using forensics in a trial, the validity of the "interpretation" part is key to it's use. The reliability of these interpretations has been historically hugely overstated.
Given this history it is absolutely reasonable to reject most forensic evidence unless there actual good science supporting the reliability of expert interpretations.
The usual example is DNA evidence. The two most obvious fallacies are actually called The Prosecutor Fallacy and The Defendant Fallacy because it's so easy for either side to mis-sell what the evidence really means.
It's basically impossible for someone who isn't pretty handy with Bayes to work out the significance for themselves.
It may very well be the case that a version of fire analysis that is scientifically rigorous can be done, but this would have nothing to do with how it is typically practiced.
> The science is generally sound (procedures and theory). Usually it's the prosecutors that present the forensics in a way that misrepresents what the results actually mean.
That’s not the same claim. The practitioners who come up with the results are not operating in a scientifically sound or rigorous way. One documentary I watched about a wrongful murder conviction (a man was found to have burned down his house to kill his entire family) had a discussion of how the fire investigators would talk about “feeling the beast” and other nonsense. It is very flattering to describe this as in any way “evidence-based.”
As soon as the analysts making judgments about the evidence are in communication with the prosecutors or police, it's junk science. You wouldn't do a psychology or medical experiment without at least single blinding, and certainly not with the experimenter telling the subjects what result they want!
Look at this damming quote - "ShotSpotter strongly insisted it had not improperly altered any data to favor the police's case". That shows they don't have a system in place to prevent data being altered or to prevent themselves knowing what the police want, so all their findings are suspect.
How is hair analysis junk science? Maybe you just mean that people misuse the facts to come to incorrect conclusions?
It's only supposed to be used to include or exclude someone, and based on the numbers/probabilities it's generally only useful for excluding someone. Of course that's not how most prosecutors use it...
Hair analysis, as in looking at two hair samples under a microscope and determining if they are a match, is definitely junk science. The FBI has specifically acknowledged this, dozens of cases have been overturned because subsequent genetic testing has shown hair samples did not, in fact, match. In at least one case someone was convicted based almost entirely on hair sample analysis, and later DNS testing showed that not only was the hair not a match, the hair found at the crime scene belonged to a dog. https://www.nemannlawoffices.com/blog/fbi-forensics-convicte...
There have been plenty of references in this post to the NAS study on the state of forensics.
Basically, the majority of forensics are based on sound theories and practices. The issues are when professionals and prosecutors try to "interpret" the facts.
I would be interested in any citations showing that forensics are generally unsound.
Not fingerprinting, but related issue for microscopic hair comparisons:
The Justice Department and FBI have formally acknowledged that nearly every examiner in an elite FBI forensic unit gave flawed testimony in almost all trials in which they offered evidence against criminal defendants over more than a two-decade period before 2000.
Of 28 examiners with the FBI Laboratory’s microscopic hair comparison unit, 26 overstated forensic matches in ways that favored prosecutors in more than 95 percent of the 268 trials reviewed so far
This isn't the case I originally recalled, and doesn't say how many points were matched, but I remember a 5 point match being used to convict a person who was later proven innocent.
Fingerprints aren't "junk science", but they do have error margins and plenty of room for mistakes. The problem is that many juries, judges, prosecutors, and even defenders forget about this.
This is the case for many types of evidence; witness statements for example can also be notoriously unreliable. It's not bad to use this kind of evidence, but you do have a problem if you consider any witness statement to be absolute truth and never consider any "reasonable doubt" there may be, and this is how you end up convicting people based purely on "I saw the suspect commit the crime from 50 metres away in the dark and I'm sure it was him!" or some such.
If you have six independent witness statements like that, or a witness statement and fingerprints then the reasonable doubt starts becoming a lot less reasonable.
This is also the problem with the case in this article: a piece of evidence over which it seems like reasonable doubt should exists is the singular piece of evidence tying this person to the crime. The problem here isn't the evidence as such IMHO, but that juries and/or defenders (depends on the case) seem exceedingly bad at sceptically examining physical evidence.
Fingerprints are directly-recorded facts. What may be junk science is what expert witnesses present about the interpretation of fingerprints.
> The problem is that many juries, judges, prosecutors, and even defenders forget about this.
Juries as triers of fact, and judges insofar as they are ruling on the facts of the case, aren't, in any case, supposed to allow themselves fo be guided by extrinsic knowledge, they are supposed to act based on facts (both direct and the expert knowledge relating to the import of direct facts) provided as evidence, both testimony and exhibits, in the case.
That's not really how it works; false positives exist, especially if you consider that a lot of times you have incomplete or smudged fingerprints and there is actually some amount of subjectivity involved in determining a match. See e.g. [1]: "only two properly designed studies of latent fingerprint analysis had been conducted. These both found the rate of false matches (known as “false positives”) to be very high: 1 in 18 and 1 in 30."
Again: useful evidence for sure, but considering them undisputable facts is not wise.
Matches (positive or negative) are not the same thing as fingerprints. Fingerprints are directly recorded facts. Matches are matters of interpretation (the fact of a match reported by a particular system is a distinct direct fact from the fingerprint itself, but again the significance is...what expert testimony exists to establish or challenge.)
The reality is that when "fingerprint evidence" is brought to court that it's not foolproof and has a sizeable margin of error that should be accounted for when considering the verdict, something that is currently often ignored. You can argue semantics all day, but that's the way things work right now. This seems like an exceedingly pedantic point to make.
> The reality is that when "fingerprint evidence" is brought to court that it's not foolproof
No evidence is foolproof. My entire point is that the issue isn't with “fingerprints” or the knowledge that juries and judges bring extrinsically (which was the explicit claim made upthread), but with the expert testimony that contextualizes fingerprints.
(This was even more clearly the problem with fiber evidence when the FBI crime lab was presenting pure bunk expert testimony in virtually every case.)
Yeah, knowing or not, comparing with too few points is misconduct - that should be the fault of someone besides the defense, but that is somewhat tangential - does anyone with more knowledge know where the line is drawn? I imagine there are some outlandish coincedences happening.
How then do you explain the success of Touch ID on Apple devices and the equivalent systems on other phones and tablets?
Random people can't just walk up and unlock a Touch ID device with their fingerprint, which suggests that fingerprints are in fact a very good way to tell people apart.
Yes, it is possible to make a fake fingerprint that Touch ID will not be able to distinguish from my fingerprint. But that fake fingerprint will only unlock my device, not your device, so it is still correctly distinguishing our fingerprints.
> How then do you explain the success of Touch ID on Apple devices and the equivalent systems on other phones and tablets?
Because on those devices the fingerprint is only used as a second factor (the first factor being physical possession of a particular device).
If Apple used fingerprints like law enforcement do, that is running a fingerprint against the big DB of all prints, then they would certainly get some arbitrary matches.
How many people have you let try the unlock your device with their fingerprint? Tens or hundreds of Millions? That'd be a closer comparison for how a search of fingerprint database work.
Searching a big DB is one way to use fingerprints. It is a crappy way, though.
Another way fingerprints are used is to compare fingerprints found at a crime scene to people who have been identified using other methods that have nothing to do with fingerprints.
E.g., if you have someone murdered in his office when working late at night, and security cameras show that five other people were in the building at the time of the murder, and you are able to get a good set of fingerprints off the murder weapon you don't need a database. You take the fingerprints from all five of those other people, and if one is a very good match and four do not match, you concentrate most of your effort on the one that matched.
It's actually quite similar to how DNA evidence has been used and misused. I don't know how they compare DNA nowadays, but when DNA evidence was first making waves getting people convicted they only compared two DNA samples at a few points. For a given sample there would likely be several people in the country that matched.
That's fine if used right, like fingerprints in the earlier murder hypothetical. Narrow it down to only 5 people who could have committed the crime, get a DNA sample from the crime scene that must be from the criminal, and if that matches exactly one of those 5 suspects it is strong evidence they are the criminal.
Have no suspects yet, run that same sample through a database, get exactly one match, and conclude that must be the criminal. Totally bogus. A lot of people were convicted in the early days of DNA matching that way.
The database method can be made sound, but only if the database includes everybody. Match against a database that includes everyone, only get one hit, and you've probably got your criminal. But if the database includes everyone you are likely to get several matches.
As I said, that was how it used to be. I know DNA sequencing has gotten faster and cheaper over the years, but I have no idea if routine forensic DNA matching now matches enough to make matches unique except in the case of identical siblings.
I'm curious if you think that an accused person could be used to setup a Touch ID, then have the prints collected at a crime scene used to unlock the device.
Do you think the forensic prints would meet the standard required by Touch ID?
Ok, but supposing what you say is true, they are junk science when it comes to forensics. There is research on capturing the full 3D structure of fingerprints to actually capture the uniqueness of a fingerprint[0]. But even if the 2D scan is enough not every scanner is capable of collecting that structure.
Given how much R&D goes into making better fingerprint scanners, it seems odd to then claim that a fingerprint taken off of a victim has anything near that level of detail. I guess its sound to say that it may be used as a piece of evidence to further narrow down the target, but its completely disingenuous to claim that it's sound evidence.
Because other surfaces are not a purpose built sensor designed to detect the features of a finger then repeatedly trained with a specific set of finger scans?
Like are you joking?
Saying TouchID validates fingerprinting is like saying MRI machines validate psychics.
It seems you two have a different idea of what is being discussed. The underlying fundamentals of finger prints are solid. The part that can make them ineffective is when they are using insufficiently tested tools or matching on too few points. And of course misrepresentating what a match actually means and how it pertains to the case.
> The underlying fundamentals of finger prints are solid.
What exactly are these underlying 'fundamentals'? Nobody even knows if everybody has unique fingerprints, it's just an assumption.
Beyond that, it's not all that relevant since as you mentioned how the matching is done is largely subjective, and it depends a lot on the number of "points" used and who does the matching, and that should really bring the accuracy even more into question. What exactly is the statistical probability of an incorrect match based on the number of points used? Good luck answering that question.
IMO the situation is made a lot worse by the fact that the public generally assumes fingerprinting to be extremely accurate (or 100% accurate).
That said, human in the loop ML can be extremely useful. Surfacing possible positive examples for human review and/or cutting down 'obvious' negatives can multiply the productivity of human reviewers in many contexts.
Not quite sure what point you're trying to make here...
Misusing science is a problem regardless of whether ML is the science in question. It's /already/ a big problem within criminal justice (polygraphs, psychopath tests, etc).
Avoiding 'obvious' mistakes is helpful in reducing the amount of work human raters/evaluators need to do. Automatically classifying 90% of data and leaving 10% for humans can reduce human workloads by 90%, or enable new use-cases where doing 100% of the work simply wasn't feasible. Using ML in this way doesn't make it 'fake AI.'
Forensics is full of gee-whiz-cool csi stuff which eventually is shown to be worthless. Polygraphs are a great example; they're still very wisely used in the US, despite piles of evidence showing they're prone to both false positives and false negatives. Sibling comments saying they aren't forensics because they've been discredited are missing the point: these are typical crime tech, and have been lately discredited, but are still in use in the US regardless.
The underlying problems seem to be bad incentives (the prosecution is paid+evaluated on its ability to put people in jail) and the court's willingness to accept insufficiently validated tech.
To add to that list: fire investigation. There have historically been a lot of myths in the field regarding forms of "proof" that a fire was caused by arson, and many of them were historically used to obtain (likely false) convictions.
It really boggles my mind how desperate people are to try and include and blame the police for everything.
>sales people ready to make a call any way their customer (police) wants it made.
You really believe people signed up for a dangerous amd thankless job with high liability and low pay so they can feverishly conspire to have this guy arrested?
Edit: This place is becoming reddit. Someone posting links which dont substantiate their statements gets upvoted, meanwhile my replies questioning this get hit by a downvote brigade.
Just fyi - Police are generally quite well paid for the area they live in. For example, in Alameda (the county I live in) it's common for police to make over $450,000 a year[1].
Also, while police to have a more-dangerous-than-average job[2] (and face unique risks - they are probably the most likely profession to die of a gunshot on the job), there are much more dangerous jobs[3].
So if you have been excusing police because you think they are accepting poverty for danger, I'd ask you to consider that Police are more like extremely well paid garbage collectors[4].
Where in source [1] does it show that it's common for police to make over $450k a year?
The 'Avg Total pay &
benefits' sorted by title has Chief Information Officer/Registrar of Voters as the entry into that level of remuneration. Police specific above that you only have the Sherrif, which is an elected official.
Police pay for the vast majority of roles is significantly below $450k.
That's a good critique! Looking at historical data it seems like "commonly above $300,000" would be more accurate. It's worth saying that listed police salaries are far below these numbers and the officers get there through overtime and other extra payments. Thus the use of 'common' instead of average because you have to look back at what people were actually paid.
Edit: I can no longer edit my original post so this will have to do.
How are supervisors and elected officials making 300k (after all benefits considered) in an area with a high cost of living equal to "police commonly make above 300k"?
Second, it would be absurd to account for extra income to say they make what you're saying. That's like saying a cashier at walmart works three jobs to make an average of 70k a year, so cashiers at walmart commonly make 70k a year.
> Second, it would be absurd to account for extra income to say they make what you're saying.
Except it's extra income that they make by being a police officer. A business that hires an off-duty police officer for security isn't hiring them because they're a big burly-looking dude; they're being hired because they are a police officer. An off-duty officer will typically wear their police uniform while working "off duty", and can take actions which an ordinary civilian could not (like conducting arrests).
> it would be absurd to account for extra income to say they make what you're saying.
It's not what I say they make, it's what the public records say they make. You can go look yourself.
For example, from my first link, Sergeant Stuart Eugene Barnes made $125,380.76 in salary and $172,330.17 in overtime. I think it fair to say that's compensation from their job as a police officer.
Kind of makes sense though, most officers start there then move to a safer place once they get enough experience. Remember seeing billboards advertising $80k starting salary 10+ years ago while driving through town.
I can't remember which department, but I remember the local SF/Bay news many years ago talking about some supervisor at that department somehow making $1.5 million the year before (or something insane like that.) They explained how it was all possible, which to me seemed like he had just been with them for so long, he knew exactly the optimal way to play the system and exploit all the loopholes in it.
As for your second part, I guess it would seem unlikely for some positions, but I know a doctor who made like 3x his base salary each year through all sorts of absolutely legal, ethical, and hospital-sanctioned means. Taking every single weekend he could on call (even offering to "relieve" other doctors who had been assigned that weekend) for example allowed him to make much more than you would ever think possible.
Many of these tricks are not secret. They are commonly known and coordinated among emergency service personnel. They cooperate to take vacation, etc. in ways that will maximize the overtime of coworkers. That overtime can then be recycled into time-and-a-half leave that will reciprocate into the maximum overtime for the first party.
An administration that squashes all overtime faces extreme wrath. They can be cutting employees' take home in half. I and plenty of others have seen people lose their houses or suffer other financial crumbles when the expectation of built-in pay far beyond their published salaries were cut off. You can argue that it is foolish to depend on such extra pay and I would agree. But when that significant extra has been their reality for all 10 years of their employment with the department, you can see how they would just plan it into their future finances.
The way I interpret that is that they're paid with the idea that they'll have two kids or a Sergeant is a "Manager" (Management $124,884, from the living wage source). I'm curious what the reason for overtime is, but I suspect that it's staffing requirements.
Debating pay to danger ratios is a pretty fruitless effort. People don't get paid for danger.
TLDR: "Service members will receive $7.50 for each day they are on duty in an IDP area up to the maximum monthly rate of $225."
Moreover, I was a 4 year Corporal living in the barracks with the equivalent of two deployments under my belt raking in about $20k a year.
To my knowledge, this is a fairly standard rate. Side note, we used to crack all kinds of jokes around, "I guess I earned my 7.50 today."
I think what people do get paid on frequently is the availability of people who can and will do the job. A limited and capable talent pool will always make more, especially when demand is high.
I do wonder if the pay negatively affects the confrontations that the individual officer will risk, e.g. let a suspect escape due to not having good enough backup.
That's because he's probably lying/exaggerating. I highly doubt your average officer makes anywhere near that. I don't make over 200k as an experienced developer, either.
People are terrible with nuance. OP did a good job at contextualizing danger of policing, without claiming it's "safe", as you'd see on reddit, but I'm guessing something is getting lost here.
I'm very sceptical someone is in the multiple sixes of figures without serious responsibility and credentials. In my area officers make similar (lower) pay than tech jobs, and every area I've seen is similar. So people on reddit will be complaining about officers making 80k-100k out west, when in reality that's just marginally better than bartenders.
So yeah, they're making like 60k-100k, or more out west. Here in the midwest it's more like 40k-80k. This compares to developer jobs I've seen, but it's not apples to apples. Cops end up years behind in pay, but end up with better benefits. So they'll be at 70k 3-10 years in, instead of 0-5 in.
In either case there's other tiers and roles, was assuming grunts. There might be other cases where "super grunt" jobs exist, and get paid very well. Just like with engineers it seems like the way to advance is increased specialization and moving to management.
You can spin your wheels at low level engineering jobs for decades if you want, too. Usually, people also have to move towards leadership roles to advance in engineering...
So I digress, but I guess it's fair to say police officer is of similar value as engineer in terms of job perks... but engineer is considered a good job.
I looked at your link and I dont see any police making over 450K a year. In fact, I dont see any making over 100k a year except supervisors and elected officials.
I think you're conflating issues with the institution of policing within the United States with the behavior of individual police officers. Without getting into the politics or philosophy of the comportment or purpose of a police force, there are some things in situations like this that are fact:
1. Police officers (and more importantly, prosecutors) are incentivized to make arrests and have those arrests stick.
2. It's a well understood psychological phenomenon that individuals are much more lenient of their own actions. They call this attribution theory in psychology [1].
3. While policing is a risky job, it comes with privileged treatment, powers, and trust that a common citizen does not have. As such, and as public servants, we hold those with these powers to a higher degree of responsibility and behavior than those without such powers. To hold police accountable, we need a minimum level of oversight and control. Historically, this has either not happened or been circumvented by way of police unions establishing ill-advised contracts with the local governments they work with. [2]
I'll agree on points that individual police officers aren't evil, but I'd need a lot of convincing to believe that many policing organizations within the United States are not being held accountable to an acceptable level; nor are they being trained in ways that encourage appropriate interactions with the public they serve.
3. This is a fallacy due to media portrayal. If you are late to your job what happens? Probably nothing or you get asked not to do it again. The same mistakes you make, police officer will get investigated by internal affairs and punished over.
This is equally a fallacy - "if you are late to your job" "will get investigated by internal affairs". No, they won't. But they will be protected by one of the strongest unions in the country, if not the strongest.
Policing is also a lot less risky a job than is commonly perceived.
Later it's presented as AI to a jury. It's basically the same scam as lots of forensics, junk science to get a desired result.