> I think you missed the point. An ML model would only fail in the way KL describes under certain circumstances. If it was indeed the case that everyone does drugs, then it would learn that traffic stops in wealthy neighborhoods also leads to drug busts. The conclusion of KL and related work is that we have to be careful when training ML models to remove sources of underperformance, not that all ML models are useless.
This assumes that the police actually _want_ to make drug busts in wealthy neighborhoods. It's hard for me not to think that using ML models is intended to be a way to insulate the decision makers from accountability; pick a model that gives the results you want, don't divulge the details, and you'll never have to explain your actions because you were "just following the model".
No it's not. I'm making a claim about what ML models are capable of in response to someone incorrectly summarizing a possible weakness with them.
If police don't want to make drug busts in wealthy neighborhoods, they don't need models to justify that. There is no jurisdiction in America where discretion has been ceded to statistical models.
> If police don't want to make drug busts in wealthy neighborhoods, they don't need models to justify that.
They may not need models to justify it, but that doesn't mean that it wouldn't be helpful for them to avoid accountability. I obviously can't say for sure whether or not the police are actively trying to avoid policing any given wealthy area, but it doesn't seem like a stretch that if that was a goal, then obfuscating the source of that decision would be helpful, even if it might not be required.
> There is no jurisdiction in America where discretion has been ceded to statistical models.
That's not at all what I said. My point is that someone who wants to influence _perception_ of where decisions were coming from could make use of ML to make their decisions seem to be the result of "objective" data rather than personal bias.
This assumes that the police actually _want_ to make drug busts in wealthy neighborhoods. It's hard for me not to think that using ML models is intended to be a way to insulate the decision makers from accountability; pick a model that gives the results you want, don't divulge the details, and you'll never have to explain your actions because you were "just following the model".