
Training Computers to Find Future Criminals - nigrioid
http://www.bloomberg.com/features/2016-richard-berk-future-crime/
======
Houshalter
I could not disagree more with these comments. Psychologists are just now
starting to study the phenomenon of "algorithm aversion", where people
irrationally trust human judgement far more than algorithms. Even after
watching an algorithm do far better in many examples.

The reality is humans are far worse. We are biased by all sorts of things.
Unattractive people were found to get twice as long sentences as attractive
ones. Judges were found to give much harsher sentences right before lunch
time, when they were hungry. Doing interviews was found to decrease the
performance of human judges, in domains like hiring and determining parolle.
As opposed to just looking at the facts.

Even very simple statistical algorithms far outperform humans in almost every
domain. As early as 1928, a simple statistical rule predicted recidivism
better than prison psychologists. They predict the success of college
students, job applicants, outcomes of medical treatment, etc, far better than
human experts. Human experts never even beat the most basic statistical
baseline.

You should never ever trust human judges. They are neither fair nor accurate.
In such an important domain as this, where better predictions reduce the time
people spend in prison and crime, there is no excuse not to use them. Anything
that gets low risk people out of prison is good.

I believe that any rules that apply to algorithms should apply to humans too.
We are algorithms too after all. If algorithms have to be blind to race and
gender, so should human judges. If economic information is bad to use, humans
should be blind to it also. If we have a right to see why an algorithm made a
decision the way it did, we should be able to inspect human brains to. Perhaps
put judges and parolle officers in an MRI.

~~~
Smerity
Stating that method A is problematic does not automatically mean method B is
better.

> The reality is humans are far worse

Citation needed - especially when comparing against a specific instantiation
of a machine learning model. Papers published by the statistician in the
article used only 516 data points. Most data scientists running an A/B test
wouldn't change their homepage with only 516 data points. There's no guarantee
the methods he is using for the parole model involve better datasets or models
without deep flaws.

An algorithm or machine learning model is not magically less biased than the
process it is replacing. Indeed, if it's trained on biased data, as you
believe by stating "never ever trust human judges", then the models are
inherently biased in the exact same way.

If you give a machine learning model a dataset where one feature appears
entirely indicative (remember: correlation is not causation), it can overfit
to that, even if that does not reflect reality.

I highly recommend reading "How big data is unfair: understanding unintended
sources of unfairness in data driven decision making"[1], by Moritz Hardt, a
Google machine learning researcher who has published on the topic (see:
Fairness, Accountability, Transparency). It is a non-technical and general
introduction to some of the many issues that can result in bias and prejudice
in machine learning models. To summarize, "machine learning is not, by
default, fair or just in any meaningful way".

Algorithms and machine learning models _can_ be biased, for many reasons.
Without proper analysis, we don't know whether it's a good or bad model, full
stop.

[1]: [https://medium.com/@mrtz/how-big-data-is-
unfair-9aa544d739de...](https://medium.com/@mrtz/how-big-data-is-
unfair-9aa544d739de#.ykd1vedz1)

~~~
Houshalter
>Citation needed - especially when comparing against a specific instantiation
of a machine learning model. Papers published by the statistician in the
article used only 516 data points. Most data scientists running an A/B test
wouldn't change their homepage with only 516 data points. There's no guarantee
the methods he is using for the parole model involve better datasets or models
without deep flaws.

516 is more than enough to fit a simple model. As long as you use cross
validation and hold out tests to make sure you aren't fitting. 516 data points
is more than a person needs to see to be called an "expert". Many of the
algorithms I referenced used fewer data points, or even totally unoptimized
weights, and still beat human experts.

>An algorithm or machine learning model is not magically less biased than the
process it is replacing. Indeed, if it's trained on biased data, as you
believe by stating "never ever trust human judges", then the models are
inherently biased in the exact same way.

We have ground truth though. Whether someone will be convicted is a fairly
objective measure. Even if it's slightly biased, it's still the best possible
indicator we have of whether or not someone should be released. If you had a
time machine that could go into the future and see who would be convicted,
would you still argue against using that information, because it might be
biased? Leaving people to rot in prison, even if all the statistics point to
them being very low risk, is just wrong.

>"machine learning is not, by default, fair or just in any meaningful way"

 _Humans are not, by default, fair or just in any meaningful way._ Nor are
they accurate at prediction. Any argument you can possibly use against
algorithms applies even more to humans. That's my entire point. You should
trust humans far, far less than you do.

~~~
pdkl95
> You should trust humans far, far less than you do.

Which is why I don't trust _the people picking the algorithm_. You still have
human bias, but now they are easier to hide behind complicated algorithms and
unreliable data.

edit: removed original editing error

edit2: You say I should trust the algorithm, but y9u seem to be going out of
your way to ignore that the algorithm itself has to be created by someone. You
haven't reduced the amount of bias; trusting an algorithm simply codifies the
author's bias.

~~~
Houshalter
You should trust the "complicated" algorithm far more than you trust the
complicated human brain of the judge, who was also trained on unreliable data.

Look it's easy to verify whether parole officers are better at predicting
recidivism than an algorithm. If the algorithm is objectively better than it
should be used.

~~~
kragen
Given an unbiased algorithm that does better at predicting recidivism, it
would be easy to _deliberately_ construct an algorithm that does almost as
much better, but is egregiously biased. For example, if you had been wronged
by somebody named Thiel, you could persuade it to never recommend parole for
anybody named Thiel. There aren't enough people named Thiel for this to
substantially worsen its measured performance.

Given that it's easy to construct an example of how you could _deliberately_
do this, and it's so easy to accidentally overfit machine-learning algorithms,
we should be very concerned about people _accidentally_ doing this. An easy
way would be to try a few thousand different algorithm variants and have a
biased group of people eyeball the results to see which ones look good. If
those people are racist, for example, they could subconsciously give undue
weight to type 1 errors for white prisoners and type 2 errors for black
prisoners, or vice versa.

The outcome of the process would be an algorithm that is "objectively better"
by whatever measure you optimize it for, but still unjustly worsens the
situation for some group of people.

A potential advantage of algorithms and other rules is that, unlike the brain
of the judge, they can be publicly analyzed, and the analyses debated. This is
the basis for the idea of "rule of law". Aside from historical precedents,
though, the exploitation of the publicly-analyzed DAO algorithms should give
us pause on this count.

Deeper rule of law may help, but it could easily make the situation worse. We
must be skeptical, and we must do better.

------
imh
I think the whole idea here is frightening and unjust. We are supposed to give
all people equal rights. What people might do is irrelevant. A person whose
demographic/conditional expectation is highly criminal should be given an
equal opportunity to rise above it, else they might see the system is rigged
against them and turn it into a self-fulfilling prophecy.

~~~
Taek
It's frightening depending on how you use the data.

A good example perhaps is that I like to horse around when I'm at the beach.
I'm more like to get hurt than others who are more cautious. I'm also more
likely to hit people accidentally.

I had some parents of younger children approach me and ask me to stay on the
far side of the beach. On one hand it felt rude, but on the other it allowed
me to be rambunctious and it allowed the parents to prioritize their
children's safety.

The world isn't flat enough for this to be a reality yet, but if you cluster
people by their morals, you don't have to throw them in jail. Put all the drug
users together. Keep the drugs away from people who don't want anything to do
with them.

Usually if people are more likely to commit crimes, it's either because they
are desperate (which means successful intervention is likely provided you can
solve their core problems), or it's because they find that
activity/action/crime to be morally or culturally acceptable. To the extent
that you can exclude that culture from your own daily life, you don't have to
punish/kill that culture.

Pollution is a good counter example. You can't really isolate a culture of
pollution because it's going to affect everyone else anyway. So there are
limits.

As long as our methods for dealing with criminals evolve appropriately against
our ability to detect them, I am okay.

Human history is full of genocide though. I don't think that bodes well for
our ability to respect cultures that allow or celebrate things we consider to
be crimes.

------
moconnor
"between 29 percent and 38 percent of predictions about whether someone is
low-risk end up being wrong"

Wouldn't win a Kaggle contest with that error rate. What's not disclosed is
the percent of predictions about whether someone is high-risk ending up being
wrong. These are the ones society should be worried about.

And these are the ones that are, if such a system is put into practice,
impossible to track. Because all the high-risk people are locked up. The
socio-political fallout of randomly letting some high-risk people free to
validate the algorithm makes this inevitable.

This leaves us in a situation where political pressure is _always_ towards
reducing the number of people classified as low-risk who then re-offend.
Statistical competence is not prevalent enough in the general population to
prevent this.

TL;DR our society is either not well-educated enough or is improperly
structured to correctly apply algorithms for criminal justice.

~~~
Houshalter
The question is whether human judges do better, and they don't. We have no
better method of determining whether someone is low risk or high risk. But
keeping everyone locked up forever is just stupid. If these predictions let
some people get out of prison sooner, I think that is a net good.

------
brillenfux
The nonchalance of these people is what really terrifies me.

They just laugh any valid criticism off and start using the references
"ironically" themselves.

I don't understand how they can do that; do they not have a moral compass? are
they psychopaths?

~~~
brillenfux
What happened here? There was a whole thread coming after that?!

~~~
dang
[https://news.ycombinator.com/item?id=12123453](https://news.ycombinator.com/item?id=12123453)

------
sevenless
The entire concept of using statistical algorithms to 'predict crime' is
wrong. It's just a kind of stereotyping.

What needs to happen is a consideration of the social-justice outcomes if
'profiling algorithms' become widely used. Just as in any complicated system,
you cannot simply assume reasonable looking rules will translate to desirable
emergent properties.

It is ethically imperative to aim to eliminate disparities and social
inequalities between races, even if, and this is what is usually left unsaid,
_judgments become less accurate in the process_.

Facts becoming common knowledge can harm people, even if they are true.
Increasingly accurate profiling will have bad effects at the macro scale, and
keep marginalized higher-crime groups permanently marginalized. If it were
legal to use all the information to hand, it would be totally rational for
employers to discriminate against certain groups on the basis of a higher
group risk of crime, and that would result in those groups being marginalized
even further. We should avoid this kind of societal positive feedback loop.

If you accept that government should want to avoid a segregated society, where
some groups of people form a permanent underclass, you should avoid any
algorithm that results in an increased differential arrest rate for those
groups, _even if that arrest rate is warranted by actual crimes committed_.

"The social norm against stereotyping, including the opposition to profiling,
has been highly beneficial in creating a more civilized and more equal
society. It is useful to remember, however, that _neglecting valid stereotypes
inevitably results in suboptimal judgments_. Resistance to stereotyping is a
laudable moral position, but the simplistic idea that the resistance is
costless is wrong. The costs are worth paying to achieve a better society, but
denying that the costs exist, while satisfying to the soul and politically
correct, is not scientifically defensible. Reliance on the affect heuristic is
common in politically charged arguments. The positions we favor have no cost
and those we oppose have no benefits. We should be able to do better."

    
    
        –Daniel Kahneman, Nobel laureate, in Thinking, Fast and Slow, chapter 16

~~~
wtbob
> It is ethically imperative to aim to eliminate disparities and social
> inequalities between races, even if, and this is what is usually left
> unsaid, judgments become less accurate in the process.

Why? Why is it 'imperative' to be wrong?

> Facts becoming common knowledge can harm people, even if they are true.

Well, they can harm people who, statistically speaking, are more likely to be
bad.

If anything, I see accurate statistical profiling being helpful to black
folks. Right now, based on FBI arrest data, a random black man is 6.2 times as
likely to be a murderer as a random white man; a good statistical profiling
algorithm would be able to look at an _individual_ black man and see that he's
actually a married, college-educated middle-class recent immigrant from
Africa, who lives in a low-crime area — and say that he's _less_ likely than a
random white man to be a murderer.

Perhaps it could even look at an individual black man, the son of a single
mother from the projects, and see that he's actually _not_ like others whom
those phrases would describe, because of other factors the algorithm takes
into account.

> If you accept that government should want to avoid a segregated society,
> where some groups of people form a permanent underclass, you should avoid
> any algorithm that results in an increased differential arrest rate for
> those groups, even if that arrest rate is warranted by actual crimes
> committed.

That statement implies that we should avoid the algorithm 'arrest anyone who
has committed a crime, and no-one else,' because that algorithm will
_necessarily_ result in increased differential arrest rates. On the contrary,
I think that algorithm is obviously ideal, and thus any heuristic which leads
to rejecting it should itself be rejected.

~~~
pc86
> _a random black man is 6.2 times as likely to be a murderer as a random
> white man_

But I bet the likelihood of a random man or random person to be a murder is so
low that "6.2 times" doesn't really tell you much about the underlying data.

------
andrewaylett
I like the proposal from the EU that automated decisions with a material
impact must firstly come with a justification -- so the system must be able to
tell you _why_ it came out with the answer it gave -- and must have the right
of appeal to a human.

The implementation is the difficult bit, of course, but as a principle, I
appreciate the ability to sanity-check outputs that currently lack
transparency.

~~~
dasboth
Interpretability is going to be a huge area of research in machine learning,
what with the advent of deep learning techniques. It's hard enough explaining
the output of a random forest, what about a deep net with 100 layers? In some
cases it doesn't matter, e.g. you generally don't care why Amazon thinks you
should buy book A over book B, but in instances where someone's prison
sentence is the output, it will be vital.

------
Smerity
As someone who does machine learning, this absolutely terrifies me. The
"capstone project" of determining someone's probability of committing a crime
by their 18th birthday is beyond ridiculous. Either the author of the article
hyped it to the extreme (for the love of everything that's holy, stop freaking
hyping machine learning) or the statistician is stark raving mad.

The fact that he does this for free is also concerning, primarily as I doubt
this has any level of auditing behind it. The only thing I agree with him on
is that black box models are even worse as they have even worse audit issues.
Given the complexities in making these predictions and the potentially life
long impact they might have, there is such a desperately strong need for these
systems to have audit guarantees. It's noted that he supposedly shares the
code for his systems - if so, I'd love to see it? Is it just shared with the
relevant governmental departments who likely have no ability to audit such
models? Has it been audited?

Would you trust mission critical code that didn't have some level of unit
testing? Some level of code review? No? Then why would you potentially
destructively change someone's life based on that same level of quality?

> "[How risk scores are impacted by race] has not been analyzed yet," she
> said. "However, it needs to be noted that parole is very different than
> sentencing. The board is not determining guilt or innocence. We are looking
> at risk."

What? Seriously? Not analyzed? The other worrying assumption is that it isn't
used in sentencing. People have a tendency to seek out and misuse information
even if they're told not to. This was specifically noted in another article on
the misuse of Compas, the black box system. Deciding on parole also doesn't
mean you can avoid analyzing bias. If you're denying parole for specific
people algorithmically, that can still be insanely destructive.

> Berk readily acknowledges this as a concern, then quickly dismisses it. Race
> isn’t an input in any of his systems, and he says his own research has shown
> his algorithms produce similar risk scores regardless of race.

There are so many proxies for race within the feature set. It's touched on
lightly in the article - location, number of arrests, etc - but it gets even
more complex when you allow a sufficiently complex machine learning model
access to "innocuous" features. Specific ML systems ("deep") can infer hidden
variables such as race. Even location is a brilliant proxy for race as seen in
redlining[1]. It does appear from his publications that they're shallow models
- namely random forests, logistic regression, and boosting[2][3][4].

FOR THE LOVE OF EVERYTHING THAT'S HOLY STOP THROWING MACHINE LEARNING AT
EVERYTHING. Think it through. Please. Please please please. I am a big
believer that machine learning can enable wonderful things - but it could also
enable a destructive feedback loop in so many systems.

Resume screening, credit card applications, parole risk classification, ...
This is just the tip of the iceberg of potential misuses for machine learning.

Edit: I am literally physically feeling ill. He uses logistic regression,
random forests, boosting ... standard machine learning algorithms. Fine. Okay
... but you now think the algorithms that might get you okay results on Kaggle
competitions can be used to predict a child's future crimes?!?! WTF. What. The
actual. ^^^^.

Anyone who even knows the hello world of machine learning would laugh at this
if the person saying it wasn't literally supplying information to governmental
agencies right now.

I wrote an article last week on "It's ML, not magic"[5] but I didn't think I'd
need to cover this level of stupidity.

[1]:
[https://en.wikipedia.org/wiki/Redlining](https://en.wikipedia.org/wiki/Redlining)

[2]:
[https://books.google.com/books/about/Criminal_Justice_Foreca...](https://books.google.com/books/about/Criminal_Justice_Forecasts_of_Risk.html?id=Jrlb6Or8YisC&printsec=frontcover&source=kp_read_button&hl=en#v=onepage&q&f=false)

[3]: [https://www.semanticscholar.org/paper/Developing-a-
Practical...](https://www.semanticscholar.org/paper/Developing-a-Practical-
Forecasting-Screener-for-Berk-He/6999981067428dafadd10aa736e4b5c293f89823)

[4]: [https://www.semanticscholar.org/paper/Algorithmic-
criminolog...](https://www.semanticscholar.org/paper/Algorithmic-criminology-
Berk/226defcf96d30cf0a17c6caafd60457c9411f458)

[5]:
[http://smerity.com/articles/2016/ml_not_magic.html](http://smerity.com/articles/2016/ml_not_magic.html)

~~~
Houshalter
> There are so many proxies for race within the feature set.

Yeah but, so what? Surely you don't believe race is a strong predictor after
controlling for all the hundred other things? Algorithms are not prejudiced
and it has no reason to use racial information when so much other data is
available.

Even if somehow race was a strong predictor of crime in and of itself, so
what? Lets say economic status correlates with race, and it uses that as a
proxy. It still isn't treating a poor white person different than a poor black
person.

And if it makes a prediction like "poor people are twice as likely to commit a
crime", well it's objectively true based on the data. Its not treating the
group of poor people unfairly. They really are more likely to commit crime.

~~~
pdkl95
> Surely you don't believe race is a strong predictor after controlling for
> all the hundred other things?

It can be if you select the right data, algorithms, and analysis method.

> Algorithms are not prejudiced

That's correct. However, the _selection_ of algorithm and input data is
heavily biased. You're acting like there is some sort of formula that is
automagically available for any particular social question, with unbiased and
error free input data. In reality, data is often biased and a proxy for
prejudice.

> It still isn't treating a poor white person different than a poor black
> person.

I suggest spending a lot more time exploring how people actually use available
tools. You seem aware of how humans bring biased judgment, but you are
assuming that the _creation_ of an algorithmic tool and _use_ of that tool in
practice will somehow be free of that same human bias? Adding a complex
algorithm makes it easy to _hide_ prejudice; it doesn't do much to eliminate
prejudice.

> Its not treating the group of poor people unfairly.

Yes, it is. The entire point of this type of tool is to create a new way we
can _pre-judge_ someone based not on their individual behavior, but on a
separate group of people that happens to share an arbitrary set of attributes
and behaviors.

The problems of racism, sexism, and other types of prejudice don't go away
when you target a more complicated set of people. You're still pre-judging
people based on group association instead of treating them as an individual.

------
ccvannorman
>Risk scores, generated by algorithms, are an increasingly common factor in
sentencing. Computers crunch data—arrests, type of crime committed, and
demographic information—and a risk rating is generated. The idea is to create
a guide that’s less likely to be subject to unconscious biases, the mood of a
judge, or other human shortcomings. Similar tools are used to decide which
blocks police officers should patrol, where to put inmates in prison, and who
to let out on parole.

So, eventually a robot police officer will arrest someone for having the wrong
profile.

>Berk wants to predict at the moment of birth whether people will commit a
crime by their 18th birthday, based on factors such as environment and the
history of a new child’s parents. This would be almost impossible in the U.S.,
given that much of a person’s biographical information is spread out across
many agencies and subject to many restrictions. He’s not sure if it’s possible
in Norway, either, and he acknowledges he also hasn’t completely thought
through how best to use such information.

So, we're not sure how dangerous this will be, or how Minority Report
thoughtcrime will work, but we're damned sure we want it, because it's the
future and careers will be made?

This is a very scary trend in the U.S. Eventually, if you're born poor/bad
childhood, you will have even _less_ of a chance of making it.

~~~
sliverstorm
On the bright side, if we can pinpoint at risk children with high accuracy, we
can also _help_ them make it.

Like the attempts to deradicalize individuals at very high risk of flying of
to syria, rather than arresting them.

~~~
seanmcdirmid
We can already pinpoint at risk children with high accuracy, just check out
any inner city ghetto. We just lack the caring needed to do anything about the
root cause (e.g. and mostly poverty) that causes kids to go bad later. It
isn't a mystery.

------
kriro
Predictive policing is quite the buzz word these days. IBM (via SPSS) is one
of the big players in the field. The most common use case is burglary, I
suspect because that's somewhat easy (and also directly actionable). You
rarely find other use cases in academic papers (well I only browsed the
literature a couple of times preparing for related projects).

The basic idea is sending more police patrols to areas that are identified as
high thread and thus using your available resources more efficiently. The
focus in that area is more on objects/areas than on individuals so you don't
try to predict who's a criminal but rather where they'll strike. It sounds
like a good enough idea in theory but at least in Germany I know that research
projects for predictive policing will be scaled down due to privacy concerns
even if the prediction is only area and not person based (noteworthy that
that's usually mentioned by the police as a reason why they won't participate
in the research). I'm not completely sure and only talked to a couple of state
police research people but quite often the data also involves social media in
some way and that's the major problem from what I can tell.

~~~
jmngomes
> IBM (via SPSS) is one of the big players in the field

They have been pitching "crime prediction" since at least 2010 with no real
results so far...

~~~
antisthenes
The results are that IBM's consulting arm is flourishing from all the crime
prediction contracts.

------
peterbonney
Here's something I really dislike about all the coverage I've seen about these
"risk assessment algorithms": There is absolutely no discussion of the
magnitude of the distinctions between classifications. Is "low risk" supposed
to be (say) 0.01% likelihood of committing another crime and "high risk" (say)
90%? Or is "low risk" (say) 1% vs. "high risk" of (say) 3%?

Having worked on human some predictive modeling of "bad" human events (loan
defaults) my gut says it's more like the latter than the former, because
prediction of low-frequency human events is _really_ hard, and, well, they're
by definition infrequent. If that suspicion is right, then the signal-noise
ratio is probably too poor to even consider using them in sentencing, and
that's _without_ considering the issues of bias in the training data, etc.

But there is never enough detail provided (on either side of the debate) for
me to make an informed assessment. It's just a lot of optimism on one side and
pessimism on the other. I'd really love to see some concrete, testable claims
without having to dive down a rabbit hole to find them.

------
conjectures
What is Berk's model? How well does it do across different risk bands? What
variables are fed into it in the states where it is used? How does prediction
success vary across types of crimes, versus demographics within crime?

This article treats ML like a magic wand, which it isn't. There's not enough
information to make a judgement on whether the tools are performing well or
not, or whether that performance, or lack of it, is based on discrimination.

Where we do have information it is worrying:

"Race isn’t an input in any of his systems, and he says his own research has
shown his algorithms produce similar risk scores regardless of race."

What?!? The appropriate approach would be to include race as a variable, fit
the model, and then marginalise out race when providing risk predictions.
Confounding is mentioned but the explanation of how it is dealt with, without
doing the above isn't given - just a (most likely false) reassurance.

------
anupshinde
This is like machine introduced bias/racisim/castism... we need a new term for
that.. and its based on statistically induced pseudo-sciences many times
similar to astrology. This is the kind of AI everyone should be afraid of.

~~~
benkuykendall
I fail to see what is unscientific about stating conditional probabilities.
Astrology is unscientific because the orientation of the heavens is for the
most part independent of the observables of a person's life. But the inputs to
Berk's system clearly do effect the probability of committing crime. A
frequentist would say that a large group of people with this set of
characteristics would yield more or fewer criminals; a Bayesian would say we
have more knowledge about whether such an individual will commit future crime.
These are scientific conclusions. The question "how should we use this data"
is a question of ethics, not science.

~~~
eyelidlessness
> I fail to see what is unscientific about stating conditional probabilities.

1\. Create conditions which disadvantage and impoverish a segment of society.

2\. Refine those conditions for centuries, continually criminalizing the
margins that segment of society is forced to live in.

3\. Identify that many of the people in that segment of society are likely to
be identified as criminals.

4\. Pretend that you're doing math rather than reinforcing generations of
deleterious conditioning, completely ignoring the creation of those conditions
that led to the probabilities you're identifying.

And science can't be divorced from ethics. These are human pursuits.

~~~
vintermann
This looks like sloppy ML. But human judges do all those things already
(susbtitute "just applying the law" for "doing math" in 4) and they can't be
inspected - their brains are "closed source".

Sure, these humans can come up with wordy justifications for their decisions.
But there are plenty of intelligent people who employ their intelligence not
to arrive at a conclusion, but to justify the conclusion they already arrived
at. Legal professionals aren't merely capable of this, they're explicitly
trained to be experts at post-hoc justification.

And legal professionals basically ignore all criticism not coming from their
own professional class. They are rarely taught any kind of statistics in law
school. Nobody wants to discuss math in debate club - answers with hard right
and wrong answers are no fun to them.

Your pessimism wrt. modeling may be justified, but you're not nearly
pessimistic enough about people or the legal system.

~~~
eyelidlessness
I frankly don't understand your response. I described a list of despicable
things humans have done, and you're suggesting that I'm not pessimistic about
people.

------
acd
Would the following be common risk factors for a child becoming future
criminal? Would it not be cheaper for society to invest in this risk children
early on rather than dealing with their actions as an adult? Minority report.
What are your observations for risk factors? Has there been Social science any
interviews of prisoners and their background feed into classification engines?

Classification ideas: * Bad parents not raising their child * Living in a poor
neighbourhood with lots of crime * Going to a bad school * Parents who are
workaholics. * Single parent * Parent who is in jail

------
nl
For those who haven't read it, Propublica article on this is even better (and
scarier): [https://www.propublica.org/article/machine-bias-risk-
assessm...](https://www.propublica.org/article/machine-bias-risk-assessments-
in-criminal-sentencing)

------
phazelift
It might be a better idea to first train computers to define criminality
objectively, because most people cannot.

~~~
jobigoud
> most people cannot

So why do we hold computers to higher standards than humans? Either it's OK to
not being able to define criminality objectively and in that case algorithms
shouldn't be disqualified for this very reason, or it's not OK, but in that
case humans should not be allowed to do the job either.

~~~
phazelift
Convicting people for criminal behaviour based on a subjective definition of
it, is what we already do wrong. Way too many innocent people end up being
punished or killed, which I expect to be (objectively) a criminal act in
itself. So, I thought we better first have a tool that solves that, instead of
a tool that amplifies it.

------
Digit-Al
I find this really interesting. I think what most people seem to be missing is
the wider social context. Think about this. If you exclude white collar
financial crime, pre-meditated murder, and organised crime - most other crimes
are committed by the socially disadvantaged. So, if the algorithm identifies
an area where crime is more likely to be committed, instead of being narrow
minded and just putting more police there to arrest people, why not instead
try to institute programs to raise the socioeconomic status of the area?

People are just concentrating on the crime aspect, but most crime is just a
symptom of social inequality.

~~~
darpa_escapee
The American thing to do is to cry "personal responsibility" and treat the
symptoms with jail time, fines and a lengthened rap sheet.

Suggesting that we treat the cause suggests we all have responsibilities as
members of communities to ensure no one is in a place where crime might make
sense despite the consequences.

------
mc32
The main question should be, like with autonomous vehicles, is does this
system perform better than people (however you want to qualify that)? If so,
it's better than what we have.

Second, even if it's proven better (fewer false positives, less unduly biased
results) it can be improved continuously.

There is a danger in that people may not like the results because if we take
this and diffuse it, has the potential to shape people's behavior in
unintended ways (gaming), on the other hand this system has the potential for
objectivity when identifying white collar crime, that is surfacing it better.

------
justaaron
gee, what could possibly go wrong, Mr. Phrenologist?

SOMEONE seems to have viewed Minority Report as an Utopia rather than
Dystopia, I'm afraid.

------
DisgustingRobot
I'm curious how good an algorithm would be at identifying future white collar
criminals. What would the risk factors be for things like insider trading,
political corruption, or other common crimes?

------
liberal_arts
consider the (fictional) possibility that an AI will be

" actively measuring the populace's mental states, personalities, and the
probability that individuals will commit crimes "
[https://en.wikipedia.org/wiki/Psycho-
Pass](https://en.wikipedia.org/wiki/Psycho-Pass)

AI may be worth the trade-off if violent crime can be almost eliminated.

or consider (non-fiction): body-language/facial detection at airports; what if
they actually start catching terrorists?

~~~
vegabook
There is a school of thought that says some crime is necessary for a healthy,
functioning society. Personally, while I would hate to be the victim of
violent crime (obviously), I actually do agree that cities with very low crime
levels are often stultifyingly uncharismatic.

[https://www.d.umn.edu/~bmork/2111/readings/durkheimrules.htm](https://www.d.umn.edu/~bmork/2111/readings/durkheimrules.htm)

------
jamesrom
What is bloomberg's MO with these near unreadable articles?

~~~
puddintane
Still rocking paint like it's 1995 - Bloomberg

The color choice is just yikes!

Did anyone else get reminded of Futurama - Law and Oracle (S06E16 / E104)?

[1]
[https://en.wikipedia.org/wiki/Law_and_Oracle](https://en.wikipedia.org/wiki/Law_and_Oracle)

I do wonder if this type of technology is something we should slowly approach
due to the very nature of the outcome of sentencing. We already incarcerate a
lot of innocent people and I truly wonder if this is something we should tread
lightly on.

~~~
nickles
Law and Oracle is an adaptation of PKD's short story "Minority Report" in
which the Pre-Cog (short for pre-cognition) department polices based on
glimpses of future crime. Orwell's "1984" explored the similar, but distinct,
notion of 'thoughtcrime'. Both works examine the implications of such policing
methods and are certainly worth reading.

~~~
Pamar
To be honest I read 1984 maybe six or seven times and I do not remeber
"prediction" to be one of the main themes. See
[https://en.wikipedia.org/wiki/Thoughtcrime](https://en.wikipedia.org/wiki/Thoughtcrime)

The oppressive regime of 1984 does not use math or computers to look for
"suspects". We might argue this was because Orwell had no idea about these
methods (book was written in 1948, after all) but personally I doubt he would
have used this because advocating decisions to an impersonal algorithm would
make the bad guys slightly less bad: in the novel the main baddie says
something like "do you want to see the future of Human Race? Think of a boot
stomping on a human face, forever...".

I.e. power for them is a end in itself, and the only way to use it is to make
someone else suffer. I don't see any place for "impersonal algorithms to
better adjudicate anything" in this.

------
niels_olson
Can someone just go ahead and inject a blink tag so we can get the full 1994
experience? Oh, my retina...

~~~
cloudjacker
[http://www.wonder-
tonic.com/geocitiesizer/content.php?theme=...](http://www.wonder-
tonic.com/geocitiesizer/content.php?theme=1&music=2&url=http://www.bloomberg.com/features/2016-richard-
berk-future-crime/)

surprisingly more readable

~~~
cJ0th
On a serious note: They original link looks okayish when you switch to article
mode in Firefox.

------
Dr_tldr
I know, it's almost as if they don't consider you the sole and undisputed
arbiter of the limits of technology in creating social policy. What a bunch of
psychopaths!

~~~
eyelidlessness
If you have (created!) a job that closely resembles a work of dystopian
fiction, laughing that off is absolutely lacking in human empathy. That's not
even the first problem with this line of work, but since you're also laughing
off the problem, it deserves a rebuttal.

If I said to you that I was going to create a network of surveillance devices
that also serves as mindless entertainment and routinely broadcasts faith
routines that non-participants will be punished for, and you told me that
sounds like something out of 1984, and I told you were paranoid, you'd think
I'm mad.

And the advance of technology unhindered is not a universal good. Algorithms
only have better judgment than humans according to the constraints they were
assigned. If there's a role for automation in criminal justice, that role must
be constantly questioned and adjusted for human need, just as the role of
human intervention should be. Because it's all human intervention.

~~~
ionwake
What is a "faith routine" ? Thanks

~~~
Jtsummers
In context, faith routines would be things like, in the book _1984_ , the Two
Minutes Hate. In reality, it might be a (imlicitly mandatory, if not
explicitly) routine such as pledging allegiance to a flag, or mandatory
participation in a moment of prayer, or something similar.

