
The hidden reality of ridiculously complicated algorithms - jonbaer
https://www.the-tls.co.uk/articles/public/ridiculously-complicated-algorithms/
======
culturedsystems
This is a pretty frustrating read. Interpretability of automatic decision
making processes is a genuine, important issue, but I'm not sure this issue is
best explained by a writer who, as far as I can tell, has made no effort to
understand basic questions like what an algorithm is. The generalized
bafflement of the author is just mystification, a vague sense of spooky
machines we can't understand, which doesn't help us understand the specific
difficulties in understanding the decision making process that depend on
processing huge quantities of data.

~~~
evrydayhustling
Charitably, I think the writing here underscores how much scarier the spread
of automation is for people who have never been involved in writing an
algorithm (or better yet a complex system). Even if you're well educated in
another field, there's a really giant gap between being told 100 times in
popular press how information systems work (or don't), versus being part of
trying to make one work.

Personally I don't see much difference between how opaque the world of
automation is and how opaque law, or government bureaucracy, or media
production, or any other human industry is from the point of view of a non-
expert. But perhaps for many people, it is easier to trust long chains of
human processing than long chains of machinery.

~~~
yters
At least with people there is someone you could possibly ask and get an
answer. No such luck with a black box algorithm.

~~~
evrydayhustling
Strongly disagree with this line of argument. Almost all human systems are
intentionally opaque about some choices or have unplanned consequences you
can't get a clear answer about. And, algorithms can be studied in the same
ways that we use to critically evaluate human systems -- and sometimes better.

Why does your local paper run stories featuring some businesses but not
others? Why do college admissions tend to overrepresent some populations and
underrepresent others? Why are women paid less on average for the same titles?
Why do prosecutors disproportionately bring charges against minority suspects
with equivalent weight of evidence?

You can't get answers about those human decision processes just by asking - it
requires research, competing theories, and sometimes legal action forcing
people to share info.

Explainability of decisions isn't about how much tech is used... It's about
how open a system's operators are to sharing info, and how much effort
everyone is willing to put into interpreting it.

~~~
perl4ever
"Almost all human systems are intentionally opaque"

I question the word "intentionally". I would suggest that human systems evolve
to be opaque, because that helps them be resilient and survive. If a human
system is truly understandable by anyone, then it can be destroyed by some of
its members/adversaries and sooner or later someone will.

~~~
evrydayhustling
Fair point about evolution - what I meant about intention is that some people
involved in a decision could tell you more than they are willing to. I think
it's worth distinguishing that from systems feeling opaque because of
consequences nobody operating them was aware of.

------
jtbayly
If I was the guy who risked my job (or more) to get this out there, I’d be
ticked that this is all the journalist managed to produce with it.

~~~
tedunangst
Yeah, I'm finding it hard to believe this guy will never work in tech for the
rest of his life.

~~~
growt
But he shared the terrible secrets that there are "functions" and "libraries"
and different colored words.

------
rahimnathwani
"And there it was: a white screen with instructions neatly arranged in a
series of boxes."

If seeing a Jupyter notebook impresses the writer so much, it might blow their
mind if they found out that the screen is made up of millions of dots. And
that each dot is actually made up of three different-coloured dots.

~~~
jzwinck
I too realized the author was looking at Jupyter when I saw In[3] and so on.
Absurd. The author is conflating a typewriter with a story. And starting from
such ignorance, I wasn't about to find out where it ended up.

------
infinity0
The algorithms are opaque but how's that different from a human's own brain?
We don't have a very good view into a judge's own life experiences from which
his own decisions are based either. At least with an algorithm we have a
chance of seeing how it really works, in principle. All the data is recorded
and the code is available.

> The reality is that if the algorithm looks like it’s doing the job that it’s
> supposed to do, and people aren’t complaining, then there isn’t much
> incentive to really comb through all those instructions and those layers of
> abstracted code to work out what is happening.”

If a judge is performing well are you going to run an expensive background
audit on him, or open up his brain to try to see how he "really works"?

The difference is that algorithms today are practically run by centralised
organisations that control it plus the data, rather than having lots of humans
all slightly different making their own decisions in a semi-centralised[^]
distributed fashion. But the opaqueness is nothing new.

[^] because most human organisations today are hierarchical, but at least most
people have some sense of self-agency.

~~~
coldtea
> _The algorithms are opaque but how 's that different from a human's own
> brain?_

Algorithms can blindly follow through into an obviously illogical decision in
ways that a human brain doesn't (and when it does, we consider that person
unfit for a job as well).

~~~
rs86
Algorithms cannot be illogical. They just can't.

~~~
coldtea
Actually they very much can. They just can. Except if you believe in some kind
of soul / body duality. Else human-like thinking is something that can be
achieved with the right (and properly complex) computing program, including
consciousness. And that would include anything logical or illogical that we're
capable of.

In other words, if you do believe that humans can be illogical, then who told
you the human thinking doesn't come from an algorithm itself (of e.g. NN
nature)?

Not that this is not needed, of course, because what I wrote is much easier to
achieve. I wrote that "Algorithms can blindly follow through into an obviously
illogical decision". That doesn't even need the algorithm to be illogical
itself, but just to arrive at an illogical (obviously for us, humans)
decision. It can still arrive at that logically. E.g. if I program a simple
algorithm that "if the car sees a zebra crossing, go full steam ahead" the
algorithm will do just that, whereas most humans would question it.

------
jimnotgym
I have to challenge the statements of Jure Leskovik

> They often have no feedback on whether they made the right decision

Of course they do. They will see people stood in front of them that have
violated their parole all of the time. They can read their case notes and use
the information to adjust future decisions. This of course is after a career
of standing the other side of the bar as an advocate/attorney of those people,
because that is what you have to do to become a judge in the first place.

> and there is no knowledge-sharing between the judges.

Except the whole legal system in the US is built on common-law. Literally the
law being applied in a common way to everyone. Literally by knowledge sharing.
If the esteemed academic is suggesting that parole decisions are being shared
less well than judgements the fix seems to be rather simpler than ML.

The idea that a ML algorithm is somehow less biased than a judge is of course
just as absurd. A ML researcher is just as capable of using race, for
instance, as a characteristic. They are just as capable of seeing a
correlation without understanding the complex underlying causes.

I choose the human judge.

------
andrewaylett
One of the interesting -- or at least, I hope it'll prove to be interesting --
effects of GDPR is that we have the right to an explanation of important
automated decisions and also the right to human review.

Before anyone comments that ML models are notoriously difficult to explain,
the whole point of the regulation is that opaque models are intolerable. Also,
[https://github.com/slundberg/shap](https://github.com/slundberg/shap) looks
like it could be very useful.

------
gjstein
> “We need”, Jure said emphatically, “to step up and come up with the means to
> evaluate – vet – algorithms in unbiased ways. We need to be able to
> interpret and explain their decisions. We don’t want an optimal algorithm.
> We want one simple enough that an expert can look at it and say nothing
> crazy is happening here.

As someone who works with machine learning, I find this statement rather
misleading. The researcher quoted here has a very strong bias about what it
means for an algorithm to be "optimal". A system cannot both "be optimal" and
also "biased" in a way that the designers don't like: such a system is indeed
not optimal. The dialog in the machine learning community has increasingly
been about how we might structure these systems in a way that they are
unbiased; it's a shame the researcher (and the author of this article) seem to
think that "simplicity" is the _only_ option.

(In case anyone is interested, I recent wrote a blog post to this effect:
[http://www.cachestocaches.com/2018/7/bias-and-
ai/](http://www.cachestocaches.com/2018/7/bias-and-ai/) )

~~~
slv77
Every model designer is going to come to the table with his own set of lenses
and biases based on this own life experiences. These biases are incorporated
into our models based on the metrics that we optimize for and the features
that we incorporate into our models.

It may comforting to think that all our biases are “baked out” of the models
during the training process but if the researcher had been a black man do you
believe that they would have arrived at identical models?

Even if the model designer found ways to compensate for his own bias the world
that these models operate in are inherently biased. Training data may be
subtly biased in ways that are difficult to detect. A model trained on
recidivism is likely to be biased if, for example, a lower class black male or
upper class white women have different risks of being arrested for the same
crime.

Assuming the models themselves are unbiased as trained they may create bias
with subtle errors when pushed into production or due to second order effects,
feedback loops or errors or manipulation of data. For example during the
credit bubble entire industries found ways to raise borrowes FICO scores.

There is also the inherent corporate bias that the researcher pointed out
which the bias of the almighty dollar. Compensating for bias is expensive and
may even impact salability if an unbiased model doesn’t “gel” with the
expectation of customers who are biased. For example a judge may feel pressure
of homogenize his decisions or abdicate his responsibility by rubber stamping
the models decisions.

Currently we compensate for this in a democratic society with checks and
balances. What are the checks and balances for a computer algorithm that is a
trade secret for a for-profit corporate entity?

~~~
canhascodez
If the system being modeled is abstract, you might get away with claiming that
it is unbiased. If the system being modeled has very much to do with the real
world, the modeler is forced to make assumptions about said putative "real"
world, because the real world is inconveniently large and complex to fit in
memory. However, one of the problems that societies and maintenance
programmers encounter is that in the long run these assumptions are invalid.

We've pretty much all read the article "Myths Programmers Believe about
Names", and various snowclones. We still write dumb name validation code every
day, because there actually isn't a perfect way to do that, and we have to
ship _something_. Even when the concepts are perfectly executed, the
fundamental assumptions of the model may change. Having separate classes for
"tool" and "weapon" may make sense one day, and the next day you need to write
the game _Clue_. Even if you're working with such boring concepts as
filtration, you may have to adjust your model when you discover that water is
compressible.

All models are wrong, in many ways, both subtle and overt, both currently and
in the future. Some may be useful. I'm not sure that there's any solution to
these problems except more and better models though.

------
ggm
Anyone remember back in the 1980s when a UK medical school tried expert
systems to do admissions and built in systematic bias?

Oh wait.. didn't that just happen again in Japan?

And here we have the tls discussing heuristics to sentencing which "out
performs" the judge...

~~~
temporallobe
We have "expert systems" now in medicine. It's called the insurance EHR.
Doctors punch in a diagnosis, system spits back what treatment and/or
medication the insurance company is willing to pay for. Even if the doctor
completely disagrees, that becomes the de facto solution for the patient.

------
sabujp
[https://cs.stanford.edu/people/jure/](https://cs.stanford.edu/people/jure/)

------
malmsteen
The terrifying reality of articles with way too much words 8x

