
Deep learning is creating computer systems we don't fully understand - jonbaer
http://www.theverge.com/2016/7/12/12158238/first-click-deep-learning-algorithmic-black-boxes
======
vardump
Wait. Are there still computer systems in 2016 we can fully understand? Except
maybe _some_ simpler embedded systems.

Last computer system I can somewhat fully understand is OCS Commodore Amiga.
CPU, chipset and software.

~~~
jimbokun
Unequivocally yes.

Obviously no one has a full mental model in their head of everything going on
in their phone (for example) at a given moment in time. But when it really
matters, and there is a specific question to answer, and a programmer with
experience writing the kind of software in question has full access to all of
the source code, its possible to get an answer to why a program behaved in a
particular way.

I'm a programmer, and this is what I do every day.

And in those rare cases where knowledge of the specific hardware is required,
there are experts who understand the hardware and can answer the relevant
questions.

So for machine learning approaches, what is the equivalent of the "source
code"? What resource is there to examine to determine how the system will
respond to a given set of inputs, and why? Evidently, there are not clear
answers, at least for some machine learning approaches.

~~~
vardump
To start with, most computers are running about 20-30 microcontrollers. No
source code, most of the time even no access to the firmware.

Phones have a bit fewer, but I think you'll find at least about 10.

A simple cheap SD memory card has an embedded core, usually ARM7TDMI. Even
most lithium batteries have an integrated microcontroller.

I'm also a programmer, I do bare metal embedded, kernel drivers and such. I
often don't have anyone else to ask from, except occasionally hardware guys.
They're also often stumped.

~~~
jimbokun
Interesting.

So in a high profile case, like the Tesla Autopilot fatality, if the behavior
was traced back to a specific micro-controller, it might be just as hard to
determine why the system behaved as it did, as with a complex neural network?
Even with a company's reputation at stake, it's not possible to hire a
consultant with a deep understanding of obscure micro controllers?

I only deal with the source code level, don't really have experience digging
into the hardware level.

~~~
vardump
We rely on many layers of more or less leaky abstractions.

Components have always a set of intended behavior and some, often unknown or
poorly understood, set of unintended behavior.

It's very important to understand that abstractions mask _both_ intended and
unintended behavior.

You deal with unintended side the best you can, with sanity checks and so on.
But some of it inevitably leaks through.

Those outcomes that result from multiple systems being _individually_ in good
state but collectively causing undesired behavior are often very tricky to
understand. On the flip side, when you do understand them, you tend to gain
new surprising insights into the whole system.

Also timing makes understanding hard. When it comes to a complex network of
systems, it's hard to concurrently reproduce the state where the effect
occurs.

> So in a high profile case, like the Tesla Autopilot fatality, if the
> behavior was traced back to a specific micro-controller, it might be just as
> hard to determine why the system behaved as it did, as with a complex neural
> network?

If Tesla has good, exhaustive, event logging facilities, a "black box", I
guess they'll be able to figure out what happened by analyzing all of the
available data with the systems knowledge they have.

Anyways, _if_ it's not systematic, why should Tesla autopilot mistake be
handled so differently from a human being making a mistake?

You can't analyze why a human being caused an accident either. The best you
can usually do is to develop scenarios "theories" based on the available
evidence, but that's it.

~~~
jamescostian
I don't think analyzing why a human caused an accident is important either.
However, being able to fix the bug is very important. If a programmer can
identify the problem in their code, they can fix it. But how can one identify
the problem in a neural network and fix it? At least right now, understanding
an (actually useful) neural network might as well be impossible

------
xkcd-sucks
Maybe we'll see the emergence of a profession kind of like clinical psychology
for diagnosing strange behavior in trained pattern-recognizing machines :)

~~~
gambler
Asimov thought there will be robopsychologists. Maybe he was right. Although,
his robots had a common foundation laid out by the Three Laws of Robotics, and
they didn't consume gigabytes of data to learn.

~~~
daveguy
> and they didn't consume gigabytes of data to learn.

Didn't they though? I'm pretty sure a couple of cameras for vision would mean
they consume a couple of gigabytes rather quickly. That doesn't include sound,
motor stress feedback and any other included sensors.

Asimov's robots would have consumed massively more data to learn. Of course,
it wouldn't have required the data be curated and classified by humans.

------
charlieflowers
I think that's exactly the point. Evolution produced our brains, and no one
understands them yet. But evolution was able to produce them because it cares
about success-testing, not understanding.

I think we limit ourselves quite a bit in the mainstream software industry by
insisting that everything be hand-crafted by humans and understandable. (Not
to mention how short we fall of making software "understandable").

~~~
TeMPOraL
By "limiting" ourselves this way we can go forward _much_ faster than
evolution can. We create designs in hours what would take evolution millions
of years to achieve. Also, it's a mistake to treat products of evolution as
some kind of gold standard. It's literally _worst design that could still
work_. We can do better, but we do it via different means.

Another thing is that understanding is a key component of _trust_. Animals
come to mind here. We can't _understand_ their firmware per se; by observing
them over thousands of years, both passively and actively (through
domestication, training regimens, etc.), we've learned to expect some things
from them but not others. So I can be sure that e.g. my cat won't kill me in
my sleep. But I can absolutely not trust it to go somewhere when I want it to
go there, or to stay in place when I want it to stay, or even to eat food when
I give it. We want to write software that will bear much greater, more
important responsibilities than any animal in history. So we need to make them
more understandable than animals if we're to ever trust them with those
responsibilities.

~~~
charlieflowers
Sure, I agree. Evolution did not take us from horse and buggy to planes and
autonomous cars in 100-ish years.

But when we push up against the limits of the problems we can solve with
software, it's usually because it has outgrown our ability to understand it.

That's exactly what is happening with this new burst of progress in machine
learning. We're "teaching" the computer to do things, and we've let go of the
ability to directly understand how the computer "represents its learnings."

------
strebler
I have heard Dr. Andrew Ng point out the "well known" fact (in the computer
vision community) that nobody, including Dr. David Lowe himself, really
understands why SIFT works either. So at least for those of us in the computer
vision industry, this isn't maybe such a surprise.

------
PhantomGremlin
Minor nit ...

Was anyone else bothered by the phrase "show their working"?

That's the kind of thing that grates on me because it's just slightly wrong.
When I grew up it was always "show your work", not "working".

Is this a regional thing?

~~~
AnimalMuppet
I think it's British rather than US English, but I'm not sure.

~~~
PeCaN
Do Brits use gerunds other than ‘working’ as nouns or is it specific to that
case? Either way, that's really weird.

------
blazespin
This sounds far fetch and sci fi, but intelligible machines are particularly
of importance in avoiding machines becoming self aware unbeknownst.

However, I think it's also possible that we may never be able to create
machines that are self aware as people are unless we do allow them to build
without our understanding of them. There could be a recursive relationship in
that an intelligence is not capable of understanding itself.

~~~
danieldk
I think self-awareness is still so far off that it is not the most pressing
problem.

The more serious problem is that once we make life-changing choices using
machine learning models, the models should be _just_ and _accountable_.
Obviously, these are vague terms, but in terms of machine learning models they
can have concrete meanings:

* _Just_ : choices should be made on what we consider (morally) relevant inputs.

* _Accountable_ : choices should be traceable to inputs and parameters (or eventually training data).

E.g. suppose that we find drones that stun people with a weapon acceptable.
Such a drone would be just if it decides to stun a person because they have a
weapon. Such a drone would be unjust if it decides to stun a person because
they have a particular skin color [1].

Now, suppose that such a drone stuns a person robbing a supermarket with a
fake weapon. Regardless of whether we consider this to be just, we probably
want to know why the drone misrecognized the gun as real (the model should be
accountable).

[1] There may be a non-uniform prior distribution _p(weapon_use|race)_. But I
hope we all agree that using such prior distributions would be absolutely
unfair towards the individual.

~~~
blazespin
Well, we hope it is. I don't think we can say that with much confidence though
unless we understand what the computers are doing. Take some massive
datacenter with petabyte sized deep networks doing neural networks. Think of
the dimensionality of those matricies. Do you really know what's going on??

~~~
danieldk
_Think of the dimensionality of those matricies. Do you really know what 's
going on?_

As someone who uses deep networks for language processing, I completely agree.
IMHO this implies that we should be careful making life-changing decisions
using such models.

Though, I am not sure whether legislation at this point is useful either. The
lack of understanding of new technology in legislative branches typically
leads to bad laws.

------
scotty79
> The humans looked straight to the windows to answer this question, but the
> machines, for some reason, looked at the beds instead.

Maybe the machine wasn't sure if most important thing in the question is
"window" or "cover", so it looked at a bed looking for bed cover?

~~~
tensor
Given that humans all recognize that there is a bed, it's almost certainly the
case that humans also looked at the bed when first scanning the image. We
don't require a human show the breakdown of it's visual processing cortex for
every decision, why do we want to require it of a machine?

------
apathy
Or, overfitting strikes again. Except that here the overfit is buried in a
bunch of layers. Still -- eventually, especially by reversing the models to
generate instead of discriminate, it is possible to be more transparent about
what's going on.

I have to agree that the EU, despite normally being a bunch of martinets, is
likely getting it right here. It's too easy to construct and overfitted DNN
and have it produce bizarre biases that affect patients, payrolls, or patents.
There needs to be an appropriate level of transparency baked in as a
precondition for licensable IP, IMHO.

------
keithnoizu
I think there's an existing analogy here to service animals. You can train a
sheep dog to manage the herd, you can train a seeing eye dog to stop it's
owner at the lights. You can improve their training, etc. but you'll never
really ever know why they sometimes don't follow the expected script.

That isn't necessarily a bad thing. You get more done with them than you would
with out them.

~~~
phee
The analogy is with anything with a brain, not just animals. Take a poorly
trained human and put him in a decision-making position, he will probably make
bad decisions. But no amount of training will guarantee you that he won't make
any. The learning data set is just too big to be sure you won't have any bias.

------
btbuildem
Isn't that the essence of this particular approach though? To "learn" a
solution in ways us humans would not conceive of? It's a clear consequence
that we cannot easily follow that path of "reasoning" easily.

~~~
guidopallemans
This is the main difference between Machine Learning and Data Mining; in
Machine Learning it's the Machine that learns, in Data Mining it's you (and
the machine).

------
ninjakeyboard
"LIKE GOOD STUDENTS, COMPUTERS NEED TO BE ABLE TO SHOW THEIR WORKING"

Is working used as a noun here or is that a typo? Just never heard that use of
the word working. Like ya, you can see I'm working vs you can see my working.

~~~
ajb
A noun. Normally used to refer to the intermediate steps produced by a student
doing mathematics.

~~~
PeCaN
I've never seen that before. Using a gerund as a noun seems very
unconventional and incorrect almost by definition.

Simply ‘work’ reads much better—is it really frequently used as ‘working’ in
mathematics‽

~~~
creamyhorror
Here in Singapore, "working" is the standard term used in schools to refer to
the written steps in solving a maths problem (as ajb said). I guess this means
it's simply BrE (which Singapore English is a cousin of). I know Americans use
"work" for the same purpose instead.

> _Using a gerund as a noun seems very unconventional and incorrect almost by
> definition._

In English, gerunds _are_ nouns (by definition). In _" his singing was
terrible"_, _singing_ is a noun.

Furthermore, gerunds do often become used to refer to physical objects/results
of actions: "writing", "reading" (e.g. readings for a class), "drawing",
"booking" (of a ticket), "crossing" (on a road), "wiring", "etching", and
others. Using "working" to refer to the written steps of working out a maths
problem is just another example in that vein.

~~~
PeCaN
You're right, I got something backwards there! I suppose it's not that
unnatural on second thought.

Your examples are good, though the only ones that are moderately common in
American English seem to be “drawing” and “crossing”.

------
nurettin
I think there is an economic threshold for the need to know something versus
the effort to reverse engineer and find out exactly how that thing happened.
(don't xkcd this, randall!)

------
quadrature
Does the Tesla autopilot system use deep learning ?. I assumed it was all
control systems and tracking.

~~~
Fishman343
I think the author has read that Tesla use "deep learning" to make their auto
pilot more accurate and presumed that that means a deep learning algorithm is
running in the car.

What I would guess what they do is run deep learning algorithms on the data
produced by the car's systems back at their HQ to design more efficient and
complete systems, which they then update all of their cars to use.

------
PepeGomez
The problem is I think that we don't actually know how to program an AI.
Neural networks only sidestep the problem by being easier to program by trial
and error, but it's nothing more than emulating a different architecture and
brute forcing its programming.

~~~
ghgfhgfhn
But you see the process of training is a solution of the problem "how to
program AI". For a machine to be intelligent it has to form it's own model of
the world. This model is necessarily somewhat opaque because it includes
representations which the machine has created for itself.

An intelligent machine that is interpretable in a straightforward manner is an
impossibility, it is a contradiction in terms.

You can teach the machine to answer questions about it's internal state in
order to analyse it's function or you can apply your own mind to painstakingly
deduce it's operation. But I contend you cannot have a generalised rubric for
determining it's operation.

------
iofj
Why would anyone use machine learning if not to create systems they don't
understand ? That's the whole point of doing it. Not having to understand
complex system to understand -or even predict- them is a very good thing
indeed. I don't know anything about how to mathematically analyze pixel values
to read text, but I can create such a formula using machine learning. After
creating the formula I know nothing more about it. So the same person can
create programs to read text, transcribe audio, predict loan repayment, ...

And when it comes to the bias in the data - bias in the system thing, well,
same thing goes for human judgement. Another way of putting this would be
garbage in-garbage out. Humans might be able to tell you of bias in the
inputs, rather than just using it, but this ability rapidly deteriorates when
the dimensionality of the inputs rises. You can tell if there's something
weird about combinations of 3 numbers, but you can't tell if there is
something off about 150 number sequences.

~~~
abtinf
Along these lines, I think these classes of algorithms might have much broader
applications and could possibly solve problems that we don't even have good
definitions for yet. I'm particularly curious to see how the field develops
new techniques for systems that start off with poor quality results but evolve
with new data to produce better results.

btw, I vouched for and upvoted the parent. It makes an excellent point and,
frankly, I'm baffled by the downvotes/dead status.

