
The Dark Secret at the Heart of AI - mitchelldeacon9
https://www.technologyreview.com/s/604087/the-dark-secret-at-the-heart-of-ai/
======
AndrewKemendo
This is not a secret and is a well known fact of deep learning.

It's also a misnomer to think that we don't know how they work. We know
exactly how they work. What is difficult and arguably will become more
difficult as we go along, is investigating the exact parameter pathway that
resulted in a specific output of a DNN.

Ok, well that's actually way easier than understanding how a person came to
their decision/conclusion because we can actually evaluate the string of
inputs and the training history of a net. If you ask someone how they made a
decision, it will be mostly FUD because we don't understand granularly how
humans reason.

Looking at outputs and moving the baseline as they improve on a tangible,
quantifiable task is really the only way to do this in the long run IMO.

I think people want a decision tree visualization that includes biases from
previous datasets etc... and I doubt if that is tractable as we inch closer to
general AI.

I'd go further to say, that even if we did have that, it would still be
intractable to evaluate. We'd need an AI to verify AI explanations - so at
some point you just have to trust it.

------
beamatronic
Perhaps the solution in this case is to put the AI in sort of a sandbox, where
it has control up to a point. But there are independent anti-collision systems
in place with override power.

