Hacker News new | past | comments | ask | show | jobs | submit login
Why are we using black box models in AI when we don’t need to? (2019) (mit.edu)
351 points by Hooke 19 days ago | hide | past | web | favorite | 193 comments



I think it's important to note that human pattern recognition is basically black-box as well.

We can't "explain" how we recognize a particular person's face, for example.

Robust pattern recognition based on 100+'s of factors is just inherently black box.

Even when people make decisions, they're generally a weighted combination of a bunch of "feelings". We attempt to explain a kind of simple, logical rationalization afterwards, but psychologists have shown that this is often a bunch of post-hoc fiction.

Black box doesn't necessarily mean bad. I think the relevant question is: how do we ensure machine learning is trained and verified in ways that don't encode bias, and only used with "intended" inputs, so a model isn't being trusted to make predictions on things it wasn't trained for?

And also: when do we want to decide to use alternatives like decision trees instead -- because despite being less accurate, they can have designers who can be held legally accountable and liable, which can be more important in certain situations?


I think it's also important to remember a lot of human models were abandoned because of the inherent bias and black-boxed nature of them (feelings, subjectivity, etc.).

This is one of the reasons the scientific method blossomed: objectivity, rigor, transparency, reproducibility, etc. Black boxes can lead to bad decisions because it's difficult to question the process leading to decisions or highlight flaws in conclusions. When a model is developed with more rigor, it can be openly critiqued.

Instead, we have models running across such massive datasets with so many degrees of freedom that we have no feasible way of isolating problems when we see or suspect certain conclusions are amiss. Instead, we throw more data at it or train the model around those edge cases.

To be clear, I'm not saying ANN/DNN models are bad, just that we need to understand what we're getting into when we use them and recognize effects that unknown error bounds may cause.

If, when the model fails to correctly classify a new data point, the result is your photo editing tool can't remove a background person properly... then so be it, no harm no foul. If the result is that the algorithm classified a face incorrectly with high certainty and lands someone in prison as a result (we're not there, yet) then we need to understand our method has potential unknown flaws and we should proceed carefully before jumping to conclusions.


> I think it's also important to remember a lot of human models were abandoned because of the inherent bias

The industry needs to stop misusing the term bias this way. Virtually every attempt to find this supposed human bias has failed. Latest public example was Amazon and hiring[1]

Bias is the tendency to consistently mis-classify towards a certain class or tendency to consistently to over or under-estimate.

Somehow the term has been hijacked to mean 'discriminate on factors that are politically incorrect'. You can have a super racist model that's bias free, and most models blinded to protected factors are in fact statistically biased.

It's not constructive to conflate actual bias with political incorrectness.

Operational decision making, whether AI or human or statistical, faces an inherent trilemma: it's impossible to simultaneously treat everyone the same way, to have a useful model, and to have uniformly distributed 'bias'-free outcomes. At best a model can strive to achieve two of these factors.

See: https://www.youtube.com/watch?v=Zn7oWIhFffs

[1] https://www.reuters.com/article/us-amazon-com-jobs-automatio...


Hold up. Bias is being used in two different manners because it has two differed meanings. When you are using bias in industry you are talking about a minor mathematical factor added to a learning rate. When we talk about human bias, we aren't. The term was never hijacked, it just has multiple meanings.


I’m not sure if he was using any of those known definitions of bias though. I’m not sure if I would define bias as the consistent act of mis-classification


Yes, mathematicians mean something different and specific when using the word bias. The average non-expert is not misusing the word. They are using the word to express a different -- and far more popular -- meaning.

Neither is wrong, but insisting that a naming clash carries any substantive significance on an underlying issue is just silly. Similarly, insisting that nonmathematicians should stop using a certain word unless they use it how mathematicians use it is a tad ridiculous.

Of anything, it's more reasonable for mathematicians to change their language. After all, their intended meaning is far less commonly understood.


If an algorithm is clearly sorting on irrelevant criterion, especially a black box algorithm, we normally assume it's a bug. It's not reasonable to reverse that, assume the code is incapable of being mistaken and say that obviously irrelevant criterion are somehow correct in an unknown way.

Amazon's problem is a bug, they even describe it's nature. And given how flawed their recommendation algorithms are, it's especially unreasonable to assume this one is infallible.

So that linked Reuters does not show a failure to find bias, if anything it shows a design error.


You say "obviously irrelevant criterion"

Data says criterion is an eigenvalue and no matter how hard amazon tried to blind the solution to that eigenvalue, the ML system kept finding ways to infer it because it was that strongly correlated with the fitness function.

This is the difference between political newspeak '''bias''' and actual bias. Amazon scrapped the model despite it performing just fine and being bias-free, because it kept finding ways to discriminate on a protected attribute which is a PR nightmare in the age of political outrage cancel culture. It's fine to explicitly decide that some attributes should not be discriminated upon, but this comes with a cost either in terms of model utility or in terms of discrimination against other demographics. There's no way around this. In designing operational decision making systems, one must explicitly choose a victim demographic or not to implement the system at all. There's no everyone-wins scenario.

The harm of the newspeak version of '''bias''' is that it misleads people into thinking that making system inputs uniform somehow makes it bias-free when the opposite is typically true. Worse, it creates the impression that some kind of magical bias-free system can exist where everyone is treated fairly, even though we've formally demonstrated that to be false.

No amount of white-boxing or model transparency will get around this trilemma. The sooner the industry comes to grips with it and learns to explicitly wield it when required, the better.


>No amount of white-boxing or model transparency will get around this trilemma. The sooner the industry comes to grips with it and learns to explicitly wield it when required, the better.

Agreed. The optima of multiple criterion will essentially never intersect.

But for Amazon, there is no evidence the tool was accurately selecting the best candidates. They themselves never said it was. After all, altering word choices in a trivial way dramatically affects ranking. On the points you mention, why should we assume their data was relevant or their fitness function even doing what they thought? If they were naive enough, they could just be building something that predicts what the staff of an e-commerce monopoly in a parallel universe will look like.

The most likely story is that they failed at what they were doing. Part of that failure happened to be controversial and so got unwanted attention. I would guess there were quite a few incredible correlations the tool "discovered" that did not get to press.

At any rate, their recommendation engine is more important and has been worked on longer yet it is conspicuously flawed. When their recommendation tool inspires awe then maybe we could take their recruiting engine seriously enough to imagine it has found deep socio-psychological-genetic secrets.


Supervised learning algorithms assume that the input data are iid of the future. This is not valid in most of the real applications. The observation that we see men more than women in programming does not necessarily generalize to the future. That's why online learning provides an exploitation vs exploration mechanism to minimize the bias in the hindsight. In many applications, people just forgot about this simple strategy and blame the bias caused by supervised learning to the black box model.

Of course, black box AI itself is not the right solution. As more and more cross domain multitask settings emerge, open box AI will gradually take off. It is about compositional capability like functor and monad in functional language. Explanable or not is just a communication problem which is parallel to the ultimate intelligence problem. It is very possible that human intelligence is bounded.


That is the fascinating thing about consciousness.

How do we "know" we see red and not green or vice versa? How do we "know" we are feeling heat or coldness? We just do. We feel different things in different, yet recognizable ways.

And these "feelings" are multi-dimensional, feeling heat and feeling color are not on the same axis. Red and Green seems to be on different points of the same axis and hot and cold are points on some different axis, other dimension.

How many different dimensions of "feelings" are there? Note they are not the same as different senses. There can be many different feelings associated with things we see and recognize that way: colors, and shapes and lightness vs. darkness etc.


Physical sensation seems simple tbf. Any robot can sense temperature or color gradient.

It's the decision making that's complicated.

To make a decision on whether it would like chocolate ice cream or not (before trying it) a brain will take all the data gathered over time, cross reference it with related data (does chocolate taste good alone? \ did I like it? \ It looked rather inedible. \ was it high cocoa content or not? \ Chocolate yoghurt seems similar, but I did not like it. \ I do like cold, but only when it's hot outside. \ My friend says chocolate ice cream tastes good. \ My other friend says it tastes bad. \ trust or discard opinions? and much more).

All within minutes, seconds even.

An AGI would need an absolutely massive database of knowledge similar to the one acquired by a human over the first 20-30 years of their life, if it's to be truly general.

It would also need bias correction, and maybe 2-5 other AGIs to form a consensus before making any decision properly.

And even that would not be enough to fit in with humans - if you want an AGI to make the objectively right decisions, it will have to ignore many human feelings/biases.

If you want it to fit in with people, it will have to make "mistakes", i.e. ineffective decisions.


> Physical sensation seems simple tbf. Any robot can sense temperature or color gradient.

It can measure them and process it as a number, but when does sensing come into play? Does a red sheet of paper recorded by a simple camera evoke the same "feeling" of redness inside the camera that it evokes in a human being?

The flippant answer is "of course not, the camera has no consciousness/perception". The question is then how and when this feeling of "redness" gets created and what are the necessary conditions for it to happen.


By sensing, I mean just that, reading input. You mean reasoning/decision, I guess.

That would come from learned data, experience, imo. A child who's never touched a red hot electric stove, for example, would not have any bias towards it. No fear, no love, unless they previously interacted with one. They would try and get close, watch it, smell it, touch it, to learn more.

The most interesting part of consciousness is how does one decide based on incomplete information? If you need to learn more, where do you find the raw information? It seems to be done subconsciously, some people have a higher affinity for learning/fact-finding than others. But all people are pre-wired to learn from others, distributed computing works best heh.

I guess you're asking the same question as me, where is this "programming" and how is it created? It's not just raw experience, it seems to be genetic/evolutionary. A sort of basic firmware to bootstrap further learning.

I find it fascinating, it would seem the brain never stops processing the data it acquires. During the day, and during the night, it always runs learning jobs.


I agree with your thinking on learning, so I have nothing to add, but it doesn't sound like we were referring to the same thing. I'll try rephrasing, though I find this subject extremely hard to communicate effectively.

When you're looking at a red sheet of paper, reading the input in the form of a signal is not the only thing happening. The signal is definitely read and passed along to other brain circuitry for processing, but somehow another thing happens: the experience of red (which is what the other poster called "feeling"). This experience of red is fundamentally different from the experience of green or blue or from the complete absence of looking. This experience is what I was referring to.

This experience seems to happen without any prior knowledge or exposure to the color red. The first time you stumble upon red light, this experience of red arises. You can tell red from green by the marked difference in experiences, but you cannot explain this difference in words to someone who hasn't experienced red or green themselves.

How does this experience get created? Does any sensor (such as a camera) have such experiences? Why not?


I don’t know if you’re asking rhetorically or not, but the honest answer is we don’t know yet. Those sensory experiences are called “qualia” in philosophy and cognitive science and they are not well understood yet. You might enjoy the paper “What is it like to be a bat?” https://warwick.ac.uk/fac/cross_fac/iatl/study/ugmodules/hum...


Yes, it was rhetorical in an attempt to get the point across. I try to refrain from naming qualia when first explaining it. I've read Nagel's paper, but I haven't thought about it for some time so thanks for reminding me.


I usually see the Turing Test as a thought experiment for showing the irrelevance of qualia for AI, among other philosophical objections (e.g. Chinese Room).

These feelings you're describing are called qualia (https://en.wikipedia.org/wiki/Qualia), in case you're not familiar with the term.


> When a model is developed with more rigor, it can be openly critiqued.

Yes. And the models developed like this haven't solved the problems the current black box models have.

> we have models running across such massive datasets with so many degrees of freedom that we have no feasible way of isolating problems when we see or suspect certain conclusions are amiss. Instead, we throw more data at it or train the model around those edge cases.

There are ways being studied to check for sensitivity to parameters, biases, etc. But in the end, reality is difficult and there's no way of dealing with that, or "getting the right answer" every time


Sure, human pattern recognition is poorly understood, but it's a system that evolved over millions of years. We should expect more from something we ourselves designed.

The question of why we would choose a black box model over something interpretable that works equally or nearly as well still needs to be asked, especially if there is a misconception of black box models working better than interpretable models due to current trends and the hype cycle.

> held legally accountable

I think this misses the point. It is not about accountability but about getting the right results. The right question is where we need an interpretable model to eliminate biases or otherwise satisfy ethical requirements. Often you can't know whether a decision was ethical without knowing how it was reached. Would you accept a trial verdict given by AI? What about a lending decision, or a job promotion? If you need to justify a decision and not merely reach a decision, you need something interpretable.


>human pattern recognition is ,,, a system that evolved over millions of years. We should expect more from something we ourselves designed.

Why? Human intelligence evolved over millions of years. And we're struggling to keep our changes to the world from killing us.


Because we design it, so we have the ability to choose its characteristics. The current prioritization of 1% improvements in artificial benchmarks over all else is a disease that is actively retarding progress in the entire field.

Political systems are also designed. If human behavior is in danger of killing us, we must look to the political systems we have created and solve the problems there.


>how do we ensure machine learning is trained and verified in ways that don't encode bias

Or, more commonly, people work to ensure their machine learning is trained and limited to not specifically notice any patterns of fact that would be politically unacceptable to notice and make use of, even if it was rational and useful for the task at hand.

The status quo moral system of our society rests on a collection of objectively false beliefs about the physical world. Most humans know enough to avoid noticing these facts because of the social cost and possible societal negative impacts of shaking those moral foundations. It's not even necessarily wrong, since agreed-upon lies can do a lot to reduce conflict and cruelty. But machines need to be specifically trained to not notice these facts. You need to teach the machine not to notice the emperor's nudity.

And what's even better is that this'll never get acknowledged in the literature. It's one of those self-hiding facts which exists if you're willing to notice it yourself, but which no authority will ever tell you (I love these, wish there was a word for them).


If we were able to provide perfect datasets you would be correct. As it is our datasets are imperfect and ML models are only as good as the training data.


> I think it's important to note that human pattern recognition is basically black-box as well.

Mammalian pattern recognition is fairly well understood. We have a good understanding of the information path from the eye to V1 in most model animals. So sure are we of this pathway that we've successfully implanted ships into blind patients and restored sight (results vary a LOT, though).

I know this may not fit pattern recognition in a super-philosophical sense, but damned if it's not a ways down that road. We're literally shocking a person's brain so that they 'see' again (!!!). How V1 feeds into the rest of the brain is VERY active research and has had a lot of successes, but we're a long ways from a 'black box' these days.

Honestly, the auditory pathways are a lot better understood, as they seem to be 'older' and specific tones are represented physically in the mammalian brain (tonotopy). As such we know a lot better (relative to sight, smell, taste, etc) how tones are encoded and how they elicit specific recognitions and responses (this work is NOT straightforward).

In Neuroscience/Biochem, we are getting a LOT closer, but yes, we are not there yet by any means. The 'black box' idea of the mammalian brain is not going to last another 500 years, likely. We're marching along with good time as a species.


Knowing how to input data to the visual cortex doesn't mean that we know how it works. It is the same with neural networks, we know how to pass data to them and how they do the math, but we can't trace back its reasoning to know why it comes to the conclusions it does.


We have a very very good idea of the mammalian eye-V1 pathway. It's not a straightforward path and describing it in a HN comment is not going to be useful to anyone. Explaining the complex, but well known, pathway is a job best done by other people than myself. We are well aware of what 'data' processing the mammalian brain does at nearly every synapse, and it is a LOT. Right form the cones and rods, data is being highly processed (synapses are VERY noisy, as it turns out). So while more research always comes up with more questions, we are fairly confident about the eye-V1 pathway (contrast that to other organs, like the vestibules, where we're fairly in the dark still).

That said, wait, what? We can't do the back-propagation calculus in a NN? When has that been true and for how long? I thought it was fairly straightforward to know the weights of the connections of the nodes in your network. It's just a tensor you grep for, right?


Black boxness isn't a property of the thing, it's a property of our understanding of the thing. Just because we don't understand the how pattern recognition in humans works, doesn't mean the process has no internal structure, or that we couldn't potentially understand it in the future.


Right, and currently we understand neither human vision, nor CNN vision - they are both black boxes. I don't think anyone said they must remain black boxes forever.

Edit: I guess you were referring to this:

> Robust pattern recognition based on 100+'s of factors is just inherently black box.

I think he meant " ... given our current knowledge."


Yes, that’s true. But human decisions have accountability - algorithms don’t have accountability. If entities who deploy algorithms would be legally required to take accountability for algorithmic decisions, there would be no issue with black box algorithms.


I agree in spirit, but a lot of human behaviors, derived from decisions, don’t have consequences. Did Nestle face consequences for their decisions to promote formula, etc?


Let's not confuse accountability of decision making entity with consequences faced by the accountability. I was referring to the former, not the latter. The former is a case for law, the latter is a subject of law enforcement. In the case of Nestlé, you have a clear assigned accountable entity. We don't have that with algorithmic decision making (yet) where organizations wash off their hands saying "Oops! It was an algorithm, we didn't do anything!" Weak law enforcement is not a reason to not have proper laws in place.


Why would organizations not be responsible for decisions made by an algorithm? After all they are still the entity who decided to use that algorithm, and to execute the decision.

Sure, they say "Oops! It was an algorithm, we didn't do anything!", but is that any different than management saying "Oops! It was just a rogue employee, we didn't do anything!" For anything sufficiently consequential or systematic the second excuse doesn't work, so why should the first excuse work?


> We don't have that with algorithmic decision making (yet) where organizations wash off their hands saying "Oops! It was an algorithm, we didn't do anything!"

That's what individuals within companies do all the time.

You're onto something, form a limited liability company for your AIs and hire them as contractors. There, a clear assigned accountable entity :)


> I think it's important to note that human pattern recognition is basically black-box as well.

Agreed. But as you note, even though humans are basically black boxes we can ask them questions in order to find out how they came to a particular conclusion. (How reliable the answers to these questions are is of course a different matter.)

So maybe we don't necessarily need fully interpretable models but simply a way to ask black-box models specific questions about their state, e.g., "To what degree does a person's age influence the output?".


> But as you note, even though humans are basically black boxes we can ask them questions in order to find out how they came to a particular conclusion.

No, you can't. If somebody treats you with suspicion, it's because of a combination of their news intake, their culture, local events, what their friends and family would think, the way you present yourself, and many other factors. You can always ask somebody to state their reason as a simple "if-then" statement, and they can make one up on the spot, but it'll be so oversimplified that it's basically a lie.

> So maybe we don't necessarily need fully interpretable models but simply a way to ask black-box models specific questions about their state, e.g., "To what degree does a person's age influence the output?".

You can already do that. Just change that number in the input and see how the output changes. To that extent, even the most black box AI model is more transparent than human decision making.


> You can always ask somebody to state their reason as a simple "if-then" statement, and they can make one up on the spot, but it'll be so oversimplified that it's basically a lie.

Well, I guess it depends on how self-aware a person is. I think the biggest danger is trying to rationally explain your decision when in fact it was based mostly on your feelings, in which case I agree that the explanation is "basically a lie". One needs to be honest when something is not based on a fact but on a feeling to prevent pointless discussions. (If I hold an opinion based on a feeling then you cannot convince me that I am wrong by giving me facts.)

> You can already do that. Just change that number in the input and see how the output changes.

Makes sense. But I guess transparent models would still be generally preferable because you can fully understand how the output is produced, whereas in black-box models you might have to ask quite a lot of questions to get a feeling for it, but even then you can't be sure that you have a full understanding of it.


> even though humans are basically black boxes we can ask them questions in order to find out how they came to a particular conclusion. (How reliable the answers to these questions are is of course a different matter.)

You punctuate the second sentence as though it were of secondary importance. But in many cases, we have little ability to figure out how we came to a conclusion, while being much better at fabricating plausible and politically acceptable answers. I put it to you that having questions answered with plausible fabrications, is actually a significantly worse situation than not yet being able to ask the questions at all. At least in the latter situation, we know what we need to be working on.


> But in many cases, we have little ability to figure out how we came to a conclusion, while being much better at fabricating plausible and politically acceptable answers.

Hmm, too me it feels like I can explain the reasons why I came to a conclusion in many (but certainly not all) cases. You "just" need to clearly identify your feelings and emotions and separate them from your rational arguments.

Anyway, these are our own shortcomings and of course don't have to be adopted by any artificially built black-box model.


This is actually false. Early psychologists thought that humans could know everything about their own brain's processes, but observational psychology proved most of their assumptions wrong. It's called introspection, and it's almost always not indicative of a person's actual inner workings.


Expert systems from the 70s and 80s had a capability similar to this. They could explain how they reached a conclusion by reporting the rules they used to get there. The problem was that interviewing experts and coming up with a huge rules database was a ton of work and didn't scale very well.

Maybe the next direction in AI will be to bridge the gap between expert systems and black box models?


> we can ask them questions in order to find out how they came to a particular conclusion.

For the most basic cognitive tasks, we typically can't.

If you show someone a picture of a cat and a dog, they can easily recognize which is which.

If you then ask, "but how do you know?", I don't think most people could say anything useful. If it looks like a cat and it walks like a cat...


I was more thinking about higher level reasoning. But yes, a lot of the lower level stuff just appears as thoughts in my mind seemingly out of nowhere.


Introspectable Higher level reasoning is irrelevant. Any introspectable higher level reasoning a person can do has already been automated for efficiency.


Wait, what? Where has this been automated?


Humans however, in addition to the immediate perception, can also more slowly recognize objects with reasoning and thus explain their perception.


But that explanation is likely full of cognitive bias, and all sorts of subjective experience.

Epistemology and phenomenology are philosophical fields that deal with this. By no means is it a solved problem.


Object recognition is firmly in the realm of Neuroscience. We've had a good idea what the eye-V1 path is doing in mammals for a few decades now. To the point where we are implanting chips into the brain to (crudely) re-establish vision in blind patients. Though the research is very much on-going, it is being better understood with each passing year.


Respectfully, unless I'm not understanding what you mean by this, that seems very far from object recognition. You're replicating the data stream that would normally be coming from the optic nerve/retina, but that's not the same thing as understanding how the data of an object's image relates recognition of that object. If I've never seen the entire Mona Lisa before (but maybe have seen portions of it), your proposed system could force me to see the painting but it can't force me to recognize it as something I have seen, nor force me to know that it is a famous painting by Leonardo da Vinci currently on display at the Louvre.


I don't think you are understanding. But Bio isn't meant to be easy.

At each synapse a significant amount of data processing occurs, right from the rod and cone cells in the retina and at nearly each synapse along the way. 'Data' is more-ish conserved into V1 where all hell seems to break loose, path-wise. After V1 and in nearly the entire cortex, these data are distributed in unique and very complicated ways throughout the brain. Most of the mammalian eye-V1 pathway is well understood, but that process is too much for a comment on HN. But, I want to stress, we have a very very good idea of what is going on.

That said, the research seems to indicate (highly debated still) that there are specific synapses that encode 'the left side of grandma's nose in dim lighting from 10 feet away' and other many other things. Again, I cannot stress enough that such 'grandma' synapses are not firm science and there is a LOT of research still ongoing. But the evidence seems to be pointing that way at this time. Such cells are fed this information from V1 and, likely, a lot of other places. When such 'grandma' cells fire, they then send that signal out to other cells they may be connected to. Such a system is likely replicated many times all over the brain; there are many copies of 'grandma' cells and they are wired up in many different ways. The physical location of 'grandma' cells in highly unique, if they exist at all. Again, research is very much on-going.


One more benefit: an interpretable model could at least give some hint when the model is not working as intended (e.g. debugging).


Sometimes I have the feeling that this is exactly why so many people prefer black-box modeling. You are not going to be criticised and you don't have to do all the hard work of software development.


It's true that humans are black boxes but there are whitebox techniques such as the scientific method and algorithms that are justified step by step.

I think the article tries to compare black box learned algorithms to explicit algorithms. Humans being a blackbox is more of a philosophical issue from that perspective.


> Even when people make decisions, they're generally a weighted combination of a bunch of "feelings". We attempt to explain a kind of simple, logical rationalization afterwards, but psychologists have shown that this is often a bunch of post-hoc fiction.

Isn’t the first sentence just an example of the second?


It's not. The point is that the explanation given does not correspond to the process through which the decision was truly reached. That is to say, we give incorrect explanations of our own decisions.

That's not the same question as whether there were feelings involved, or whether the process was conscious.


"The point is that the explanation given does not correspond to the process through which the decision was truly reached."

Sure, but that doesn't mean "a weighted combination of feelings" is an accurate description of the true process. That's tendentious and speculative. What I think we can be sure of is that there are parts of our decision making that we are not consciously aware of, but precisely because of that, one should be cautious of assuming how it works even if you're not doing it by reflex. Psychological explanations are the rationalizations that people make after the fact.


Parent wasn't saying that the weighting process was rational. The weights are also feelings.


As long as we don't know what the weighting process is, we shouldn't make unfounded statements about how it works. You're asserting that it exists, but nobody knows anything about what it is. It's a program that you haven't read the source code for.


At this point we seem to be reasoning by analogy. We'd do better to just study the fact of the matter. Psychologists do not tend to discuss this topic in terms of weightings.

https://www.patheos.com/blogs/tippling/2013/11/14/post-hoc-r...

http://www.theunconsciousconsumer.com/consumer-psychology/20... (regrettably thin on citations)

https://academic.oup.com/scan/article/6/4/460/1648209

https://en.wikipedia.org/wiki/Rationalization_(psychology)


I'm not reasoning by analogy; I'm disputing that it's useful.

If you look at the physical world with objects that have relationships and characteristics, and describe a mental world that also has discrete entities that interact in designated ways, then I think you are reasoning by (unfounded) analogy.


> Even when people make decisions, they're generally a weighted combination of a bunch of "feelings". We attempt to explain a kind of simple, logical rationalization afterwards, but psychologists have shown that this is often a bunch of post-hoc fiction.

Does it matter? If a brain counstructs post-hoc argument for given decision and the argument is defensible against other arguments, then the argument itself is good enough substantiation of the decision, even if it is different from the actual decision process.


> because despite being less accurate, they can have designers who can be held legally accountable and liable

Are you implying you cannot be held accountable for black box models?


An anecdote, but this is a criticism being aimed at YouTube currently for it's demonitisation scheme.

The argument being made is that even though the demonitisation of certain classifications of content is immoral (e.g. demonitising any and all LGBT content, even when family friendly), it is tacitly being allowed to happen due to YouTube's stance of "the algorithm" making the decision, not them.

Which may lead then to have a position of "supporting" LGBT as a company while simultaneously being able to demonitise said content, which could hurt their business in actively hostile to LGBT countries.

This isn't proven, but it's an example of how it could be used to absolve yourself of blame and point it at an inscrutable system.


I believe this is correct. Assume a black box model fails. To the point that the author is removed from the system (e.g. executed). How do you transition accountability to another party in a reasonable way?


I'm not following. If the author is gone, then who decided to use the model? Who decided which safeguards were put in place in case the model failed? Liability is a chain.

AI systems aren't activating themselves, they're being used because a chain of people are authorizing them based on promises made by other people. So if an AI system in a courtroom wrongly jails people, you can still hold judges accountable for using the system. You can still hold manufacturers and businesses accountable for promising a degree of accuracy that their system couldn't meet.

In the same way, if my dog bites someone, and I get sued and argue, "animal motivation is really hard, we don't know why the dog chose to do that", I'm not going to win the case. The reason it happened is because I didn't leash my dog.

The reason the black box hurt someone is because an accountable human improperly used it or inaccurately advertised it.


This is tort law, the worst defined laws we have. Liability is far more complex that and usually governed by irrational humans or arbitrary bureaucratic rules contrived by powerful people protecting their interests.


But how do black boxes change that? It's still the same people operating under the same laws.

A few commenters here posit that without a black box, designers might be held accountable. If there's a system where software manufacturers would be held accountable for bugs, is any judge going to say, "well, the software is bugged, but they don't know how to fix the bugs, so they're off the hook"?

If there's proof that Twizzlers are poisonous and kill people, the Twizzlers manufacturer can't say, "but we don't know why, we just threw a bunch of chemicals in a vat at random without writing down the labels. So therefore, it's not our fault."

I don't know, can they? I'm not a lawyer, maybe everything in our legal system is way more horrifyingly broken than I assume.


There are payment systems based on human facial recognition. If the model failed, it means losing money. Apparently you can not deny liability because of using a black-box model.


Not all human thinking is the same. How you decide the amount to tip your waiter is not at all the same as how your doctor diagnoses you. And at we have a zillion years of evolution towards understanding and judging and conversing with each other's black boxes.


This is an incredibly salient point. Others have also pointed out that there are numerous applications in which black box models appear to offer significantly greater accuracy than interpretable models, bolstering the notion that this article is a bit overstated.

However, in this article and elsewhere Professor Rudin has cited compelling evidence of cases in which black box models have been demonstrated to be no more accurate than interpretable alternatives. I feel this fairly justifies the question in the title of the article. For example, based upon available evidence, it appears reasonable that some onus should lie on the creators and buyers of COMPAS (a proprietary black box recidivism model) to demonstrate COMPAS actually is more accurate than an interpretable baseline. While it may not be the case, as the article seems to suggest, that in all modeling cases there is an interpretable alternative with comparable accuracy, in cases which there is, there doesn't seem to be any justification for using a black box model.

On the matter of "human-style" interpretability, we are brought to the difference between "interpretability" and "explainability." Humans have a complex capacity for constructing explanations for the thoughts and actions of ourselves and others (among other things). As OP points out, a lot of famous psychological experiments by Kahneman and others have shown how much of our reasoning appears to be post-hoc, often biased, and often inaccurate (in other words, human explanations are not actually true transparent interpretations of our thoughts and actions). However, we humans do have a powerful capacity to evaluate and challenge the explanations presented by others, and we are able to reject bad explanations. For those interested, a great book on this topic is "The Enigma of Reason" by Mercier and Sperber (https://www.hup.harvard.edu/catalog.php?isbn=9780674237827), but the gist here is that we must understand that while explanations are not the same as transparent interpretability, they are still useful.

I would conjecture that at some level of complexity (which some predictive tasks like pixel-to-label image recognition seem to exhibit), true end-to-end interpretability is not possible -- the best we can do is to construct an explanation. However, two very important points should be observed when considering this conjecture: 1. (Professor Rudin's point in the article) In cases which are not too complex for interpretable models to achieve comparable accuracy to black-box models, we can and should use them, as they offer super-human transparency at no cost in accuracy. 2. Constructing no explanations (or bad explanations) is not the same as reaching the same level of semi-transparency that humans offer. If we want to use human interpretability as a benchmark, black box models with no explanations are not up to par.


Was the fact that objects fall down a "black box" before Newton came up with some equations?

I don't think its fair to describe CNN's and human vision as being black boxes in the same way.


The fact that objects fall down was well understood, obviously. What wasn't understood was the mechanism that makes this happen. Why did that apple fall down instead of up? Where exactly is down a round earth? Where is down when I stand next to the Himalaya? Why don't planets fall down?

In my eyes this is very similar to our lack of understanding about the inner workings of a CNN, or human vision. We understand the input and the output and their rough relationship, we just don't understand how the move from input to output happens.


I see it differently.

The distinction I would make is that even before Newton, people could make useful predictions about the trajectory of falling objects. My dog does it well, every time he catches a ball. He even seems to grok the effect of wind on a frisbee.

We have no way of predicting how a CNN will react to an input. GANs really illustrate this point. I think this is an important distinction.


> And also: when do we want to decide to use alternatives like decision trees instead -- because despite being less accurate, they can have designers who can be held legally accountable and liable, which can be more important in certain situations?

Decision trees are black box models and are highly accurate. (Think XGBoost, LightGBM, Random Forest, etc)

Not sure what point you are trying to make here.


Those models are ensembles of decision trees. A single tree is interpretable, but sensitive to overfitting.


They never got to the results of the competition and how the interpretable model did!

From https://www.fico.com/en/newsroom/fico-announces-winners-of-i...: "The team representing Duke University, which included Chaofan Chen, Kangcheng Lin, Cynthia Rudin, Yaron Shaposhnik, Sijia Wang and Tong Wang, received the FICO Recognition Award acknowledging their submission for going above and beyond expectations with a fully transparent global model and a user-friendly dashboard to allow users to explore the global model and its explanations. The Duke team took home $3,000."

Cynthia Rudin is one of the article's authors.


But how did their model compare against others? The article only mentions how their interpretable model compared against their own ML attempts


Their model didn't win. IBM's model won, based on actual metrics around useful insights.


The IBM team got $5,000 and the second place/honorable mention NYU got $2,000. So going by prize amounts, the Duke model was still pretty good.

IBM turned the model/paper into a toolkit: https://www.ibm.com/blogs/research/2019/08/ai-explainability... Their model seems to be a variant of decision trees that has a knob controlling how complicated the trees are.

And the evaluation was completely subjective, so there's not any meaning to the Duke people losing besides that the judges didn't like them.


> And the evaluation was completely subjective, so there's not any meaning to the Duke people losing besides that the judges didn't like them.

That's what you get if you use a black box for judging :-)


Reading "subjective" to mean "nonexistent" is a potentially big mistake.

Objectivity is more accurate, sure. The winner of an objective contest is always objectively better against objective criteria. But, objective criteria are generally narrow. This works well if one is either (a) seeking fundamental principles like in physics or (b) the narrow objective criteria is the definite goal.

In this area, we don't exactly know how to define narrow, objective goals & subsequent criteria. We can definee goalposts, but not goals. These are guesses at useful markers of success, useful to the larger goal of useful/novel ai.

Subjective goals have their own (massive problems), but since we can't objectively define the goals of ai research... we need to fall back on human subjectivity to define our subgoals.


All objective criteria are chosen, directly or indirectly, based on subjective criteria.


Also true. Subjectivity is unavoidable as long as we are relevant.. or so it seems circa 2020


The main advantage of a blackbox ML solution is shorter development time to a useful performance level. Creating a transparent, explainable solution typically takes more time, more work, and a higher level of expertise to get to the same performance level. If the problem is complicated and the cost of a mistake is low, then your best approach today is likely to be blackbox. If the cost of a mistake is high, you should not even be considering a blackbox approach (at least not without transparent safeguards). There is a large gray area between these two extremes that requires good engineering judgement to choose well.

I think we definitely need more R&D dedicated to creating easier-to-use, lower-cost approaches to transparent, explainable ML. There is way too much effort devoted to blackbox R&D today. Ultimately, transparent, explainable ML should almost always beat blackbox ML due to a better ability to find and resolve hidden problems, such as dataset biases, that may be holding back performance.


I would also add in that blackbox ML solutions are deceptively "simple" approaches. Like a miraj in the desert for software developers: No domain expertise neccessary - or even the counter narriatives may come in... "domain expertise holds you back - you need to be an outsider with a new method", where as many problems require domain expertise for a realistic understanding of the complexity of the problem-solution.

Other narriatives that can creep in "once we get enough data..." which for some problems may as well be never.

Also, for some systems pertubing the system changes the problem meaning your dataset and model may not reflect changes in a dyanmic / complex system.

Also, blackbox ML approaches struggle to combine data from multiple modalities which is often relevant to many real world problems (at least with the majority of algorithms which are realistically implementable off the shelf).


> The main advantage of a blackbox ML solution is shorter development time to a useful performance level.

I think that's kinda true but kinda false. Its true that deep learning often makes feature engineering moot. however, a lot of deep learning projects takes machine learning engineers and applied scientists along with a host of other support engineers + hardware costs ( I seen people at work say: I only used a 16 GPU instances over a couple weeks of training )

meanwhile, I consider gradient boosting fairly interpretable and they can get pretty close results with a lot less tweaking and training time. If you want to go full non-black box, logistic regression with l1 penalization and only a little bit of feature engineering often does really well, probably a lot less development time / cost compared to those high cost PHD research scientists.


> Its true that deep learning often makes feature engineering moot. however, a lot of deep learning projects takes machine learning engineers and applied scientists along with a host of other support engineers + hardware costs

DL's reduction of the required feature/model engineering is a big deal for difficult problem domains where the cost of a mistake is low. That doesn't mean you don't still benefit from adding development resources, it's just that the development cost/performance tradeoff is still typically better than with a similar-performing explainable solution. I hope this will change in the future.

> ..., logistic regression with l1 penalization and only a little bit of feature engineering often does really well,...

While I agree that Lasso is far more explainable than DL, its explainability rapidly degrades as the useful feature dimension increases and it requires significant feature engineering for good performance on difficult problems.


gradient boosted trees are black box


I would disagree with that statement. There are many ways at getting to what the model is doing, feature importances, SHAP, etc. It's not as clear as logistic regression or a single decision tree, but its not a black box either.


This isn't how it works, at all. You're going to get better results with 1,000 parameters than with 100 explainable ones. There's a limit to how much humans can understand. We use machine learning to surpass that limit


> You're going to get better results with 1,000 parameters than with 100 explainable ones.

This is not always true.

Many problems are modeled very well with less than 100 parameters and adding more is of little-to-no benefit.

Many problems are naturally hierarchical such that simple models can be combined to yield a large number of explainable parameters. If done well, this can result in a high-performing solution. Admittedly, this is usually harder than just applying a blackbox.

In critical applications, an explainable model with benign failure modes (even if it has worse overall performance), can be far preferable to a blackbox with wildly unpredictable failure modes. From a utility standpoint, the explainable results are better.

> There's a limit to how much humans can understand. We use machine learning to surpass that limit

We can also work to improve our ability to discover and understand. I think that holds far more promise than improving our ability to do things we don't understand.


My understanding is that interpretable models, especially for neural networks, are very far and away from state of the art performance. Work that e.g. tries to approximate neural nets with decision trees have yet to be applied to very large models [1].

Even in computer vision, which is where I think they've been most successful, the visualization techniques used seem more suggestive, then explanatory [2].

[1] http://www.shallowmind.co/jekyll/pixyll/2017/12/30/tree-regu... [2] https://distill.pub/2019/activation-atlas/


> Work that e.g. tries to approximate neural nets with decision trees have yet to be applied to very large models

Well, that and a decision tree isn't any more interpretable if its nodes are all filled with `Pixel[i,j] > Threshold` -- or is the idea that you would somehow extract logical predicates by tracing paths through the tree or glean interpretation from those predicates?


I agree that large decision trees are also hard to interpret. The idea in the above paper is that we regularize with respect to average tree path length, based on the intuition that smaller trees are more interpretable. (But the issue about pixel thresholds probably still stands.)


I think it's a real open question whether the interpretable models are actually worse, or merely worse in competition/benchmark problem sets. The more deep models I build, the more I'm convinced that behind every inscrutable parameter hides a certain amount of overfitting, with maybe a few notable exceptions. E.g., can you build a decision tree that's not obviously overfit but is also susceptible to adversarial perturbations?


It’s also a real open question whether any of the interpretable models are actually interpretable, or even if they are in any well-defined sense more interpretable than “black box” alternatives.

In practice the answer is a massive “no” so far. Some of the least interpretable models I’ve had the misfortune to deal with in practice are misspecified linear regression models, especially when non-linearities in the true covariate relationships causes linear models to give wildly misleading statistical significance outputs and classical model fitting leads to estimating coefficients of the wrong sign.

Real interpretability is not a property of the mechanism of the model, but rather consistent understanding of the data generating process.

Unfortunately, people like to conflate the mechanism of the model for some notion of “explainability” because it’s politically convenient and susceptible to arguments from authority (if you control the subjective standards of “explainability”).

If your model does not adequately predict the data generating process, then your model absolutely does not explain it or articulate its inner working.


> If your model does not adequately predict the data generating process, then your model absolutely does not explain it or articulate its inner working.

That's a very dynamicist viewpoint. I don't necessarily disagree.

However, in what sense to the prototypical deep learning models predict the data generating process?

I tend to agree that a lot of work with "interpretable" in the title is horseshit and misses the forest for the trees.


I agree that there's probably a good question about to what extent looking at benchmark sets is biasing our judgment.

But it's also unclear to me how to get a decision tree to perform as well on image recognition tasks the same way that a CNN does. (Of course, as you mention, the CNN will likely face adversarial examples.)


If it can be explained in terms of precise rules, that's called programming.


The distinction between ML and programming is mostly propaganda, in the sense that it's not flat out wrong but is mostly used to win money/power. It's not actually a helpful way of understanding... anything.


I think the main reason is that creating a so called "Black Box" model is faster and easier to some degree.

Its very similar to the argument kids make "Why do I have to learn how to do long division when I have a calculator?" or "Why do I need to show my work on the exam if I got the right answers?"

I tend to agree with the articles premise that any model being used in critical decision making should at the very least have a list of parameters and the weight given to each, no matter how long and complicated such a list might be.

I think eventually this will be the end result of most critical implementations of machine learning applications as they make their way through the courts, as I can't imagine a judge accepting the argument "The machine made a mistake, we are not sure why or the reason why is proprietary, but we are not responsible because the machine made the error".


A little off topic, but also not:

I once worked at data science consultancy. They worked on a project that they ended up using random forest. But they also ran a simple single layer net to validate the results, scoring a similar accuracy (it was like 70%). They gave a presentation on a Friday to the company (it was a regular show and tell demo slot).

At the end, I asked them if they double checked whether the net weights corresponded to the random forest and almost everyone in the room just looked at me like I was an idiot.

I ended up causing a heated discussion in the pub later on because I’d actually done something similar for my masters thesis (it was for modelling XOR operations on binary inputs).

The company in question is notoriously short term focussed, so it shouldn’t really have been a surprise that they didn’t even try to think about it.

And that’s kind of the problem. Short term speculative projects don’t care about interpretability. A director at some clothing firm doesn’t care about the super nerdy maths, they like getting their numbers.

But, on the flip side, Explaining loss functions and backprop is also a no go for the directors. If you can’t help them understand an algorithm in 2 sentences then they often don’t buy it. So then explainability becomes a big thing (hence why random forest was implemented and not the net).


Ofcourse the scary thing is when decisions are being made that will affect a persons life but will never make it to the courts. Like the example of predicting loan defaults or parole releases, those decisions are made with little to no explination. As much as I hate regulations I think this may be a good place to legislate in order to protect those wrongly affected.

It makes me think about the mess the derivatives markets made in 2008


It’s not like your bank manager used to give you a phone call to explain in depth why your loan was declined. Most of these situations were already impenetrable from a human standpoint.

Humans are far more ‘black box’ than neutral nets, we just happen to be able to construct plausible explanations in parallel.


No, they were impenetrable from the customer perspective, but either the human making the decision or the software being used can answer. With a black box model, nobody can answer.


From the post I responded to:

> the scary thing is when decisions are being made that will affect a persons life but will never make it to the courts. Like the example of predicting loan defaults or parole releases, those decisions are made with little to no explination.

I was pointing out that it’s already like this. Whether it’s a black box ML model or a capricious bank employee makes no difference if there’s no transparency either way.


Bank managers, however, can at least theoretically be hauled in to court and questioned.


That's true, but as mentioned above, the explanations humans give of how they reached their conclusions are parallel construction rather than genuine introspection. We really don't understand how our brains work, either when they're generating their conclusions or when they're generating an explanation for those conclusions.


Yes, I'm (somewhat) aware of the research. I'm unaware of any legal jurisdiction that recognizes it, let alone tries to address it somehow.

There's reality, there's legal reality, and then there's what legal reality does to reality.


The killer feature with "black box" models is that you can do illegal discrimination with regards to eg. race and gender and then just blame the computer since you can't see the weight as easily as in component analysis.


> Its very similar to the argument kids make "Why do I have to learn how to do long division when I have a calculator?" or "Why do I need to show my work on the exam if I got the right answers?"

Exactly. We included mathematicians, computer scientists and engineers in the design of those calculators. They use algorithms backed by rigorous proofs, run upon circuits with rigorous tolerances such that the answer is correct (to a reasonable amount of precision). Kids with their calculators are intellectually lazy -- we should expect that, but continue to teach otherwise. To see this from full-grown adults in academia is an embarrassment. But, god, would I enjoy a ML salary...


Like I have pretty much always thought this when I read about ML or uses of it. The process and end result is just math so it has to be explainable in some way. I feel like not enough effort is given to understand the important variables to ML models or how are results derived. When I see uses of it it's more like the following: "Oh, it works, that's good enough."

For-instance when a model starts being over fitted, one can say it's pretty much converging to something akin to a LUT (Look up table). While over fitting is generally undesirable it still might be interesting. Principally, if someone could figure out how basically it is indexing the data. Perhaps you could create a simpler hash function or find some rules for creating useful hash functions for that type of data.


The whole point is that your models should perform well on data that they haven't seen before. You can't create a hash function for that. Let alone one that works for 100s or 1000s of dimensions.


I'm confused by what you're saying... by data that hasn't seen before, do you mean new variables which need to be hypothesis tested? Or just a larger sample?

I also don't see why you can't create a "function" hash or otherwise, that works for 1000s of dimensions. The model itself, if it was perfect would be an equation... the number of dimensions really doesn't matter that much.

Sorry, I'm just real confused by your statement.


Say you're building a self-driving car. You need to take an image of what the car sees, and figure out whether/where there is a pedestrian/car/bicycle/etc in the image. So your input/variable is the image itself.

Almost every image the car sees is going to be different. I'd guess that's why GP said that it doesn't make sense to use a hashing function - there's little value in mapping inputs to results, because your input images are pretty much always going to be different. So pretty much every time you look up an image in your hashmap, you wouldn't find a match.

That's the point of ML models - find patterns in data so that when you see a new example, you can predict what you're seeing based on what you've seen in the past.


It seems like you might be deep in the ML rabbit hole. Zoom out a bit. A hash function is "just a function." Every image labeling black box can be though of as a hash from images to label vectors.

OP's comment effective asks whether there is another, more grokkable function that maps/hashes inputs to the same labels.

Granted, that question boils down to "can we create human-understandable models?" which is the whole point of this discussion.

It's a good question, though. If we had black-box-like spaghetti code performing the same task, I predict that the comments here would be very different.


Thanks for explaining that perspective. I was seeing things a little too narrow-mindedly for sure.

Based on your phrasing of the issue, it seems like we could think of the problem as: can we reduce the number of parameters in an ML model to the point where humans can understand all of the parameters? That's related to an active research area - minimizing model size. ML models can have billions of parameters. It's unfeasible for a human to evaluate all of them. Research shows that (sometimes) you can reduce the number of parameters in a model 97% without hurting it's accuracy [0]. But, 3% of a billion parameters is still way too many for people to evaluate. So I think the answer so far is no, we can't create human-understandable models that perform as effectively as black boxes.

[0] https://openreview.net/forum?id=rJl-b3RcF7


By data it hasn't seen before I mean a sample which doesn't exist in the training set. Maybe I'm not understanding something in the parent's argument.

For example, you train an image recognition model to tell cats and dogs apart using images from the internet, then you take out your phone, snap an image of your dog and give it to your algorithm to determine the species. When preprocessed this picture is equivalent to one row in a table with thousands of columns and this specific combination of pixel values doesn't exist anywhere else. Where is the hash function looking and for what in that case?


Unless there is a self-modifying feedback loop you can indeed replace any ML model with a hash function. The problem is one of size rather than any fundamental limitation of a look-up table.


Shouldn't this fall into the "trust by verify" category of things?

Whilst your model may not be able to explain to you today why it can operate on future data, if it can explain to you why it works on any data you've already given it, it makes it easier to explain behaviour.

You have to trust it on new data.

You can always verify why it made the choices it did on old data.

That seems like something incredibly helpful, to me.


> You can always verify why it made the choices it did on old data.

This makes no sense. Your model typically isn't asked to make predictions on "old data" because it was trained on it. And understanding why it made certain prediction isn't any different on "new" vs old data.

You can look into bias-variance tradeoff, train-test splitting and cross validation to get a better picture of this.


> You can look into bias-variance tradeoff, train-test splitting and cross validation to get a better picture of this.

Or, like in the article, you can make use of introspective models that can tell you what dimensions were fitted. This can more easily show you where overfitting is occurring.

You can see the weights of the model and follow the decision network, which allows you to explain why something has occurred in a much clearer fashion.

An introspective model gives you a much clearer picture of why certain choices were made, rather than the fuzzy picture you get running statistical analysis against a black-box'd algorithm.


Uh, yeah you can. Hash functions handle data its never seen before.


Indeed. It would be nice to read an article about NN introspection/debuggers. I don't see why you couldn't use an introspected ML model as the basis for a behavior tree in some cases.


It's an old idea (22 years at least), the black box method can be used to remove noise and find a none bayes/compression/simplest info theory model, then the model is used to generate clean training for a decision tree or association rules.


I am worried about the recent trend of "ethical AI", "interpretable models" etc. IMO it attracts people that can't come with SOTA advances in real problems and its their "easier, vague target" to hit and finish their PhDs while getting published in top journals. Those same people will likely at some point call for a strict regulation of AI using their underwhelming models to keep their advantage, faking results of their interpretable models, then acting as arbiters and judges of the work of others, preventing future advancements of the field.


> IMO it attracts people that can't come with SOTA advances in real problems and its their "easier, vague target" to hit and finish their PhDs while getting published in top journals.

I'm also pretty wary of interpretability/explainability research in AI. Work on robustness and safety tends to be a bit better (those communities at least mathematically characterize their goals and contributions, and propose reasonable benchmarks).

But I'm also skeptical of a lot of modern deep learning research in general.

In particular, your critique goes both directions.

If I had a penny for every dissertation in the past few years that boiled down to "I built an absurdly over-fit/wrongly-fit model in domain D and claimed it beats SoTA in that domain. Unfortunately, I never took a course about D and ignored or wildly misused that domain's competitions/benchmarks. No one in that community took my amazing work seriously, so I submitted to NeurIPS/AAAI/ICML/IJCAI/... instead. On the Nth resubmission I got some reviewers who don't know anything about D but lose their minds over anything with the word deep (conv, residual, variational, adversarial, ... depending on the year) in the title. So, now I have a PhD in 'AI for D' but everyone doing research in D rolls their eyes at my work."

> Those same people will likely at some point call for a strict regulation of AI...

The most effectual calls for regulation of the software industry will not come from technologists. The call will come from politicians in the vein of, e.g., Josh Hawley or Elizabeth Warren. Those politicians have very specific goals and motivations which do not align with those of researchers doing interpretability/explainability research. If the tech industry is regulated, it's extremely unlikely that those regulations will be based upon proposals from STEM PhDs. At least in the USA.

> faking results of their interpretable models

Jumping from "this work is probably not valuable" to "this entire research community are a bunch of fraudsters" is a pretty big jump. Do you have any evidence of this happening?


> If I had a penny for every dissertation in the past few years that boiled down to...

This is very, very accurate. On the other hand, I oftentimes see field-specific papers from field experts with little ML experience using very basic and unnecessary ML techniques, which are then blown out of the water when serious DL researchers give the problem a shot.

One field that comes to mind where I have really noticed this problem is genomics.


> Those same people will likely at some point call for a strict regulation of AI using their underwhelming models to keep their advantage

While this is probably true (and IMO, possibly the right choice), this:

> faking results of their interpretable models

is taking a huge leap.

Why are they more likely to fake their results than companies selling a black box model?


Leaving interpretability and conspiracy theories about softy PhDs aside, for a bit, "SOTA advances" are not progress. e.g., in the last seven years since AlexNet, results have kept creeping upwards by tiny fractions on the same old datasets (or training times and costs have gone down) all of which is achieved by slight tweaks to the basic CNN architecture and perhaps better training techniques, or, of course more compute. But there has not been any substantial progress in fundamental algorithmic techniques- no new alternative to backpropagation, no radically new architectures that go beyond convolutional filters.

But I'll let Geoff Hinton himself explain why the reliance on state-of-the-art results is effectively hamstringing progress in the field:

GH: One big challenge the community faces is that if you want to get a paper published in machine learning now it's got to have a table in it, with all these different data sets across the top, and all these different methods along the side, and your method has to look like the best one. If it doesn’t look like that, it’s hard to get published. I don't think that's encouraging people to think about radically new ideas.

Now if you send in a paper that has a radically new idea, there's no chance in hell it will get accepted, because it's going to get some junior reviewer who doesn't understand it. Or it’s going to get a senior reviewer who's trying to review too many papers and doesn't understand it first time round and assumes it must be nonsense. Anything that makes the brain hurt is not going to get accepted. And I think that's really bad.

What we should be going for, particularly in the basic science conferences, is radically new ideas. Because we know a radically new idea in the long run is going to be much more influential than a tiny improvement. That's I think the main downside of the fact that we've got this inversion now, where you've got a few senior guys and a gazillion young guys.

https://www.wired.com/story/googles-ai-guru-computers-think-...


> no new alternative to backpropagation, no radically new architectures that go beyond convolutional filters.

Attention

My point wasn't about lack of investment/propagation of fundamental research that is not trendy, it was about hijacking what should be science by "softy PhDs" that found a niche in less demanding areas and will likely impose their will over the ones who are doing hard science and not politics, like how CoCs were recently used to take control over open source/free software licenses by some fringe non-technical groups. It's a pattern that is repeating across all industry and academia in the past, the ones that move field forward are often displaced by their "soft-skilled" and less-capable peers.


In your view, it's better to answer the wrong questions optimally than the right questions suboptimally? That's not the kind of AI want putting people in jail or denying them housing or healthcare or shooting missiles at them.


There are two parts to my opinion:

1) I would prefer something that works most of the time (blackbox) than something interpretable that doesn't

2) if my DL converged to e.g. solving some complex partial differential equation, how does it help me that I could "interpret that" if 99.9999% human population has no clue what is going on regardless? "Your loan was rejected, because the solution to this obscure PDR with these starting conditions said so; these Nobel prize economy winners created this model with these simplifications and we deem it good enough to keep you where you are."


Interpretability is probably one of the most misunderstood topics in AI. Interpretability is fundamentally not a math or statistics challenge, it's a Human Computer Interaction (HCI) problem and needs to be studied in a similar manner, with user studies. The fundemental goal of interpretability is to help users achieve some concrete goal such as detecting bias, identifying bad models, and debugging. This is similar how we can better design keyboards to help users achieve faster typing speed. Unfortunately, most interpretability studies don't actually bother testing if their methods actually work and I am highly skeptical that things like "white box models" or salience maps or attention weights or whatever latest methods can actually achieve the aims that interpretability proponents claim they do. I would love to see a simple randomized user study that would emperically show a concrete improvement in something like bias detection or error detection.


> nterpretability is fundamentally not a math or statistics challenge

No, I keep on seeing AI/ML/DS people keep on downplaying statistic.

Statistic interpret things. The majority of the models out there have a one to one, predictor to response, holding all other predictor constant (linear regression, logistic regression, arima, anova, etc..). Statistic inference is a thing. Inference is interpreting. Descriptive statistic is interpreting. Parsimonious is a thing. Experimental design is a thing. Degree of freedom is a thing in statistic.

If you want interpretability do statistic. One of it's tenant is to quantify and live with uncertainty not fitting a curve and lots of coefficient to just predict. Not just classification.

It's a reason why biostat or econometric is a thing. Statistic.

Even the blog cited statistic papers even though it barely mention statistic models in it. ~~And Rudin is a statistician and contributed a lot in statistic.~~ Wrong person (I'm thinking of Rubin for casuality and missingness)

This is not a tribal fight between statistic and ML. This is pointing out that ignoring statistic is a detriment to AI/ML/DS as a field.

I predict that 2020 to 2030 statistic will be coming to AI/ML much more so regardless how much people downplay statistic.

~~Seeing on Dr. Rudin is coming over.~~ I've seen other statistician too. Dr. Loh works took decision tree and added ANOVA and Chisqaure to build parsimonious decision tree.


I've happen to be in that workshop room and had chosen the 2% robot, not the 15% surgeon. If I remember correctly, the point of the question was determining willingness of society to use black box models, not challenging the need for models interpretability.

Interpretability obviously doesn't hurt accuracy. But it is costly to engineer. And not always possible to make. Not always possible, because human capacity (and willingness to put in the effort) into understanding the explanation is limited.


>> Interpretability obviously doesn't hurt accuracy.

Why do you say this? From what i have seen it certainly can and does. For some industries finding a trade off where the magic is.

https://link.springer.com/article/10.1007/s10664-018-9638-1

https://www.oreilly.com/ideas/predictive-modeling-striking-a...


You can take nearly any black box model and add some interpretability into it, without modifying this model. No modifications to the model, so accuracy stays the same.


> [...] these black box models are created directly from data by an algorithm, meaning that humans, even those who design them, cannot understand how variables are being combined to make predictions.

So uhhh, isn't this like not science? Like my biggest problem with "machine learning" is people assume the data they have can correctly answer the question they want to answer.

If your data is off, your model might be incredibly close for a huge portion of the population (or for the biased sample you unknowingly have), but then wildly off somewhere else and you won't know until it's too late because it's not science (e.g. racist AI making racist predictions).

A model cannot be accurate if it doesn't have enough information (like predicting crime, or the stock market). There are an insane amount of statistical tests to detect bullshit, and yet we're pushing those and hypothesis testing right out the window when we create models we don't understand.

Like I just don't get how some folks say "look what my AI knows" and assume it's even remotely right without understanding the underlying system of equations and dataset. You honestly don't even know if the answer you're receiving back is to the question you're asking, it might just be strongly correlated bullshit that's undetectable to you the ignorant wizard.

I find it pretty hard to believe we can model the physical forces inside a fucking neutron star (holla at strange matter) but literally no one in the world could pen an equation (model) on how to spot a fucking cat or a hotdog without AI? Of course someone could, it would just feel unrewarding to invest that much time into doing it correctly.

I guess I can sum this up with, I wish people looked at AI more as a tool to help guide our intuition helping us solve problems we already have well defined knowledge (and data) of, and not as an means to an end itself.


> your model might be incredibly close for a huge portion of the population (or for the biased sample you unknowingly have), but then wildly off somewhere else and you won't know until it's too late because it's not science

But all science works like this. All scientific models can have errors, and presumably they all do. Even really basic empirical science is only reliable to the extent that our models of how photons and our visual systems work, and those models have errors we know about and probably others we don’t know about.

Fallibility does not mean that something is not science, on the contrary, denying that some theory or model is fallible is profoundly unscientific.

But of course, that doesn’t mean we should accept “black box” algorithms as the end of the story. We should strive to develop explanations for those things just like for all other things.


> So uhhh, isn't this like not science?

Very little of technology has anything to do with validating hypotheses.

> meaning that humans, even those who design them, cannot understand how variables are being combined to make predictions.

The intention is to not rely on the explanation to evaluate the effectiveness of the model. This does not preclude any of the infinite narratives that might explain the model.

This is fundamentally a cost saving mechanism to avoid hiring engineers to code heuristics useful to business. There is nothing related to science here at all. A "black box" model is fashionable to those who prefer to observe and not create meaning, even if the observed meaning is deeply flawed from a human perspective. After all, people spend money based on less all the time.


At our company, we are working with "mechanistic" or mathmatical models rather than stastical approches. I have observed :

1. It is hard than it should be to explain the concept to people (particularly VCs) 2 . people struggle to understand that a mechanistic model could have more utlity than a machine learning black box 3. people think you are doing something wrong if you are not using a neural network 4. The less people understand about neural networks, the more they seem to believe they are appropriate for all predictive / modelling problems 5. There is generally quite a low understanding of scientific method in the startup / VC space (speaking as someone who has worked in and around academia for years) vs how "scientific" people believe they are because it sounds good to be data driven and scientific about running startups and funding them.


Do you mean like symbolic regression or algorithmic information theory based stuff?

If so, I'd love to get in touch, shoot me an email


We are using biophysics based approaches. Will send an email over :)


Send me one, too, if you don't mind. I like collecting these techniques to give to researchers or practitioners wanting to try new things.

Machine Learning is about solving a task by automatically learning patterns based on examples and trying to generalize to new data. Granted, it's not necessarily the best approach to understand deeply how a phenomenon occurs.

You could use a black box model if you're more interested in predicting correctly images of handwritten digits than in understanding how the pixels relate to each other.

Of course, usually people want both accuracy and interpretability. It boils down to understanding what's more important for the problem at hand and making the compromises accordingly.


The goal of science is to produce a testable hypothesis. The world is full of things that people can 'explain', or even prove with mathematics, but don't hold up to basic experimental testing or reproducibility.

Every new theory has a plain english explanation that's easy to understand. Few of them have the raw accuracy or reliability of top ML models

https://en.wikipedia.org/wiki/Replication_crisis


>> I guess I can sum this up with, I wish people looked at AI more as a tool to help guide our intuition helping us solve problems we already have well defined knowledge (and data) of, and not as an means to an end itself.

Problem is, most machine learning algorithms cannot incoprorate background knowledge except by hard-coding inductive biases (as, e.g. the convolutional filters in convolutional neural nets). Unfortunately, this is a very limited way to incorporate existing knowledge.

This is actually why most machine learning work tries to learn concepts end-to-end, i.e. without any attempt to make use of previously learned or known concepts: because it doesn't have a choice.

Imagine trying to learn all of physics from scratch- no recourse to knowledge about mechanics, electomagnetism, any kind of dynamics, anything. That's how a machine learning algorithm would try to solve a physics problem. Or any other problem for which "we already have well defined knowledge (and data) of". We might as well be starting at around before the stone age.


> could pen an equation (model) on how to spot a fucking cat or a hotdog without AI? Of course someone could

I had a philosophy lecturer who would vehemently disagree that it is even possible to construct an algorithmic AI to decide what is and isn’t a cat.

For a start, do you mean domestic cats, or the cat family? What about photos, sculptures and other representations?

I mean, is Garfield a cat?

People use statistical machine learning over algorithmic AI because trying to model the real world with algorithms is an endless and often pointless exercise.


I think you are missing the key part of the appeal here (or framing it as a negative).

Let's look at your question. Writing an equation for a cat is hard, actually really hard. Humans cannot reliably explain their decisions here. If I ask a person to tell me how they classify between cat and not cat, the answer will invariably be something along the lines of "well, it has the general shape of a cat". Which is actually just a huge combination of heuristics it took about 10 years to work out. There is quite a lot of work in neuroscience suggesting that the actual decision you make when you classify a cat happens before a rational is developed.

We could encode a function for that, but it relies on us knowing a lot about cats, which takes time and only works for toy examples.

If you use a convolution neural network, you can get close to human level performance on much more complex topics with little domain specific insight. There is no universal law for classifying hand written letters - they are an individual's interpretation of some symbols we made up. This task will always be 'non-rigorous' because the very underlying thing is not actually well defined. When does a 3 become an 8?

So we could have a person toil away and come up with a bunch of heuristics that we encode in a regression, but why is this better than having a machine learn those heuristics? Most problems are not life or death. What is the real added value in having people hand crafting features for predicting traffic jam delays or customer retention, when the end use is probably just to have a rough indication.

As somebody who does research using a huge range of models, I object that we should be guided by our intuition- our intuition is mostly wrong about non trivial problems.

Basically any "equation" somebody has discovered for what happens in a neutron star is "simple". There is a large amount of observational data, it is a consequence of some already well proven theorem, or relies on something well established to narrow the range of possible descriptions immensely, or (most commonly in my experience) the equation is basically a human version of deep learning, where grad students toil away making tweaks and heuristics until a point that the description fits the data somewhat well, and then there is some attempt to ascribe meaning after the fact.

For example, we can describe the trajectory of a comet using a "few" lines of high school level math. This means it is actually feasible for a person to have a reasonable intuition about what is happening, as the problem is actually dominated by a handful of important variables. Good luck getting anywhere near simple to describe cats (again, in a domain where the line of what is and isn't a cat is actually not even a property of it's physical attributes, so the problem is not properly defined under your requirement). To tell if something is or is not a cat, would require a DNA sequence. That is how we define the cat. So by your own definition, we do not have sufficient data in our dataset to properly do this classification.

I'm not sure you really understand the point you make about "statistical tests for bullshit". Most statistical tests are themselves ivory towers of theory and assumption which nobody ever verifies in practice (which is as unsciency as anything you accuse machine learning of). And people do actually use well grounded ways of evaluating machine learning models. Cross validation is very common and predates most machine learning, and has various "correctness" results.

For any model we build, if we do not have data that encodes some pathological behaviour we can test it out on, there is no test, no statistical procedure to tell us that model is flawed. If we have that data, we can run the exact same test on a black box model.

You should not conflate science with formalism or complexity. Running a statistical test is pseudoscientific unless you do it correctly and appropriately.

Saying something is not scientific because the data may not contain enough information to fully answer the question is flat out wrong.


This article seems to misrepresent a number of important issues, and as a result significantly overstates their claims. I'll pick just one illustrative (but important) example:

> "For instance, when ProPublica journalists tried to explain what was in the proprietary COMPAS model for recidivism prediction, they seem to have mistakenly assumed that if one could create a linear model that approximated COMPAS and depended on race, age, and criminal history, that COMPAS itself must depend on race. However, when one approximates COMPAS using a nonlinear model, the explicit dependence on race vanishes, leaving dependence on race only through age and criminal history. This is an example of how an incorrect explanation of a black box can spiral out of control."

The concern about the strong relationship between race and COMPAS predictions is not largely based on a concern about whether there is an explicit dependence in the model. The concern is whether there's a relationship either explicitly or implicitly. And in particular, whether such a relationship results in unfair outcomes. The findings of the ProPublica study (https://www.propublica.org/article/how-we-analyzed-the-compa...) strongly suggested this was the case:

"- Black defendants were often predicted to be at a higher risk of recidivism than they actually were. Our analysis found that black defendants who did not recidivate over a two-year period were nearly twice as likely to be misclassified as higher risk compared to their white counterparts (45 percent vs. 23 percent).

- White defendants were often predicted to be less risky than they were. Our analysis found that white defendants who re-offended within the next two years were mistakenly labeled low risk almost twice as often as black re-offenders (48 percent vs. 28 percent).

- The analysis also showed that even when controlling for prior crimes, future recidivism, age, and gender, black defendants were 45 percent more likely to be assigned higher risk scores than white defendants."

I understand the desire of the MIT researchers to promote the value of their work, but in this case they appear to be doing so in a potentially damaging way.


No, they're not misrepresenting this at all. ProPublica's article https://www.propublica.org/article/machine-bias-risk-assessm... was pushing the claim that somehow, the COMPAS black box was implicitly deducing defendants' race from the "137 questions" input to it and labelling them likely reoffenders based on it in a way that was indepdendent of the key factors known to affect reoffending rates, such as age and gender. The paper in question seems to demonstrate the exact opposite is true: ProPublica were inadvertently using race as an imperfect proxy for age at sentencing, which is what the COMPAS algorithm really cared about, because their attempt to control for age at sentencing didn't work. After controlling for the actual age factor, COMPAS results didn't have any relationship to race anymore. (It's not even a weird weighting factor: predicted reoffending risk falls off rapidly with increasing age at first, then more slowly in a smooth fashion. It's just not linear.)

Now, it's of course possible to argue that judging reoffending risk based on age is in fact unfair and racist because it has disproportionate impact on certain racial groups, even though it's strongly predictive across all racial groups. That's not the argument ProPublica made, though. Their argument was about the supposed perils of black boxes, and they kind of acknowledged that age probably wasn't a racist criteria - or at least that it would be a lot harder to justify calling it one - by attempting to strip out its effects in the first place. It's also a different kind of argument entirely, one that revolves not around whether the algorithm is somehow treating people differently based on their detected race - because it isn't - but around what it means for a decision like this to be fair in the first place.


Whether a variable is latent or explicit isn't really relevant to the question of algorithmic fairness.

The link I provided gives the actual details of the method and findings; this is probably a more useful source for the details. The claim that the actual source of the difference is 'age' doesn't really make sense. There isn't enough of a difference in the number of young people between black and white populations to result in the differences found in the analysis.

(I do agree that the actual attempt to control for age was poorly done; it really shouldn't have been done at all, since it had nothing useful to add to the analysis or results.)

PS: It's 'COMPAS', not 'COMPASS'.


“Explainability” is a subjective idea that allows people to act as gatekeepers about the use of a model through arbitrary politics.

“The Mythos of Model Interpretability” is good reading on this.

https://arxiv.org/pdf/1606.03490.pdf


Ultimately, a lot comes down to the accessibility of tools and know how. The more widespread and general a technique or tool, the more widespread and general it becomes... because people already have it and know how to apply it.

OTOH, these authors (and old school heavyweights like David Ferucci) are not wrong. Interpretability, explainability & interoperability with human intelligence is not something to just give up on.

I like "challenges" as ways of exploring these areas. Good luck to the authors. Female team, btw.


It's not that accuracy will always be sacrificed if one wants an explainable model. The point is: if interpretability is an important constraint, it could prevent improvements on accuracy.

Sometimes, the best interpretable model is as good as a black box, and that's great.

When this is not the case, the trade-off is that one should see what's more important for the actual problem. Perhaps interpretability is not a big deal.

Another solution is to try to extract interpretability from the more accurate black box model with something like SHAP.


This is a great point. There is a general lack of understanding about what it means for models to be interpretable & explainable. These words get thrown around often by people who don't understand the definition, and also the trade off with accuracy.

Some papers i found interesting on the subject:

https://arxiv.org/abs/1606.03490

https://arxiv.org/abs/1707.03886

https://arxiv.org/abs/1806.07552

https://arxiv.org/abs/1702.08608 (i found this was a good sumary of the issues)


A couple salient quotes from "Interpreting AI Is More Than Black And White" in Forbes [1]:

"No matter the definition, developing an AI system to be interpretable is typically challenging and ambiguous. It is often the case that a model or algorithm is too complex to understand or describe because its purpose is to model a complex hypothesis or navigate a high-dimensional space, a catch-22. Not to mention what is interpretable in one application may be useless in another."

"Underspecified and misaligned notions of interpretation impede progress towards the rigorous development of understandable, transparent, trusted AI systems."

[1] https://www.forbes.com/sites/alexanderlavin/2019/06/17/beyon...


The interest in black-box models is partly for the reasons given, that it is presumed that their inexplanability makes them more powerful, but also because business people are sick of dealing with programmers and analysts as a class, people with arcane knowledge limited replacability, and a penchant for blowing deadlines.

The dream of machine learning is to fire the programmer and break up the last trade (baring doctors and lawyers).

If it was just business people being alienated, I'd say fuck it they deserve it. But the fact is everyone, and many of the programmers too, are alienated too when much of the code makes sense to no one. I would rather fix programming to be comprehensible to more people even though it will take down the walls between programmer and non-programmer.


> there are actually a lot of applications where people do not try to construct an interpretable model, because they might believe that for a complex data set, an interpretable model could not possibly be as accurate as a black box. Or perhaps they want to preserve the model as proprietary.

I think they missed an important reason here. It's why I decided to study AI and Machine Learning: The ability to "just throw some data at it and see if it figures it out" is just so exciting, mysterious and intriguing.

I remember actually being a bit disappointed when I learned how (classic, 3 layer, early 2000s) neural networks worked. That it was just something super simple with derivatives, and resulted in something that seemed a lot like a more complex form of statistical regression. Kind of took the magic out of it for me, a little (didn't stop me of course, it was and still is an exciting field of research).

I know it's not a good reason, but surely I'm not the only one who thinks it's extremely cool that one is even able to build a black box that does useful things with what you feed it, but you don't know how it works, and yet you built it.

It's just something mystifying, and I believe there is also a tinge of fear for that disappointment, to figure it out and find out it's not really doing as clever things as you hoped it might.


When using ML models to deal with natural science such as physics, working well+being interpretable actually implies new discovery. Working well only, on the other hand, cannot win you a Nobel Prize.


This is a pointless article. Obviously everyone would prefer fully interpretable models if it was possible to get comparable performance with the leading black box/deep learning approaches. No one chooses to make black box, non-interpretable models, it's simply the fact that the models that give the best performance tend to be black box. There is a lot of work happening on trying to interpret deep learning models, because no one prefers non-interpretable models and being unable to pin point and explain why some prediction was wrong.


The article explains that their fully interpretable model actually got within 1% accuracy of the best black box model. Which was within cross validation error. That was at a data science competition whose explicit goal was to encourage people to explain black box models. They also give other examples of cases where people have used black box models unnecessarily.


> The article explains that their fully interpretable model actually got within 1% accuracy of the best black box model. Which was within cross validation error. That was at a data science competition whose explicit goal was to encourage people to explain black box models.

That's awesome, but this was a single model for a single application being measured against presumably by a limited test criteria/benchmarks. The point still stands that the recent narrow AI "renaissance" is largely due to deep learning, which is inherently black box. There is a lot of work going on in making deep learning more interpretable precisely because it's so prominent today and because its lack of interpretability is a huge con.

Machine vision for example has come a long way due to deep learning. A lot of autonomous car companies are relying on it for perception. All of these companies hate the fact that there is no way to tell when the classifier will fail or why it'd fail. When using deep learning in finance setting, its lack of interpretability is a huge downside.

Despite this deep learning is being used because it provides an appreciable improvement in performance over the alternatives. Certainly there could be alternatives to deep learning that may perform just as well or even better. But finding those alternatives for the vast number of applications deep learning is being used for today is much easier said than done.


Human cognition itself is a black box model. We routinely make snap decisions first and then justify them after the fact. Anything that requires serious consideration is darn near insurmountable for a human mind in the presence of incomplete information, emotions, and conflicting motivations.

So while I'd like neural nets to be more interpretable, to me it'd take a distant second place. The first would be to get models that actually work better than humans for practical tasks, even under limited circumstances.


I share this sentiment.

It occurs to me that human decisions making can be either explained a very high level, or a very low level(neurons firing). But the magic in between is too complicated to draw direct lines between stimuli and decisions.

AI seems to be the same way. We have statistical models that explain how most everything works at a low level. That a simple math formula can approximate just about anything. How we arrive at weights that separate data sets in different dimensions and etc. For convolution networks we also understand how layer "decisions" flow through to other layers. But, it's too complicated to look at an image and explain exactly how the input pixels will result in a classification output.

I'm not sure how much I should care about that.. Seems like a problem for mathematicians/statisticians. Having the doctor explain himself gives me no more true insight into how his brain works than having somebody explain at a high level how the robot was trained and reacts in different situations.


The convenience to be able to say: "we didn't take the decision, the algorithm did" is the holy grail of non accountability.

I wouldn't be surprised if someone, especially in business, is actively opposing transparent AI.


Interpretable models are an exciting new entry to the ML toolkit. While they are likely sufficient for everyday ML tasks, they might not be expressive enough vs deep networks to tackle complex tasks like fraud and anti-money laundering classification. As with any tool, interpretable vs black models have optimal target applications that data scientist can apply them for. As interpretable models get wider tooling for training, expect them to play a larger role in simpler ML tasks. Black box models are however here to stay and will always have their place with their versatility.


I don't know how many of you are familiar with Kaggle competitions, but the winning scores on the leaderboards can come down to 0.0001 points of accuracy or less. Winning something like that with an explainable model is like having one hand tied behind your back.

If tiny fractions of a point matter (the rare case), then making the model explainable is adding a ton of complexity for little practical gain. It sounds nice, but the numbers are unforgiving


What a dumb question is it? The same logic is behind sports and nobel price awards.

It's easier to say to people who make financial decision that this is great artificial intelligence that will solve all your problems then explain mathematical equations behind it that 90% people won't understand. The more complicated the stuff is the higher probability it will be sold. That's the standard in IT world from quite some time.


We have all these tools to build neural networks and we use them to peform feats where a simple C4.5 would have sufficed.

To continue innovation in the algorithmic field, I think we need a branch of scientific research or a competition where researchers are constrained to 100MHz CPU clocks and no GPU cheats when actually running their algorithms (not when loading and parsing data)


I feel like we're ignoring the obvious answer: because most people in AI right now are overwhelmed by the necessary mathematical background and comfort necessary to produce first-principles generative innovative algorithms and models, so all they're left with is following the latest trend, which is locked on deep learning right now.


While I can definitely see the value of black box models in engineering and commercial applications (including clinical ones), I am quite troubled by the new hype of applying to deep learning to so many basic scientific problems. The 'deep' models do not tend to result in deeper insights - which idealistically ought to be the goal.


When I want to predict something, I need the highest accuracy, and I am not interested in interpretability of an individual parameter.

and interpretability is only a half measure. If i really want to understand the model, I will conduct causal analysis with randomized controlled trial and will find the root cause and impact of the factor of my interest.


I think this argument fails to understand that purpose of black-box models is to be impossible to understand, as it shields the operator from pre-emptive accountability.

Then, since they are supposedly the 'only way' to provide whatever function the market is 'demanding', then clearly we must abandon any notion of accountability, as this need 'must' be met.

This logic is perhaps most evident in google/facebooks content moderation dilemmas, where both companies refuse to define any sort of non-vague actionable standards about what they censor (or fail to promote) as 1) they can't provide them and 2) they don't want to be held responsible for what they actually are.

Then, as outrage has grown over facebook/google's terrible content moderation and censorship policies, and the need for accountability has grown, both companies have been forced to hire ever more moderators and censors, because the technologies as claimed just don't work well enough.


I don't think it's worthwhile assuming this level of malice.. Ultimately we choose models that are the most performant. Interpretability is just a nice bonus. I can assure you that I am held accountable for my model performance regardless of whether I can explain its decisions satisfactorily. All the business cares about is whether we're burning or making money.


> purpose of black-box models is to be impossible to understand

This is just outright wrong. And the argument about moderation is completely unrelated.


Sounds more like a useful side effect than anything someone would design on purpose. But that's an interesting point none-the-less.

Occam's razors the easier way to explain the actions of the big guys and their own internal responses to cultural pressures to not allow things they don't like from happening or being tolerated in the world.


Well maybe also because black box models can be applied to a wider range of systems and can help to understand general properties of a class of algorithms. I’m sure it’s mostly all a conspiracy, though.


It's true our own cognition is also a black box, e.g. how do we recognize? It's as much a mystery to introspection as digestion. But the goal of science is to understand. Whereas with DL, we're just chimps in the dirt playing with sticks.


I think there is a lot of scope for interpretation in black box models. E.g. when doing transfer learning with a frozen pretrained model it should explain the new classifications in terms of pretrained classes.


There are probably dozens of holes one can drill into this sensationally titled click-baity study. It's surprising what gets through Nature's review system these days.


And yet you provide none. What do you base this on?


Can you train a white box solution with only a tagged data set? It seems obvious to me why we have black box solutions even if we could have spent man hours engineering an algorithm.


You actually can - for example https://arxiv.org/abs/1806.10574


Off topic, apologies, but why are we putting (2019) in the title when this article is barely two months old?

Makes is seem like it's an out-of-date article. Is it?


Absolution of liability. Why else use the term "AI" to describe your heuristic?


why include 2019 in the title of this post? was it automatically added (is that a bug)?


There's a (human-enforced) HN rule to do that for old articles, so they're not mistaken for news. I'm not sure what the standard for "old" is, but it isn't 2019 anymore, so I guess it's technically correct.


2 months old is usually not considered old news on this site... I think that their algo is buggy on year-change


As far as I know, there's no algorithm doing it. People put it in the title themselves, or mods do if they forgot.


For god sakes. Enough of this black-box—we don’t understand how this works narrative. Neural networks are JUST computational circuits. Unfortunately there’s been a lot of laziness in cleaning those circuits up (W_ijk=O) to better reveal the topological circuitry.




Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: