People want to know whether some mathematical formulas can work. Then how do they work? Then what can make them work in a different way.
Explanability or interpretability leads to controllability at the end.
I rather see NNs with semantic meanings instead of semantic meanings from NNs. If human would like to control NNs, why not make them meaningful modules that can be composed like a regular program.
For example, instead of using CNNs or RNNs, we simply make a model by stating the definition:
:: (p: Person, scene: Image)
:= p in scene
& exist s: Street in scene,
walk_cross(p, s) in scene
& not exist z: ZebraCross in scene,
inside(p, z) in scene
Here predicates, walk_cross and inside, are neural network modules that might be used in many different problems. We can identify cases where the model make wrong predictions and modify the definition accordingly.
This is much human friendly development than tweaking parameters. After all, not everyone is fond of programming in NNs directly.
The distinction is this: explanations can only be made ex post facto, about why the model acted a certain way based on specific inputs; interpretations can be made based on the model's parameters themselves, i.e., "feature X is very important and feature Y is almost always ignored and I know this because my NN is one layer deep and all the weights for feature X are large in magnitude and all the weights for feature Y are small in magnitude." This does not require specific inputs to be fed, and specific outputs to be studied, so is a different concept and why I am suggesting we make the distinction explicit.
Capsules are groups of neurons that represent discrete entities in different contexts. For example, a 4x4 pose matrix is a capsule representing a particular object in different orientations seen from different viewpoints. Similarly, a subword embedding can be seen as a capsule with vector shape representing a particular subword in different natural language contexts. More generally, a capsule can have any shape, but it always represents only one entity in some context.
In certain new capsule-routing algorithms -- e.g., EM routing[a], Heinsen routing[b], dynamic routing[c], to name a few off the top of my head[d] -- each capsule can activate or not depending on whether the entity it represents is detected or not in the context of input data.
Models using these algorithms therefore make it possible for human beings to interpret model behavior in terms of capsule activations -- e.g., "the final layer predicts label 2 because capsules 7, 23, and 41 activated the most in the last hidden routing layer."
While these new routing algorithms are not yet widely used, in my humble opinion they present a promising avenue of research for building models that are explainable and/or enable assignment of causality at high levels of function composition.
[d] If you're aware of other routing algorithms that can similarly activate/deactivate capsules, please post a link to the paper or code here.
If capsules work wonders for you, my first guess would be that you can improve your training of the standard network to make it work equally well.
In general, my hunch is that capsules are still too low level and too much of a local change to make a strong difference.
To give an example, all of the state of the art optical flow AIs are based on building cost volumes and then resolving them. There are edge cases, where one can prove mathematically that reducing the cost volume to a flow direction will make it impossible to produce the correct result. So to make a significant contribution, it doesn't help to use capsules in the feature processing stage, but you need to replace the entire architecture.
FWIW, I've had the most success with EM/Heinsen-type routing algorithms -- that is, those in which each output capsule is generated by a probabilistic model (such as a Gaussian mixture), and the output capsule activates only to the extent its model can explain (i.e., generate) its view of input data better (in some quantifiable manner) than other output capsules. The notion that an output capsule "must explain input data better than other capsules in order to activate" is very appealing to me as a mechanism for inducing per-layer "explainability" in models.
In my experience so far, routing tends to work better on top of conventional architectures, e.g., use a ResNet for feature detection and stack two or more routing layers on top for classifying into hidden factors and then into training labels. Also, to get models to converge, I have found it helps to apply a nonlinear transformation to the features and then at least two routing layers on top. (I don't have a good explanation as to why two or more tend to work better than only one.) Finally, I usually feed only the capsule activations to the loss function -- that is, during training I let the capsules themselves "do whatever they want" to learn to explain input data.
FYI, Vaswani-style query-key-value self-attention mechanisms can be understood as a type of capsule-routing algorithm -- one in which the capsules are in the form of vector embeddings (each representing a token in a context), the activations are in the form of attention heads (representing which input tokens are most active for each output token), and the number of input and output capsules is the same (for every input token there is an output token).
Here, I'm talking more generally about using capsule-routing algorithms in which the capsules can be of any shape (they can be vectors, matrices, or higher-order tensors), the activations can be computed via different proposed mechanisms (including self-attention of course), and the number of input and output capsules need not be the same (e.g., with some algorithms it's possible to have a variable number of input capsules and a fixed number of output capsules).
As I wrote elsewhere on this thread, the routing algorithms I find most interesting are those in which each output capsule is a probabilistic model that "must explain input data better than other output capsules" in order for the capsule to activate.[a]
This doesn't work for humans. Sure, they'll give an explanation, but they don't fully understand their own decision making process so they can't reliably explain it. I am not sure which paper you're referring to, but how did the researchers address this issue?
Disclosure: I support the team that developed Captum.
Because fundamentally neural networks really have a debuggability problem. It's impossible to say if the program/code is actually correct and I'm not sure how visualization is actually going to solve the problem of correctness. If someone explained something to me I'd want to know why it actually addresses whatever problem they claim it addresses and their reasons of appealing to distilled models would not convince me because as long as we are looking at a compressed version of the program then why would we conclude that the larger program is actually correct and never misclassifies a pigeon as a stop sign? So if I can't be sure that a pigeon will never be classified as a stop sign then what has been actually explained and of what value is it?
> Deep neural network (DNN) is an indispensable machine learning tool for achieving human-level performance on many learning tasks.
Not to be pedantic, but words matter. Is anyone actually claiming that deep learning achieves true “human-level performance” on any real world open-ended learning task?
Even the most state of the art computer vision/object classification algorithms still don’t generalize to weird input, like familiar objects presented at odd angles.
I get that the author is trying to write something motivating and inspirational, but it feels like claiming “near” or “quasi”-human performance, with disclaimers, would be a more intellectually honest way to introduce the subject.
No, but the text you quoted doesn't say that.
Human level performance in this context means humans perform no better than some algorithm on some specific dataset.
Incidentally, that's also how you get to claim superhuman performance on classification tasks. Just include some classes that aren't commonly known in your dataset, e.g. dog breeds, plant species, or something like that. ;)
Uh, it says DNNs are indispensable for achieving human level performance. That clearly implies that this level of performance is achievable, despite all evidence to the contrary.
Buffalo buffalo Buffalo buffalo buffalo buffalo Buffalo buffalo
>Even the most state of the art computer vision/object classification algorithms still don’t generalize to weird input, like familiar objects presented at odd angles.
"some x are not y" does not invalidate "many x are y"
Deep learning has surpassed human level performance on many tasks ... (could add more you get the point).
Let deep learning do what it is good at, without explaining their performance and errors to anyone: invading your privacy on social networks, helping hedge funds make more money by analyzing Elon Musks tweets, and building military surveillance.
Leave the justifications and explanations to inherently white box models (they are nearly as good in performance as black box now, at least for structured data), and hold off on firing radiologists for a few decades, even though your train set performance is overfitted to be on par with "human-level".
Somehow, somewhere, the deep learning revolution started to drink its own kool-aid and became alergic to critique or solid verifiable computer science. Explainable deep learning does not exist, since half of the time the engineer that build the system can't even explain why it works in the first place. "Strong inspectable feature engineering is hard and time-consuming, so here we shook a box of legos a million times, burned six holes in the ozon layer, and out comes a deep net optimized with gradient descent". End-to-end learning is supposed to be really end-to-end, including the explanation.