Any insights on how it's generated? Markdown, Rst, Latex -> HTML? I would love to produce my documentation in this way.
Edit: I was too hurried. Everything is explained in https://distill.pub/guide/, the template is at https://github.com/distillpub/template
Ludwig is fantastic and put an incredible amount of work into polishing this article. And my co-editor Shan (who used to do data vis at the New York Times) seems like he has super powers half the time. We also get lots of outstanding advice from Ian Johnson and Arvind Satyanarayan.
I like this metaphor.
That said, after skimming the piece and thinking about the amount of work that went into it, one question popped up in my mind: do you think it would be possible to train a DNN to learn to visualize the "most important" neuron activations / interactions of another DNN?
> do you think it would be possible to train a DNN to learn to visualize the "most important" neuron activations / interactions of another DNN?
That sounds like a really hard problem. I'm not entirely sure what it would mean even, but it would not surprise me at all if there was some refinement that could turn into an interesting research direction! :)
In terms of what it could mean, one idea I just had is to take a trained model, randomly remove (e.g., zero out) neurons, and then train a second model to predict how well the trained model continues to work without those removed neurons. The goal would not be to 'thin out' the first model to reduce computation, but to train the second one learn to identify important neurons/interactions. Perhaps the second model can learn to predict which neurons/interactions are most important to the first model, as a stepping stone for further analysis?
Check this out
Can you elaborate on what you mean by "directions in activation space" ? If I understand it right:
You take a few neurons in a layer, and you follow some linear combination of their weights; you are then walking along a random direction. You take a single neuron and walk along its weights; you walk along this neuron's direction. Is this correct?
Also another comment: Szegedy et al  (https://arxiv.org/pdf/1312.6199.pdf) has the following abstract:
> First, we find that there is no distinction between individual high level units and random linear combinations of high level units, according to various methods of unit analysis. It suggests that it is the space, rather than the individual units, that contains the semantic information in the high layers of neural networks.
This is one of the predictions of the following paper, even though I think it came later: Opening the Black Box of Deep Neural Networks via Information (https://arxiv.org/abs/1703.00810). Here: https://youtu.be/FSfN2K3tnJU?t=1h19m23s, Tishby says: "One of the consequences of this theory is that single neurons in a layer don't tell us much."
Also, If you extend this idea to training examples, you get Mixup (https://arxiv.org/pdf/1710.09412.pdf)
distill.pub is really a really amazing place to learn.
(a reason why I'm moving into scientific visualization and GIS instead of backend work...)
The visualizations of image gradients was really fascinating, I never really thought about plotting the gradient of each pixel channel as an image. I take it these gradients are for a particular (and same) random starting value and step size? It's not totally clear.
(I have to say, "second-to-last figure.." again.. cool presentation but being able to say "figure 9" or whatever would be nice. Not everything about traditional publication needs to be thrown out the window.. figure and section numbers are useful for discussion!)
When we do feature visualization we do start from a random point/noise. For the diagram showing steepest descent directions, however, the gradient is evaluated on an input image from the dataset, shown as the leftmost image.
There's no real step size either as we're showing the direction. You can think of the scale as arbitrary and chosen for appearance.
Section numbers are on their way—and figure numbers also sound helpful! I've added a ticket. (https://github.com/distillpub/template/issues/63)
For now you can already link to figures like this: https://distill.pub/2017/feature-visualization/#steepest-des...
And how can e.g. layer3a be generated from layer conv2d0? By convolving with a linear kernel? Or by the entire Inception Module including the linear and the non-linear operations?
Thank you. Outstanding work breaking it down.
Here's another paper people might enjoy. The author generates an example for "Saxophone," which includes a player... Which is fascinating, bc it implies that our usage of the word in real practice implies a player, even though the Saxophone is an instrument only. This highlights the difference between our denotative language and our experience of language! https://www.auduno.com/2015/07/29/visualizing-googlenet-clas...
Also, for those curious about the DepthConcat operation, it's described here: https://stats.stackexchange.com/questions/184823/how-does-th...
Edit: I'll be damned if there isn't something downright Jungian about these prototypes! There are snakes! Man-made objects! Shelter structures! Wheels! Animals! Sexy legs! The connection between snakes and guitar bodies is blowing my mind!
 Not technically a neural network, but a CPPN. Which is something like a neural network with many different mathematical functions as activation functions. This allows things like a neuron with a sine wave activation that can repeat a pattern across the image.
hazarding a guess with no knowledge of the subject, I wonder if that is because no other class in ImageNet can be defined by orange and black. The net simply doesn't need to learn anything more about orange and black, because on 100% of the samples it trained on, orange and black meant "school bus". Every time. So no need to learn any other features -- if you see orange and black, it MUST be a school bus, at least in the context of this data set.
I wonder if we introduced other "orange and black" classes to ImageNet, it would need to learn more features about the school bus in order to identify it.
It doesn't necessarily mean that color is the only feature it uses to classify school buses. Just that that feature alone is enough.
Neural nets are doing something that the mind does too - it takes visual input, then it "predicts" and "fills in the blanks" (aka hallucinates) so that you "see" what you expect to see. Its why you dont notice your blind spot.
Its also interesting to note that this process appears to change as we age: https://www.npr.org/2016/04/17/474569125/your-brain-on-lsd-l...
It seems to be a natural speculation that the further layers of visual cortex/brain and NNs are subject to similar optimization, and thus they would have similar kernels. I think psychedelics somehow stimulate these deep layers to massively fire without waiting for the normally required input from the eyes to propagate to those layers to selectively fire neurons in them.
The neural structures in human brains that recognise edges/textures/surfaces are the same ones that generate trippy images when exposed to psychedelic drugs (or flickering light: http://journals.plos.org/ploscompbiol/article?id=10.1371/jou...): http://www.math.utah.edu/~bresslof/publications/01-3.pdf
I believe for anybody who has ever tried psychedelics seeing these was a watershed moment. It seems almost impossible that an algorithm could so faithfully reproduce the experience without somehow having recreated some fundamental structure of the human brain. That is compounded by the fact that this wasn't the result of actually trying to do so, but of an open-ended experiment of creating feedback loops in NN layers.
This question isn't about feature visualisation, but I though I'd take the chance to ask you, what do you think of Hinton's latest paper and his move away from neural network architectures?
While I agree that the practicality of these visualizations mean that you have to fight against this high-frequency "cheating", I can't help but shake the feeling that what these optimization visualizations are showing us is correct. This is what the neuron responds to, whether you like it or not. Put in another way, the problem doesn't seem to be with the visualization but with the network itself.
Has there been any research in making neural networks that are robust to adversarial examples?
edit: ok, someone already tested it, and it does not seem to help that much: https://github.com/jaesik817/adv_attack_capsnet
(Though maybe not as symmetric?)
E.g. like if I want to scan areas around me to see if there are any perspectives in my environment that light up the "snake" neurons or the dog neurons???
Inspecting layer activations in real-time is trickier, but presumably possible.
Could they train on white noise from a television and see if the CBR shows a structure similar to the structure of the observable universe when examining the feature layers?
Something about dissecting intelligence, and the potential that our own minds process things similarly. Creepy how our reality is distilled into these uncanny valley type matrices.
Also, I suspect it says something that these images look like what people report seeing on psychedelic trips...
With AI it's trivial to get access to the inner workings of the intermediate layers.