I should have explained how these are made. They train another neural network[1] to produce an image that most maximizes each class. This acts as a prior that the image must have a very simple and regular structure. And so the results seem to be very simple and even abstract, instead of pixel vomit.
[1] Not technically a neural network, but a CPPN. Which is something like a neural network with many different mathematical functions as activation functions. This allows things like a neuron with a sine wave activation that can repeat a pattern across the image.
the school bus one really sticks out to me. it seems all the net cares about is seeing orange juxtaposed with black. No shapes, no vehicle features, just orange and black.
hazarding a guess with no knowledge of the subject, I wonder if that is because no other class in ImageNet can be defined by orange and black. The net simply doesn't need to learn anything more about orange and black, because on 100% of the samples it trained on, orange and black meant "school bus". Every time. So no need to learn any other features -- if you see orange and black, it MUST be a school bus, at least in the context of this data set.
I wonder if we introduced other "orange and black" classes to ImageNet, it would need to learn more features about the school bus in order to identify it.
That's a good observation. But when I see that image I definitely think of a school bus. The color pattern is very distinctive (in the US anyway where all school buses are painted the same color and style.) So I can't say the NN is wrong.
It doesn't necessarily mean that color is the only feature it uses to classify school buses. Just that that feature alone is enough.
would be interesting to try these pictures, the stripes would compete with the other features of the picture, so it might not result in seeing a schoolbus.
