
The Neural Network Zoo - hgarg
http://www.asimovinstitute.org/neural-network-zoo/#
======
cs702
Good job.

Alas, the proliferation of different kinds of neural net architectures _that
work_ , over the past few years, is a sign that we lack a decent unifying
theoretical framework that can explain, from fundamental principles, what
works, what doesn't, and why. We're not there yet.

~~~
Houshalter
Look at computer science. They have invented so many different algorithms that
work! This is a sign that computer science lacks a decent unifying theoretical
framework.

Having different algorithms for different purposes is fine. For instance,
autoencoders can do unsupervised learning, GANs can learn generative models
that sample from the entire distribution, recurrent neural networks can handle
time series, etc. Also while there are many different types of networks,
research has shown which ones work and which ones don't. Few people pretrain
autoencoders or bother with RBMs anymore, for instance. And I think we have
good theoretical reasons why they aren't as good.

But to continue the analogy to computer science, imagine all the different
kinds of sorting algorithms. They will each work better or worse based on how
the data you are sorting is arranged. If it's already sorted in reverse,
that's a lot different than if it's sorted completely randomly, or if it was
sorted and then big chunks were randomly rearranged.

There's no way to prove that one sorting algorithm will always do better than
another, because there's always special cases where they do do better. The
same is true of neural networks, it's impossible to formally prove they will
work, because it depends on how real world problems are distributed.

~~~
LeanderK
I don't think that this is a sign that computer science lacks a decent
unifying theoretical framework.

We can even proof that there is no optimal sorting algorithm. "For every
sorting algorithm x exists an input i for which an sorting algorithm y exists,
so that y sorts i faster than x" is provable

~~~
joe_the_user
I think the GP was being sarcastic and casting doubt on the need for or
importance of a unifying theoretical framework in CS.

Which is a shame. I think there are unifying frameworks in CS but think we
could use more and I think we would benefit if, when people looked at a body
of knowledge and saw just a list of techniques, they would say "hey, this
could benefit from a more unified treatment".

~~~
LeanderK
well thats a typical case of editing without reading the answer if was
referring to first. I read Andrews answer and then changed my comment without
re-reading the answer i was referring to.

I normally hate when this happens, so i have to apologise.

------
gugagore
The explanation about MC is too wrong to be useful. What on earth do the
"nodes" (states) in a Markov chain have to do with the "nodes" (neurons) of a
neural network?

