
When to assume neural networks can solve a problem - george3d6
https://blog.cerebralab.com/When_to_assume_neural_networks_can_solve_a_problem
======
folbec
"That’s why building a product recommendation algorithm was a hot topic 20
years ago, but nowadays everyone and their mom can just get a WordPress plugin
for it and get close to Amazon’s level."

Well, I haven't had a remotely relevant Amazon recommendation in 5 to 10
years. Unless you count read a book by the exact same author as a useful
recommendation.

~~~
scrooched_moose
At least on my account, it seems like Amazon has given up on recommendations
and is just suggesting I buy the same things again.

I wouldn't be shocked if it's actually quite a bit more accurate then real
predictions, but it does lead to funny things like "You already bought this
book? How about a second copy?"

~~~
pareshverma91
I wonder if the same book recommendations are happening because people might
just be buying products like these for gifts. Something like I liked this
book, so you should read it to.

------
ljlolel
I've made a neural network where you feed it a problem and it will tell you if
a neural network will solve it. I call it a Neural decider. Getting weird
results when I feed it to itself though.

~~~
keiferski
Neuroboros?

------
makach
This reminds me of an oldie, but goodie:
[http://thedailywtf.com/articles/No%2C_We_Need_a_Neural_Netwo...](http://thedailywtf.com/articles/No%2C_We_Need_a_Neural_Network)

~~~
kelvin0
Wow, I had never read this gem before. Made me laugh, thanks for sharing this
piece! I'm surprised they did not ask him to log everything in a blockchain :)

~~~
teraflop
Check the date -- the article predates the Bitcoin white paper by two years.
(It also predates most of the modern "deep learning" hype.)

------
jonesn11
Here is a take from Google AI on when to use ML:
[https://developers.google.com/machine-learning/problem-
frami...](https://developers.google.com/machine-learning/problem-framing/good)

"You should understand the problem well enough to list some potential
solutions to test first to find the best model" and "it shouldn't be used for
a problem that conventional programming can solve" like hard coding a symbolic
rule.

~~~
rahimnathwani
Unless it's fizzbuzz:

[https://joelgrus.com/2016/05/23/fizz-buzz-in-
tensorflow/](https://joelgrus.com/2016/05/23/fizz-buzz-in-tensorflow/)

------
manthideaal
it seems that neural networks have problem for computing the maximum function,
and a human can compute the maximum easily, so it seems that the three
heuristic rules don't work in this case.

(1) [https://datascience.stackexchange.com/questions/56676/can-
ma...](https://datascience.stackexchange.com/questions/56676/can-machine-
learning-learn-a-function-like-finding-maximum-from-a-list)

~~~
6gvONxR4sf7o
You're referring to about whether a generic one-size-fits-all model will do
well, but ML is full of bespoke models. It would be simple to build _a_ neural
network that can compute (and differentiate through) the max function to
within some arbitrary epsilon, even though the most generic model (feed
forward network) won't do great.

~~~
george3d6
> l (feed forward network) won't do great.

See my answer below, in the case of this problem a generic feed-forward
network, even a simple one, will work.

Not any ffn, but assuming you are using an efficient architecture search it
will probably find one that works.

There's other numerical problems where this doesn't hold but that's another
story.

~~~
6gvONxR4sf7o
You demonstrated it for a reeeeeeeallly constrained version of the problem. Do
you expect your solution would generalize to many lists? Because it would be
easy to make a neural network that does, while your toy example (and larger
generalizations) probably won't generalize super well.

x_i = ith list element from list x

y = sum(x_i * softmax(k * x)_i)

This one parameter, arbitrarily wide network one will get arbitrarily close to
the max function.

This is a super toy version of why attention is so effective. It can pick
stuff.

------
mooshmoosh
> 3\. A neural network can solve problems that a human can solve with small-
> sized datapoints and little to no context

I like this rule, because it is actionable. Its easy to take a sample of your
data, (in the case of images) lower the quality to 20x20 pixels and get see if
you, or another person can do the classification task. Audio could be handled
similarly.

------
imvetri
When a neural network solved a problem, it means it found the algebraic
function it needs to solve the problem.

> Once it found the algebraic function, there is no need to run the problem to
> the neural network. We can sideload the problem to a simple program that
> takes input and gives an output. This strategy can help to free up the GPU
> for next set of problem.

~~~
ludwigschubert
How would one extract an algebraic function from the network’s weights? Why
would the function the network approximates necessarily be algebraic?

~~~
Plough_Jogger
The function would simply be a chain of multiplications and (typically) non-
linear transformations.

For classification problems, there is often a final non-linear transformation,
like a softmax.

Example with ReLu activation:

Output_i = max((input_i * weight_i),0)

~~~
jawarner
I don't understand. The example you gave is not an algebraic function.

~~~
Plough_Jogger
You are correct. It is possible to construct a linear neural network, where an
algebraic function could be extracted, but in practice, almost all networks
use non-linear activation functions.

In the case of a linear network, the function would be a dot product between
the input and the weights: 𝑥1𝑤1+𝑥2𝑤2+𝑥3𝑤3 ... for all inputs (xi) and weights
(wi)

------
lidHanteyk
I don't understand why the author thinks that the PAC learning framework is
not useful. PAC gives quick back-of-napkin math for estimating how long it
might take to train a network, what kind of quality level is acceptable for
inputs, etc.

~~~
eugenhotaj
I disagree, PAC bounds are usually way too loose to be useful in practice when
it comes to NNs. I think part of the reason is that the way we measure model
complexity for NNs does not correspond very well with their ability to
generalize, as we've seen with recent papers like deep double descent [1].

Anecdotally, I was of the main authors of the AdaNet framework [2] where we
used (approximate) Rademacher complexity (closely related to PAC) to bound the
complexity of our learned ensemble models. We never got good results using the
complexity measure and would basically turn it off for any practical problem
we solved.

[1]: [https://openai.com/blog/deep-double-
descent](https://openai.com/blog/deep-double-descent)

[2]:
[https://github.com/tensorflow/adanet](https://github.com/tensorflow/adanet)

------
ramoz
What is the current state of NN in the context of AI state of the art?

------
gambler
I've been hearing about deep learning revolutionizing "everything" for the
past 8 years. So, can someone name me any significant impact it made on
figuring out what to do about the recent epidemic? If not, I think it's worth
reflecting on what value we get out of the technology that sucked up so much
of our intellectual, financial and computational resources.

~~~
malux85
If you're listening to people who claim any one technology solves "everything"
\--- and believing them --- then _you_ are the fool.

There's no silver bullet. Deep Learning has made huge strides in noise
reduction, image and video processing (self driving cars), lidar processing,
medical imaging, etc, but it's not made any progress in knitting, tooth
brushing, dog walking, or many other tasks ill suited for deep learning - so
clearly not everything.

~~~
yters
That's been the general vibe the popular tech press has been spouting for
awhile. That NNs are somehow a major step towards AGI, which'll 'capture the
lightcone of the future' to quote Sam Altman.

At some point, we'll realize that AGI is the modern alchemy. Ironically,
alchemy's penultimate goal was to construct a homoculus, which is essentially
the goal of AGI.

[https://en.wikipedia.org/wiki/Homunculus#Alchemy](https://en.wikipedia.org/wiki/Homunculus#Alchemy)

"The allegorical text suggests to the reader that the ultimate goal of alchemy
is not chrysopoeia, but it is instead the artificial generation of humans."

~~~
criddell
_Penultimate_ means the second last thing in a set. In the alphabet, 'y' is
the penultimate letter.

~~~
yters
The last thing is 'profit'!

~~~
criddell
Ha! Makes sense then. :)

