
Surprising importance of spontaneous order and noise to how we think - dnetesn
http://nautil.us/issue/68/context/why-the-brain-is-so-noisy
======
crdrost
A very similar effect is harnessed in current machine learning models and is
called "dropout". The basic idea in context is:

\- you take your reference data and reserve a random chunk of it to figure out
how good you're doing, this data is _only_ used to grade performance -- the
rest is for training.

\- you notice that as you train on the training data, you start improving your
performance on the evaluation data, but only up to a point: then as you train
more, the neural net starts getting worse on the evaluation data.

\- this problem is called "overfitting", your neural net is now trying so hard
to get the "details" right that it is losing accuracy on the "big picture".

\- so we create a circumstance where the neural net cannot properly do
overfitting because those "details" cannot really be resolved, by just
dropping nodes at random out of our neural network as we're training it. It
will have to pick up a level of redundancy in the nodes and the associated
noise of losing nodes should stop the later stages from really having a
"focused view" of the training data, such that it can overfit that data.

If you're interested see e.g. Adrian's Medium post about it, here:
[https://medium.com/@bingobee01/a-review-of-dropout-as-
applie...](https://medium.com/@bingobee01/a-review-of-dropout-as-applied-to-
rnns-72e79ecd5b7b)

~~~
zozbot123
Dropout is computationally expensive though. It does address overfitting (by
doing something like training an ensemble of weaker models, none of which will
individually overfit, but all of which capture different "views" of the
dataset), but it's not without its drawbacks, and people seem to be using it
less these days.

~~~
sdenton4
Qua? I was under the impression that it was just going into architecture with
less remark - ubiquitous, rather than dying. It's a great step towards
building sparse models, as well, which is important for client side
deployments.

------
cabaalis
> When a new image is sufficiently different from the set of training images,
> deep learning visual recognition stumbles, even if the difference comes down
> to a simple rotation or obstruction.

I'm about as far from an ai expert as you can get.

When I see and recognize a school bus, it seems that object remains a school
bus to me until there is very significant evidence otherwise, whether it is
ahead, beside, tipped over, or behind as referenced in the example.

It would seem ai on a single image is problematic, and needs classification
over time to gain "confidence" instead of a single attribution.

Edit/additional thought: It also seems to me that I know and accept that it's
a "bus" before I know it's a "school bus" while another person might
immediately recognize a "school bus" and then think "that's a type of bus."
How wonderful to think of how those arrangements of hierarchies leads to
differing opinions and creative abilities in humans.

~~~
jerf
"It would seem ai on a single image is problematic,"

It means that whatever it is that these latest models are doing, it still
isn't what we are doing as humans.

What exactly that difference is... well, if you could confidently and even
more importantly, _correctly_ tell me, in a way so detailed and correct it was
implementable, you'd be able to become very rich.

~~~
taneq
Well for starters, humans work on continuous video streams rather than still
images, so there's a ton more information there. Even when we're identifying a
still image, we're looking at a video stream of an object showing a still
image (which is why a photo can look "exactly like the real thing" but we're
never in any doubt that it's a photo and not the real thing.)

------
rdlecler1
I once built an evolutionary algorithm to evolve the neural network of a
virtual robot. When I saved to disk and tried to rerun it produced a different
result. Turns out the performance of the network was sensitive to floating
point precision errors. I added some slight noise to the input values which
made it much more robust.

------
perseusprime11
Brain is lot more complex. Trying to equate Brain to machine learning or
neural network is not correct. The key ingredient of the brain is the reptile
brain which is based on fight or flight and highly optimized for survival.
Everything uses this part. In order to build similar, we have to build a
neural network based on this survival instinct.

------
ggm
The most important sentence is the last sentence:

 _There’s still a considerable gap between real intelligence and so-called
artificial intelligence_

My Tl;Dr on the article is that it's a lot of maybe, we don't know, we don't
know why, we are not sure, maybe.

But I certainly agree with the final sentence.

