
"Memory foam" approach to unsupervised learning - bpolania
http://arxiv.org/abs/1107.0674
======
hacker_9
_We propose an alternative approach to construct an artificial learning
system, which naturally learns in an unsupervised manner. Its mathematical
prototype is a dynamical system, which automatically shapes its vector field
in response to the input signal. The vector field converges to a gradient of a
multi-dimensional probability density distribution of the input process, taken
with negative sign. The most probable patterns are represented by the stable
fixed points, whose basins of attraction are formed automatically. The
performance of this system is illustrated with musical signals._

Could anyone eli5 this?

~~~
jamessb
They estimated how likely particular notes were, using a kind of averaging
procedure that corresponding to simulating an elastic foam. They used a
fourier transform to identify the notes in a piece of music: they then
simulated the physical procedure of adding a stone to the foam at a location
corresponding to the frequency, and time corresponding to when the note was
played. The final shape of the foam then gives a density estimate of the data:
the more often a particular note was played, the deeper is the dip in the foam
at that location ('the foam evolves into a time-averaged density of the
input').

Once you've done this, you have a curved surface. You can imagine a ball being
released at some point and rolling around: it could get stuck in the bottom of
a well ('a stable fixed point'), which corresponds to the 'most probable
patterns' in the training data.

Edit to add: _" The vector field converges to a gradient of a multi-
dimensional probability density distribution of the input process, taken with
negative sign"_ You can think of the "vector field" of the dynamical system
describing the movement of the ball as a bunch of arrows, giving the direction
the ball would move if released from each possible starting point. If you
trace these arrows end-to-end, you can see the trajectories that the ball
could take.

Since the ball will roll downhill, the vector field of the system is (minus)
the gradient of the potential energy/height. Also, because of how we formed
the surface, its height 'converged to the probability density of the input
process' (the more frequent an input was, the lower the corresponding dip in
the foam). Thus, the vector field of the system converges to the gradient of
the probability density of the input.

------
jamessb
Another arxiv preprint by the same authors has more detail, and seems to be
the essentially the same article but fleshed out more:
[http://arxiv.org/abs/1111.4443](http://arxiv.org/abs/1111.4443)

~~~
nharada
Thanks for this. I was amazed the authors offered no comparison to any
existing unsupervised DBN/ANN work.

------
dsfsdfd
Yay, someone with the sophistication required is working on this! As I've been
thinking for a few years, intelligence is more about absorbing pattern from
the world than it is about the deliberate process of constructing pattern from
scratch within the system. It's is better to create a space where the pattern
of the world can live, subject to some measure of objective utility, than it
is to deduce some pattern from scratch using that same objective function.

God I would love to work in this field.

~~~
visarga
Cognition is pattern recognition, and we don't ever access the raw data, just
the output labels - this is so Buddhist. In a sense we create our (inner)
world because it is just patterns.

~~~
baobabaobab
Recognizing patterns in data is so easy to do, that how easy it is, turns out
to be a major problem in machine learning. You can always find some
explanation that perfectly fits all the data you see, but that doesn't mean it
will fit the data you haven't yet seen.

The challenge is finding the simplest patterns that will generalize to explain
the most data, while wasting as little effort as possible on the irrelevant
patterns. In high dimension data, the number of possible relationships to
analyze explode, you can find patterns everywhere you look, so it's deciding
where to bother looking with your limited resources that's hard.

That patterns are nothing special doesn't seem obvious to us because,
evolution has done a pretty good job solving this problem(in the domain of
inputs we evolved to deal with), and we only perceive those patterns that are
likely to generalize.

------
im3w1l
This sounds similar to Kernel Density Estimation. How does it compare to it?

------
webmaven
An implementation to play with would be very nice...

