
Learning Concepts with Energy Functions - stablemap
https://blog.openai.com/learning-concepts-with-energy-functions/
======
nerdponx
From the abstract of the article they linked:

 _Energy-Based Models (EBMs) capture dependencies between variables by
associating a scalar energy to each configuration of the variables. Inference
consists in clamping the value of observed variables and finding
configurations of the remaining variables that minimize the energy. Learning
consists in finding an energy function in which observed configurations of the
variables are given lower energies than unobserved ones. The EBM approach
provides a common theoretical framework for many learning models, including
traditional discriminative and generative approaches, as well as graph-
transformer networks, conditional random fields, maximum margin Markov
networks, and several manifold learning methods._

 _Probabilistic models must be properly normalized, which sometimes requires
evaluating intractable integrals over the space of all possible variable
configurations. Since EBMs have no requirement for proper normalization, this
problem is naturally circumvented. EBMs can be viewed as a form of non-
probabilistic factor graphs, and they provide considerably more flexibility in
the design of architectures and training criteria than probabilistic
approaches._

Seems like a really interesting unification of the wide variety of techniques
out there in statistics and machine learning, analogous to the "everything is
a computation graph, as long as it's differentiable" revolution. I like it
when this kind of thing has its day. Would be interesting to see how well it
works non-robotics problems.

~~~
BucketSort
See Yann's tutorial on EBM:
[http://yann.lecun.com/exdb/publis/pdf/lecun-06.pdf](http://yann.lecun.com/exdb/publis/pdf/lecun-06.pdf)

We actually ran into them when doing research in our startup. It is a really
powerful perspective.

~~~
lucidrains
Thank you for this!

------
IIAOPSW
Setting up some energy function and then finding the lowest energy state
sounds a lot like adibatic quantum computing. Assuming this research lives up
to the hype, quantum computers might be able to run this algorithm faster.
Quantum machine learning is already a thing, but its nice to see it fit so
congruently with a classical counterpart.

~~~
discoball
Quantum Simulated Annealing?

------
arashout33
I have no idea what's going on in this article. Is there a good resource or
video for understanding this stuff?

------
yters
Every new technique is able to "quickly learn X." Something most not be so
quick, otherwise why aren't these techniques turning into AGI?

I think the problem is the goal is not well defined. So, increased velocity
has no bearing on increased velocity towards the target.

A side question, why is there no research into whether human intelligence is
computable? The assumption in AI is that human intelligence is computable, but
I've never seen any good argument or evidence that this is true. Seems very
unscientific to exert so much energy into this research direction without
validating the fundamental assumption.

For example, the one instance I know of that defines AGI in a quantitative
manner is Solomonoff induction (SI), but it is not computable. If SI is
representative of human intelligence, then AGI is impossible.

~~~
jerf
Solomonoff induction is not representative of human intelligence.

At least, speaking for myself, I do not exhaustively search literally every
hypothesis and match it against my data. Your mileage may vary. Dunno. It's a
diverse world out there, right?

~~~
yters
Solomonoff induction is a way to quantify the human ability to induce general
principles from limited observations. All the computable methods are unable to
achieve this ability.

~~~
jerf
With respect, no, that is not what it is. My summary is glib and phrased in
non-mathematical terms, but closer.

A true Solomonoff Inductor would be wildly, wildly smarter than a human being,
if it could get over the problem that such a machine would also consume super-
exponentially more resources than the universe has.

~~~
yters
It's not a matter of resources. Solomonoff induction is not computable, since
it needs to calculate the Kolmogorov complexity of the data.

If that is what is required for induction, it is surprising that humans are
able to do so well at identifying concise descriptions of the data we observe.
This seems inexplicable with a computational view of human cognition.

