
Local Minima, Saddle Points, and Plateaus - hellerve
http://www.theorangeduck.com/page/local-minima-saddle-points-plateaus
======
jpfed
This made me think about code. I've long thought of incremental refactoring as
"following the gradient" of good design principles, but this essay opened up
new connections for me.

For example, I've heard elsewhere that great programmers possess the skill of
being able to decompose large changes into smaller commits, making the code no
worse as they go. Having the ability to see the small changes that get you
where you want to go corresponds to finding the direction of change that gets
you past the saddle point.

Random restart is a standard tool in the kit of stochastic gradient descent.
But if local minima are rare, it may be overused. You don't have to rewrite
your code if you can see the refactors and gradual improvements that get you
where you need to go.

------
vnorilo
I enjoyed this article. It got me thinking that one of the problems in life is
that the loss function is unknown: we don't always know beforehand what would
make us happy or satisfied. However, it's a lot easier to gauge one's current
gradient: whether things are getting better or worse, or stagnant, as in
plateau.

~~~
blt
Life is reinforcement learning, the reward signal is dopamine :)

~~~
solveit
Side effects may include an opioid epidemic :(

------
ncmncm
I just wish they would say "minimum" when they mean minimum.

In real high-dimensional spaces, like life, exploring away from local minima
and plateaus ("plateaux"?) is expensive. Figuring in the expense of
exploration distorts the reward function intriguingly.

------
Chirono
This translates well into advice for startups: Don't obsess over finding the
perfect solution, but focus on getting _something_ then making constant
improvements to what you have. Don't try to jump to a global optimum straight
away. There are always a huge number of options to choose from, and you can
always improve.

------
james_s_tayler
I love this analogy. There's a great lesson here in assuming the saddle point
case and trying to eke out the additional gains you know could be close by.

The tricky part in real life is figuring out the dimensionality and then
reasoning about how changes in the gradient of one dimension appear to affect
changes in the gradient of another.

------
MarsAscendant
While I believe to understand the general idea behind the theory as applied to
real life, I'm struggling with a few concepts.

What are local minima, in relation to the real life? Are those the "places",
mentally speaking, where one feels like they're failing?

What, exactly, are saddle points in this context?

~~~
coldtea
> _What are local minima, in relation to the real life? Are those the
> "places", mentally speaking, where one feels like they're failing?_

They are places where you're doing OK but can't seem to make progress to
"better than OK" by moving a little away from them (e.g. for many steps away
from that situation, things are getting worse -- you need to travel a lot to
get out of the "pit" you're in).

Consider someone obese. They feel worse when doing exercize/diet (compared to
e.g. munching a pizza), but the feelings they have are not the best they could
have related to their weight situation -- they'd feel best if they could
persevere and eventually feel fit/healthy.

Saddle points are those where you can't seem to make easy progress on one
direction (one aspect, e.g. how you feel about your health) but you can on
another (e.g. how much money you make).

It's a little more involved (because if you're obese you're always best to
lose, whereas a function can try both increasing and decreasing values to get
out of a local minimum) but that's the gist I think.

~~~
MarsAscendant
Thank you.

I'm still having trouble visualizing it, though. I reckon it's the kind of an
idea where, if you don't give me a clear-enough image, I'm stuck no matter how
well you pick your words others.

Not to put any kind of pressure on you, of course. I'm grateful for the
explanation. It's my brain that does its tricks.

