
Life Lessons from Machine Learning - nkurz
https://outlookzen.wordpress.com/2015/03/15/life-lessons-from-machine-learning/
======
u23KDd23
I'm worried about living in a society where we've become overly dependent upon
statistical methods for important decision making especially when we aren't
allowed to question their integrity, precision, and limitations. As a
scientist I know the reality is there is a lot of terrible data being produced
through various methods and I've met a lot of PhD students in the field of
machine learning who argue against starting from scratch even when it's quite
apparent their prior abstractions are incorrect. The reality is there are a
lot of promises being made by ML researchers and dishonesty about their
competency and ability to achieve those results. This is absolutely not unique
to ML and plagues a lot of other fields as well. We really need more ML
researchers to be actively critical about research being performed in the ML
field if ML is going to achieve it's full potential.

~~~
Houshalter
>I'm worried about living in a society where we've become overly dependent
upon statistical methods for important decision making

That's very unlikely as people are incredibly biased against algorithms:
[http://lesswrong.com/r/discussion/lw/lsc/link_algorithm_aver...](http://lesswrong.com/r/discussion/lw/lsc/link_algorithm_aversion/)

There are many areas where even really simple algorithms have been shown to
outperform humans for decades:
[http://lesswrong.com/lw/3gv/statistical_prediction_rules_out...](http://lesswrong.com/lw/3gv/statistical_prediction_rules_outperform_expert/)

Even when humans are given the predictions of the algorithm and allowed to
take that into account, they still do worse than just the algorithm on it's
own. Sure the human might fix an obvious error of the algorithm in one case,
but then makes 10 other worse errors elsewhere.

Despite that organizations are very slow and hesitant to adopt them. There are
massive regulatory and liability issues in many areas. And people are just
generally biased and scared of them. Even on tech friendly places like hacker
news, your comment is at the top of the thread. I remember a post awhile ago
about using machine learning to detect fraud in loans in the third world, and
half the comments were about how evil and racist such and unfair such an
algorithm would be. Not realizing humans are all of those things.

People are very overconfident in human ability despite overwhelming evidence
we _suck_ at predicting things and doing anything statistics related. Human
error is just ignored or seen as an inevitable fact of life.

~~~
u23KDd23
"People are very overconfident in human ability despite overwhelming evidence
we suck at predicting things and doing anything statistics related. Human
error is just ignored or seen as an inevitable fact of life."

Programs don't program themselves. Algorithmic biases often reflect human
biases. If we want people to accept technology and give us opportunities to
pursue our visions of what technology can offer society we need to be
cognitive of ethical and moral challenges especially when there is so much at
stake. Yes there are fields where there are regulatory and liability issues,
but I'm more worried about the fields where there isn't as much oversight and
transparency.

~~~
yummyfajitas
_Algorithmic biases often reflect human biases_

I've been doing this for a while and I've literally never met a human who told
an algorithm to overweight x[23] ("good looking"), x[48] ("is white") and
x[873] ("is wealthy"), for x a 1,100-dimensional feature vector.

Algorithms do have biases, but they are almost always orthogonal to the human
ones. Witness, for example, all the recent "we can fool deep learning image
recognition systems" papers.

[http://arxiv.org/abs/1412.1897](http://arxiv.org/abs/1412.1897)

[http://arxiv.org/abs/1312.6199](http://arxiv.org/abs/1312.6199)

At this point I'm 90% sure you are a layperson who's never actually programmed
such a system.

------
mellavora
Nice summary/intro to ML.

I challenge one point, near the end "Life is the greatest epistemological
problem of all... We arrive into this world not knowing anything, and using
only these mountains of data, we try to put them together in a way that makes
this massive, immensely complex world, slightly more understandable and
predictable."

Your point is of course correct, but perhaps not complete. Two observations:
1) The purpose of the nervous system is to allow more coordinated movement. 2)
Rhythm is the fundamental computational element of life (including brains).

Conclusion: The only purpose of it all is to dance.

~~~
codehalo
Rhythm, Resonance, and Recursion.

------
yummyfajitas
My statistics/ML life lesson: the harder a decision is, the less important it
is to get the right answer.

If choice A is vastly better than choice B, you don't need much information to
determine that. So if, after gathering lots of information, you still can't
determine which is better, don't stress too much about choosing the best one.

A professor of mine pointed this out to me when I was stressing about which
grad school to go to. He then pointed out that at my age, his life choices
were CUNY (math dept) or West Point - that was a decision that would have had
a far larger effect on his life than my choice of Rutgers vs Brown vs Austin.

~~~
NotableAlamode
Does this life-lesson hold in an adversarial world, where the counterparties
actively try to deceive you, e.g. when making financial investments?

~~~
yummyfajitas
That's an interesting question. My entire context and thinking about this is
probabilistic, not adversarial, so I have no idea.

------
raverbashing
Yeah, but you don't know if, for the first graph a straight line is
underfitting or not.

Or if it's really a line or something with, for example, proportinal to T^1.1
(and not T) for example.

(that is possible if you reduce the noise of the experiment, either through
better procedure or more experiments)

Oh and nature may have a different concept of "simple". Solving Maxwell's
equations for a variety of cases is "simple" for nature. Folding a protein?
Simple.

Machine learning is not actually about the scientific method (to find a model
that gives a certain prediction, given the most amount of samples possible)
but to find an ok answer from a limited dataset.

------
darkhorn
Life lessons from machine learning? These are topics taught in Statistics
(Bachelor of Science).

~~~
whazor
Machine learning is mostly applied Statistics. It is a different view of the
same thing, but now the machine is doing the work. This is why avoiding
overfitting is important (as always).

------
SideburnsOfDoom
> We were all born without any knowledge whatsoever of how the world works

Is that really true? A google for "babies are hardwired to" brings up a
enormous number of results.

~~~
hedgew
Not at all true. The simple fact that we're born with two legs is proof that
massive amounts of information about our surrounding world is already encoded
into us before birth. It took millions of years for life to develop this novel
mechanism for traversing this planet.

This, of course, extends to our minds as well. Simply the structure of our
brain has an immense impact. Even if you don't believe that our neurons are
pre-trained to know some things, you'd have to admit that our brain is wired
in a way that allows us to learn quite efficiently in this world. But
psychology is quite complicated, so it's hard to say what exactly is nature
and what nurture.

Technically, when first life began, it really was without knowledge. Over
time, organisms learned and adapted more and more to our environment, both
physically and mentally.

~~~
cheatsheet
This really depends on where you draw the line between knowledge and non
knowledge. Our entirety scientific body is the best explanation we have for
the phenomenon we as a social species have observed. Much of what we call
'knowledge' \- like maths or physics, is actually human produced culture (in
that it is taught after it has been discovered). It is subject to being
incorrect (although it would take a very powerful set of observations,
deductions, definitions, and measurements to prove it incorrect).

I think honestly, the line between nature and nurture really depends on what
you believe makes you human, conscious, and existing with awareness. You can
technically define the entire existence of the universe as a process of
changing states of information (some of which may not be measurable).

------
politegoose
Similarly, studying reinforcement learning raises awareness of the tradeoffs
you need to make between exploration and exploitation (studying to became a
generalist vs. an expert, switching careers vs. continuing in your field, etc)

------
lukas
Really interesting post.

I think working on Machine Learning algorithms early in my career had a pretty
significant effect on how I think about running my business.

The biggest factor by far in a model's quality is the amount and accuracy of
the training data. In the same way, people seem to learn in direct proportion
to the amount and clarity of the feedback they get.

I think groups of people (i.e. companies) improve as fast as the amount of
clear feedback people are giving each other and the amount and accuracy of the
feedback they are getting from customers.

------
kolinko
Two counterarguments:

\- we are not a blank card when we're born

\- but more importantly, for some decisions the amount of data you need to
have an outcome will never get close enough to what you can obtain. That's why
cs majors have problems with real life - they look for algorithms amd data
where only educated guesses are possible.

------
raddad
Dr. Peter H. Diamandis — Intelligent Self-directed Evolution

[https://www.youtube.com/watch?v=1H68gX_uCj4](https://www.youtube.com/watch?v=1H68gX_uCj4)

------
jerf
Further insights I got from machine learning:

1\. There is a mathematically determinable maximum rate of learning. In the
real world, this often comes up at small scales, and once you start learning
to see the world through a machine learning lens you can see people routinely
making enormous leaps off of data that not only doesn't support the leap, it
literally is mathematically incapable of supporting it. As the amount of data
scales up this starts mattering less, and the dominant factor becomes the fact
that as humans, we actually aren't all that great at pulling learned truth
from large amounts of data, though some of this is also that the universe is
pretty darned noisy and if we _were_ "good" at it, we'd learn an awful lot of
untrue stuff. (If that sounds like a description of the current reality... no,
I mean _orders of magnitude_ moreso than today's real science problems.)

2\. The maximum rate of learning is heavily dependent on two things: The
ability of the agent to control input into the environment, and the latency of
the feedback. First, it has been well established in both theory (machine
learning mathematics) and fact (various psychology experiments with cats in
very strange visual environments) that active learning is radically faster
than passive learning. Passively sitting in a room and listening to someone
attempt to present facts is perhaps not the worst approach to teaching, but
it's definitely radically suboptimal.

Second, latency is _huge_. Trying to learn from a latent signal is
intrinsically harder than a less latent one, and it has absolutely nothing to
do with willpower or moral fiber... it is intrinsically, mathematically,
irreducibly more difficult. The maximum speed of learning continues to
increase all the way down into the sub-second response times.

The amount of learning that can even _in theory_ be done under the "listen to
a lecture, two days later get quizzed on it, receive marked-up quizzes back a
week later" model is _shockingly low_ compared to what is theoretically on the
table. If I were in the educational startup field, my near-number-one priority
would be _speed of feedback_. I would literally be happy to hear from one of
my programmers that the yes/no feedback on the arithmetic drilling went from
.5 to .4 seconds. (If you can get your hands on it, try Big Brain Academy or
any of its brethren that have the arithmetic drilling in it, and imagine just
how much faster you could have learned basic arithmetic with this incredibly
responsive tool around when you were a kid. Reminds me, my oldest is getting
near where I ought to be pulling that back out....) Between what's on the
table with lower latency and integrated spaced repetition, there ought to be
almost unbelievable gains on the table for anyone who can put this all
together into the right package. (The aforementioned "Big Brain Academy" line
of games being the closest I've seen. Latency for putatively "educational"
games is often just _awful_.)

On that note, if you are in the educational startup space, it behooves you to
take a machine learning course. Whether or not you are able to apply the code
techniques themselves, the insights about learning itself will more than pay
back your time. And, frankly, I'm virtually desperate to see someone get this
right soon.

(On a sidenote, I'm open to people's suggestions about such programs that do
take latency into account for elementary school age children. Most real-world
programs seem to go in _exactly_ the wrong direction and assume that the
children are not discriminating, so who cares anyhow, let's not spend money on
quality software, and the result is slow, slow, _slow_... agonizing. My
feeling is that if we learn from Big Brain Academy and keep things moving
along, the mere act of learning itself can feel good enough to keep most users
engaged just fine, and that if you make learning boring, no amount of
graphical frippery, loud noises, or encouraging-sounding recorded sound clips
of people saying "very good" can make up for it.)

