
Bill Gates is naive, data is not objective - cs702
http://mathbabe.org/2013/01/29/bill-gates-is-naive-data-is-not-objective/
======
JumpCrisscross
Requiring absolute certainty as a subjective being yields an incomprehensible,
absurd universe. The scientific method is based on testing models, i.e.
systems of measures, against each other.

I volunteer at two institutions: (1) as a reading tutor at a Harlem charter
school and (2) as a mathematics tutor for disadvantaged youth in Lower
Manhattan. At 1 the administrators regularly assess students' reading levels.
The process is highly subjective but allows for individuals' effectiveness to
be tracked, as well as tutors' and methods'. At 2 the administration is more
ad hoc, giving each tutor more freedom but having no system for measuring
progress. 1 has a good idea about what works and a programme with demonstrable
effectiveness. 2 is a more challenging environment as every problem calls for
starting from square one. I set my own measurement criteria but due to a
small, variable sample they have low statistical power.

In both cases the measurement is subjective. Strictly codifying the measure
and then lording over it would be unproductive. But having a measure and
optimising processes to the measure creates a feedback loop. Simultaneously
remaining vigilant of exceptions, e.g. a child who could read complicated
words so long as they don't contain 'q', creates a meta-loop that optimises
the measure-process system. Together one has a system that evolves, that
learns.

" _We see that cars are safer for men than women because the crash-test
dummies are men._ "

Crash dummies have made cars safer for women. Perfect is the enemy of good.

~~~
jtbigwoo
_Crash dummies have made cars safer for women. Perfect is the enemy of good._

I know you just threw this in at the end, but "Perfect is the enemy of good"
does not apply here. I'd argue the better quote would be "Good enough is the
enemy of better." NHTSA has been around since 1970 but they didn't start
testing with different-sized dummies until 2003. I don't think the author is
suggesting that we needed 200 different-sized crash dummies in 1970, but why
did it take 33 years to notice that our dummies didn't match most drivers? We
could have tested smarter and better and we didn't.

~~~
JumpCrisscross
I read the post as criticising Gates for (1) not being transparent about the
models he's using, and, (2) using models that aren't "objective", taken to
mean not accurate, and being adopted arbitrarily because of his authority. 1
is a valid point if true, but 2 occupies the bulk of the post. The standard
for accuracy appears set too high, illustrated by calling the crash dummy
testing process un-objective for having missed a nuance. Worse, the proposed
solution to a perceived lack of objectivity is rejecting the system. This is
essentially nihilism, whose conflict arises because it rejects any model that
isn't perfectly "objective".

33 years is too long to notice an exception to 50% of the population. That,
however, doesn't reduce the "objectivity" nor utility of said model. It just
means it could have been better. Similarly, valid criticisms are to be had of
the Gates models. That does not mean we say it's not objective and dismiss it
as the manifestation of a rich man's ego.

~~~
ScottBurson
You've missed her point entirely. She's not saying that only "objective"
models should be used; she's saying that there's an _irreducible_ subjectivity
in the _choice_ of model, and that this needs not to be forgotten. She's not
saying that measurements should not be taken; she's saying that we need to
discuss what to measure, and make sure that what we're measuring is what we
actually want to see improved. Her point is not nihilist in the slightest.

------
brudgers
_"Gates also brings up the campaign to eradicate polio and how measurement has
helped so much there as well. Here he sidesteps an enormous amount of politics
and debate about how that campaign has been fought and, more importantly, how
many scarce resources have been put towards it. But he has framed this fight
himself, and has collected the data and defined the success metric, so that’s
what he’s focused on."_

I was looking for some meat to back up the author's claims in regard to
mosquito nets etc. Instead I read the above nonsense about polio and realized
that the author was either fabricating claims from whole cloth or grinding an
ax with Bill Gates.

The campaign against polio is in its 25th year. Gates involvement was a
challenge grant to Rotary International's Polio Plus campaign in 2009 He never
set the metrics. They had been in place for twenty years. I was active in
Rotary when it was announced. Several local members had been volunteers in far
corners of the world on vaccination drives. Some more than once. One more than
a few.

The grant was sized to allow the existing effort to finish its work. The Gates
foundation didn't set the agenda, let alone Bill Gates. The foundation employs
experts is my understanding.

[http://www.gatesfoundation.org/speeches-
commentary/Pages/bil...](http://www.gatesfoundation.org/speeches-
commentary/Pages/bill-gates-2009-rotary-international-speech.aspx)

<http://en.wikipedia.org/wiki/Polio_eradication>

~~~
majc2
Absolutely right. Furthermore, regarding the claim that Gates has sidestepped
the politics of polio eradication is also incorrect - Gates has been in the UK
talking about this to the UK media this week.

"This plan says that if the world supplies the necessary funds, political
commitment, and resolve, we will certify the eradication of polio by 2018." He
continued: "Funds, commitment, and resolve… These are the key variables.

[http://www.telegraph.co.uk/health/children_shealth/9835691/B...](http://www.telegraph.co.uk/health/children_shealth/9835691/Bill-
Gates-world-can-eradicate-childhood-polio-within-six-years.html)

------
kenjackson
The author seems to have constructed a strawman. And I think the point she
misses most is transparency of data and the feedback loop. If you have these
two things it puts a light on the problem points she discusses.

That is, having a model is not sufficient. You need a model and then a way to
measure that a model is a success, and a feedback loop. All of these things
are open and can be updated. Things like seatbelts being better for men become
more obvious, and the model can be improved to include gender weights, etc...

At the end of the day a model is a model. Almost always imperfect. But if the
model is open, as is the metric of success, and how the feedback loop works, I
think you'll tend toward improvement and better discussions.

~~~
gvb
The title is wrong, data is objective but the _models_ are not and _which data
is used_ is not (and she argues both those points).

She links to Gary Rubinstein's blog[1] that is mind boggling in its analysis
of the results of an actual model of teacher's effectiveness and the resulting
data.

1) The model that NYC used to evaluate a teacher's effectiveness is not
available. It appears that the input data is not available either, only the
output data of the model. That isn't transparency.

2) The output of the model is random. Having a feedback loop on random output
data isn't going to improve the model.

Maybe Bill Gates' model and data will be better and will be transparent. I'm
skeptical.

[1] [http://garyrubinstein.teachforus.org/2012/02/26/analyzing-
re...](http://garyrubinstein.teachforus.org/2012/02/26/analyzing-released-nyc-
value-added-data-part-1/)

------
moistgorilla
Keep in mind guys that this is the same person who wrote about Nate Silver
confusing cause and effect. Which was also thoroughly wrong (you can find the
reasons why it is wrong in the first post of the hn link).
<http://news.ycombinator.com/item?id=4949103>
<http://mathbabe.org/?s=nate+silver>

~~~
yummyfajitas
This is an ad-hominem attack, and pointless. A person can be wrong on some
topics without being wrong on every topic.

~~~
moistgorilla
Pointless?

Two articles attacking two people that are held in high regard with link baity
titles. I mean, come on, "Bill Gates is naive", "Nate Silver confuses cause
and effect". This person is obviously trying to grab page views with whatever
they can come up with. These articles could have been worded in much less
offensive and aggressive ways but that wouldn't be conducive to the authors
goal.

------
vph
The author doesn't get what Bill Gates is saying. I doubt if Bill is naive
enough not to understand what the model of measurement is important and never
100% objective. Bill is saying (1) teachers' performance needs to be measured;
and (2) college ranking is based on a wrong criterion.

And we must do something to fix this. Bill isn't saying he has the perfect
model for measuring these two things. But he is right in saying both of these
things.

Teacher performance in K12 needs to be measured. It's hard, maybe expensive,
and certainly subjective and biased in certain ways. But it needs to be
measured. In college, teachers' performance is constantly measured. No
professor will tell you that the student evaluation is a perfect indication of
his/her ability to teach. But most professor would agree that such a thing is
necessary.

~~~
wes-exp
College professors tend to be recruited based on their ability to do research
and get grants, so measuring them on teaching performance is non-threatening.
However, the situation is different in K12.

Bill Gates has been able to apply his measurement philosophy at Microsoft, and
the results have been disastrous by most accounts.

Here's a paper by an HR grad student slamming the practice:
<http://stevegall.wetpaint.com/page/Human+Resource+Management>

Money quote: "Stack ranking provides questionable value as to insight into an
individual’s actual job performance. Its use highly politicizes an
organization. The rank number is most often based on unsubstantiated
subjective judgment by an evaluator who may feel pressured to respond
according to a narrow set of guidelines."

I love science. But measuring employee or teacher performance is not science.
Managing an organization by pseudo-science should be called out for what it
is: B.S.!

~~~
vph
Teaching colleges of course take teaching seriously. Research universities
place higher importance in research, but do take teaching also seriously
simply because undergraduate tuition contributes to a big chunk of their
revenue.

>But measuring employee or teacher performance is not science.

This assertion is baseless.

------
krutulis
I don't think that Bill Gates claims decent measurement is sufficient for the
kind of progress he seeks but rather claims that (in most cases) it is
necessary.

I suspect one answer to Mathbabe's question _"what can a non-academic
mathematician do that makes the world a better place?"_ might be to highlight
and explain the kinds of logical dependencies that are so often obscured by
polemics.

I cringe at the suggestion Bill Gates is naive. Anyone who has spent the time
he has in corporate America will be well aware of the challenges and
politicization of measurement.

------
VeejayRampay
I like how the title introduces a HUGE bias against the author.

I know it's all trendy to say whatever can bait some viewers to blog posts and
stuff, but come on, he's one of the richest men in the world, one of the
greatest entrepreneurs of recent history and currently solving world-wide
health issues, let's have some decency.

~~~
wnight
The author called him naive, not a monster.

Also, Bill might be rich but under his command Microsoft faked evidence in
federal court, etc, to get and keep it. If the law worked the same for him as
for everyone else he wouldn't have so much money to give away.

~~~
VeejayRampay
He did some questionable things, true. He's not a saint, true as well. But
then again, name ONE great entrepreneur that never did any shady stuff...

And at least he's not hoarding the money like other CEO douchebags and doing
what he thinks is best for mankind. I like to think that's commendable.

------
yummyfajitas
tl;dr;

Data is objective. But a decision process incorporates both data and a utility
function (a goal [1]), and the utility function/goal is based on values.

The successes that Bill Gates ascribes solely to data collection are actually
due to a choice of utility function, together with using data to measure it
and make decisions based on it.

However, the claim made in the title does not agree with the content of the
article. Data is objective. Your choice of goals is the thing that is not.

Bill Gates chose as his goal (# of children who can read) - others prefer (#
of teachers with lifetime job security) or (3 x # of children who can read + 2
x # of children who can do math). It is absolutely true that no amount of data
will change your fundamental goals, and we should recognize this.

[1] <https://en.wikipedia.org/wiki/Utility>

~~~
lutusp
> Data is objective.

Almost never true. One of the responsibilities of science is to gather data in
a way meant to minimize sources of bias in the data collection process. Any
number of studies have come to a very predictable, and wrong, conclusion,
based on biases and errors in data collection.

> It is absolutely true that no amount of data will change your fundamental
> goals, and we should recognize this.

Yes, and one point of science is to separate the process of data collection
from any particular goal or outlook.

~~~
yummyfajitas
Bias and error are not the same as a lack of objectivity.

A loaded coin may come up heads 75% of the time. Using that coin to make a
decision yields a biased procedure. It's also objective, since the bias is
unrelated to the experimenter.

~~~
lutusp
> Bias and error are not the same as a lack of objectivity.

Say what? Bias _is_ a lack of objectivity, that's how the word is defined. And
errors can be managed by a combination of procedural discipline and peer
review. In other words, the essentials of science.

> A loaded coin may come up heads 75% of the time. Using that coin to make a
> decision yields a biased procedure. It's also objective, since the bias is
> unrelated to the experimenter.

No, never. The outcome of the study is not objective if the experimenter
believes the coin to be fair. And if the experimenter knows the coin is
unfair, then it's not objective for a different reason.

Objectivity is not a debating point as in post-modernism, it's a prerequisite
for science, and with sufficient rigor, it can be established. This is not to
argue that this is always true, but it is always possible.

~~~
yummyfajitas
_Bias is a lack of objectivity, that's how the word is defined._

No. According to a quick google search, "objectivity - judgment based on
observable phenomena and uninfluenced by emotions or personal prejudices."

There are many types of error, and they aren't all the same.

a) Bias - an estimator's expected value differs from the estimatee's expected
value.

b) Inaccuracy - an estimator's variance is large.

c) Lack of objectivity - the experimenter is applying incorrect methods for
reasons other than lack of skill.

The first two are properties of a statistical method, the third is a property
of the statistician. All are orthogonal to each other.

~~~
lutusp
>> Bias is a lack of objectivity, that's how the word is defined.

> No. According to a quick google search, "objectivity - judgment based on
> observable phenomena and uninfluenced by emotions or personal prejudices."

You just tried to deny the accuracy of a definition by quoting the definition
of the word's antonym. Language doesn't work that way.

> All are orthogonal to each other.

Also false, and obviously so. In your list of points, (a) and (c) are
identical, because "objective" means "a lack of bias".

Source: <http://www.thefreedictionary.com/objectivity>

Quote: "Objectivity ... lack of bias".

Source: <http://www.vocabulary.com/dictionary/objectivity>

Quote: "Objectivity is a noun that means a lack of bias, judgment, or
prejudice."

And your point (b) confuses inaccuracy and variance, which are distinct
factors. All your points (a through c) result in an easily definable and
scalar error factor, not at all orthogonal.

~~~
yummyfajitas
You realize that "lack of bias" comes from the thesaurus, not the definition
of the word, right?

If you want to use an unusual definition of a word in order to make it apply
to all errors (rather than only some), be my guest. There is no point
disputing definitions. By your definition, you are correct, by the common
definition, you are incorrect.

<http://lesswrong.com/lw/np/disputing_definitions/>

~~~
lutusp
> You realize that "lack of bias" comes from the thesaurus, not the definition
> of the word, right?

What distinction do you think you're making? Theasuri list synonyms and
antonyms. Also you're mistaken:

Source: <http://dictionary.reference.com/browse/objective>

Quotation: "not influenced by personal feelings, interpretations, or
prejudice; based on facts; unbiased: an objective opinion."

Source: <http://www.thefreedictionary.com/objective>

Quotation: "undistorted by emotion or personal bias".

> If you want to use an unusual definition of a word ...

I just proved that I am using _the_ definition of the word. If you're not
happy with what dictionaries have to say on this issue, then begin a campaign
to change the meaning of "objective".

> There is no point disputing definitions.

So stop doing that. I'm not disputing the accepted definition, I'm simply
posting it. Copy, paste.

~~~
yummyfajitas
All the definitions of objectivity describe a property of the experimenter,
not the method.

A method can be biased too. Take, for example, the estimator S^2 = (1/n) sum
(x[i]-mean(x))^2. This is a biased estimator of the standard deviation, but
nevertheless it is objective. It is not influenced by the state of the
experimenter at all.

Personal bias contradicts objectivity, but not all bias is personal bias.

Feel free to conflate all errors under one label - those of us who care about
getting our measurements right don't have that luxury.

------
lutusp
The author makes the point that data are not enough, but then doesn't drop the
other shoe. The "other shoe" in this case is that data collection can only
_describe_ , but cannot _explain_.

If I collect data about all those points of light in the night sky, I can then
say, "There are many points of light in the night sky". But that is only a
description -- it lacks an explanation, for which the data collection effort
can only be a preliminary step.

The social sciences are famous for describing things they cannot explain. When
we make an effort to explain what has been described, we cross the line into
science.

> ... behind every model and every data set is a political process that chose
> that data and built that model and defined success for that model.

 _Not in science._ This is precisely what science is meant to avoid. All we
need to do is practice science rather than perform naive data collection
followed by shallow conclusions.

------
gentlegiant
I was also averse to Bill Gates and his slide rule.

There's a whole warm and fuzzy area missing from his vision, just like his
book The Road Ahead had him standing on an asphalt road.

When children aren't terrorised at home by their own stressed out parents,
then we have a chance at raising an enlightened generation.

How to measure that?

~~~
muoncf
Psychological measurements are taken all the time. There's nothing special
about that.

------
warmfuzzykitten
I suppose MathBabe provides a service by constantly reminding us that human
motivation is impure and leads to all source of bias that taints the numbers
and the models used to evaluate them, but it does get tiresome to read a blog
supposedly about math which is, in fact, largely written to support her
political agenda. (Even if you agree with her agenda.) She recently attacked
Nate Silver for "defending corruption" because he didn't use his book to
attack evil, greedy Wall Street, and now Bill Gates for being naive, because
he didn't use his annual report to state the obvious, about which little can
be done, but instead chose to promote measurement, where progress can be made.
Perhaps she should change her moniker to PoliticalScienceBabe?

------
twoodfin
Where's the evidence in this rant that Bill Gates doesn't understand the
importance of a model's assumptions?

------
SoftwareMaven
If you define the model arbitrarily, I agree, but it you build a model based
on observed data, you are much better off.

Choosing thin crash test dummies is an arbitrary decision. Measuring sets of
real people and building crash test dummies around them is not. Unfortunately,
it would also cause the costs of testing to skyrocket because you don't get
two tries for a larger and a smaller person in one car.

Tackling education can follow the latter model, but just like any process, the
real risk is that the end goal becomes following the process and not
accomplishing what the process is trying to achieve. A sufficiently
complicated model could, in theory, address that, but I think trying to turn
educators in robots will have far more negative effects because that will
impact the best teachers the most.

------
linuxhansl
So what is the author proposing? Maintaining the status quo, because we cannot
expect to measure anything accurately.

I do agree with the author that the current problem with education in the US
will not be solved with measurements alone. The problem seems to be systemic.
But it would be a start.

------
RockyMcNuts
Goodhart's law - When a measure becomes a policy target, it ceases to be a
good measure.

<http://en.wikipedia.org/wiki/Goodhart%27s_law>

------
yarou
As my old economics professor used to say, quoting George E.P. Box:

"All models are wrong, but some are useful."

This is very much true in any discipline.

------
adrianbg
Academic self-proclaimed "babe" correctly points out minor flaws in a major
insight, misses the forest for the trees.

------
gesman
Give child a chance ...

