
New Study: Rewarding Good Teachers and Firing Bad Ones Accomplishes Nothing - curtis
https://www.motherjones.com/kevin-drum/2018/06/new-study-concludes-that-rewarding-good-teachers-and-firing-bad-ones-accomplishes-nothing/#disqus-container
======
turingcompeteme
My sister is a teacher, and had been recognized as "high performing". She was
even given some type of award. She always taught in pretty well off areas
where students came from supportive families.

A few years ago she moved to a school on the edge of a native reserve. She is
now officially a "low performing" teacher. She insists she works twice as hard
as before and has much more positive impact on the lives of her students.
She's been robbed and beat up by students. Yet she now devotes her life to
these kids, and would probably be fired in this type of experiment.

My guess is that the 'good' and 'bad' teachers in the study were almost
totally dependent on the kids they had.

~~~
jackfraser
This sort of research is amusing in a sad way. IQ studies on large populations
have revealed data for nearly a century that would easily have predicted the
outcome of your sister's efforts. The scores for teachers have to be
normalized against the best available data for the demographic composition of
their classroom, if we're going to bother with it at all.

Unfortunately, people don't like what the IQ scores show, and so instead the
Gates Foundation has to spend millions on a grand experiment that "teaches us
nothing" as Mother Jones would put it, mostly because they refuse to
acknowledge just how precisely it confirms the uncomfortable things we already
know.

~~~
turingcompeteme
The parents and grandparents of the children in her class were taken away from
their families and put in residential schools where they were abused and
mistreated their entire lives. They were then released back into the world
with a very poor education, and obvious mistrust for the government.

Friend of hers have taught in schools where a cycle of poverty inflicts the
population. Most children are raised by single mothers because the father is
in jail. When the father is released he has no opportunity to make a honest
living for himself and the cycle continues.

The uncomfortable things we already know have nothing to do with IQ. It has
everything to do with centuries of mistreatment of groups of people.

~~~
jackfraser
> The parents and grandparents of the children in her class were taken away
> from their families and put in residential schools where they were abused
> and mistreated their entire lives.

Let's keep in mind that residential schools were absolutely the most
progressive thing going at the time. Let's reach out to these poor beleaguered
people and give them the benefit of our modern school system! You can see how
easily the story could be sold to people as a very moral act before the actual
results of the way it was handled were known, generations later.

> The uncomfortable things we already know have nothing to do with IQ.

Why establish a false dichotomy here? You've obviously pointed out an
uncomfortable truth which I was already well aware of. There's no reason we
can't also be aware of the implications of IQ and any number of other factors
determining school performance of any intersectional cohort against
standardized metrics.

~~~
roywiggins
It's one thing to teach people, it's another thing to beat their native
language out of them, which is what these schools were designed to do from the
beginning. The whole point was to forcibly Christianize them and exterminate
their culture.

> The Carlisle Indian Industrial School, founded by the US Army officer
> Richard Henry Pratt in 1879 at a former military installation, became a
> model for others established by the Bureau of Indian Affairs (BIA). Pratt
> said in a speech in 1892, "A great general has said that the only good
> Indian is a dead one. In a sense, I agree with the sentiment, but only in
> this: that all the Indian there is in the race should be dead. Kill the
> Indian in him and save the man." Pratt professed "assimilation through total
> immersion." He conducted a "social experiment" on Apache prisoners of war at
> a fort in Florida. He cut their long hair, put them in uniforms, forced them
> to learn English, and subjected them to strict military protocols.

[https://en.m.wikipedia.org/wiki/American_Indian_boarding_sch...](https://en.m.wikipedia.org/wiki/American_Indian_boarding_schools?wprov=sfla1)

~~~
jackfraser
So does this unfortunate and regrettable historical event mean that we can't
use research, demographics, and psychometrics to determine likely outcomes and
inform policy decisions?

------
theothermkn
One of the canonical motivating sketches for total quality control, developed
by Edwards Deming et al, is a mythical factory with three production lines,
each with their own line manager. The lines are performing equally well,
staying within a bell-shaped distribution about approximately the same mean.
(Widgets per hour, or defective unit rates, or what have you.) As factory
manager, in an effort to improve the situation, you decide that, at the end of
each month, you will promote the top performer and fire/demote the laggard.
Congratulations! You promote and demote/fire a line manager every month for
six months. However, you notice that the performance of your lines has
remained the same. What happened? You were essentially using the performance
numbers of your three lines as a random number generator. The system was in
control, producing no results outside of three standard deviations. You
promoted and fired at random.

In order to improve the output of a system, you have to understand it. You
can’t just blindly incentivize on the randomly distributed results. If this
research leads to a more humane work environment for teachers, freer than
before of misguided incentive programs, then that will be a good thing,
indeed.

~~~
jeremyt
Your comment assumes that educational achievement is a "randomly distributed
result". It may or may not be, but you can't just assume it.

~~~
Tade0
What holds true is that the teachers' effect on their pupils' educational
achievements is not under full control of the former, so rewarding and
punishing them on that basis alone is ineffective.

Edit: grammar.

~~~
prostoalex
This seems to be a semantic argument with “full control” implying 100%,
whereas in reality it could be 98% or 76% or 59% or 11%, who knows.

What’s the actionable item here? Should we fire all teachers and replace them
with hamsters who would coincidentally also not exercise full control?

~~~
theothermkn
In the sense of Total Quality Control, a process is "under control" or "under
full control" when the output of the process conforms to a statistical
measure, such as every data point falling within three standard deviations of
the mean. To a first approximation, anything outside of that range implies a
"special cause of variation," as opposed to a "common cause" of variation. In
this case, think of a special cause as something like a layoff at the major
industry in town, causing something like stress at home or missed meals. Those
would affect the performance of students, outside of the "common causes" of
the usual variations in ability, interest, temperament, and so on.

So, no, the answer is not to throw away the experience and insight embodied in
the faculty. The answer is to try to understand and remove (or compensate for)
the special causes of variation in the population of students, as well as the
special causes of variation within the faculty. One of the other things Deming
said that I find very insightful is that the interesting thing about geography
is not where the borders are, but why the borders are where they are. Applied
here, the interesting thing about the variation in grades or test scores is
not the variation itself, but why the variation is there, and why is it shaped
the way it is. These 'why' questions only have answers that are close to the
data and contingent upon the systems, in contrast to blanket incentives that
try to sweep an understanding of the system under the rug of punishment and
reward.

------
jedberg
As anyone who’s been a teacher knows, the parents are the most important part
of a kids education.

The teacher is there to set up the framework and guide the parents as to what
to teach.

The parent has to bring it home.

The single biggest indicator of future student success is how many of the
parents at the school are married with only one working parent. I.e the
parents with the most free time to help their kids.

~~~
Eridrus
I'm left with two thoughts after reading this, either teacher quality doesn't
matter or our ability to measure teacher quality is garbage.

The latter is no more satisfying because when you can't tell the difference
between good and bad you get a market for lemons, so you end up with a pile of
bad teachers despite the fact that good ones would help.

~~~
ncphillips
The quality of the teacher does matter, but OP says it’s not the constraint
for most children.

This is basic theory of constraints: a chain is only as strong as its weakest
link. The quality of a students education is a function of many variables, and
the teacher is just one of those variables. This is why rewarding a teacher
based on their classes performance is unfair. By luck they may either get a
class filled with exceptional and privileged students, or a class of children
with abuse and attention problems.

The only way to really get a good measure the quality of a teacher, would be
to control the quality of the parents, and all the other variables too.

------
rossdavidh
Not super-surprising to me, but I agree with the postscript: "kudos to the
Gates Foundation for running an honest test. They designed it well; they
funded it properly; and when it was over they hired a third party to provide
an honest assessment of what happened. That’s the way it should be done."

------
neel8986
Can it be the case that the constant fear of being fired is not an ideal
environment for a teacher? Teaching needs a long-term commitment which needs a
different kind of incentive than a bonus. I have similarly observed that
researcher who wants to do fundamental research are not very comfortable in
startup environment which lacks security.

~~~
briandear
We could argue that the fear of being fired ought to motivate as well.
Professional athletes operate under this model. Perhaps we could implement a
teacher’s “draft.” Lower performing but still competent enough teachers could
be “traded” and stars could result in bidding wars. At the end of each
“season” the lowest school gets first pick in that summer’s draft. Substitutes
could be like free agents. And winning “seasons” could be financially rewarded
with bonuses. There could be a system of measurement based on improvement of
students rather than just raw scores, so even lower performing students could
help the team “win” by improving even if they are still not meeting the
overall standards. You might find teachers wanting to go to tougher schools
because bringing a kid from a 20 to a 50 earns more “points” than bringing a
kid from an 80 to an 85.

I haven’t fully thought this out yet, but we do know one thing: if you are in
a hole, stop digging. If something isn’t working, brute force rarely helps. We
need innovation in education but not just pedagogical— but with the entire
incentive structure for both teachers and students.

Crazy idea, I know. But sometimes crazy works.

~~~
neel8986
In my opinion carrot and stick model works in areas where 1) participants are
extremely competitive 2) End reward is huge (fame, money). That's why such
model excels in areas like competitive sports, investment banking, startups
etc. Teaching has neither component. If you give great teachers rockstar like
salary may be it will work. Similarly most of the researchers I meet are fed
up with publish or perish model of research

------
sizeofchar
Every teacher knows that learning performance depends primarily on students.
The very same lecture, with same methods, teacher, books, resources, have
drastically distinct performance with different classes.

~~~
Viliam1234
There is a huge informational asymmetry when debating education: almost
everyone was a student once, but most people have never been teachers.

This is how people know that the quality of education in the same class can
change dramatically when you replace a teacher. But most of them don't know
that the same is true in the opposite direction: the quality of education can
change dramatically when the same teacher teaches different classes (or just
when the most disruptive child in the class happens to be sick at home today).

There is also no teaching method that would make everyone happy. Some students
prefer to work in teams, some prefer to work alone. Some students prefer to
have explained how things work, some insist on receiving a list of stuff to
memorize. Most people who have a recipe "how to fix education" simply want
everyone to do it their preferred way.

(So perhaps the best approach would be to have different kinds of school for
everyone? Problem is, people would not want to honestly admit some of their
preferences. If you listen to internet, everyone prefers understanding to
memorization. Then you start explaining stuff, and students follow every
sentence you said with "do we have to remember this for the exam?")

------
mabbo
I feel like the real problem is in how we measure teachers. It shouldn't be
about how well the students do, it should be about how well the students do
_relative to expectations_.

Imagine a school in a well-to-do area where we expect all the students to do
well and get A+'s. If an entire class of students gets A- on a standard test,
that means the teacher actually _negatively_ impacted the students. But
because they all got A-, we reward the teacher for having done such a great
job.

I propose this: for each student, far ahead of time, guess how well they're
going to do on a standardized test based on previous tests, family situation,
zip code they live in, etc. Predict based on all the things _other_ than the
teacher. Now measure a teacher on how well they did _compared to
expectations_.

To do even better, don't rate teachers based on how well the students do on
the test the same year they teach them, but rate them based on the following 3
years. Now you're asking what the _long term impact_ the teacher has on
students, on average. That's a useful measure to pay bonuses on.

The problem is that politically, it's hard to explain to the general public
(or to politicians).

~~~
shanghaiaway
That is not a useful measure because there are an unlimited number of factors
that influence the outcome.

------
jupiter90000
It seems like the primary purpose of school is to make sure kids have been
well trained to sit still and follow the rules, learn how to jump through
societal hoops, and have a place to be for parents who can't/won't deal with
them all day. How much can a teacher really do, trying to enforce all this and
get across a subject to 15+ kids in a room, who have various degrees of
motivation (sometimes none) to want to learn anything?

It seems like there must be a better way to help kids acclimate into adults
and give their parents a break/opportunity to work.

~~~
bitwize
Sitting still, following the rules, and jumping through societal hoops are all
critical life skills for that cushy job adding to the world's pile of webshit
in JavaScript. So in some sense schools are doing the job of educating the
young to prepare them for adult life.

------
maxander
I've seen more sophisticated schemes for teacher evaluation, in which the
metric takes into account the ability of the students as they enter that
teacher's class. So, for a student who got a B, say, in grades N < 5, there's
an expected grade value for them to achieve in 5th grade; that student counts
towards the teacher's "score" only insofar as they do better or worse than
that expected value.

Theoretically, this puts teachers on even footing, even when one might be
teaching a class of Harvard-prep elementary schoolers and another teaching a
class of inner-city kids who have been educationally neglected their entire
lives. And data from student performance over time in the current educational
system should be entirely sufficient to train the model.

It's not clear if the study referenced controlled for student variance this
way; if so, it would readily explain the lack of results.

~~~
AmericanChopper
>It's not clear if the study referenced controlled for student variance this
way; if so, it would readily explain the lack of results.

This headline is pretty click baity. A “better” teacher is one the produces
better outcomes for their students. Firing the “bad” teachers, and rewarding
the “good” teachers will obviously lead to better outcomes for students. This
experiment was only testing whether the metrics they were using truly measured
the quality of the teachers, and whether their insentive structure was
productive. Turns out they weren’t.

------
catchmeifyoucan
This title is really misleading. There are many other factors that this
article does not cover or examine, and it reaches a premature conclusion.
Furthermore, standardized exam measures are poor way to understand teacher
impact. From what I remember from school, the best teachers made me
understand, the poor ones helped me memorize.

------
aetherson
Neat, glad to know that we don't ever need to give teachers raises.

------
maxxxxx
Maybe they should put more effort into figuring how to create an environment
where teachers and students can perform well. It's not like the teacher is the
only player here. Students and their environments play a big role, curriculum
and so on.

Reminds me of the 10x programmer nonsense. Give people a decent work
environment where they feel respected and most will perform well.

------
curtis
I find it hard to believe that teacher quality doesn't matter at all. I'm not
sure that that's what this study is telling us. Maybe it's just telling us
that there is no significant (or at least measurable) difference between
teachers that are a little bit below some threshold and teachers that are a
little bit above.

It surely isn't telling us what would happen if you replaced all the teachers
in a school with teachers recruited from the top 5 or 10 percent of all
teachers in the country. I really want to know what would happen if you did
that. On the other hand, suppose it works really well. We still can't replace
all the teachers in the country with the top 5 or 10 percent of all the
teachers in the country, so it's not like that would be actionable.

~~~
turingcompeteme
> replaced all the teachers in a school with teachers recruited from the top 5
> or 10 percent of all teachers in the country.

My Sister and SO are both teachers and we've had this discussion. They believe
that if you took a low performing school, and replaced all the teachers with
the best teachers in the country, you would find zero difference.

My sister has been beaten up by students, breaks up fights nearly every day,
deals with child protection services, assault from parents, weapons in the
classroom, drugs, gang activity, students who haven't eaten or bathed all
weekend, etc. She teaches Grade 6. It doesn't matter in the slightest how good
of a math teacher she is.

She's has it bad, but I think many people here may be surprised just how
common her situation is.

~~~
shoo
That's a pretty strong argument for increasing the baseline level of social,
community, heath, employment services, in addition to a focus on education.
likely need to invest in those changes and then keep the programs funded and
running with careful adjustements for multiple generations.

i agree, there's only so much a single teacher, or a school full of teachers
can do if a lot of other basic needs aren't being met.

~~~
tellarin
This again links back to the Finish system. It doesn’t stand alone by itself.
All the other social services are also in place.

------
hueving
If it accomplishes nothing then the 'quality' of the teachers is irrelevant.

------
Alex3917
Perhaps the least surprising finding ever, given the previous research:

[http://www.citevault.com/#vam-efficacy](http://www.citevault.com/#vam-
efficacy)

[http://www.citevault.com/#vam-statistical-
problems](http://www.citevault.com/#vam-statistical-problems)

[http://www.citevault.com/#vam-implementation](http://www.citevault.com/#vam-
implementation)

~~~
lend000
All this means is that better methods will need to be invented to score
teachers, not that it's impossible. For example, one thing I've seen missing
in these studies is a system design that attempts to control for the
distribution of students in a given incoming class, which seems like the
single most predictable metric for end-of-year performance.

~~~
Alex3917
> All this means is that better methods will need to be invented to score
> teachers, not that it's impossible.

No, it is impossible. That's the whole point. If it wasn't impossible it'd be
a great idea.

~~~
lend000
I suspect some in the HN community assume that because public school
curriculums were generally so easy and aligned with textbooks such that they
got all A's regardless of the teacher, that the teacher does not matter. No,
the teacher definitely matters, as does the inflexible and oversimple
curriculum and a million other factors.

Switch 'teacher' with 'professor' and I highly doubt you would argue that
quality no longer matters. The difference is that professors are less bound by
stringent curriculums and teach more advanced material. This is where everyone
really feels the difference. Whereas for public grade school teachers, an
academically inclined student will only struggle under exceptionally bad
teachers, less well academically inclined students will suffer without decent
teachers even in grade school. The first quarter/semester at a good college
typically provides a reality check for students who breezed through primary
education.

------
lawnchair_larry
Alternate title: we thought money is what good teachers find rewarding, but we
were wrong

~~~
jVinc
That's not the story at all. They meassured actual performance and fired
teachers who showed bad performance, and rewarded those showing good
performance. This was not just a test of dangling money in front of teachers
and expecting them to improve their performance.

What you could get out of this study is however that it doesn't matter if
teachers find the job rewarding or not. It doesn't matter if you keep on bad
teachers, in fact the study might point to the fact that teachers are not the
bottleneck of student performance at all.

~~~
fancyfish
Good point here, that we ought to expand the scope of how we look at student
performance. We're quick to blame teachers, and indeed teachers probably have
as much an impact as anything else, but measuring up teachers and expecting
big changes isn't the greatest idea.

Yes, it gets more complicated looking at more nuanced measures. Like resources
available for the students, their home lives and personal situations, their
time available to study and focus.

Then again maybe they should look at discrepancies between teachers in
districts that pay well and districts that pay less. Perhaps the answer isn't
to pay some more and fire others, but rather pay all of them more!

I'm a fan of the Education Endowment Foundation(1) on this which looks at all
the measures you can think of to try to improve education practices (in the
UK).

[1]:
[https://educationendowmentfoundation.org.uk/](https://educationendowmentfoundation.org.uk/)

------
chvid
Next up: fire the educated teachers and just let random grown ups run the
class ...

~~~
protonfish
Sudbury Schools
[https://en.wikipedia.org/wiki/Sudbury_school](https://en.wikipedia.org/wiki/Sudbury_school)
have no curriculum, they let the students do whatever they want. Graduates
don't differ significantly from traditional schools in standard educational
assessment testing.

------
michaelscott
This is unfortunately a complex, multivariate system problem and won't rely
solely on the "quality" of a teacher in it.

In addition to the behind-the-scenes work most teachers have to deal with,
such as departmental objectives, I think great teachers enter the profession
in order to benefit children, not to get that fantastic paycheck . It's a
socially focused decision, not an financial one, so if I were designing a
system to grade a teacher's quality I would focus on attending to their
efforts with students, not throwing more money at them.

~~~
exotree
Money may not be the end all for people who go into this profession, but it
most certainly does matter. See the protests over the past couple of years in
areas where teacher pay is low.

------
randyrand
This conclusion is amazing! Finally we have a study that says hiring good
teachers does not matter. And that going to teaching school to become a better
teacher is not important!

If this conclusion is correct, teachers can be replaced low wage unskilled
laborers!

/s

This (possibly MotherJones simplified) conclusion must clearly be bogus.
Perhaps I should take a look at the actual study, because this article is
terrible.

------
derpsauce1
Mother Jones is an extremely left leaning website.

Teachers Unions are among the core groups supporting democratic and liberal
candidates.

I love a good study but it is possible that they are misinterpreting the
results of this study on purpose.

During the last election, I became very sensitive to this type of media
manipulation. Tread carefully.

------
asafira
What are good resources to learn more about what works/ what doesn't in
improving education?

At the end of the article, they add that the study deserves a lot of credit
even though it didn't work. Kudos! I was pretty happy to read that =).

------
Gimpei
Anyone check this for the quality of the study?Doesn't look like this was
randomized. Also would have been nice to see more pretreatment years for
evidence of a trend but maybe that was in the actual report.

------
dqpb
If teacher salaries were $120k and up I bet you'd get a whole new class of
teachers.

~~~
lobotryas
Would we really? At some point a higher cost will stop resulting in better
outcomes.

~~~
zeth___
And yet we keep paying CEO's more and more.

At least teachers are only in control of one class room. The most mischief
they can do is to buy animatronic dinosaurs for the one lesson where they
mention prehistory to kindergartners.

------
candiodari
Someone should make a study about management theories. The issue is that the
major examples of management theories were ... fraud.

Taylor ... was a fraud [1]. Lied about his data in order to get paid more.
Even paid the people he "tested his theory on" to act out the outcomes he
wanted to see for company higher ups. And he acted like what I've come to
expect from managers: first he commits scientific fraud, then writes a book
and titles it "Principles of Scientific Management".

Because when you go for bullshit, the big lesson is: don't just bullshit a
little. Go all out.

Weber and Fayol were no different, and even Ford ... let's just say that if
you focus on his technical accomplishments and skip all the management
bullshit you will not be missing much (besides a big part of Ford's ideas were
high wages, so I doubt many managers will be repeating it).

As can be seen, among many other places, on wikipedia [2], the fact that
Taylorism was an outright fraud has not stopped pretty much every management
school in the country from expanding and pushing on with his "work".

It's like most of the famous economists. The top ones were simply frauds, who
got important positions like chairman of the Fed, or secretary of the Treasury
and just happened to be in office during big events. Their theories were wrong
(really fraudulent, as they didn't believe in it at the time, which makes it a
fraud rather than just being wrong, and Keynes, Hayek, Friedman and Greenspan
have admitted as much, and let's face: they were "the greats". If they were
frauds, then people like our current Mnuchin, Powell or Geithner aren't even
frauds : they're ignorant and actually proud of that (Mnuchin has said in an
interview that not understanding much economics makes him a better secretary
of the treasury). Yellen actually had credentials, and seems to need another
year or two before she'll confirm she was a fraud too, she has already
admitted her theories were wrong, so all she has to do is also admit that she
knew they were wrong when she decided to implement them. And boy, she was,
like Keynes, more than a little bit wrong).

Once money or power gets involved, humans, "scientist" or not, become lying
bastards.

At least this research was honest. But it's not saying what people want to
hear, and so it will be ignored, as it wouldn't contribute (and that's being
polite) to the careers of the decision makers implementing these policies.
Meanwhile, Trump is happily pushing on Obama's efforts in this regard, who
simply continued from Bush's policies, ... and so on and so forth. Don't
worry: Trump is not about to let such a detail as science stand in the way of
looking good to a few people, and let's just not kid ourselves that Hillary or
whoever gets elected in 2020 will be better.

[1] [https://www.theatlantic.com/magazine/archive/2006/06/the-
man...](https://www.theatlantic.com/magazine/archive/2006/06/the-management-
myth/304883/)

[2]
[https://en.wikipedia.org/wiki/Scientific_management#Relation...](https://en.wikipedia.org/wiki/Scientific_management#Relationship_to_Fordism)

~~~
darepublic
> At least this research was honest. But it's not saying what people want to
> hear, and so it will be ignored

I think you underestimate how much teacher unions want to hear this result,
and their political influence

