
Gates Foundation study: We’ve figured out what makes a good teacher - eggspurt
http://www.washingtonpost.com/national/gates-study-weve-figured-out-what-makes-a-good-teacher/2013/01/08/05ca7d60-59b0-11e2-9fa9-5fbdc9530eb9_story.html
======
the_bear
It's common for people who received good educations to say that test scores
aren't an acceptable indication of learning. It's probably true that if you
already have an acceptable level of education, test scores shouldn't be too
important, but it seems like the people criticizing test scores haven't
experienced what education is like for the children being "left behind."

In middle school, the last three weeks of each semester were spent watching
movies (the same two or three movies over and over). In high school, I was
taught that the big bang theory was when the continents divided. I was taught
that the most direct route between two points on the globe is through the
north pole. My Spanish and woodworking teachers taught math because they knew
that the math teachers weren't doing it.

When you're talking about schools like the ones I went to (which weren't
nearly as bad as many other schools in America) I think that test scores are a
decent indicator of the education a student is receiving. You're measuring
things like "can this kid read English" and "can they multiply two numbers
together." Standardized tests are certainly capable of measuring things like
that. Testing probably breaks down for measuring intelligence/education as the
level of education improves, but I encourage everyone to consider just how
dysfunctional many schools are before ignoring testing entirely.

~~~
roguecoder
Except that even those things don't predict later success. Any kid with a
family that can afford books will score higher on "can this kid read English".
Any kid who's been drilled can learn to multiply two numbers together. Without
cognitive skills behind them, though, those are not useful abilities.

The best predictors of future success are self-control, emotional intelligence
and a grasp of functional algebra (which I believe, but don't have evidence to
support, is because of the importance of computational and systems-level
thinking.) Only one of those, self-control, is even indirectly measured by
current standardized tests.

~~~
sopooneo
From my work teaching I am very skeptical of the notion that it is a family's
ability to "afford" books that makes the difference. It is whether the family
knows that books are important and chooses to get them and read with their
children. While monetary poverty is horrible, it's effect on educational
outcome is dwarfed in comparison to cultural dysfunction and despair.

~~~
w1ntermute
Yeah, the monetary poverty theory doesn't hold up when it comes to the case of
poor Asian (or African) immigrants. Although it certainly plays some role when
all other things are equal, I think the major issue to be tackled is the
glorification of thug life and demonization of education in contemporary
African American culture.

------
cbsmith
I'll have to look at the prior reports, but this 3rd report is somewhat
disappointing in terms of its statistical rigor.

They do provide weightings based on optimal predictions for maximum accuracy
with predictions for state tests scores, although those weights appear to
assume a simple linear correlation. Even then, the data suggests that in class
observations are almost a complete waste of time (single digit weightings),
and that student surveys decline in value as the kids mature (no shocker: with
younger kids, how much they like the teacher has a much more profound impact
on their success at learning). In all cases the kids seem to be a more
reliable indicator than the observations of professionals! This may just be an
indication that assessments based on more frequent observations are more
accurate, but the only way we can really get a sense of that is if they also
had peer observations from team teaching environments... which are completely
absent.

That's not the worst of it though. The worst is the rest of the data. It is
based on some fairly arbitrary presupposed models for combining these
different assessments. No breakdown of the individual components of these
assessments (other than "English" & "Math") and no attempt to discern a model
from first principles.

There are some other factors as well. Aside from some outliers, the accuracy
of predictions seems kind of suspect. Most of the outcomes are within +/-0.05
standard deviations, and a majority of them are within what looks like
+/-0.025 standard deviations. These are pretty minuscule variances (0.25 was
supposed to represent a year of schooling, so 0.025 represents maybe 15 school
days worth of progress at best), and it is not hard to see that without some
of those outliers, the regression line doesn't look so pretty (particularly
for English). I also didn't see any metrics on the variance of the data within
a classroom, which I suspect makes any predictive value look limited at best.

Ugh. I really hoped that Gates would make sure there was more statistical
rigor with an analysis like this. :-( I hate to say it, but the conclusions of
the study are not well supported by at least what is in the final research
report.

~~~
wheaties
Yes, this is also the same foundation that spent 1 billion to make schools
smaller, because they didn't do simple statistical analysis to find that
smaller schools showed much larger variability. Now they're coming back to say
"oops, we were wrong."

[http://marginalrevolution.com/marginalrevolution/2010/09/the...](http://marginalrevolution.com/marginalrevolution/2010/09/the-
small-schools-myth.html)

~~~
danielweber
He thought his theory was right, and after implementation, continued to test
it, and found out it was wrong.

Good on him.

------
nathan_long
I can't say whether the findings are correct or not, but I like this:

>> The teachers who seemed to be effective were, in fact, able to repeat those
successes with different students in different years, the researchers found.
Their students not only scored well on standardized exams but __also were able
to handle more complicated tests of their conceptual math knowledge and
reading and writing abilities. __

To everyone who keeps saying "standardized tests are bad", please cut it out.
A _particular_ code benchmark may be useless, but benchmarking overall is
good. _Existing_ standardized tests may be bad, but in general, having some
way to make objective measurements and adjust accordingly is the basis of all
human improvement.

And yes, we should reward good performance, if we can determine what good
performance is. How else do you expect to get more of it? It __is __incredibly
difficult to make incentives that actually do what you want rather than
encourage gaming of the system, but that doesn't mean we shouldn't try.

In short, let's stop debating _whether_ we should measure and incentivize, and
start debating _how_.

~~~
calibraxis
Then why not use standardized tests for job interviews, or why don't decent
universities use them? Standardized tests have serious disadvantages to
consider, and aren't normally considered best-by-default.

There at least a couple obvious groups qualified to rate educational quality:
teachers and students. If they happen to choose a standardized test to
evaluate the education, then fine. Otherwise, I don't see standardized tests
as obviously virtuous.

~~~
pflats
Don't confuse the government-mandated standardized tests with the SATs.
They're not really out there to measure individual students. These tests are
to ensure individual schools, districts, and states are giving the proper
instruction to their students. They exist to make sure that Springfield High
and Franklin High are both teaching the students the required material.

Universities do use standardized tests: they use the SATs and the ACTs, the
GREs, LSATs, MCATs, Miller Analogies. They use AP exams to determine who gets
college credit for courses the school never administered. They use their own
standardized entrance exams for placement.

~~~
yummyfajitas
Further, many foreign nations (e.g. Japan, I think Singapore) have admission
systems comprised entirely of standardized tests. Top 300 scorers get into
college #1, 301-600 get into college #2, etc.

------
gnosis
There are many important factors which teachers (no matter how good they are)
can't control:

1\. How wealthy, poor, educated, or uneducated a child's parents are.

2\. How many resources the school and teacher are given.

3\. Whether a child's parents are abusing them at home.

4\. Whether a child is witnessing violence, hopelessness, despair in their
neighborhood.

5\. The attitude of a child's family and friends towards education.

6\. Whether a child's parents are active in their child's education and in a
PTA.

Study after study has shown that these factors greatly influence a child's
attention and attitude to school work, their attendance, and ultimately their
performance at school.

No child's performance on a standardized test, and no evaluation of a
teacher's performance is going to pick up on the above factors, yet they can
critically impact how well the child ultimately learns and what the child
learns.

Standardized testing of a child that is unfortunate enough to live in a
situation where the above factors are against them will likely notice a poorly
performing student, one who gradually does worse and worse on the tests until
they finally flunk out.

Then, instead of getting extra aid, the funding to the school that has a lot
of such children will be cut (thanks to programs like No Child Left Behind),
and their teachers penalized.

These factors, which are strongly associated with poverty, are things a given
teacher is in no position to influence, but which society as a whole could
address. If it has the will.

~~~
crusso
And at my job, there are many factors out of my control. I don't control the
home life of my engineers. I don't control my customers' decision making
processes. I don't control my competitors' actions in the market.

But yet I'm judged on the small slice of that overall pie that I can control.

What this study indicates is that there are teachers who can reproducibly
create success levels given the control of the situation that they do have.

How is it any different than the rest of the working world where you can
control some things and you can't control others yet your job is to do the
best you can with what you can control?

~~~
gnosis
That's a very legitimate question. How can teachers do the best with the
resources they're given and the the children they're working with, no matter
how threadbare the resources are, and no matter how traumatized and
disadvantaged their students are?

Whether standardized tests have a role in answering that question or in
evaluating teacher performance is hotly debated right now.

But the purpose of my original post was to draw attention to a different
question: How can society as a whole ensure that every child gets the best
education possible?

In schools that have no problem attracting resources, great teachers, and
children with mostly fortunate backgrounds, perhaps this second question can
be answered by asking the first: how can we improve our teachers?

But in schools that lack resources, whose students live in abuse, poverty, and
despair, what even an exceptional teacher can do will be radically limited,
and society would probably be better off focusing on how to eradicate poverty
-- the real cause behind the failure of too much education in this country.

Unfortunately, seriously tackling the issue of poverty is much harder (and
even contrary to some dominant American ideologies) than simply demanding more
standardized tests be given and blaming teachers and their unions.

~~~
crusso
_How can society as a whole ensure that every child gets the best education
possible?_

Step one, define "best education possible". Step two, learn to measure whether
or not a child is getting one (or the closest you can come to it). Step three,
maximize some aggregate or regionalized version of your measurements through
restructuring of school systems, rewards for good schools and teachers, etc.

The study in this article is just gets us to step two. It looks like you're
wanting to discuss step three. That's a different can of worms.

~~~
randomdata
It is not even clear if the article gets to step one. Some of the teachers who
I feel had the greatest impact on my education taught classes that I did not
necessarily score well in. If I am a statistical norm, those teachers would be
consider poor teachers according to the metrics, contrary to reality.

~~~
crusso
This is where I have an issue with this discussion. You're throwing up
anecdotal data in the face of an actual study. Why is that okay in a community
like this that understands the Scientific Method and should have a basic
understanding of statistics.

~~~
randomdata
The only point I was trying to make is that test scores do not necessarily
define the success of a teacher. I suggest, to be able to apply scientific
method and get meaningful results will require far more data than is seemingly
being collected.

~~~
yummyfajitas
What data do you feel is needed, but not there?

------
aaronharnly
I, like many, hope the MET project yields fruit.

However, see this post for a strong critique of one of the key claims of the
MET study:

[http://garyrubinstein.teachforus.org/2013/01/09/the-50-milli...](http://garyrubinstein.teachforus.org/2013/01/09/the-50-million-
dollar-lie/)

Namely, in the scatterplot correlating predicted effectiveness and actual
effectiveness, they pooled teachers into 5% groups. This naturally reduces the
variance, yielding a much straighter line of points than would an ordinary
scatter plot. This is not, to my knowledge, an accepted or ordinary technique.

~~~
crusso
Ugh, right from the get-go, Rubinstein throws away any attempt at statistical
rigor by using an outlier example to try to prove something.

The 5% groupings doesn't evaluate the metric. It just means that if they plan
to use this actual study as a guideline, they should use the same technique in
grouping teachers.

------
arocks
Actually they have figured out how to evaluate teachers based on statistical
models based on their repeatability of success. Finding a 'great' teacher is
much more than finding the top rated teacher in my opinion. What qualities
makes such a teacher good is a totally different matter. The latter was what I
was hoping to find.

------
axelfreeman
"We’ve figured out what makes a good teacher" They've figured out that a
"good" teacher can repeat the process of success (whatever this is), there is
no information how he made it. The headline is slightly misleading.

~~~
jonknee
The good news is they have video of good teachers (and bad) at their craft for
years. It will take more analysis, but they have all the data at the ready.

------
pflats
They haven't figured out a damn thing regarding what makes a good teacher. At
best, they've figured out how to find which teachers are good. That's not
nearly as helpful. It's barely helpful.

What happens when you find the bad teachers? Fire them and hope the fresh-
faced college grads do a better job?

So much of this debate on teachers is leans towards the carrot or the stick.
Wanting to reward the best teachers because it makes you feel like you're
helping is fine, but don't fool yourself into thinking it'll hugely improve
instruction. Teachers aren't going to suddenly go, "Oh, wait, you'll pay me
more if I'm a good teacher? I'm going to start trying!" (One or two might, but
those outliers exist in any field, and are hardly the issue.)

In every district I've taught, very little time is given for teachers of the
same subject to collaborate, observe each other, and improve their own
teaching. Every year, we have to get 20 hours of professional development.
Tell me, are 6 minutes and 20 seconds of development per teaching day going to
improve instruction?

------
tokenadult
The article kindly submitted here links to the project homepage

<http://www.metproject.org/>

at which the latest project report can be found. I've got some reading to do
before I think about what else to add to the discussion already underway here.
I get the impression that one observation that has motivated much of this
research is how persons educated overseas have fared in United States society
after immigrating here in early adulthood. My own experience living in another
country (twice, in two different three-year stays) does much to prompt my
interest in education reform, which is what drew me here to Hacker News.

<http://news.ycombinator.com/item?id=4728123>

------
murbard2
If this becomes policy, it will most likely become another example of
Goodhart's law. <http://en.wikipedia.org/wiki/Goodharts_law>

You can find a measure that correlates to performance, but once you use it to
actually evaluate teacher, teacher will adapt and the measure will lost all
relevance.

Rely on test scores? Teach the test, not the material. Rely on student
evaluations? Bribe them with easy material and good grades. Rely on principal
evaluation? Get on its good side.

------
edtechdev
This work is valuable, but I personally would like to put more focus on
_improving_ teaching and learning - helping teachers do better, helping
students do better, creating better learning environments and curricula.

For example the budget of this study came out to about $15000 per teacher.
What if instead we spent $5000 of that for professional development / training
for the teacher, $5000 for a classroom set of chromebooks, and gave the
teacher a $5000 raise. That alone would cause some significant improvements, I
believe (not that that is ALL that is needed, and there are so many bigger
issues that need to be tackled as someone else mentioned, such as poverty and
inequity).

But when we talk about "good" teaching and "bad" teaching, we are treating it
like it is a simple craft that is mastered or not mastered. Teaching is like
engineering, it involves design, and design is about making things better (see
this prezi on learning design: <http://prezi.com/b44jwdgvs8nl/olds-mooc-
introduction/> )

~~~
jonknee
Because this was a study, the money spent has little to do with the individual
subjects. If pollster calls me for a a national opinion poll I would rather be
paid 1/1000 of whatever the budget was than answer the questions, but
obviously my participation is more helpful overall.

The goal was to identify what actually works, which is something you need to
know before improving the situation. Paying a bad teacher more will not make
the education provided any better (despite whatever the teacher's union says).

------
brudgers
The study identifies teachers who are effective against a particular metric.
Most identified are likely members of the subset of good teachers. Yet, the
three R's as measured by standardized test outcomes are not the only measure
of effective education, just one that is cheap to measure.

Dewey believed that the goal of education should be to produce good citizens.
Systems with metrics based upon the model of student as consumer are more
likely to produce outcomes biased toward producing consumers.

The entire set of good teachers includes those who inspire a lifetime of
learning. It includes those who teach kindness, sportsmanship, and good humor.
It includes those who teach subjects for which there are few students and in
which standardized tests make little sense such as art and shop.

Maybe the study can point us in the right direction. But few districts will
bring outside evaluators from Cambridge's famed institutions. Few districts
will fund ETS analysis or purchase their tools.

What we are likely to get are processes localized to the political reality on
the ground.

------
MaysonL
Evaluating teachers is all well and good, but the key to improving our
educational system is not that. What is needed is _improving_ teachers. See
"Building a Better Teacher"[0]

[0][http://www.nytimes.com/2010/03/07/magazine/07Teachers-t.html...](http://www.nytimes.com/2010/03/07/magazine/07Teachers-t.html?ref=magazine&pagewanted=all&_r=0)

------
loucal
Is this the same gates foundation that I saw on 60 minutes pumping GMO seeds
and fossil fuel fertilizer to third world countries to "help" them out of
their food crisis?

In fairness, I think I heard they realized something like "what works in
america doesn't work everywhere" (bit of a cop out to save face) and moved
toward more sustainable organic methods, but I'm not completely sure.

All I know is, if its that hard for them to figure out the most practical way
to grow a plant, we should probably take their advice on growing the minds of
our children with a grain of salt.

~~~
atomical
There's nothing wrong with using GMO seeds to increase yield.

~~~
loucal
It is opinion either way but in my opinion not only isn't it necessary, but it
clearly threatens global biodiversity. Proper sustainable farming techniques
produce an abundance and a variety of food, so the extra 1% increase on a huge
monocropped field isn't even something that makes sense for third world
communities.

On the biodiversity thing, viewing your website leads me to believe you
appreciate nature, so I'm just scratching my head that you haven't made the
connection. Ohh well, downvote me all you want buddy. You are entitled to your
opinion also.

~~~
atomical
It's much more than 1%. I haven't down voted you. Liking nature really has
nothing to do with anything else.

------
dspeyer
If I'm reading the linked page correctly, these metrics were verified by
correlating them with individual student improvement on a variety of
standardized tests and "more cognitively challenging assignments in math and
english". This is better than pure circular logic, but until _those_ metrics
are validated, these are suspect. In particular, everything seems to test for
shallow understanding.

------
blueprint
The title of this article, and perhaps the researchers' conclusions from their
data, is wrong. They didn't find the criteria of a great teacher (that is,
what makes a good teacher). They just found some correlations among the
_qualities_ of good teachers. It's the total opposite meaning. They found
results of good teachers, not causes.

------
robdoherty2
I believe that studies like this are well-intentioned in that those conducting
the research are truly trying to improve the state of education in the US.
However, the result of their work I'm afraid is the 'quantification' of
teaching as a profession.

------
UnoriginalGuy
Using test scores in order to judge the quality of a teacher is like using
brush strokes in order to measure the quality of a painter.

~~~
crusso
So where part of the study showed that with randomized redistributions of
students, certain teachers were able reproduce higher test scores -- your
countering evidence shows otherwise.

Great. Where's your evidence?

My own evidence (experience going through school and generalization of how
ability works in most professions) tells me that the good teachers in my life
helped me to learn the information better and more deeply. My memories of
those classrooms are of the subject matter and my desire to achieve in that
teacher's environment. The crappy ones were noticeably so in that my memories
of those classrooms was of misbehaving with my friends, having no desire to
help the teacher teach me, and cramming to learn material for tests because I
hadn't absorbed anything beforehand.

~~~
sethammons
I just realized that I keep replying to you, crusso (lol). Cheers. I'm more
interested in the demographics of the student population than the 'random
distribution (1)' of students in that population to different teachers. What I
mean by this is that I would expect 'good' teachers to be able to demonstrate
their abilities with with relatively homogeneous groupings of students where
those students are from relatively decent neighborhoods. If, however, this
study included teachers in inner city schools with huge variance in the
'quality' of student that a teacher would receive at often vastly different
levels of understanding in the teacher's discipline area, then I think the
results would be interesting insofar as to allow others to dig into what makes
these teachers effective.

1: If we are looking at a teacher who teaches a course with a pre-requisite,
the grouping of students becomes more similar (ie, a trigonometry teacher is
likely to have students both motivated to do well and to have passed the
requisite lower level math classes, and their success is less surprising to
me).

~~~
crusso
Cheers... I'm over-posting to this thread. I feel fairly passionately about
education. It's one of those things that I feel is grossly unjust to kids of
all socioeconomic and aptitude levels. We spend so much money on education but
entrenched interests use their louder megaphones to hold kids hostage.

A study comes out from the Gates Foundation that tries to tackle some of the
education problems head on. It produces some statistically interesting
results, but here on HN, I keep reading knee-jerk responses about
"standardized testing" from people who apparently didn't even read the
attached article.

How are we going to solve these problems if no one even bothers to understand
our options?

Regarding your point - I agree. Any solution would need to account differences
in what you referenced as a "variance in 'quality' of student".

------
jnazario
kudos to the WaPo for giving a link to the study. too often, many publications
discuss a study but never link to it, which then makes it harder to go and
review it for yourself.

------
madaxe
"Test scores"

Brilliant. Well, yes, that is the best way to measure a teacher's efficacy, if
your goal is to teach students to jump through hoops and pass standardised
tests rather than to think.

The best teacher I ever had was my grandmother - she taught me the most
important lessons I'd ever learn, such as:

"Observe the world around you."

"Reach your own conclusions."

"What was fact yesterday may be fiction tomorrow."

Tests are a distraction from learning.

~~~
crusso
I don't really understand the visceral reaction against testing in this HN
thread.

You're tested throughout your working life. Almost every job interview is a
test. Giving a customer presentation is a test. Finding bugs in your software
for a release date is a test. Memorizing that big Cheesecake Factory menu so
you can wait tables is a test. Getting your General Contractor's license is a
test. Getting certified as a Pilot requires passing a test.

There should be more to education besides testing, but being tested is a part
of life and it's easy to measure success - that combination means that it's a
useful tool for education.

~~~
Mvandenbergh
Job interviews are a good example of a kind of testing that we know doesn't
work all that well but we keep around for lack of a better alternative.

The others are not good examples because they are useful in-and-of themselves.
Giving good presentations and finding bugs are useful because they are useful,
not because they predict something else.

Pilot's licenses, like driver's licenses are based on tests where you more or
less do the thing that you're being tested on. The test and the thing that
it's supposed to measure are very similar and we can therefore use it to
predict performance. In cases like that "teaching to the test" doesn't exist
because practicing for the activity and for the test is the same thing.

I'm not denying that educational testing is useful to quantify progress but
such tests are often different enough from the underlying activity they're
intended to measure that effort spent preparing for the test has little other
purpose.

