
Why Kids Should Grade Teachers - jseliger
http://www.theatlantic.com/magazine/archive/2012/10/why-kids-should-grade-teachers/309088/?single_page=true
======
tokenadult
This is one way this process has been validated, from the submitted article:
"The responses did indeed help predict which classes would have the most test-
score improvement at the end of the year. In math, for example, the teachers
rated most highly by students delivered the equivalent of about six more
months of learning than teachers with the lowest ratings. (By comparison,
teachers who get a master’s degree—one of the few ways to earn a pay raise in
most schools —delivered about one more month of learning per year than
teachers without one.)

. . . .

"The survey did not ask Do you like your teacher? Is your teacher nice? This
wasn’t a popularity contest. The survey mostly asked questions about what
students saw, day in and day out.

"Of the 36 items included in the Gates Foundation study, the five that most
correlated with student learning were very straightforward:

1\. Students in this class treat the teacher with respect.

2\. My classmates behave the way my teacher wants them to.

3\. Our class stays busy and doesn’t waste time.

4\. In this class, we learn a lot almost every day.

5\. In this class, we learn to correct our mistakes."

Here is earlier reporting (10 December 2010) from the New York Times about the
same issue:

<http://www.nytimes.com/2010/12/11/education/11education.html> Here is the
website of Ronald Ferguson's research project at Harvard:

<http://tripodproject.wpengine.com/about/our-team/>

And here are some links about the project from the National Center for Teacher
Effectiveness:

[http://www.gse.harvard.edu/ncte/news/NCTE_Conference_Using_S...](http://www.gse.harvard.edu/ncte/news/NCTE_Conference_Using_Student_Surveys.php)

LAST EDIT: I'm amazed at how many of the comments in this thread appear to be
about issues thoroughly discussed in the submitted article, but unresponsive
to what the submitted article said. On this kind of issue, it's an especially
good practice to read the fine article before assuming what is being
discussed. We all know about school, but specific proposals for school reform
have specific details that make some worse than others, and can be empirically
tested.

~~~
jseliger
>I'm amazed at how many of the comments in this thread appear to be about
issues thoroughly discussed in the submitted article, but unresponsive to what
the submitted article said.

I'm not, unfortunately. It seems like most people read a headline and perhaps
a paragraph or two, then active their pre-existing beliefs about whatever the
subject happens to me, and move on from there. That's certainly been my
experience with commenters on my blog, anyway, and it's been experience in
observing both online communities and in reading student papers.

------
hooande
Basing teachers' pay and job security on surveys from students seems like a
good idea, especially given the numbers mentioned in the article. One problem
is that it might give too much power to students.

I was a dick back in high school. The hacker I was back then would have
figured out exactly how the testing and metrics were set up (public
information) and organized a union of students to manipulate it. I can't do
much with standarized test scores, they reflect on me. But a teacher quality
survey? That's just a weapon.

Things like this make me wish that we had some kind of Hacker in Chief, to
figure out how to circumvent new systems _before_ they get implemented.

~~~
Alex3917
"But a teacher quality survey? That's just a weapon."

Only if it's used as part of a (simplistic) algorithm.

And besides, why would students want to fire the good teachers? That doesn't
really make sense. (As someone who was sent to the principal's office on a
semi-regular basis.)

~~~
ryankey
I agree that other factors would prevent students having too much power, but
"good" is a very subjective term. Even "effective" depends on how the students
learn. Just because I think a teacher is good for challenging me, another
student might hate them for being so tough.

~~~
wtallis
Even if students are colluding to give poor ratings to teachers they dislike,
it won't significantly alter the relative rankings of the teachers - the
better the teacher, the fewer students that are willing to systematically
exaggerate the teacher's weaknesses, and the best teachers (the ones who make
learning _fun_ ) won't have their scores hurt at all. Students colluding will
make it hard to set a threshold for "good enough" to not fire, but even if the
students catch on, it should still be a good tool for identifying the best
teachers.

------
dubiousjim
I know more about student evaluations at the university level than at the
grade levels discussed in this article. Here is an excellent overview of some
of that research: <http://home.sprynet.com/~owl1/sef.htm>

The message one takes away from that is that (i) yes, student evaluations are
a good predictor of some objective properties of a class (and other measures
don't even achieve that much), but those properties aren't what teachers
should be optimizing. I'd grant that (ii) it does seem worthwhile for students
to see that their interests make a difference to what happens in the
classroom. I'd also grant that (iii) some classroom situations may be so bad
that optimizing student satisfaction may, even if not educationally ideal,
still be a big improvement. And for all I know, this may be widely true at the
pre-university level; but on the other hand, for all I know, giving these
evaluations a big institutional role at the pre-university level could also be
counter-productive...the evidence cited in the article hardly enables us to
say. Any deliberation about giving student evaluations an institutionalized
role should take the evidence behind (i) seriously.

One promising message from the research reported in the Atlantic article is
that (iv) the specific tests being discussed have been designed in ways that
seem novel and especially revealing. But the article mixed that together with
an indiscriminate enthusiasm for student evaluations quite generally. And I
think many people will read this and say, "Duh, that's a no brainer." Yes, it
is a brainer! These kinds of policy issues aren't settleable from the
armchair. Even if we cleared all the political hurdles and made someone the
educational policy dictator, he or she isn't going to be able to tell just
from the armchair what the results of rolling out one policy rather than
another is going to be. So I get frustrated with articles like this one, that
report some interesting evidence but mix it together with the kind of
insensitivity to the details exhibited in comments like "That research had
shown something remarkable: if you asked kids the right questions, they could
identify, with uncanny accuracy, their most—and least—effective teachers. The
point was so obvious, it was almost embarrassing."

Neither does this inspire confidence: "Some studies...have shown that
professors inflate grades to get good reviews. So far, grades don’t seem to
significantly influence responses to Ferguson’s survey: students who receive
A’s rate teachers only about 10 percent higher than D students do, on
average." I hope that readers of this site don't need an explanation of why
the clause after the colon is only barely relevant to whether grades get
inflated because that leads to better evaluations. It's almost irrelevant. In
the first place, teachers needn't be aware of the cited fact; they may
experience grade-inflation pressures differently. Also, the cited fact is
compatible with the majority of current A-getters scoring their teachers in
ways that are largely insensitive to the grades they get, but a minority of
current A-getters and a majority of current B-getters being extremely
responsive to the grades they get. The cited fact is just not what we need to
know.

~~~
pdeuchler
Forgive me for being glib, but this seems like a lot of words to simply say
"You can't implement it because, well... you just can't!"

~~~
dubiousjim
Thanks for your comment. I'm sorry that---despite using so many words---I
didn't make myself understood. I didn't intend to argue that student
evaluations can't or shouldn't be widely deployed. I was criticizing the
sloppy, casual attitude towards the evidence displayed in the Atlantic
article. And I thought I had linked to a literature review, and pointed out
passages in the article, that supported that criticism.

------
PaperclipTaken
I am worried about adding another metric to the way that we measure
performance in schools. It's already been shown that there are problems with
the standardized tests. Not that they are terrible things, but students in the
US tend to disagree with their proliferation. (Perhaps they would feel less
this way if there were less tests)

Having students grade the teachers I think is also the same way. One of the
problems is that students don't know what makes a good teacher. The tests can
gear students in that direction (ie "I feel challenged but not overwhelmed in
this classroom"), but if we look for too much insight from the students I
think we will be misdirected. Just like we are misdirected when we pay too
much attention to standardized tests.

My fear is that some schools would start to look at these performance
measurements as golden bullets sort the way that we've started to look at
standardized tests as golden bullets. Most of jr. high was geared towards
getting perfect scores on the state exams. My Junior and Senior years of high
school were almost 100% (a few teachers went outside of the scope, but it was
a teacher decision and not an administrative decision) geared around AP tests
and the ACT.

The ACT and the AP tests have their place. And I think that student
evaluations of teachers have their place as well. Both are very useful when
applied appropriately.

I just don't want to see the system (d)evolve in such a way that too much
emphasis is placed on empirical data.

~~~
wtallis
"One of the problems is that students don't know what makes a good teacher."

Students don't need to know what makes a good teacher. They only need to be
able to assess whether they learned, and whether they had fun in the process.
Whether they learned _enough_ is pretty much an orthogonal issue, and one that
can and will be dealt with through standardized testing. Students also don't
really need to give any thought to a teacher's specific methods in order to
offer useful information.

------
rumcajz
As with any self-sustaining system (politics, economy, ecology) you need a
feedback loop to form among the parties with conflicting interests. If the
feedback loop is broken, the system deteriorates and ultimately falls apart.
If it's inefficient, the system tends to be inefficient as well. (Examples:
soviet-style planned economy, rabbits in Australia etc.)

This kind of thing (teachers grade students, students grade teachers) could
improve the efficiency of the feedback loop and thus efficiency of the
education system as a whole.

------
philwelch
This is probably the root cause of grade inflation in American universities.
Terrible, terrible idea.

~~~
Alex3917
Actually that would be the Vietnam war.

~~~
possibilistic
Are you being facetious? If not, would you mind expanding on this? I'm greatly
interested in the recent trend towards grade inflation.

~~~
Alex3917
No that's the actual reason. During the Vietnam war you would get drafted
unless you were in college, which means that anyone who failed out of college
was basically put on the next plane to the jungle to get killed. Since most of
the faculty were anti-war (or at least relatively liberal) most colleges
either dropped the grading system entirely or else made grading significantly
easier.

~~~
philwelch
OK, but the Vietnam War has been over for almost 40 years now, but grade
inflation keeps _getting worse_. Why?

~~~
Alex3917
\- Grades don't mean anything, so there's no reason to give people bad grades.
I know I quote this all the time on HN, but only because it's accurate: "A
grade can be regarded only as an inadequate report of an inaccurate judgment
by a biased and variable judge of the extent to which a student has attained
an undefined level of mastery of an unknown proportion of an indefinite amount
of material."

\- Despite the fact that grades don't predict workplace performance at all,
companies still use them to hire people. (Basically because it's a legal way
to keep out black people.) Because of this, schools can boost their US News &
World Report rank by giving people better grades, since starting salary
factors into the rankings in many years.

\- Schools are basically run as businesses these days, and that's what the
customers want. Asking why schools keep inflating grades is kind of like
asking why McDonald's keeps making their chicken nuggets taste so fucking
delicious.

~~~
mc32
-I don't know about you, but if I had to hire between 50 A students and 50 D students, I'd hire the A students.

-Schools in Africa grade students, who are they trying to keep out?

-Some schools might be run as a business; I think most aren't (they're subpar and inefficient). If they are, they're run as a mom and pop. If San Francisco schools were businesses, they'd have gone bankrupt and displaced a while ago.

~~~
Alex3917
"I don't know about you, but if I had to hire between 50 A students and 50 D
students, I'd hire the A students."

The science suggests that if you we're assigned both groups at random, you
wouldn't be able to tell the difference.

~~~
Alex3917
"Any citations for this?"

There are only a handful of studies because it's considered kind of a dumb
question to ask. But Alfie Kohn references some of them in his books, I think
in What Does It Mean To Be Well Educated? But the ones that exist consistently
show the same thing.

This includes the huge internal study that Google HR did:

"Unfortunately, most of the academic research suggests that the factors Google
has put the most weight on — grades and interviews — are not an especially
reliable way of hiring good people."

"When all this was completed, Dr. Carlisle set about analyzing the two million
data points the survey collected. Among the first results was confirmation
that Google’s obsession with academic performance was not always correlated
with success at the company."

[http://www.nytimes.com/2007/01/03/technology/03google.html?e...](http://www.nytimes.com/2007/01/03/technology/03google.html?ex=1325480400&en=e71cadb22a20a3c4&ei=5088)

~~~
lbo
The article you link to and quote, while saying grades are less important than
Google thought, goes on to say that progressive Google after doing this study
and at least partially implementing its new system "Last week...hired 6 people
who had a below 3.0 GPA." 6 out of an estimated 200 weekly hires. That's a far
far cry from D student = A student and nobody can tell the difference. Also,
all of the direct quotations from Google people in the article talk about
years spent in academia, degrees like doctorates, and interviews being the
poor predictors. Some of the best programmers I know dropped out of high
school. Doesn't mean I couldn't tell apart a room of A students from D
students or that I couldn't use the A students better for many jobs I might
need to fill.

~~~
Alex3917
"That's a far far cry from D student = A student and nobody can tell the
difference."

That's what the results found. Google just didn't change their hiring policy
accordingly.

------
jakejake
Some kids would be excellent at grading teachers. Unfortunately some would
just have a grudge, or just have a power trip and try to sabotage teachers. I
think on average they would be a good reflection of how well-liked the teacher
is, but maybe not how effective they were at teaching.

For example the tough math teacher would get a bad evaluation, but the cream-
puff teacher who never assigned homework and gave everyone A's would get a
good evaluation.

~~~
anigbrowl
_For example the tough math teacher would get a bad evaluation, but the cream-
puff teacher who never assigned homework and gave everyone A's would get a
good evaluation._

I always hear this objection, and it's usually brought up to discourage even
experimental data gathering. I don't buy it at all, and I think it says more
about the objectors' mentality than about the students'.

~~~
tptacek
It's not a crazy objection; in other environments (for instance, inside
corporations), peer and customer review regimes do foster CYA cultures. Also:
the best teachers tend not to be the friendliest or easiest, and the issue is
not just "will those teachers get bad performance reviews", but rather "will
the act of measuring disrupt the behavior we're trying to measure?"

~~~
mgkimsal
In many other situations that foster CYA cultures, the people doing the
evaluating have as much to lose (might lose my job, lose a bonus, etc).

Student/teacher situations are somewhat unique in that students don't choose
to be there, and will usually be gone in 1 year, and definitely in a few
years. Students have far less to lose in this situation. That may make some of
the be more harsh, but it likely (based on the article) will just make more of
them honest.

I can't be honest with my boss because he might make life a living hell or
fire me. I can certainly be honest with a teacher in a school who I'll never
see again in my life.

~~~
rmc
Teachers can have a very big impact on someone's life. "Oh I see you want to
go to this college, wouldn't it be an awful shame if that B turned into a
C...."

~~~
mgkimsal
the evaluations wouldn't be done until after the class is over, from how I
read it.

------
LH_Chapman
This survey was part of the larger study funded by Bill Gates. The claim of
"prediction" is not supported by the evidence.. the correlations with test
scores were low.. and in any case, this is another example of reifying test
scores and stupid concepts such as "a months worth of learning" as if all
learning is the same, regardless of the subject, the grade level, the prior
experience of the kids and so on. The very low correlations with test scores
are not surprising and a by-product of the survey questions, designed to check
whether the "good" teacher gets kids to comply with rules, defer to the
teacher's authority, think of learning as not making mistakes (and for the
sake of Gates, stay on task all the time). Perfect conditioning for students
being taught that education is a matter of doing well on fill-in-the bubble
tests that Gates and this researcher seem to value as the single best measure
of great education. See this and the links within it.
[http://voices.washingtonpost.com/answer-sheet/guest-
bloggers...](http://voices.washingtonpost.com/answer-sheet/guest-bloggers/the-
biggest-flaw-in-the-gates.html)

------
alttag
If the student evaluations correlate strongly with the standardized tests'
measures of student progress, then:

a) What's the purpose of having both?

b) If the evals are preferred over the tests, will "good" teachers continue to
teach to a predictable, standardized curriculum?

c) Is the correlation additional evidence in favor of "differential
compensation", that is, a compensation program based at least in part on exam
scores?

d) Even if the information supplied is similar, doesn't this extra test/survey
administration detract from instructional time? Is the information gleaned
sufficient to compensate for the loss of instructional time?

e) Atlanta (Georgia, USA) is still reeling from a years-long cheating scandal.
If such evaluations become "high stakes" (and there will likely be a push to
do so, despite likely union opposition), won't these results be exploitable as
well? (And perhaps even more so, through campaigning, social engineering,
etc?)

~~~
wtallis
"What's the purpose of having both?"

What's the purpose of looking at multiple polls when you're trying to predict
the outcome of the upcoming election? More evidence gives you higher
confidence and lower margin of error. And as the article says, these student
surveys provide clean, stable data that doesn't fluctuate very much from year
to year and doesn't require much correction for race and family income.

These surveys take on the order of 10-15 minutes. That's nothing compared to a
battery of standardized tests. They wouldn't have to be very informative at
all in order to be worth the small sacrifice of instructional time, and if
they're the second-best predictor of class achievement, then they're certainly
worth the time (if the results are actually used).

I don't think there's going to be much movement to stop paying attention to
standardized tests and curriculums, since these surveys don't measure the same
thing - roughly speaking, the standardized tests seek to measure how much was
learned, and these surveys add a dimension of _why_.

~~~
alttag
The point being, if they're strongly correlated, the "why" can already be
inferred (assuming adoption of this particular theoretical causal chain).

I agree the movement toward testing is likely to continue unabated. The point
was the futility of additional questions; when a strong correlation is already
known, the responses can be predicted with a high degree of accuracy, so the
additional information is not really "additional". Yes, social scientist
prefer multi-item measures and avoiding a single source bias (particularly
when publishing theory), but in the end, if the correlation is maintained, not
much new is really learned.

(And one of the reasons for multiple election polls is that the results change
over time, and leading up to an election, that's relevant. Additionally, the
entire electorate isn't polled each time, so the "cost" to the system is
lower, relatively speaking.)

~~~
wtallis
Let's make some simplifying assumptions: the survey identifies two causes of
poor performance, a lack of academic rigor, and a poor classroom environment
(such as a teacher that's mean and unresponsive to requests for help or
clarification). Those two categories are weighted equally on the survey - a
teacher who gets 100% on the survey is good in both categories, a teacher who
gets 0% is bad in both categories, and a teacher has more than one way to
score 50%.

So, if both of those factors affect student performance on standardized tests,
then there will be a strong correlation between the overall survey scores and
the test scores. But analyzing the details of the survey results can offer
actionable guidance that the test scores can't - the survey _does_ provide
useful information for how mediocre teachers can improve, even when it matches
the test scores in predictive power for future test scores.

------
camus
next step , let kids grade there parents , and the administration calls social
services when scores are low.

