
Why Your Boss Is Wrong About You - KeepTalking
http://www.nytimes.com/2011/03/02/opinion/02culbert.html?src=twrhp
======
jtbigwoo
The biggest problem I've seen with performance reviews is that most managers
spend very little time working with or observing their employees. Couple that
with performance review processes that don't actually measure results and it's
a wonder anybody knows anything.

I was once a manager at a medium-sized consulting company. I had an employee
we'll call "Roy." He was always involved in the critical parts of large,
profitable projects. He generally got 3.5 to 4 out of 5 in his peer reviews.
There was another guy, "Jim." Jim was mostly on smaller, simpler systems and
maintenance work. He generally got similar scores in his peer reviews. For
their first performance review, I gave Roy a 4.2 and Jim a 3.8. Roy got a 7%
raise and Jim got a 5% raise. Later on, I was the lead dev on a project with
Roy and Jim as my team. Once the project got rolling, getting Roy to actually
produce code was like pulling teeth. I talked to the other leads and found out
that Roy talked a good game in front of clients but required a baby-sitter to
actually get anything done. He was always on the critical components not
because he was a great developer, but because the lead or architect was
already paying extra attention to the critical components and could more
easily manage the babysitting. Jim, of course, got all his work done on time
with minimal fuss and even stayed late to finish some of Roy's work.

I went back and looked at those peer reviews... Roy's lead dev had given him a
review that averaged to 3.5. The soft skills were mostly 4's, the technical
skills were mostly 3's and 4's. (He was an OK developer when you actually got
him to work.) My only clue would have been a 2 in "Works Independently." Jim
didn't have other devs on his projects so he had Project Managers giving him
3's and 4's for soft skills and 4's for technical skills.

A poorly designed performance review was actually worse than no performance
review.

~~~
tomjen3
>Roy talked a good game in front of clients but required a baby-sitter to
actually get anything done

Move him over in sales, he will properly do very well there since he knows the
technical part very well but has human skills as well.

~~~
dwc
This is an awesome suggestion. Except for very small companies there's usually
a place to move someone where they will excel, if management is willing to
take the time to think about it for a moment. Once in a while you have someone
who doesn't work out at all, but at least as often you can go for the win-win
of moving them somewhere they like and perform well.

------
yummyfajitas
Summary: performance reviews are not very effective, due to the injection of
subjective opinions of the boss. Therefore, we need a variant of performance
reviews which gives the boss incentives to inflate performance estimates.

Of course, the author completely overlooks any solutions besides his own pet
method, suggesting "taxpayers can't ask for more than that".

They certainly can. For teachers (the situation he uses as a lead in and fade
out), they can use Value Added Modeling. In fields with more subjective
performance, 360 reviews (reviews by your peers, underlings, overlings and
self, discarding any singleton viewpoints) are very effective.

But I guess it's more fun to push your own toy method (and hopefully get hired
as a consultant) than propose serious solutions.

~~~
jbooth
Well, for whatever it's worth, when they're at the negotiating table, the 2nd
most important issue (after comp of course) to the teacher's unions tends to
be "assorted protections against a psycho principal". So there's a lot of
cruft in the typical collective bargaining agreement that was initially aimed
at taking personalities out of the evaluation process but then turns into it's
own monster.

~~~
yummyfajitas
No, teachers tend to oppose having an evaluation process of any sort. They are
just as hostile to objective measures of their performance (standardized
testing) as they are to subjective ones.

~~~
ebiester
The problem is that the objective standards are harder to measure then you
would think.

How are you judging the urban teacher against the suburban one? Is the kid who
is getting abused at home ever going to match the one with two great parents?
What percentage of your students eat breakfast? What percentage of your
students have English as their first language?

What percentage of the individual class are troublemakers? What tools are
teachers allowed to use to subdue them? Principals differ on this.

Are there bad teachers? Certainly! Do the metrics identify which teachers are
the bad ones? Often, no.

~~~
yummyfajitas
_How are you judging the urban teacher against the suburban one?_

From my first post: _For teacher..., they can use Value Added Modeling._

Come up with a statistical predictor of performance, based on observable
quantities such as income, ESL, race, free lunch, and previous year's
performance.

Teacher alpha = actual performance - predicted performance.

All the factors you described are either highly correlated with easily
available quantities (low income), or they are directly measurable. Thus, the
predictor performance is likely to include them.

I'd be curious to see if you could even hypothesize a systematic error (as
opposed to sampling error) with a large effect that is not highly correlated
with with easily observable base quantities (income, demographics, prior
performance).

~~~
jbooth
I do think that's the general direction things should go in..

But that statistical predictor of performance will always be horribly flawed.
Applying a rigid set of a dozen statistics to human beings always misses the
mark in a ton of cases. The question is whether it's better than the current
system -- personally I'd want to see several iterations experimented with
before I'd feel confident at all that it wasn't making things worse. "First,
do no harm" and all that.

One huge problem a lot of good urban teachers have with testing is that the
tests as written really don't "speak to" poor urban and minority students. The
teachers complain that they spend more time teaching the students how to take
the test and less time teaching them the material in a way that's relevant to
their lives. I haven't ever taught, so I'm just repeating this, but I can see
where they're coming from.

~~~
yummyfajitas
_Applying a rigid set of a dozen statistics to human beings always misses the
mark in a ton of cases._

<http://en.wikipedia.org/wiki/Law_of_large_numbers>

 _The question is whether it's better than the current system..._

Without objective measurements, how could we ever answer this question?

~~~
jbooth
Teachers don't see that many kids in a year. Breaking them down into a
combination of a few demographics is oversimplifying things to such a degree
that I really don't get how anyone can be so blase about it.

I mean, just to start with:

* Are our tests measuring the right things? * Are we measuring the right things about the kids? * Are we capturing all of the data for the things we do measure about the kids?

Those are _huge_ unanswered questions to just be like "oh yeah just average
the results for the income quintile, bang, done, we know who the good teachers
are".

If a student's being sexually abused at home, and test scores dramatically
decline year-over-year, is the teacher a worse teacher for it? Just one edge
case. Throw in 4 more and you've got a significant fraction of the scores that
the teacher will be evaluated on. These are students' futures and teachers'
jobs we're talking about, remember.. I think general teacher resistance to
having some sort of oversimplified system shoved on them from above is a
feature -- that system needs to be good enough to buy them in before being
implemented.

~~~
yummyfajitas
Lets crunch some simple numbers.

Suppose 1 in 100 children is sexually abused _for the first time_ in any given
semester [1], and a teacher has 4 classes of 30 students each. Then in 2.5% of
classes, a teacher will have 4 sexually abused children (vs an average of
1.2). When this occurs, the percentage of children failing will increase by
2.3%.

So lets say on average, a teacher is expected to have a failure rate of 20%.
2.5% of the time (roughly once in their career) they get unlucky and have a
failure rate of 22.3% instead of 20%. This will occur 2 semesters in a row
about one time in 1500, i.e. it will happen to 1 in 50 teachers (assuming a 30
year teaching career).

Alternative numbers, to show that I'm not using cherry picked numbers:
P(abuse)=0.001, P(4 abuses in class of 120) = 2.56e-6, slightly greater impact
(failure rate goes up to 23.3% over 20%). If sexual abuse happens at a higher
rate (e.g. 2%), then we get into the rather implausible situation that 48% of
children are sexually abused, resulting in them failing school.

[1] I don't know the relative proportions, but this seems like a high number.
Among other things, it would imply that at least 24% of children are sexually
abused, resulting in them failing school. We are also assuming the child has a
good upbringing and good grades until the sexual abuse, which I imagine is not
the most common situation. The assumption that it is the first instance of
sexual abuse is important, because if it occurred in the prior year, it would
likely affect their prior year grades and hence their current predicted
grades.

~~~
jbooth
Well, my point wasn't that child abuse happens at whatever frequency, my point
is that a whole universe of stuff happens that isn't captured by those
statistics.

Again, I do think that some sort of unified performance measurement is
important. But it's really easy to screw something like that up. Teachers
spend between 5-35 hours a week with a student depending on schedule and we're
summing up that interaction with information that fits on half of the back of
a business card? That doesn't tingle your spider sense at all? "Warning, lots
of information loss here"?

When you base performance on a metric, then you re-orient everything around
the metric. It is very very very important to get that metric right and a
right answer probably doesn't fit in an HN post. I mean, nobody thinks
engineers should be measured by kloc, let's not oversimplify for teachers
either.

------
viggity
like the old saying goes, "its the worst system out there until you look at
all the others".

I'm sorry, but a subjective review by a principal definitely has problems
(personal biases), but it is a HELL of a lot better than judging teachers
solely on how many years experience they have and how many meaningless masters
degrees they have (French literature).

------
fleitz
Whenever measurements are involved one should diligently research the values
expected and optimize accordingly.

If you care about such things as getting a good performance review you should
know that it measures likability far more than productivity.

If one of the KPIs on the CEO's dashboard is hits to the website, you should
drive that up, if it's acquisitions then drive that up.

Performance reviews and KPIs are highly instructional as to what you should be
optimizing for. It's the company telling you what they value most. If you find
yourself in disagreement with the things being measured it's probably an
indication that the value systems you use and your company uses are out of
alignment. It's usually much easier to find a company that shares your values
than change your companies' values to suit your own.

