
Two Kinds of Judgement - dfranke
http://paulgraham.com/judgement.html
======
gyro_robo
There seems to be some confusion here with "the usual distribution of
ability".

>"There will be a few stars who clearly should make the team, and many players
who clearly shouldn't. "

Rather, there are a few who clearly should, a _few_ who clearly shouldn't, and
many in-between.

Moreover, this assumes there is a quantity called "ability" and that the
evaluator is accurately measuring it. I note the later downplaying of
accuracy, but before that, it says:

>"Probably the difference between them will be less than the measurement
error."

How do you arrive at an estimate of "measurement error" in order to claim
that? Did you first measure the evaluator's "measurement ability"? With what?

Even if we naively assume that we can order a group by "ability" with regard
to start-ups, and that the difference between adjacent slots is small, it
_does not follow_ that your evaluation comes anywhere close to the presumed
omniscient set-ordering.

The data you have is your self-selected subset, and even there you find that
choice #30 is actually in the top few, which is a rather large measurement
error; you might argue this doesn't matter in aggregate, but it still argues
against any notion of small measurement error. If you can be off by almost
your entire selection size then who knows what you missed in the numbers up to
twice the chosen cut-off?

The statement "it matters least to judge accurately in precisely the cases
where judgement has the most effect" seems self-contradictory. What I would
_guess_ you mean is that if someone is clearly good, they make the cut, and
you don't need to scrutinize to find out _how_ good; but it still matters that
you put them on the right side of the cut!

It matters _the most_ to judge accurately where judgement has the most effect
-- given the nature of start-ups, your _missed_ borderline case could be the
one that would have been larger than all the rest put together. Remember
_your_ perceived borderline is not the same as _the_ actual borderline. See
Bessemer's anti-portfolio: <http://www.bvp.com/port/anti.asp>

The article seems to focus on accuracy initially, then semi-mea-culpa its way
out of it later.

Reading hundreds of applications, your eyes glaze over, and you may only
notice someone when they happen to push one of your buttons by chance.

It's fine to say you think you selected an "optimal subset" of people
according to your checklist, biases, or opinions; but don't confuse that with
"ability". You have an idea of what you want and pick according to resemblance
to it: small teams that remind you of Viaweb's structure.

Using the game theory definitions, I don't find it discouraging that you chose
an "optimal" set, which is clearly different in this case from "maximal" ;)

Just don't confuse maximal individual ability with optimal set selection,
which is what you appear to do with your example of selecting players for a
national team.

Tanenbaum would've failed Torvalds due to his own _bias_ , which is clearly
separate from Linus's ability.

~~~
wordsthatendinq
I don't know the consequences of posting in a 5-day old thread, but I just
felt like practicing my side interest in econ. My only experience with game
theory was a half-term graduate course and I don't follow everything you're
saying, but my gut feeling is that these definitions aren't important to the
spirit of the article.

Suppose you're buying stocks; you choose a portfolio of stocks to maximize
your return. Of course you'll make mistakes and pick bad stocks, but as long
as you have enough good stocks in your portfolio, you'll still turn a profit
(and companies won't come and argue "why didn't you buy my stock?"). The
difference between YC and the stock market is that being chosen by YC can
potentially increase your value; they are more like the coaches that train
athletes. So whatever portfolio they pick will actually end up with a higher
value than if it were just left alone. Also, it makes sense to evaluate people
by things not so one-dimensional as "ability" (which by gyro_robo's argument
we should maximize), otherwise we ignore other possible synergies between
coach and trainee, such as similar personalities, that can cause the trainee
to train more productively, independent of ability.

The problem with the stock market analogy is that all you need to do is to
beat the market, and to do that you just need to perform better than average.
Actually YC could do that if all they wanted was to make some money. Perhaps
similarly, a sports coach could be lax about picking athletes if the second-
best team was way behind. This is where I see the game theory coming in. If
there was another team, similar to YC, that was similar in performance, then
YC would have more of an incentive to minimize the measurement error. I'm not
saying they don't already, I'm just saying it would make sense to do so.

Anyway, I don't know if this makes any sense but the essay is fun and probably
a good piece to show to some younger people.

------
whacked_new
This curiously reminds of "You and Your Research" posted here yesterday:
<http://www.paulgraham.com/hamming.html>

The Nobel Prize committee, for instance, has to pick a handful prizewinners
out of hundreds of first-class scientists. The ones that don't get picked are
still first-class, but if the "Nobel Prize Effect" holds true, the ones that
do receive the prize are the ones who don't sustain first-class work.

Of course this works differently for a sports team, because the feedback of
your efforts are more or less immediate (from game winnings). But for things
like startups then, if we really digest both these articles, I'd say the Nobel
Prize analogy holds better. The ones who experience failure, or a lack of
success, and obstinately jab at their problems, will still make a big kill.
And the rejects are always far more numerous. For every Nobelist, there are
probably a thousand others doing comparable work.

I once met a scientist who was considered Nobel-worthy. He came off mostly as
a grumpy old man though, because he kept lamenting how his classmate got one
but he didn't. If I was half as intelligent as him I would be doing cutting
edge research as a hobby! Perhaps the world is better now that he didn't get
the prize.

------
jason13
I disagree with the writers assumption that the committees judgements are only
inaccurate for the borderline cases. I think there are plenty of examples of
committees judgements being wrong about candidates, that should have been
viewed as outstanding.

~~~
danw
They're not judging the candidate, they're judging the application. The
candidate might be outstanding but if the application does not convey this
then the committee won't know.

With any application the you write take a step back, leave it for a few days
and then look at it as if it was writen by someone else. You'll quickly see
where your assumptions lie and what you've missed.

~~~
jason13
Yes, committees can do no wrong!!

------
vlad
Great essay.

I disagree that you can or should only protest those judgements about you
specifically, but I do agree with the entire second half of your essay.

I agree with the idea behind the essay, that "If it doesn't matter to someone
else whether they choose you, then the other party has the leverage. You want
to put yourself in positions where you have the leverage." For example, if you
apply to a college, and thousands of applicants are just like you, then you
don't have leverage.

Your examples just ignore the fact that people get rejected from about 4
colleges a year, while they have a chance to correct their tests and homeworks
about 1,000 times a semester.

This means that the numbers simply don't match up. If you appealed 1,000
college decisions, you would have a great chance to get into a lot of them. If
you appealed 1,000 questions on a homework to one busy and egotistical
professor, they would likely start avoiding you after the 10th question.

Therefore, people SHOULD appeal everything they want to, whether the judgement
is specifically about them or not.

I saw that this was written originally as a post to a question about YC
decisions. As far as advice about YC, I think the best explanation of why some
people were rejected is that there is a hidden bias (though very well earned)
to hang around with people you think you'll find interesting or ideas you
think you'd love or have a lot to contribute to over the summer.

~~~
shiro
PG is just pointing out a general fact, which is well known to those who are
constantly being judged. It doesn't change whether PG wrote this as a response
to the post, or as a general advice. If you doubt that, ask anybody in the
profession where being judged is in its nature (I sometimes audition as an
actor so I think I know that business a bit.)

Your mentioning about hidden bias is somewhat true, except that it's not
really hidden, but just implicit---in most of the judgement there are always
some implicit criteria the choosers have, and good players know that. Assuming
otherwise is too naive.

About appealing: As far as you're not in the few of the obvious top group or
in the obvious fall-behind groups in terms of the primary factors, any
arbitrary secondary factors can put you in either side. Those factors are so
random and it's waste of time to appeal---sure, you may have some chance of
the judgements reversed, but it is much wiser to spend that time to make
yourself excel in the primary factors, so that next time such randomness will
affect less to the outcome, because if you like to live a life taking
challenges, you will be constantly judged, and you won't have time to deal
with those randomness.

~~~
vlad
Great post. The point of my post is that I don't think there are two specific
categories of judgement, where in one you appeal, and in the other, you don't.
Reversing a rejection to 1 or 2 or 3 extra colleges, since you already paid
the application fee, could be worth it. That's just one example. I'm not
disagreeing with Paul' overall message, I just think the examples are very
bad. I think sending a letter of appeal could be worth it in a lot of cases if
the trade-off or risk-reward is worth it.

~~~
shiro
I see. I understood the original article's two categories maps to the case
whether those "random factors" exists or not; I'm not familiar with school
admission process in US, so I don't know if that particular example is good or
bad.

------
ced
I think that the distinction between the two kinds is rather artificial, from
the applicant's perspective. Both involve estimators of your ability, be it
your ability to take an exam, or your ability to write an application. Both
are more-or-less correlated with something important (being good in math;
being a good potential entrepreneur). And both estimators are rewarded for
being good. Teachers get a raise, investors get money. Why should one
rejection be personal, and the other not?

In fact, the incentive to be accurate seems much larger for investors than for
teachers. Thus, I would be much more bummed about being ignored by a smart
investor, than about being given an F by a 9-to-5 teacher.

------
danielha
This was part of Paul's Stanford talk. It fits in nicely in regards to
decisions from venture capitalists on investment and even Y Combinator on the
founders program.

------
febeling
I have clearly always somehow felt that it was like this, but I never really
realized that there were these two distinct types of "judgement," or perhaps
better (but uglier) types "being judged". It is certainly an important
(however small) idea, and one without capitalism and freedom in general must
really be hard to deal with. Put it into the "Zen of Business" teachings.

------
Alex3917
I'd downvote this just to be ironic, but in his infinite foresight pg removed
the option.

~~~
nostrademons
That must be one of the things he can do to influence the outcome. ;-)

------
brlewis
This is a really well-written essay, which actually surprises me a little
given how timely it is to the YC application process. When did you start
writing it?

~~~
gyro_robo
To me, it seems muddled; I don't see the usual thanks to the usual suspects so
I assume PG single-foundered it instead of running it past RTFM et al.

~~~
pg
If you can point out a specific muddle I can fix, let me know.

~~~
ced
"if the difference between the 20th and 21st best players is less than the
measurement error, you've still done that optimally."

I thought that the coach was responsible for minimizing the measurement error.
Then, the best he could do is to pick the best team. How do you define
measurement?

------
dfranke
Experiment: let's see if this gets a higher score here or on Reddit:
<http://reddit.com/info/1idib/details>

~~~
Sam_Odio
Of course it will get a higher score on reddit, just because huge volume of
traffic going to reddit.

But the scores themselves mean nothing, on absolute terms. What really matters
is the article's relative rankings - which (roughly) determine the article's
placement on the main page.

And that's what makes News.YC so appealing. Even though the community's
smaller, we don't have to wade through pages of higher-ranked flickr photos to
find the quality content.

