

Ask News.YC: Proposed experiment to test "The PG bias" - randomhack

It has been frequently commented by people outside the news.yc community that news.yc is too pandering to PG. To test this, I propose the following experiment which requires PG's co-operation. I propose that PG also post comments / submissions etc under some other username and then compare the average upmods for posts under the "pg" username vs the random anonymous username.
This can be started at an unspecified time in the future and run for lets say 1 month. Only PG will know what username he is using other than pg.<p>I also propose the same for nickb and other prominent users. It may turn out that people upmod stuff from more recognizable names when they might have left it at neutral if some other user had submitted the same content.<p>Well actually it may already be the case that they are  posting under multiple names.<p>Might be an interesting social experiment.<p>edit : Or is it already underway? nickb is pg :P?
======
coderrr
I'm surprised no one has mentioned this yet. If you are going to do this
experiment, it needs to be _double-blind_. Meaning PG cannot know which
username his post is going to show up under until after he has written it.
Same goes for all the comments he writes (if you're going to measure those
too).

If PG knows before hand what name his content is going to show up under there
is a high likelihood of a bias on his part which he would be _unable_ to
ignore even if he consciously tried. This is standard psych experiment stuff.

~~~
attack_forgotpw
This is easy to protect against. Just obscure pg's real name from a portion of
the YC userbase.

This is really an extremely easy test to get right if we think for a minute.

~~~
coderrr
That's a great idea. I think it would probably work, but I can think of one
potential case where it would introduce some complications.

If user1 sees 'pg' as the submitter and user2 sees 'blah' user1 might comment
something which gives away that it is actually pg to user2. Now user2 might
realize this and act on their pg bias.

Can you think of a way to prevent this?

~~~
kirubakaran
Way:

group1 that contains user1 should be isolated from group2 that contains user2.
They can't see each other's replies.

But I wouldn't want such things to happen here except on Apr 1.

------
bootload
_"... Only PG will know what username he is using other than pg. ..."_

Unfortunately I suspect more than a few will recognise his fist. Just like old
school morse signallers can recognise individuals by listening. I'm pretty
sure individual writing styles are recognisable.

My idea would be to make user names invisible on submissions to individuals,
until after you vote. Only then revealing who they are to the person who
votes.

------
dangoldin
An idea I just thought of would be to keep the poster anonymous until the post
achieves a certain rating - after that show who the poster is.

If there actually is favoritism this might solve that issue somewhat and yet
give people credit where it's due.

~~~
llimllib
yes, but sometimes the reputation is valuable. Given a poster's prior
probability of intelligent comments, you have more information on which to
vote.

~~~
attack
You also allow him to relax his quality while enjoying his automatic points.
Plus group-think lock-in for free.

I can only reducing this making the overall quality here improve.

------
skmurphy
Persistent identity is an important building block in community formation.
Clay Shirky goes into some detail on this in "A Group is Its Own Worst Enemy"
<http://www.shirky.com/writings/group_enemy.html>

"If you were going to build a piece of social software to support large and
long-lived groups, what would you design for? The first thing you would design
for is handles the user can invest in. [...] It's pretty widely understood
that anonymity doesn't work well in group settings, because "who said what
when" is the minimum requirement for having a conversation. What's less well
understood is that weak pseudonymity doesn't work well, either. Because I need
to associate who's saying something to me now with previous conversations. The
world's best reputation management system is right here, in the brain.[...] If
you want a good reputation system, just let me remember who you are."

I am not sure what the problem is that you are trying to solve, but full
anonymity will undoubtedly cause the HN community much more headache than
whatever you believe is afflicting is now. Whether there is a "PG bias" or
not, he is accountable for his words because they are associated in a
persistent fashion with his login.

One final point, I don't know about a pg bias, but I certainly have a positive
nickb bias, because he has submitted so many good articles, I will normally
follow whatever he posts even if I am not thrilled about the title. Doesn't
mean I vote him up no matter what, but making the submissions anonymous would
remove an important signal/attribute.

~~~
coderrr
I think you raise an interesting point. There are actually TWO problems here.
One is that there is possibly a bias based on the content submitter/creator.

But the other is that, people who are followed will naturally get more votes,
just because they get more views. Is that a bad thing? I dunno. But what if it
got to a point where there were enough different people being followed that
the only way onto the first page is by being one of them. That kindof sounds
like a problem too.

~~~
JacobAldridge
I understand that some of Digg's changes relate to this second concern. Their
issues (I'll admit to limited understanding, so feel free to correct me
somebody) relate to their rating algorithm being linked to how 'popular' a
contributer is. This naturally weights power users more and pushes their
contributions to the front, where they get more views, more votes, and
therefore more power.

My (similarly limited) understanding of the HN ranking system is that it's
more a factor of time and votes - ie, the question of who contributed is not
relevant to the ranking system, only to all of us users who decide which
articles to read and vote on, which brings us back to the original question.

------
Alex3917
"Might be an interesting social experiment."

It's not an interesting experiment unless you learn something from it. There's
virtually nothing that can be learned from this.

Plus, I like being able to follow the comments of pg and others on their
threads page.

~~~
brett
Being able to follow pg on his threads page is part of the bias too. He gets
more votes because his comments get read by more people.

~~~
coderrr
This is true. But the experiment could take this into account. It could only
count the votes which occur directly from the main page.

------
lacker
Just because people upmod more recognizable names, it doesn't mean they are
pandering.

For example, I will be more likely to read a comment if it was written by any
user that I recognize. If I'm more likely to read a comment I'm more likely to
upvote it as well.

~~~
attack
But voting like this is screwing up the entire merit system we have here and
enforcing group-think.

It's like movie stars getting elected to political office just because they
are more recognizable...

~~~
fallentimes
No, it's like movie stars being paid more based on past performance. People go
with brand names and things they recognize whether it's clothes, fast food,
detergent or people. Always have and always will.

~~~
attack
...which is detrimental. They won't if we don't let them.

~~~
fallentimes
Actually, most the time it's not detrimental. As the post above mentions, it's
a very useful heuristic.

------
pg
Here are the users with the top median comment scores:

    
    
      ("pg" 4) 
      ("theoneill" 3) 
      ("mechanical_fish" 3) 
      ("garbowza" 3) 
      ("kkim" 3) 
      ("paul" 3) 
      ("cperciva" 3) 
    

If there is a difference, it's not huge.

~~~
attack_forgotpw
You fail at statistical analysis.

~~~
brlewis
If you think average would be a better measure than median, say why. You might
possibly be right, but given PG's track record re. Bayesian filtering, "You
fail at statistical analysis" just doesn't seem like the right response. It
also sounds like attacking the person rather than the statement.

------
kirubakaran
When browsing through <http://news.ycombinator.com/threads?id=pg> I notice
that his comments won't make sense without knowing that it is pg who is making
them.

Most are clarifications as Hacker News Maintainer and Furry Investor. (or
Furry Capitalist?)

Knee jerk vote-ups I am guilty of are for: edw519, mixmax, rms, pg and
nostrademons. But I'm very sure I'd have voted those comments up even if I
didn't know the commenter's id. ID gives more context... it is like I hear it
in a particular voice.

~~~
mechanical_fish
+1, knee-jerk vote-up. :)

No, actually it's the phrase "it's like I hear it in a particular voice".
Precisely correct. Sites like this one aren't collections of anonymous robots
talking to other anonymous robots.

~~~
attack
Yeah, lets forget careful thinking and just support whoever had a good record
previously regardless of what they say today.

~~~
kirubakaran
The experience of reading a comment knowing the commenter's id makes up (a
little bit at least) for the lack of face2face. For example, based on a
previous discussion, I know that mixmax had sex with a porn star :) When I
read his comments now, my brain does some signal processing magic using this.
Please note that I don't give him a +1 coz of this history. But this knowledge
puts warmth in an otherwise cold place (cold = place filled with strangers).

~~~
mixmax
I should never have written that - how can anyone here take me seriously now?

:-)

~~~
kirubakaran
:-) IMO your words carry more weight because of it... (more knowledge,
experience etc)

------
jkush
I'm not entirely sure what problem you're trying to solve. Are you trying to
"fix" the impression that PG-pandering is present so more outsiders make their
way into the community?

I don't particularly care if PG gets tons of karma. I love the submissions,
comments and community here.

I think it's best to just let it be exactly the way it is, even if there's
some pandering.

------
attack
It could be made so that a nickname is automatically selected per-page. This
way threads he comments in could be followed. A small pool of 5 or so names
could be reused so that he couldn't be exposed by finding the user that has
just been created.

Example implementation:

    
    
         unames=['bob1','bob2','pg']
         lu=len(unames)
         secret_seed=309580435
    
         from binascii import crc32
         def hash_eq(number):
             return crc32(str(number),secret_seed)
    
         def get_uname(page_id):
             return unames[hash_eq(page_id)%lu]
    

A checkbox for manual override could also be added.

~~~
jcl
Why stop there? Why not have a week where _everyone's_ user id is displayed as
a randomly generated syllables based on the hash of their current user id and
the current topic id?

------
icey
Find a psychologist, or even a psychology major with enough classes under
their belt, and they can tell you that there is most certainly a bias towards
upvoting those with a familiar name.

I don't recall the name of the principle, but doing some googling around about
Robert Cialdini's work should get you all the information you need.

------
mechanical_fish
I would _hope_ that, if PG and I wrote exactly the same words about (e.g.)
Lisp, his post would be upmodded an order of magnitude higher than mine. PG
knows what he's talking about as a Lisp programmer, and we all know that. (The
published books, the reputation, the history with Yahoo Store, and the
existence of this site are big, big clues.) I don't know much about Lisp
beyond SICP, and if I claimed I did you would have no way of knowing whether
or not I'm telling the truth.

Context, in the form of background information about the author, makes a post
more valuable. That's a no-brainer.

~~~
simen
That's an argument from authority, plain and simple. Whether or not you know
anything about lisp, your comments about it should be judged on their merits
alone. If you truly don't know what you're talking about, it should be
apparent to those who do know what they're talking about.

A master and a novice might sometimes come to the same conclusions, but seldom
because they followed the same thought process. And I sure hope you aren't
upmodding posts that don't share any of the thought process behind what
they're saying, the reasons why they conclude what they conclude.

Knowing context might make a post more valuable, in the sense that you
understand better how it fits into the wider world, but it won't generally
make the content any smarter, more valuable, or more correct.

~~~
mechanical_fish
_That's an argument from authority, plain and simple._

Have you ever wondered why, thousands of years after the ancient Greek experts
on rhetoric noted the existence of the "argument from authority" fallacy,
we're still awash in authorities? Are we all hopelessly stupid, that we cannot
learn to avoid this mistake?

Perhaps. But another problem is that argument from authority is almost never
"plain and simple". There's no bright shining line between an argument from
authority and an argument based on an overwhelming accumulation of evidence.
At some point it comes down to a judgement call.

Suppose a mythical person named Graeme Paul (let's call him "GP" for short)
arrived here on news.YC and submitted a one-sentence essay: "Most of the smart
programmers that I've met would rather work in Lisp than in Java." My reaction
to this would be straightforward: GP is a troll. A very boring troll.

Now let's suppose that GP's post is a bit longer: It begins with his
autobiography, in which he talks about his young days as a programmer, and his
first encounter with Lisp, and describes the various things he has built in
both Lisp and Java. He then gives a detailed, prioritized breakdown of the
design of the Java language, comparing Java's features with analogous features
in two or three different varieties of Lisp. During this discussion, it
becomes clear that GP understands the technical compromises, that he's seen
the insides of many real-world Lisp and Java projects, and that he's a gifted
explainer. Finally, GP wraps up by summarizing his personal discussions with
the designers of Scheme, Java, Ruby, and C.

GP is no longer a troll. He's now making a serious and detailed argument, with
many facets, that goes on for pages and pages. He may still be wrong --
history is replete with examples of obsessed experts who were wrong -- but
he's got a much better argument.

Now let's suppose that, instead of typing all that information in one
monolithic 193-page post, GP spread it out over several hundred blog posts and
news.YC comments, posted over several years. At the end of that time, he comes
along with a one-sentence post in some language-war thread: "Most of the smart
programmers that I've met would rather work in Lisp than in Java."

Because this single sentence is a summary of a long-standing, well-established
series of arguments from GP, I might (in the right context) mod it up. Then,
no doubt, I would be accused of "not judging this sentence on its merits
alone".

The puzzle question here is: Since GP's huge body of published work arguably
establishes him as "an authority", is my deference to GP an "argument from
authority"? Well, if I've never read any of GP's work but I upmod him anyway
because he's famous, the answer is "yes". If I've read GP's earlier work and
know that it backs up his latest claim, the answer is "no". Can you, gentle
reader, tell the difference between these two cases? Not unless you know _my_
reputation, and/or ask _me_ to provide up to a dozen pages of rationale and
clarification.

And there's a huge grey area: If I've read GP's writings on Lisp, and he comes
out with a statement about (e.g.) Ruby, how shall I value his expertise when
weighing the credibility of his latest statement? It's a judgement call.

~~~
simen
_Are we all hopelessly stupid, that we cannot learn to avoid this mistake?_

Many of us are, though I wouldn't say it is necessarily only stupid people who
do it.

I consider it a good principle that the words in a reply in forums like this
stand by themselves. If you or pg or anyone else wants to say something about
lisp, you got to back it up. If you have already backed it up in "several
hundred blog posts and news.YC comments, posted over several years", it should
be easy for you to reference some of this material in support of your views.

You saying that the exact same argument, presented word-for-word the same, is
not worth as much as the same thing when it comes out of an authority's
keyboard, is an appeal to authority. However, if this other person has already
defended this view, all he has to do is reference it to make the post worth
more. But then the posts aren't word-for-word identical. Then one contains a
reference to a justification and one doesn't.

Your example involves a question of how much to trust a person. This is one of
the cases where it matters who's making the comment. It's a straightforward
question of whether you want to trust that the person who says "Most of the
smart programmers that I've met would rather work in Lisp than in Java" is
speaking the truth or not, and it's clear that a habitual liar isn't as
trustworthy as someone known for their honesty. It's not even an argument,
it's just a truth claim.

However, the moment it becomes an argument (e.g., "lisp is better than java"),
trusting it on authority becomes a fallacy. If Paul Graham wrote, "Lisp
rocks", his comment is not worth any more than your comment if you wrote the
same. But if he wrote, "Lisp rocks, for reasons I have detailed in this essay
here: <...>", it would be worth more. But then it would no longer be "exactly
the same words".

------
mcxx
Hey, who the hell cares? I come here because of good articles and intelligent
community, that's all that interests me.

------
JacobAldridge
Does anonymity defeat the purpose of social bookmarking (and social networking
in general)?

I agree that it would be an interesting social experiment, though I'm not
concerned if people are pandering to pg, nickb, or anyone else.

~~~
amichail
Submissions could be made anonymous for a few hours. I think you would get
more interesting items on the front page.

~~~
JacobAldridge
I agree - look past who posted something and judge it on merit.

If the pandering is a problem, you'll still get some of the same sites being
voted up (paulgraham.com, techcrunch etc), but some other more interesting
ones might break through.

Still, I would hope most of us look past those things anyway.

------
andrewparker
Even if you implement this experiment, I bet there will still be an inherent
bias for PG-related material, most notably his essays.

I really don't have a problem with it... that's what this community is all
about. Hacker News is PGs world, and if you don't like it, then go somewhere
else.

~~~
pg
_Hacker News is PGs world, and if you don't like it, then go somewhere else._

I don't think this is true. I certainly don't want it to be true.

~~~
attack_forgotpw
Wishing and ignoring amounts to nothing.

------
technoguyrob
I admit I am guilty of this. Often I realize it after I clicked vote up, but
alas, by then it is too late.

~~~
Jesin
Yes, the ability to undo a vote would be useful. Shouldn't be that hard to
implement, either, given that the site already keeps track of who has upvoted
what.

I have also noticed that pg's comments have unusually high ratings, whether or
not the actual content was unusually good.

------
andreyf
> It may turn out that people upmod stuff from more recognizable names when
> they might have left it at neutral if some other user had submitted the same
> content.

I think when it comes to social interaction, the ability to build
relationships of trust is a feature, not a bug.

------
bretthoerner
I think the PG bias was proven when a picture of "asterisk man" made it to the
top of _Hacker_ News...

------
Harkins
Easiest way to test this would give the class 'user' to links to users. Folks
could tweak their browser's CSS with a.user { display: none; } real easy.

------
amichail
Also interesting would be to see whether users who have had an application
rejected/accepted vote up pg as much as people who have not applied yet.

------
TrevorJ
Ok, guys. isn't it working as intended? People build a reputation over time
and are judged according to it. It's nothing untoward or unexpected.

------
byrneseyeview
It would be a good idea to quickly delete this and email pg directly about it,
so few people are aware of the experiment.

~~~
Hexstream
Ah, I thought security by obscurity was out of fashion by now. I'm admittedly
out of touch with the "real" world.

~~~
byrneseyeview
I thought good experimental design involved not telling people what you're
testing if they have any means of altering the test.

------
jraines
Posts & discussions like this make me cringe a lot worse than actual praise of
pg.

------
zkinion
all the accounts here are bots. Theres like 3-4 real people total. ;)

