

Ask PG: Would it be possible to do an A/B test on HN? - jacquesm

One week (or even a day) with comment and article points hidden from view to see what that does for the quality of the site ?<p>I'm very curious about this because I suspect that the visibility of the points is what is starting to create a negative undercurrent.<p>Or maybe leave them on articles but drop them on comments...
======
rarestblog
Actually, one of the biggest Russian IT social news sites (habrahabr) has this
for quite a while - you can't see a rating of article until you vote (up, down
or "zero", but you can't upvote or downvote if you vote "zero", you just get
to see the rating). It's working pretty well ever since it was implemented.

On the contrary note - bash.org.ru, which is kind of a clone of bash.org,
tested this idea, but then they got back to showing the rating beforehand.

~~~
raquo
Yep, but bash.org.ru did not give you the option to just see the score without
voting. Given how high the trash / not trash ratio is there, I think it's only
reasonable that many people wanted to see a quote's score before reading it
(and upvoted the quote to see it).

------
prakash
Sounds like an interesting experiment, but how do you measure _quality_?

~~~
pavel_lishin
Subjectively, I'd wager.

------
oldgregg
What if you could view the down votes made by any particular user from their
profile page? A little passive accountability...

Personally I think the comment voting system is working alright. The overall
s/n is more of an issue for me-- it would be nice to see some kind of karma
based throttle for posting new articles.

~~~
jacquesm
that's a good one!

There is a downside to that though. Plenty of people use the 'up' as "I agree"
and 'down' as "I disagree". I know that's not the way they are intended to be
used, but it opens the door to 'I don't like you' and 'I like you' votes as
well.

Without having them split out like that you will then only be able to view
those votes through your own perception of what was voted on.

This will eventually lead to meta-moderation (not necessarily a bad thing),
that's a lot more work than a simple switch, which is one of the reasons I
asked this, it's dead simple, takes 2 minutes to implement and we'll have
results in a couple of days.

~~~
tokenadult
_Plenty of people use the 'up' as "I agree" and 'down' as "I disagree". I know
that's not the way they are intended to be used_

Darn, I wish I would have bookmarked the comment in which pg said it was okay
to vote to indicate agreement or disagreement. SearchYC is my friend:

<http://news.ycombinator.com/item?id=117171>

(And gojomo was my friend too, as I first found his comment linking to pg's
comment when I did my SearchYC search.)

------
mkuhn
Interesting idea, but:

You have to worry about measurability. How would you measure the quality of
the site? And how would you do it objectively?

~~~
paraschopra
Total number of comments and clicks on the articles would be the metric. Time
spent on site would be a great measure too.

~~~
mkuhn
I think that both those KPIs are quite a good idea, but they can alos lead to
quite wrong measurements: What if this leads to people just clicking on every
comment link to find out if there is a discussion or if there is a new post...
etc.

It doesn't meassure the quality of the discussion and the submissions.

~~~
jacquesm
How many comments there are could be visible without any detrimental effect.

I simply suspect that the 'points' system has a shadow side and that as the
site grows the shadow side starts to overpower the positive portion. By
temporarily switching it off and asking the community what they felt about
being 'point blind' for a short period and if they thought the quality
improved or not you can make a 'metric' that is much easier to measure than
some of the more technical tricks you could pull:

    
    
      Customer Satisfaction.
    

A simple poll after the experiment would suffice. Or you could make it
switcheable on a user basis if it is a toss-up or too close to call a very
clear preference.

If on average more people feel better without the points visible than with
them after a short trial period then it's something that you could consider
doing permanently.

Another option would be to keep author and points hidden until after you've
voted for a comment, but that may have other side effects.

I think the content should stand on its own, regardless of what the voting
history and the author are it is what you think about it that counts.

Right now the choice to vote or not to vote is made plenty of time based on
the current number of votes, which leads to plenty of feedback loops. I've
seen 'flip-flops' (bi stable and tri-stable, 0,1 and -1,0,1), positive
feedback loops, and negative feedback loops.

By breaking the loop we could end up with a more balanced view.

Imagine what the effect would be of a running tally during an election, it
would completely affect the outcome, and not necessarily in a positive way.

Again, it's just a gut feeling but I think there is some truth to it, and it's
a very easy to do experiment, worst case we will learn that it did not work.

~~~
mkuhn
Thank you for further elaborating your idea.

I think the experiment would be a great idea and doing it the way you propose
with empirical data to back up the machine generated statistics could provide
very valuable insights.

Also one could start experimenting with the approach/idea e.g. let the number
of votes appear after voting or not displaying a number at all

------
req2
Any 'solution' to the 'quality problem' that fails to acknowledge that some
people are obviously voting up the (politics|techcrunch|zed|etc.) articles you
perceive as low quality is going to fail.

You need to account that some people love the hyperbole that you always find
in (politics|techcrunch|zed|etc.) articles, and some people don't. This
suggests implementing 'cliques'. If you upvote a submission, you grow one vote
closer to the clique of people who also upvoted that submission. People whose
vote history match yours very well will very easily influence your front page,
and the articles of a kind you rarely upvote will be less likely to cloud your
front page.

(More feasibly, you could just add some 'coolfinders' manually, whose upvoted
articles automatically jump to your page.)

~~~
jacquesm
I've sent you some mail about this. Very interesting indeed!

~~~
req2
I haven't had time to look it over in full yet, but I will respond when I can.

~~~
jacquesm
thanks!

------
anamax
Is it time to start treating posts about points like political posts?

~~~
jacquesm
Apologies for crossing the first rule of HN (The first rule of HN: do not talk
about HN on HN) but there seems to be no other good spot to have a discussion
about an idea like this.

What would you suggest I do instead ?

~~~
stonemetal
I believe the feature request link at the bottom goes to an appropriate place
to have such conversations.

~~~
jacquesm
That thread looks quite dead to me.

~~~
req2
You could fork the source and run your own A/B testing.

~~~
sanswork
Except that wouldn't be sampling the HN community which would make the test
unable to answer the original question.

~~~
req2
Code in hand goes further than request in hand.

------
kashif
<http://news.ycombinator.com/item?id=169479>

~~~
kashif
<http://news.ycombinator.com/item?id=169662>

