

Ask HN: How much transparency for an online reputation system? - sachinag

In light of the new orange usernames, it made me think about something that's been bothering me for a while.  Please bear with me - this makes more sense as a blog post, but I'm not going to get the comments there as I will here.  :)<p>At Dawdle, we have a substantively different approach to how we evaluate sellers versus eBay and Amazon.  At first, we didn't want to rate sellers at all, given that we have a 100% money back guarantee like StubHub.  However, it became clear that <i>users</i> just weren't comfortable with no seller ratings given the type of merchandise on Dawdle - new and used video games, systems, and accessories.<p>So when we decided that we needed to have a seller rating, we tried to figure out the best approach.  We thought that eBay/Amazon-style ratings aren't the greatest because it's all about accumulation of points (+/- 1, similar to Karma here on HN), where each additional point has diminishing value.  So we decided just to <i>start</i> with buyer feedback, but layer a ton of other things on top of that.  Thing is, we don't say what all those layers are: http://www.dawdle.com/help/index.php/sell-feedback/<p>We wanted to make sure that people weren't setting up shill accounts and inflating their ratings, and we like how Google gives guidance but not the exact algorithm for their rankings.  The general idea is to encourage good behavior and to allow us some leeway to fix things as we see issues.<p>So what do you think?  Should we tell people how we calculate it so that everyone knows it's on the up and up and we're not secretly favoring people?  Or can we get people to try us out knowing that they're on a much more level playing field as a new user versus older users?  (We're going to make it even easier to pick what seller you're buying from - you couldn't in our beta, just like StubHub doesn't tell you who the ultimate seller is - and these sorts of things are very important to both buyers and sellers.)
======
subpixel
These Clay Shirky articles might interest you. In fact one of them might have
been linked to from HN recently, I'm not totally certain:

<http://is.gd/iv84>

<http://is.gd/NZu>

~~~
sachinag
Awesome, reading now. There's a lot of stuff about implicit rankings that
we've thought about (see my standard presentation on this:
[http://www.slideshare.net/dawdledotcom/actually-useful-
trust...](http://www.slideshare.net/dawdledotcom/actually-useful-trust-
metrics-2008-08-09)), but having written words helps a lot.

------
CalmQuiet
I think there's a fine line to walk: tell enough about the _principles_ that
you use for contributors to earn reputations, but not with too precise a
detail. Not necessarily to keep people from trying to "work the system" - but
because you may need to tweak how things are weighted without always having to
make a formal commitment to weights, etc.

Besides: telling people which factors will affect reputation ratings is a
powerful statement about the principles and values you want your site to
support. That make sense?

~~~
sachinag
Thanks; do we do enough to tell people what principles and values we support?

~~~
CalmQuiet
I'm not sure that I can answer that (not familiar enough with the gaming
population). So I certainly wasn't trying to suggest that you are expressing
it enough. For that matter, I think that sometimes such expressions get laid
on too heavily/preachy. That's part of why I think it's powerful (for sits on
whose opinions I depend) to "show not tell" And the criteria a site uses for
karma/reputation is a strong way of "showing."

------
ErrantX
> so that everyone knows it's on the up and up and we're not secretly favoring
> people?

Trust has to start somewhere. The people that think the above will in the most
part either a) not be using your site or b) not convinced by your "open
algorithms" anyway.

On the other hand opening them up fully makes it easier to game the system:
unless it is a bulletproof system I would avoid full disclosure.

------
diN0bot
reminds me how i prefer rating schemes over point accumulation ones.
especially if the rating average is based on latest actions more than older
ones. point accumulation with exponential experience levels (think rpg) is
better than linear point accumulation.

i vote for you to keep the exact algorithm secret. make it work. that's more
important than transparency. your real goal is trustworthiness, which i think
you can accomplish in other ways. (i will never get used to voting on the
_quality_ of a post rather than in agreement/disagreement, which is what
voting is historically intended for)

------
RobGR
Everyone should know the rating system. People tend to ignore rating systems
that are hidden or not easy to understand. An example of a rating system that
suffers from many hidden features is the "credit score" system; of course it
is not designed to be useful to consumers, it is designed to be useful to the
companies that know it's hidden factors. An example of a scoring system that
is defined, but generally hated for being opaque and complecated, is the BCS
college bowl system.

You should make it so it is not easy to game; the way to go about this, is to
make it so that the easiest way to "game" the system, is to do something that
you want to encourage anyway. So instead of making a fake account and making
fake transactions back and forth to up your score, give sellers a bonus for
allowing new, unrated buyers to pay after receiving the item, for example.

Think about how the standardized tests for private pilot licenses and etc
work. They publish thousands of example questions with the correct answers,
and then for your test they select a few at random. If you "game" the system
by studying all the questions, you achieve their goal anyway, which is to
learn the material as well as they can reasonably measure.

Now, you could take out the reputation and knowing the seller part of it, but
I think then you would have to step in and guarantee things for the buyer.

I would do an ebay-like accumulative rating system, but have points be
weighted with age, so that a good rating from a week ago is worth more than
one from a year ago. You might fade them linearly to zero, but I think your
formula should give people who have had an account for a long time some weight
just for that.

I am experimenting with a scoring / rating system now. it is for my site
slackerfactor.com, which is mostly functional. The site allows you to keep
track of who paid which costs among roommates or a similar group; if someone
is a slacker, and doesn't pay his fair share for a while, his "slackerfactor"
score shows it, and the people who are doing his share similarly have a better
score.

I decided to make the score fade with age, linearly to 3 years, after three
years it doesn't matter any more. Since recent debts / credits matter more,
someone who is up for re-evaluation as a housemate at the end of a lease or
whatever, and has been slacking all year, has an incentive to clean up their
act and pay a bunch of bills or buy groceries or whatever. The main site
doesn't display the slackerfactor yet, only the beta site, because the formula
probably still needs to be tweaked according to how people will use it.

Another approach you might want to put in your bag of tricks is per-user
modifications of the score. On Slashdot you can say you want to ignore
"offtopic" moderations when deciding what comments to show, or to rank "troll"
as a plus instead of a negative. Similarly, you might offer users the ability
to have some checkboxes such as "ignore rankings more than a year old" or
"negative rankings count as double if the person making the negative ranking
has a score above 50" and so on. Giving people the ability to tweak knobs
often quiets the most persistent complainers.

