

Hacker News Analysis - wslh
http://blog.echen.me/2011/03/14/hacker-news-analysis/

======
tptacek
You're working from a different definition of "good comment" than Graham is.
The concern with "bad" comments isn't simply that they exist, but that they
have deceptively positive scores.

~~~
bermanoid
...which is a problem with a crystal clear cause: if you allow purely
democratic voting, then you'll end up with really popular crap reaching the
top of the pile fairly often. Imagine how useless Pagerank would be as an
algorithm if it considered links from every drooler with a domain name to
count for the same amount as those coming from the frontpage of Yahoo - that's
essentially what you're doing when you tally up votes on a site The only
reason it's not worse than it currently is is that there's no real incentive
for people to game the system, so we don't end up with massive comment
spammers. But we still have douchebags, fools, and all other sorts of sheeple,
and in large enough numbers they end up diluting the votes that should count
the most.

Several solutions are probably obvious to everyone here and I won't go into
them, but the point is, you'd have to be willing to throw fairness aside in
order to fix the problem - if a brand new account's vote is worth the same
amount as mine, which is worth the same as tptacek's, then something is wrong.

~~~
Joakal
Your last suggestion seems to make things less democratic understandably, but
how would it be implemented?

~~~
jamesbritt
_Your last suggestion seems to make things less democratic understandably, but
how would it be implemented?_

Weighted votes based on karma. Or restrict the number of votes a user can cast
based on karma or account age.

A problem with these kinds of things is determining what does or does not work
and picking up on unintended side-effects, and doing so in a timely manner.

For example, if "young" accounts have very limited votes available will new
users be less likely to stick around long enough to become more active voters?
How long would it take to see this? What if by the time you recognize an
undesired side effect you've already sent the site down the road of ruin (or
something)?

For example, how long will pg keep comment vote hidden? How many people have
found this to be sufficiently detrimental that they have, or soon will, leave
HN and not come back?

How do you craft meaningful site experiments like this while keeping risk to a
minimum?

~~~
Joakal
I suggest similar to Reddit's anti-spam mechanism, fuzz the voting but give
the impression that karma is somewhat important.

~~~
andreaja
Maybe all we need is 'bad, average, good, great' to be shown alongside the
comment. Or something along the lines of what slashdot do with allowing you to
vote for comments as 'insightful' or 'funny' etc.

------
lt
I've been thinking lately about running some analysis of my own, but couldn't
find a data dump and didn't feel like scraping. Why has the dump been pulled?

I'm thinking of classifying entries as temporal/atemporal. Temporal are
industry news, press releases, rumors and alike. Atemporal are reference
materials, essays, howtos, analysis, links to libraries/frameworks/products.

Essentially, I feel that while temporal entries provide a more immediate
reward and larger discussions, atemporal tend to be more technical, more
interesting, have more valuable discussions, and overall are what sets HN
apart of other places. I'd rather see a front page full of erlang stories than
pieces about what Apple has just announced (I'm pretty sure I'll see these in
many different places).

A good balance is probably ideal: a few most interesting breaking stories, and
the most thought-provoking atemporal materials users have ran into lately. The
perceived decline of the front page and comment quality could be attributed to
a shift in this balance towards temporal stories.

As to how to classify, I'd start as marking stories with multiple duplicates
in a short period of time as temporal, and the ones with duplicates spread
apart over long periods as atemporal, and go from there.

~~~
rflrob
It's not obvious to me how to _automatically_ classify stories as (a)temporal.
My understanding of the way submissions work is that if Bob submits something
that Alice also recently[1] submitted, then it just gets counted as an upvote
for Alice's story.

Then, of course, there are the "this happened recently, which inspired me to
think about some broader trend". For example, during the AWS outage, Coding
Horror had a story about Netflix's Chaos Monkey[2], which one could easily
classify as both temporal and atemporal.

[1] For some definition of "recently" that I haven't yet figured out. [2]
[http://www.codinghorror.com/blog/2011/04/working-with-the-
ch...](http://www.codinghorror.com/blog/2011/04/working-with-the-chaos-
monkey.html)

~~~
lt
I believe your understanding is right, recently being a month, provided it's
the exact same URL. Popular news are usually submitted from different sources
so there are actual duplicates. I agree that it's certainly not easy to
automate the classification.

The Coding Horror example is a very good one. Jeff write very atemporal
stories, I'm pretty sure that happens intentionally. It's not unusual to see
one of his posts from several years ago pop in the front page every now and
then. Same with Joel's. It may be inspired by a recent event, but the lesson
is meant to stand.

I think a good heuristic would be: given a random, highly voted story from the
archives, would it get upvoted if reposted today?

------
tokenadult
Alas, the date of the submitted article is 14 March 2011, so all the data are
from before the recent round of experimentation announced by pg.

<http://news.ycombinator.com/item?id=2434333>

Considering that pg had asked just before then (28 days ago, also after the
date of the submitted blog post) how to "stave off decline of HN,"

<http://news.ycombinator.com/item?id=2403696>

it may be that there are differing opinions about the quality of posts to HN
in the early part of 2011. Writing then, he said, "The problem has several
components: comments that are (a) mean and/or (b) dumb that (c) get massively
upvoted."

------
DTrejo
I would love to see something like this from PG.

In specific, I'd like to see information on the following in relation to the
recent changes in HN:

    
    
        - increase/decrease in activity of users with highest karma
        - increase/decrease average in comment score, normalized 
            by time after post of OP
        - amount of time the highest rated posts stayed on the front page
        - trends for # of flags
    

Also, it would be great if he put the guidelines on the submission page.

------
duck
_So Hacker News has probably gotten worse if you like to read every single
post, but better if you only like to read the front page._

It is hard to define what a "quality" article is and not sure if I agree with
his assumptions, but I personally have seen exactly this over the past year
compiling my weekly Hacker Newsletter.

------
zdw
No graph of Apple in the "mentioned companies" section? I'd be very curious
how they were trending.

~~~
topomorph
D'oh, Apple always strangely slips my mind when I think of software companies.
I added a graph (<http://dl.dropbox.com/u/10506/blog/hn-
analysis/pct_apple.png>). Interestingly, it's the company with the clearest
upward trend.

------
eru
Interesting analysis.

I strongly disagree that what he defines as "Quality of Posts" measures
quality of posts, though.

~~~
notJim
If coming up with an objective measure of quality was feasible, the voting
system could be scrapped entirely, and instead of making a site that allows
users to submit links, pg could have written a web crawler.

~~~
eru
Sorry, I don't get your comment.

~~~
notJim
I'm just pointing out that it should come as no surprise that the author of
this blog post didn't come up with a good metric for judging quality. As in,
if the author had come up with such a metric, he would have started a company
and become fantastically wealthy applying this magical quality-quantifying
algorithm, rather than writing a blog post.

~~~
eru
Perhaps. But they are degrees of usefulness, and it's probably possible to do
better. Especially if you do an analysis in hindsight, where it's no longer
possible for the participants to game the system.

------
vtail
An interesting analysis (which only pg could do) is to multiply each vote by
ln(user's karma) or something similar. That could arguably deal with "noobs
who don't follow the rules" problem.

The above analysis will assume, though, that the problem of decreasing quality
of votes (or discussions) is caused by newcomers and not by old high karma
users getting nastier.

------
palish
Perfect example of how graphs can be subtly deceptive:

1) "Variance in number of comments" is visualized as a circle's radius.

2) ... But a circle's _area_ increases as (pi)r^2. That's an exponential
increase.

A fair visualization would show a linear increase, not an exponential one.

EDIT: To be more clear, the goal is "more pixels" = "more variance". Growing
the visualization by a radius is inherently unfair.

It's precisely these kind of subtleties that make statistics such a fallacy
minefield.

~~~
masterzora
> 2) ... But a circle's area increases as (pi)r^2. That's an exponential
> increase.

No, no, no, no, no. "exponential" != "has an exponent". Exponential means the
variable _is_ the exponent, as in 2^n. The word you're looking for is
"quadratic".

~~~
palish
It doesn't matter whether it's unfair by a factor of n^2 or by 2^n. If it's
unfair, then it _cannot be looked at whatsoever without drawing incorrect
conclusions_.

Or are you trying to educate me out of good will? If so, thanks.

~~~
masterzora
Kind of the latter, really. While the true meaning was clear from context
here, it's very easy to make the same mistake in other contexts without the
true meaning being so clear. And, unlike a number of other common mistakes,
this is one that can really bite you in the ass in some contexts, so it's good
to set the record straight before your gluteals are masticated.

------
tomkinstinch
Your plots look great; what tool(s) did you use to create them?

~~~
topomorph
I used R's awesome ggplot2 package.

------
JCB_K
I'm surprised there's little under 4000 users. Always thought it was much
more.

~~~
topomorph
Just to be clear, that's the number of users with at least one submission in a
given month. (Not the number of users with at least one submission _or
comment_ , as I sleepily wrote originally, and definitely not the number of
distinct visitors.)

------
ignifero
regarding the comments, what are the types of comments that get lots of
upvotes but are not constructive? is it the funny ones?

also, what about having a multiparametric approact to voting like this:
[http://textchannels.com/?mode=page&pg=about](http://textchannels.com/?mode=page&pg=about)
(shameless plug i know, but hopefully on topic)

