
Hacker News as a case study to test the wisdom of the crowd theory - snake_case
https://venngage.com/blog/why-you-need-to-stop-obsessing-over-comments-on-hacker-news/
======
danso
Not sure if I agree with the conclusion:

> _Such a specialized community is a great place to get feedback on the
> technical details of your product, but not to validate your ideas. The
> responses from people in these communities tend towards scepticism to
> negativity, even for products they use._

Or with the methodology. Take for example the Dropbox thread [1] which the
OP's sentiment algorithm classifies as having a "negative top comment".

This is what the purportedly negative comment says:

> _The only problem is that you have to install something. See, it 's not the
> same as USB drive. Most corporate laptops are locked and you can't install
> anything on them. That's gonna be the problem. Also, another point where
> your USB comparison fails is that USB works in places where you don't have
> internet access._

> _My suggestion is to drop the "Throw away your USB drive" tag line and use
> something else... it will just muddy your vision._

> _Kudos for launching it!!! Launching /shipping is extremely hard and you
> pulled it off! Super!_

In what universe, besides the one of a sentiment-analysis-algorithm, would
that be considered a "negative" comment? Moreover, even if that last,
congratulatory line didn't exist, the rest of that comment is very well-worth
listening to. It might have been _wrong_ (has Dropbox penetrated corporate IT
as well as USB keys? USB keys have definitely taken a beating in reputation
and in convenience...but has Dropbox been an easy transition in corporate IT?
At Stanford, we have Box)...but it definitely wasn't non-constructive
criticism.

If a submission gets a lot of upvotes, to me that's a positive-enough sign of
validation. Is it really helpful to the submitter to see a dozen/hundred
comments that are merely, "Awesome! I like it!"? I often like reading the
comments before I check out the submission, because I don't want to have to
parse Press-Releasese to understand what the product does, or who it may
compete against, or what its flaws might be...Even if the comments were
completely devoid of constructive and insightful criticism -- I'm sure after a
big launch, it's helpful, yet annoying when people immediately nitpick grammar
and typos -- if you're a founder of a great product, the hemming and hawing of
HN is probably the least of your obstacles on the way to success.

[1]
[https://news.ycombinator.com/item?id=8863](https://news.ycombinator.com/item?id=8863)

~~~
jasode
_> Or with the methodology. Take for example the Dropbox thread [1] which the
OP's sentiment algorithm classifies as having a "negative top comment". [...]
In what universe, besides the one of a sentiment-analysis-algorithm, would
that be considered a "negative" comment?_

To clarify, they said they _manually_ classified the top comment instead of
leaving it to the judgement of the algorithm: _" , we manually categorized the
top comment for each thread."_

As for why their human eyes judged it as "negative", I'd speculate it was
phrases such as _" The only problem is", "That's gonna be the problem.", "your
USB comparison fails"_

So yes, the top comment's criticism is _constructive_ , but at the same time,
it can also be subjectively classified as "negative." The positive back pat at
the end of the post was _not about the product itself_ but for the
accomplishment of launching something. (You could say it's the attaboy "trophy
for participating" consolation prize to soften the previous paragraph's
criticism.)

~~~
mreiland
that sentence is technically negative, but not conceptually. constructive
criticism is a thing.

------
jasode
Interesting article. I've thought a lot about the slashdot/HN mispredictions
of future successes and my random thoughts:

1) A flawed Theory of Mind applied to the general public: I think it's fairly
safe to say that the technical crowd (which includes commentators of
slashdot/HN) will lean towards the Asperger/autism spectrum. It's easy for
geeks/nerds/experts (especially vocal ones) to misjudge how products could be
accepted by the masses (who are _not_ geeks/nerds/experts). The famous
examples being the slashdot dismissal of the iPod in 2001 and the iPad in
2009. (And Sara McGuire's article that this thread is about has more examples
from HN.)

2) Skepticism/negativity is easier to itemize and write, and as a strange
bonus, it is _perceived_ as smarter and more insightful analysis. Looking at
all the things that are "wrong" with a product might be described as a type of
"closed-ended" thinking. On the other hand, imagining the different ways a
product could succeed involves more "open-ended" analysis. The problem of
writing open-ended thoughts is that it looks like naive futurism, or
unsophisticated cheerleading. (E.g. haha, Back to the Future's Bob Gale
thought we'd have hoverboards in 2015! Dean Kamen thought Segways would sell
millions of units and would revolutionize the world! etc).

Because of the combinations caused by faulty Theory of Mind and biases towards
the shortcomings of products, the commentary on new business ideas will almost
always end up being negative.

~~~
scrollaway
> It's easy for geeks/nerds/experts (especially vocal ones) to misjudge how
> products could be accepted by the masses (who are not geeks/nerds/experts).

That's a pretty bad conclusion. Is it any easier for non-"geek/nerds/experts"
to judge the iphone on its mass market appeal? I don't think so. In fact
that's pretty much what you figured in the BTTF "predictions" failing.

The thing is a lot of people here on HN, and in the tech community, like to
judge products based on how easy or hard they are to build, and that has zero
relation to how good the product is or how successful it will be. For a non-
techie, it's in the realm of the impossible either way, so that stuff doesn't
matter at all.

~~~
VLM
"to judge the iphone on its mass market appeal"

Note that you can make a lot of money out of a product that flops in the mass
market like an iphone. Out of any group of ten Americans, roughly ten drink
water, use indoor plumbing, and have access to utility generated electricity,
all rather successful products. Its also true that nine out of ten Americans
don't have or use an iphone, however, its possible to make an enormous stack
of money out of stuff 90% of the population is uninterested in.

For other examples of the same phenomena of making fat stacks of cash off
practically no one, see pop music, television, professional sports, AAA video
games...

Its a failure to understand the market. Around 9 out of 10 comments about an
iphone should be somewhere on the spectrum of "eh" to "that sucks" because a
random sampling should indicate iphones are only about twice as cool as
Congress. People on /. complained about the ipod because they're normies, not
because they're nerds. Normies don't like that stuff, look at the actual sales
figures, therefore normie hangout like /. is not going to like the product.

Denigration and name calling of people who don't like a niche product is just
Apple fanboyism.

~~~
eric_h
> Its also true that nine out of ten Americans don't have or use an iphone

[1] suggests that one in four americans own an iPhone.

1) [http://fortune.com/2014/01/16/npd-better-
than-1-in-4-adult-a...](http://fortune.com/2014/01/16/npd-better-
than-1-in-4-adult-americans-now-own-an-iphone/)

------
allencoin
>Certain companies have succeeded despite a sceptical specialized audience.
Why is that? The first and most obvious reason is because there is simply not
enough diversity in the audience for the wisdom of the crowd theory to hold
true.

Right, this almost seems like common sense. I see the Hacker News community as
mostly entrepreneurially minded devs who are very invested in the Silicon
Valley startup ideal. Obviously there's nothing wrong with that, and there are
a lot of really smart people here. But geniuses in a particular field are
often inept in others. Hacker News readers don't represent the average
consumer, for instance, so I wouldn't put too much stock in their opinions
about a primarily average-consumer-facing product like Airbnb. I would,
however, put a lot of stock in their opinion about my new dev-facing app like
Heroku.

------
FreakyT
I'm glad to see that this article mentioned my all-time favorite HN naysayer
thread -- the one for Dropbox. Namely because it contains this gem:

 _> For a Linux user, you can already build such a system yourself quite
trivially by getting an FTP account, mounting it locally with curlftpfs, and
then using SVN or CVS on the mounted filesystem. From Windows or Mac, this FTP
account could be accessed through built-in software._

Yes, completely trivial. Every user will definitely have no trouble doing
that. (Also, accessing your files with an _FTP client_? Really?)

~~~
thomasahle
> then using SVN or CVS on the mounted filesystem.

I didn't know Dropbox had versioning control?

~~~
Robadob
I actually went to try and revert a file in Google Drive this morning and was
shocked to find it didn't have this option after using Dropbox for years.

~~~
mkobit
I think you can do this, at least on some of the Drive services. For Sheets,
go to File -> See Revision History and you should be able to view
changes/revert to a previous revision.

~~~
Robadob
This was a .tex document that I had been editing in a text editor, not a
Google doc. I tried context menus on the file in both Windows and the Google
drive website to no avail, could spot any other relevant buttons in the rest
of the Google drive websites interface either.

\----------

Just tried it again now, and found the option on the website as 'manage
versions' under the context menu of a file (no idea how I missed it the first
time). Still had to download the old version and replace the file manually
rather than an explicit revert button.

------
minimaxir
Relevant: if you're looking for a way to scrape all Hacker News submissions
and comments, I have a GitHub repository containing Python scripts showing how
to do just that: [https://github.com/minimaxir/get-all-hacker-news-
submissions...](https://github.com/minimaxir/get-all-hacker-news-submissions-
comments)

With respect to the article, I've done data analysis on Hacker News myself
(e.g. [http://minimaxir.com/2014/10/hn-comments-about-
comments/](http://minimaxir.com/2014/10/hn-comments-about-comments/) ) and I'm
a little skeptical. I was not a fan of using NTLK to classify
positive/negative because the models used for training sentiment are based on
reviews, not internet site comments, and when I had used NTLK, the results
were skewed significantly. I used an alternate model mentioned in the linked
post which IMO worked better.

The examples used in the article seem more like correlation-implying-causation
and cherry-picking. There are many, many counterexamples of startups people
loved on HN which died a painful death.

------
codingdave
Cherry picking posts about the successful companies from the glut of "Show HN"
posts that have come through over the years is practically the definition of
survivor bias.

Not that I am arguing with their larger point - if you believe in what you are
doing, do not let comments on HN stop you from pursuing your vision.

~~~
cwilkes
Exactly. While an interesting read in showing how people can be wrong about
later hits, this study is meaningless unless the failures are included. It is
like looking at all the winners of the superb owl, finding some negative
comments about them, and then saying "see, you shouldn't listen to anyone".
What about all the teams that didn't win and had good comments? Or bad?

------
danso
Obligatory rebuttal of statistical claim: does this analysis suffer from
survivor's bias? OK, so these are the few startups and projects that have had
reasonable-to-great amount of success...what about all of the startups that we
_can 't_ think of off the top of our heads [1], because they died? If the
number of those is much greater compared to the group OP selected, and they
also faced a majority-negative bunch of comments...why shouldn't we conclude
that those founders _should_ have obsessed over HN skepticism?

[1] I'll start: Fab
[https://news.ycombinator.com/item?id=2444387](https://news.ycombinator.com/item?id=2444387)

~~~
acqq
Moreover, it's easy to argue that counting the number of positive and negative
comments on any subject on any "social" platform can't be treated as more than
immensely biased sample of the sentiment of the users of the platform
momentarily interested in commenting the specific topic.

------
pkorzeniewski
What irritates me most with HN crowd is that there're several "sacred" topics
that you either agree with, or you better shut up because you're clearly wrong
and you'll be downvoted to hell. I understand that the voting system can
represent your agreement or disagreement with someone, but downvoting valuable
opinions just because they're controversial is immature.

~~~
tajen
I used to think feminism was very a sensitive topic, thus untouchable like you
describe. It appears not. Most times I've posted skeptical comments against
the _current form_ of feminism, I expected to be told off and maybe excluded
from HN. I have seen a workmate being fired for such a discussion. In reality,
my upvote balance was between 4 and 35 points. It doesn't mean I'm the one
who's right, it means it's possible to gain upvotes by echoing part of the
people's opinions.

That said, I still feel I'm earning a Godwin point for evoking this topic.

~~~
drzaiusapelord
Being critical of feminism on a forum full of young single guys isn't the big
contrarian display you seem to think it is. HN and reddit are the only places
I see this hyper critical attitude and its generally well accepted. Hell,
reddit is famous for its popular "red pill" forum.

~~~
jazzyk
The problem with controversial topics (not just this one) is that people who
disagree with you are much more likely to down-vote you than the crowd who
agrees with you to up-vote you.

------
carlosrg
>But Hacker News was wrong about Airbnb

Well, about these concerns about security, they weren't totally wrong.[1]

[1]: [http://www.thelocal.es/20150821/airbnb-to-beef-up-
security-a...](http://www.thelocal.es/20150821/airbnb-to-beef-up-security-
after-madrid-assault)

------
mbell
> Three companies, three different responses on Hacker News, and three
> ultimate outcomes. [speaking of Codecademy, Meteor and Homejoy]

The only one on this list with an 'ultimate outcome' is Homejoy. Codecademy
has no business model that I know of, and Meteor isn't even close to
profitable.

Quora is also used as a 'success' case and they also don't have a business
model.

I find this post does a pretty poor job providing any sort of supporting
evidence for it's claims.

------
bananaoomarang
The relentless cynicism of the Hackernews community is a large part of the
fun. There is usually a lot more thought put into the comments here than other
_cough_ Reddit _cough_ places on the ol' internets. A lot of the comments
which are criticised here for being later proven wrong are valid concerns.

The degree of 'magic' in Meteor, for instance, is something still discussed a
lot, despite it's obvious benefits and success.

~~~
mattmanser
Call me cynical by your example is bad, from experience a lot of us know that
excessive magic like meteor is a recipe for great pain later on.

(Boom, boom)

~~~
oldmanjay
That's pretty definitively cynicism. I don't get the objection.

------
mfoy_
There's an XKCD for this, actually:
[https://xkcd.com/1497/](https://xkcd.com/1497/)

~~~
vatotemking
HN and reddit in a nutshell

------
netcan
I've never read "The Wisdom of Crowds" so maybe I'm missing some stuff, but
from the general statement I think it would be a terrible mistake to take it
as meaning "commenters are right" or "consensus is right." It's much more
subtle than that.

Crowds, in certain structures that yield a useful dynamic, produce wisdom.
People transacting in a market allocate resources effectively. Gamblers and
bookmakers betting on football matches make good odds. Wiki-editors nitpicking
away in wiki-world produce an incredible encyclopedia. Open source, etc. There
are also lot of examples of crowds being stupid, and structures that yield a
stupid dynamic.

------
eevilspock
Wisdom of the crowd? The HN post that this article cites first had 14
comments. Not a crowd.

It categorizes the first comment on DropBox[1] as negative. What? That comment
points out a legitimate potential hurdle to user adoption, gives some
marketing advice, and then praises them for launching.

While the article's instinct may be correct, I don't see that it has shown
data that supports it.

-

[1]
[https://news.ycombinator.com/item?id=8863](https://news.ycombinator.com/item?id=8863).

------
ss6754
This is an interesting article but very one sided. To really assess "the
wisdom of HN community" the author should have assessed all four cells:
Successful companies with positive comments, Successful companies with
negative comments (which the author did), Unsuccessful companies with positive
comments and Unsuccessful companies with negative comments. This would provide
a broader and more objective picture of HN community

------
im3w1l
A positive comment is not a prediction that something will succeed, and a
negative comment is not a prediction that something will fail.

------
Rambunctious
Is there any data file to show what comment has been marked as positive or
negative?

For example, if you look at the Quora thread (
[https://news.ycombinator.com/item?id=1197146](https://news.ycombinator.com/item?id=1197146)
), a comment could say something good about the site and also provide some
feedback (e.g.
[https://news.ycombinator.com/item?id=1197230](https://news.ycombinator.com/item?id=1197230)).

Is this comment marked positive or negative? If feedback is somehow clubbed
with negative comments, I would doubt the accuracy of the analysis.

~~~
josefresco
I had this same thought - as many times I comment and then realize that it
could be "taken" several ways.

It also amuses me to no end that someone is collecting and analyzing data from
HN based (in part) on my sometimes idiotic comments and silly debates with
Internet strangers.

------
junto
I would have thought this was relatively obvious.

Many new ideas and applications are marketed on HN quite validly under Show
HN.

However, the first pass at marketing an application may not reflect its form
further down the line.

For example, we often see poor landing pages, or simply landing pages that
don't convey the problem succinctly enough to quickly satisfy our short
attention spans.

A few months or years down the line and these development teams have homed
their product, ironed out the bugs and built landing pages that are easy for
anyone to understand quickly.

Interesting none the less.

Hence, what is launched on HN != its final form and cannot determine future
success.

------
gwern
> The HN crowd, or any expert community, don’t have any greater a chance of
> predicting the success of a startup than a more general crowd.

And what 'more general crowd' are you so confidently comparing to, exactly?

------
speg
Is it telling that I come to the comments first on a story about comments?

~~~
mccracken
I even did it on this thread, then realised the irony immediately

------
ebbv
I agree with the premise of this post, but it extends beyond HN; don't listen
to the internet in general. We commenters don't know as much about whatever
you're doing as you do. Maybe you might get some good insight from a random
commenter on the web, but usually you won't.

However, their specific example is sort of wrong. AirBnB has indeed blown up,
but this has been due to two major factors:

\- Traditional landlords, rental properties and yes Bed and Breakfasts moving
onto Air BnB, embracing a new, popular platform.

\- Extremely high real estate markets in some areas making it attractive and
almost necessary to rent out room(s).

It's not like AirBnB has created a change where most or even a significant
portion of Americans as a whole rent out their rooms. The reality is that the
objections the commenters brought up are real, and most people aren't
interested in the hassle the comes with renting out a room, and most people's
rooms aren't necessarily attractive to rent.

In the end the original idea people had of how AirBnB would have to be
successful; convincing _most people_ to rent out rooms, an convincing
travelers to rent rooms in random people's houses, is what was wrong. They
only needed that to happen in certain markets, and for landlords in general to
realize the advantages they had by renting out their properties on AirBnB.

------
teaneedz
A lot of thought about measuring an audience that for the most part lives in a
bubble. In view of the fact that it's often the same audience which
continuously produces bug ridden UX conundrums and ad tech supported
mentalities locked in a tech bubble focus, I think that HN might be the wrong
place for this study.

I love HN, but one still needs to remember that the audience, although
influential, is quite different then a truly diverse mainstream crowd.

------
xixi77
I don't see how analyzing comments on ex-post known successful businesses can
tell us anything about anything -- it (at least) needs to be compared to how
comments on average startup proposals do, to see if there is any predictive
power. If, for example, successful startups had 20% positive comments vs. 10%
positive on average, and the difference was significant, that would suggest
that HN shows awesome crowd wisdom -- even if 80% of people here hate
everything :)

Accounting for confounding variables would be nice too (for example, if field
X is generally more liked by HN than field Y, equally positive feedback on a
startup doing X would tell us more than that on a startup doing Y).

Still an interesting idea, I just think that people should not forget that
"data science" is a subset of statistics, and any kind of data analysis is not
really useful without at least some familiarity with it.

------
rebootthesystem
This, in a strange way, echo's some of what I said on the thread about
Amazon's product review star system: In a crowd the people with the most
motivation to participate are those with negative views. This is a problem
with voting/rating/forum systems going back decades, HN is no exception.

~~~
jmnicolas
"In a crowd the people with the most motivation to participate are those with
negative views."

I'd had the ones that are paid to say good things about a product they
probably never used. If we're lucky they compensate each other ;-)

------
yeukhon
* Most startup will fail.

* Most of the successful startups will be acquired and merged with other more successful startups or established companies.

* Most ideas are repeated and competed aggressively.

* Most of the successful startups primary winning demographic / target audience turns out to be younger crowd, especially with college students, and the young crowd tend to be better at advertising startup ideas.

* Many successful startups you can name right away simply reinvent the existing platform by applying media attraction and sharing among crowds.

* Only way win is make noise. A lot of noise. Yeah. Back to pg's top recommendation, selling your product to users.

* Unless you have billions to burn on your own, don't try to be SpaceX or Telsa. You will never be them (or the same guy who founded both entities).

My own unscientific conclusions.

------
rayiner
What's the saying? They ridiculed Einstein? But they also ridiculed Bozo the
Clown?

~~~
TillE
Indeed. From Carl Sagan's wonderful _The Demon-Haunted World_ :

"They laughed at Columbus, they laughed at Fulton, they laughed at the Wright
brothers. But they also laughed at Bozo the Clown."

------
tlb
Here's a mechanism by which initial HN sentiment can be negatively correlated
with eventual success:

\- Great founders describe the MVP accurately, even self-deprecatingly: "Sleep
under my kitchen table." They mention the flaws, because they want the most
useful feedback.

\- Less-confident founders describe the gold-plated, fully-featured potential
future version of the product and don't mention the flaws.

At least on a superficial read, the great founder's description seems less
promising, but they usually end up with a better product.

I certainly see this dynamic at work in pitches I've been given.

------
giarc
I would love to see what they considered a "negative" comment. Did they clump
negative and neutral? Something seems off with the zenefits assessment.

Read the Zenefits post
([https://news.ycombinator.com/item?id=5242681](https://news.ycombinator.com/item?id=5242681)).
I don't see a single negative comment as implied by the second graph on the
venngage.com page (looks like 7 positive, 6 negative). Unless the negative
comments have been removed between venngage pulling the data and today.

------
smikhanov
The article says that HN failed to predict the success of AirBnb. While
factually correct, this says nothing about HN (or anyone in general). Nobody
was able to predict the success of AirBnb, even the savviest of investors.
AirBnb wasn't the only company of its kind, it wasn't run by the smartest
people out there, it wasn't launched at some particularly good time, or
anything similar.

They have just survived and succeeded. Why? Who knows.

But obsessing about HN comments is truly not worth it.

------
Spooky23
There's definitely heavy cynicism here. But, many of these points are real and
are risks that these startups need to factor in.

The AirBnB thread is a great example of this. The risks associated with having
strangers in your home is something that AirBnB has to market around and have
protections from. (ie. the stories about crazy parties that trash apartments)

------
cuchoi
I agree with the general analysis but the AirBnb example is not accurate.

Aside from survivor bias, people were just showing their concern with
security. And Airbnb success depends mainly on proving a feeling of security
for its users (that why they have ID and facebook verification, for example).

------
Mz
_The HN crowd, or any expert community, don’t have any greater a chance of
predicting the success of a startup than a more general crowd._

For the most part, the purpose of posting something to HN is not to take a
vote of that sort. Thus trying to view it through that lens is kind of silly.

------
hartator
If you believe that 99% of a success of a startup is about execution and 1% is
about the idea, sharp guesses about the ideas shouldn't matter that much to
gauge one's wisdom.

------
dmalvarado
That "wisdom of the crowd" thing only works when the crowd isn't influencing
each other. Otherwise - groupthink.

------
venomsnake
Survivor bias at its best. You must include all companies that have announced
themselves on HN.

------
dandare
Isn't the sample size too small? I invoke the law of small numbers.

------
Vendan
Utter crap on mobile. There's a side "share" panel that covers the left side
of the article with no apparent way to remove it.

~~~
noarchy
Not sure if this is meant to be a sarcastic reply that imitates the usual
negativity of HN, due to the subject of the article, or a legit reply.

~~~
Vendan
Quite legit really. I was unable to read the article on mobile, despite the
fact that it had a "responsive design", because of a sidebar covering the left
side of the article. I hate how the new standard for "mobile friendly" is to
start permanently using portions of the screen for pointless shit.

------
IIAOPSW
I think this article will do poorly.

~~~
Maken
I think this comment will do well.

~~~
mhurron
I think you shouldn't care so much about it.

------
lighthawk
tl;dr: HN presents a variety of opinion.

Also: misspelled neutral in one of the graphs.

Nice post, though. Keep it up.

------
copperx
Brace yourselves for the onslaught of negative comments. I can't wait for the
statistical analysis to be trashed.

~~~
pjc50
Well, they started out by picking successes. A more valid approach would be to
list _all_ startup ideas mentioned on HN and see how they turned out. That
includes every acquishutdown mentioned.

But that's vulnerable in the other direction. If 90% of startups fail, it's
very easy to predict failure.

An interesting non-statistical inquiry would be to look at the problems people
are complaining about and seeing whether they're still present in the
successful projects (e.g. dropbox privacy).

~~~
bachmeier
Homejoy is in the article but is no longer in business.[1] So I'm not sure if
the point is that there weren't enough negative comments on their launch.

[1] [http://blog.homejoy.com/homejoy-says-
goodbye/](http://blog.homejoy.com/homejoy-says-goodbye/)

