
You Should Downvote Contrarian Anecdotes - tylerhobbs
http://thobbs.github.com/blog/2012/06/17/you-should-downvote-anecdotes/
======
retrogradeorbit
In a world were 47 of 53 landmark cancer research papers were found to not be
replicable... where one scientist admitted (after the meta-researchers tried
50 times to replicate the result and failed) that they had run the experiment
6 times and got the result once and reported that because it made a better
"story"... where the act of suppressing refutations of other's bogus research
is "culturally commonplace"...

Something is rotten in the core of science. These days I give the research the
same weight I give the anecdote... and that is no weight at all.

Science has been supplanted by money and politics... At least anecdotes admit
they're anecdotes!

[http://news.yahoo.com/cancer-science-many-discoveries-
dont-h...](http://news.yahoo.com/cancer-science-many-discoveries-dont-
hold-174216262.html)

~~~
polyfractal
This is utter hogwash.

I'm as critical as anyone (probably more so, check my comment history) of
academic biology because of my background in it. There are certainly things
wrong it. And due to the nature of biology, replicating results _is really
hard_. It's a fact of life when you deal with systems that are not perfect,
not identical and very opaque.

But to say that "Science has been supplanted by money and politics" is
stretching the problems of biology into a mountain of conspiracy.

Furthermore, I'm reading your "source" and it reads loudly as "I'm an
underfunded big-pharma research who has neither the time nor the resources to
properly replicate studies". Did you know that most big pharma labs do not
have access to the academic literature? They mostly read abstracts because
there is little budget to actually purchase the required papers.

How much do you trust labs that are A) only trying to recreate data so they
can make a drug out of it and B) aren't even reading the original data? While
academic labs can have grad students toil away on hard experiemnts for
literally _years_ before they perfect them...how long do you think Pfizer or
Merck or Glaxco-Smith is going to let their paid researchers fiddle away on a
project that is probably low priority anyway?

Because, of course, the high-priority projects are the reformulations of
penis-enlarging drugs or cholesterol medication...you know, the ones that
actually make money.

If you are looking for snake oil and shady research, I dare you to read any
research paper that comes out of big pharma labs. We would routinely read them
just for laughs because they are (often) downright terrible.

~~~
refurb
While I agree that "Science has been supplanted by money and politics" is
going way overboard. I'm going to take the other side of the argument because
I think you are way off base. I worked in both an academic lab and a "big
pharma" lab(4 years and 6 years respectively).

To say "most big pharma labs" do not have access to the literature is
laughable. We had better access than most academic institutions. If we needed
a paper we didn't have access to, it took a few hours to get it. The company
was more than willing to pay the $50 to get a copy of whatever paper, since we
would often blow $50 running one experiment. Many of the smaller biotech might
have poor access to journals, but even then, if you could justify the cost,
you could get it.

Second of all, yes I trust labs that are trying to recreate data to make a
drug out of it. You have to remember that these attempts to recreate data were
a very important data point on a potential multi-million (billion?) dollar
investment in a new target, these are NOT low priority projects. They WANT the
data to be true. They have zero incentive for the data to not be reproducible.

Having worked in both academic and commercial labs, I would say the incentive
to "tweak" results in much great in academic labs for the following reasons:

1) Often results are never double checked in an academic lab unless the work
is use in a later project. Contrast this with a pharma lab where if the data
is positive, you'll have to prove it again and again. 2) Academics (both profs
and students) live and die by papers, not so in academic (in fact, in the
company I worked in, they preferred if you didn't publish) 3) Work in academic
is often performed by relatively inexperienced ungrad and grad students, while
big pharma scientists often have years of experience.

~~~
polyfractal
Fair points. My thoughts:

> _To say "most big pharma labs" do not have access to the literature is
> laughable. We had better access than most academic institutions. If we
> needed a paper we didn't have access to, it took a few hours to get it. The
> company was more than willing to pay the $50 to get a copy of whatever
> paper, since we would often blow $50 running one experiment._

I'll admit that my knowledge of big pharma journal access is colored by those
in big pharma that I've talked to (anecdotal evidence, oh the irony). Perhaps
they just had poor departments or bad access, I don't know.

However, every university that I've been at has _instant_ access to journals.
I never had to wait hours for a paper...we had free reign of just about every
journal. Even at my relatively small and poor undergraduate institute.

> _1) Often results are never double checked in an academic lab unless the
> work is use in a later project._

99% of projects in academia are building off some previous grad student or
post-doc's work. Sure, there are projects which are nearly impossible to
replicate (I should know, I spent 1.5 years of my life trying to replicate a
previous grad's project). But it's equally laughable to say that data is never
double-checked - professor's career is a long string of projects building on
previous projects.

> _2) Academics (both profs and students) live and die by papers, not so in
> [industry]_

I'll concede that there is often pressure to publish positive results in an
academic setting. However, as you rightly mentioned, academics live and _die_
by their papers. It just takes one lab refuting your paper to have a burned
career. While I agree that many academics prefer to just ignore papers they
can't recreate, there is still a lot riding on publishing replicable data.

> _3) Work in academic is often performed by relatively inexperienced ungrad
> and grad students, while big pharma scientists often have years of
> experience._

This is a pretty baseless statement? I know plenty of techs working at big
pharma that just graduated with an undergrad degree and have zero of wet-bench
experience (just like I know of plenty who did the same in academia).
Conversely, I can't even count the number of post-docs and senior scientists
that work at various universities, with literally centuries of experience
between them.

~~~
refurb
To address your points:

1\. The big pharma guys have instant access to journals. When I say we had to
wait a couple hours, it was because I was looking for a paper from "The
Russian Journal of Chemistry" from 1912. We had a vendor who could track down
anything. For any of the big journals, we had the same access as academia.

2\. We agree on this point. If a lab experiment is used in a later project, it
HAS to work or else the future work can't occur. However, lots of projects
have "arms", where the experiment is an interesting observation that is never
pursued. These are often "one-off" experiments that are published, but never
repeated in the same lab.

3\. I am by no means painting academics with a broad brush here. I think most
academic research is done on the up-and-up and the results are valid, if not
hard to replicate (this is research!). I think one issue is the one pointed
out in the parent comment. You run 5 reactions, two fail and the three that
work produce yields of 50%, 70% and 80%. What gets published? 80%. The devil
is in the details. In big pharma, you are trying to make a drug and the
science better work or else you can't bring it to market. Much higher
standards for reproducibility.

4\. I guess my thought here is based on the fact that big pharma typically
hires from academic labs. All those post-docs and senior scientists with years
of experience? That's who big pharma hires. So overall, I would imagine that
the level of experience in big pharma is greater than the average you would
see in academia (which makes sense since academia is training for working in
places like big pharma).

Once again, I always shy away from descriptions that put all "big pharma" or
"academic" researchers into one pile. There are brilliant people on both sides
and crappy people on both sides.

~~~
polyfractal
Ok, I'm with you on all your points. I suppose I over-reacted to the
grandparent post - it felt like useless sensationalism and conspiracy-
mongering.

Thanks for the useful counter-points...I'm now armed with some more anecdotes
(hah!) on the other end of the "big pharma" spectrum.

=)

------
Udo
No. Contrarian anecdotes are good. They may turn out to be without merit, but
then again so may the article itself. Having a real discussion is a good
thing.

Also, I would like to propose a logarithmic scale for weighting such things.
Say, if the article in question found something extraordinarily significant
with 100 out of 100 samples resulting in A, then it's still not rational to
weigh a contrarian viewpoint resulting in B with 1/101 - it should maybe be
closer to 1/3 or something.

Consensus culture and worship authority are not desirable in my opinion.
Arguments should be weighed on their merits and it's appropriate to explore
other viewpoints or explanations even if they turn out to be dead ends most of
the time.

~~~
loup-vaillant
Let's take an example, since they seem to work best: vaccines.

Vaccines protect you from the risk of contracting particular diseases, some of
which are crippling, lethal, or incurable. Plus, most are extremely effective:
once you take your shots, you are effectively immune. That's good.

There is a downside however. Sometimes, vaccines have side effects. Most side
effects are quite benign, but if you're unlucky enough, they can be crippling,
lethal, or incurable. That's bad.

From a medical point of view, vaccines are a net good (let's leave aside
logistic considerations, or the effort required to go to the doctor). When you
look at the stats, you stand a much better chance at life and health if you
take the shot. Even for relatively minor illnesses like the flu.

Now, let's say someone post a heartbreaking comment about how her 9 year old
daughter died of a vaccine shot, with all the gory details about the
suffering, how she couldn't participate in her school's festival, the size of
the coffin… I'm quite sure there _are_ stories of the kind. Given the sheer
amount of readers here, maybe one of you will more or less directly relate to
that. My apologies to those who do.

Nevertheless, what makes a good story doesn't necessarily make good evidence.
When you know of reliable statistics, and you read a contrarian anecdote, you
should shift your belief in the direction of the anecdote by a precise amount,
which is almost always tiny. What your brain will actually do behind you back
however, is shifting your belief by a _significant_ amount, often crossing the
"reasonable doubt" line. That's not rational, but that's what will happen.
Nameless statistics feel abstract, remote. An anecdote on the other hand feels
concrete, real, close. Worse, you can spend far more time reading about the
salient anecdotes than learning about the end results of reliable, but boring,
scientific studies.

Another example: you don't win the lottery. Period. You don't know of any
close family of friend that ever did. But maybe one of you readers do. Maybe
that one could comment and say "Hey, but my cousin _did_ win the lottery!".
Would that prove me wrong? Not at all. It's just that when the sample size is
huge enough, even the tiniest chance can actualize.

~~~
retrogradeorbit
Are you actually arguing against giving anecdotes weight by using...
anecdotes?

~~~
archangel_one
Those are examples, not anecdotes.

~~~
loup-vaillant
Yes, but I did make use of strong emotional story-like elements to make my
point.

------
lhnz
The simplification in the title is dangerous without a 'when' attached.
Mindlessly downvoting contrarian anecdotes will lead to a hive-mind. Beware.

Opinions overwhelm all other forms of material in a discussion. Anecdotes are
actually one up from opinions as they are concrete. You should weight them
like so:

Statistical Evidence + Logic > Statistical Evidence > "Common
Knowledge/Wisdom" > Anecdote + Logic > Anecdote > Opinion + Logic > Opinion

------
pseudonym
In my experience, some of the best advice I've ever received has been from
anecdotes that run counter to the text of an article.

...I kid. It's actually an interesting idea. I don't know about scrapping them
entirely, but I think a lot of sites (say, Reddit) could benefit from moving
the anecdotes elsewhere. Partially because of this, and partially because
anecdotes in any form tend to derail the discussion pretty darn quick.

~~~
necubi
This is what makes r/AskScience so good. "Free of anecdotes" is a rule for
posts, and it is vigorously enforced.

------
grandalf
The idea that there was an impending credit crisis was a "contrarian anecdote"
in 2008.

The idea that the US Government conducted illegal eavesdropping/wiretapping
operations against US Citizens was a "contrarian anecdote" in 2003.

This blog post condescendingly claims that HN readers are not sophisticated
enough to balance out the sources that are input to their rational decision
making process.

In reality, contrarian opinions sometimes turn out to be correct, and
mainstream opinions sometimes turn out to be wrong. Often, the benefit of a
contrarian opinion is that it causes people to ask more questions, which is
rarely a bad thing.

People are sheeplike enough without having to be encouraged to follow the
herd!

I suggest everyone find a few contrarian theories and imagine what it would be
like if you rearranged your life as if you expected them to be totally true.
Most people are unwilling to go that far, and yes, in that way contrarian
stories can weaken rational processes.

~~~
ubernostrum
_The idea that there was an impending credit crisis was a "contrarian
anecdote" in 2008._

As the saying goes: they laughed at Galileo. They laughed at Einstein.

And they laughed at Bozo the Clown.

------
beagle3
You shouldn't downvote contrarian anecdotes. You should just take them at face
value for what they are.

His notion that in the field of medicine you should disregard contrarian
anecdotes because there's statistical evidence is horrible advice. If you
actually looked at said statistical evidence, you'd realize it is very rarely
strong, and often only relevant to 70% or so of the population (which is great
for 2 out of 3, but useless for the third one).

"Best practices" often aren't, and common sense is not at all common in
medicine. A significant number of published results are plain wrong (see
<http://saveyourself.ca/articles/ioannidis.php> and the paper it references).
A lot of medical advice is wrong, harmful or useless; The archives of seth
roberts' blog are an enlightening read.

~~~
iy56
>You should just take them at face value for what they are.

The problem is that this is impossible for most people.

~~~
beagle3
That's true, and they also get to vote (both on HN, and in real world
elections). Imagine the consequences! Or just observe them in the real
world....

------
drostie
If the comments below are too laborious for you to read, here is the TL;DR of
them:

(1) Use your personal discretion. Some anecdotes are funny. Others can be
perfectly cogent rebuttals, especially when people make overbroad
statements.[a]

(2) The author is right in some respects: you need to be aware of the ways
that stupid stories can bias you. Even being aware of this fact is sometimes
not sufficient, so use downvoting to try to protect others.

(3) Lots of people believe in anecdotal responses to anecdotal original-
claims. [b]

(4) A lot of people tried to be funny by offering anecdotes to be downvoted.
Unfortunately, people haven't consistently taken the advice above, so they are
not all in one place (at the bottom). That is a pity -- this would have been a
fun and interesting use of downvotes.

[a] Actually, the discussion is full of overbroad statements of this form like
"no universal truth" and someone presenting "nothing should be above
questioning" as above question.

[b] I would like to formally respond that this is generally stupid -- you
don't clean up a house by flinging crap at the crap. Your mileage may vary.

------
DanielBMarkham
I had to vote this up because of the meta-nature of the article. If anybody
makes a comment disagreeing with it while sharing an anecdote they only play
into's the author's theme! It's like accusing somebody of being in denial:
there's nothing they can say in return that doesn't somehow support the idea
that they are in denial. Love it.

Having said that, this is another really bad article in what seems like an
endless series of bad articles on HN. Here are a few of the more obvious
flaws:

\- There is no universal truth as the author seems to imply. Simply because
some publication or source you may like has performed some sort of statistical
study doesn't have a lot of meaning on it's own. Yes, 100% of the people who
eat bananas are dead within 120 years of their consumption. No, that does not
mean bananas are bad. The study or scientific reporting is simply the
_beginning_ of a much longer conversation society has over many decades that
leads us to higher-fidelity models.

\- The purpose of a social site is to behave socially. While places like HN
have (or used to have) a lot of different guidelines for the types of
behaviors that are encouraged or not, being social means sharing stories,
anecdotes. We are not robots.

\- The idea that people are unable to sort out personal anecdotes from other
forms of information. The follow-up idea that since they are not able to do
this, we should prevent ourselves from sharing such stories. This is bad, bad,
bad, bad, bad, bad .... bad(n). We are humans. We share stories. Anybody who
says "people are so stupid" can justify just about damn near anything as long
as they keep emphasizing the stupidity and danger some people's actions
represent.

Every now and then, gasp(!), published research is either wrong or doesn't
show anything near what the reporter claims. Anecdotes don't help with this,
of course, but they serve to remind us that even well-known scientists working
at the highest quality standards available are still just sharing with us a
very specialized form on anecdote. We did these things in this way and this is
what we observed. Here is how you also can observe this. The really "good"
part of the story they are sharing is talking about hidden assumptions,
population variance, reproducibility, and so forth. Anecdotes don't do this,
but they help us brainstorm ways in which we can improve the discussion, take
the next experiment to an even better place.

I'm very uncomfortable with the line of reasoning that goes somewhat like
this: people are broken in some way, therefore we must somehow control what
they read, say, or think for their own good. To me the beauty of western
civilization is that really broken people can do these amazing and awesome
things. The fact that we're deeply flawed _is_ the magic. Science and human
advancement work _because_ of our flaws, not in spite of them. This is a very
important thing to understand! Setting up some ideal of perfection, no matter
how well-intended, and then mucking around with the way societal interaction
works in some effort to improve on things is heading down a very dark path
that has a very unhappy ending. This attitude seems rife in the technology
community, however, perhaps because we are such analytical people.

I don't want my fellow man to be irrational and distrustful of science and
knowledge. But I'll take that any day over silencing contrarian articles and
dissent. We've done the math on this: wrong people who share emotional stories
and persuade crowds about all sorts of illogical things are a price that a
dynamic community pays for progress.

~~~
dmauro
The author was referring to contrarian anecdotes in reply to statistical data,
not someone's opinion.

\- He is only arguing that there is universal truth in proxy by arguing that
the scientific method is more valuable than anecdotal evidence. If you don't
agree with that, you probably disagree with most of HN (pure conjecture, does
anyone else here think anecdotal evidence is more valuable than statistical
analysis?)

\- I agree with you on this point. I don't think we should downvote comments
because a fun discussion is what the comments are for, not to try to prove or
disprove a study.

\- Statistical analysis is not a specialized form of anecdote. That's a
stretch.

~~~
beagle3
> He is only arguing that there is universal truth in proxy by arguing that
> the scientific method is more valuable than anecdotal evidence.

The strawman here is in equating (published) statistical analysis with the
scientific method. Of course the scientific method is more valuable, but
that's not necessarily relevant.

Please have a look at <http://xkcd.com/882/> if you haven't already - what
this comic describes is a very valid statistical analysis, according to the
"scientific method", (only neglecting base rates like 99% of published papers
do).

This is (unfortunately) very commonly practiced in the life sciences,
including medicine -- sometimes knowingly but mostly unknowingly. Bad
reporting not required for a horrible, long lasting effect on the future.

~~~
Symmetry
Scientists' use of statistics is often problematic, but that doesn't mean that
its valid to counter statistics with anecdote unless you have some larger
argument.

~~~
beagle3
Right. But it is also not valid to bring up statistics unless you can properly
qualify their relevance, which is almost never the case. This requirement sets
much higher bar for anecdotes than published statistics, when the latter
rarely deserves that high bar.

As a result, most arguments about science are invalid from a scientific-method
point of view. But the claims brought up -- including anecdotes -- are often
interesting and informative.

------
brudgers
By this logic, HN'ers should downvote any anecdote describing successfully
exiting a startup because statistics show that the vast majority of startups
fail and these anecdotes describing success unduly influence the thinking of
entrepreneurs.

~~~
Symmetry
Only if its offered as a counterexample to the proposition that most startups
fail.

~~~
brudgers
One might suggest that that such a proposition is an underlying premise of HN.

------
hasenj
You should question everything. This includes studies published and labeled
"scientific".

Contrarian anecdotes are important (but they too should be questioned).

Nothing should be above questioning; even prevailing wisdom.

~~~
flyinRyan
>even prevailing wisdom.

Especially prevailing wisdom, since it is least likely to be questioned
normally.

------
pgsandstrom
Are people joking when they defend anecdotal stories with "once this anecdotal
story proved to be true and the study was faulty"?

The proposition that the article makes is not that anecdotal stories must be
false, but that they might influence the reader more than they should. Thus
they should not be encouraged.

~~~
JoeAltmaier
Yeah, thanks for deciding how gullible I am, and please do sanitize what I get
to read for my own good. That's how open dialog works, right?

~~~
pgsandstrom
What do you think the voting system on HN does? It promotes good content, thus
also demotes bad content. The author is saying "Here is a reason that content
X is bad. If you agree with me, downvote it".

~~~
JoeAltmaier
Doesn't make sense. If the content is 'bad', moderator could expunge it. Or
writer could be counseled on giving better posts. Or any number of things.

The voting system on HN sometimes demotes bad content. Far more often
(discussed widely elsewhere) its used to assert agreement or disagreement.
When used in this sense, its a sort of instant-poll.

In the case of anecdotal argument, it forms an ad-hoc 'scientific' experiment.
The HN population weighs in using their own experience. Those who's memory (or
ok, memory-triggered emotion response) aligns with the author may upvote etc.

To 'bad content' is uniformely suppressed instead, this social experiment is
lost, and the community loses. The merit of instant-polls is debatable, but
other such polls are supported here. In fact the test group on HN may exceed
the original 'scientific paper' group by an order of magnitude. Its
statistical significance may exceed a graduate student's narrow study.

~~~
carbocation
> Its statistical significance may exceed a graduate student's narrow study.

And thanks to selection bias, its external validity is likely zero. "Our HN
poll found that 1 in every 10 people has founded and sold a company."

------
verroq
Rebuttals to articles (ones that are voted up anyway) in the comment section
are almost always well reasoned refutations. This article is addressing a non-
problem.

------
dclowd9901
Contrarian anecdotes are an ill-conceived method of contributing to a
conversation, and doing so in some sort of attention-getting way. They appear
to all but undermine the entire premise of the original content, and yet, are
based on absolutely nothing but one's individual experience with the matter.

I see this kind of stuff all the time on Reddit, and it's become so numbing
that you know, no matter how well you construct an argument or how many facts
you cite, someone will always come out of the wood works and tell you how
you're wrong because it didn't happen that way for them. It wouldn't be so bad
if people didn't take that as a genuine counter-argument.

------
Goladus
Contrarian anecdotes can be very useful (and logically sound) when they are
contrary to a generalization. If someone asserts "f(x) = 5 for all x in A",
"f(a) = 3 and a in A" is a logical contradiction proving the assertion false.

And, frankly, if we're talking about public discussion of scientific papers,
inappropriate generalization is as big a problem as contrarian anecdotes, if
not bigger. Scientific papers often cover highly specific observations that
are useful primarily to other researchers, and often even then not for many
years. People then try to apply that specific knowledge to practical day to
day situations.

~~~
Symmetry
Have you ever seen that sort of generalization in an article on Hacker News?
Even when you have situations like, e.g. nobody has ever recorded a case of a
person with a certain genotype being infected by a norovirus, you don't see
papers claiming that its impossible.

~~~
Goladus
Yes. Realize that often the generalizations are a lot more subtle than your
example, and of course anecdotes aren't always the best or only counter-
arguments you'll see, even if they are valid. Start looking and I am sure you
will find examples.

------
CookWithMe
I'm all in when it comes to call bullshit on something. But because we
(hackers) are used to boolean decisions, we are looking for counter-examples.

But these are just non-arguments when the finding is "In 9 out of 10 cases,
...". The anecdote is the 1 out of 10 case.

Let's try to question anything, and when it is suspicious - by all means, say
so! But, I guess what the author is trying to say (beyond the whole Downvote
discussion) is that we should try to reason and make a proper argument.
Especially one that is not already addressed in the research.

~~~
TeMPOraL
> But these are just non-arguments when the finding is "In 9 out of 10 cases,
> ...". The anecdote is the 1 out of 10 case.

The whole point of the article is that people won't judge that anecdote as
only 0.1 relevant, but much much more (I'd say close to 0.9 if they personally
prefer what anecdote says to what research says). It's a general human flaw
that's very visible in day-to-day interactions with people.

------
keithpeter
"But in the presence of statistical evidence, don’t tolerate contrarian
anecdotes, and don’t make them yourself, knowing the exaggerated impact they
can have."

When you start to design a statistical experiment, you have already made an
important methodological choice. See

<http://en.wikipedia.org/wiki/Positivism>

<http://en.wikipedia.org/wiki/Grounded_Theory>

I've noticed that in my own field which is education, there appears to be a
fondness for sophisticated statistics, even though no manager ever allocated
students to teachers on a double blind random basis. An excellent example is
the way the UK Education ministry has decided that 'phonics' is _the_ way to
teach reading.

<http://www.bbc.co.uk/news/education-18462214>

I think that this general tendency might be an example the 'white coat
syndrome' in action; belief that using formal statistical techniques might
increase the meaningfulness of the results. I suppose that is a form of cargo
cult.

This is hacker news, a forum aimed at people with novel business proposals and
new software to try out. Should you be trying to find 'the Truth' or should
you be building some grounded theory that tells you what to do next,
provisionally, now, today?

------
Spooky23
I guess it depends on the story. If HN is supposed to be about startup culture
and technology, the anecdote is essential to being vibrant. What is a startup
if not a story whose premise is unproven?

Now if you're telling me that your sister stuck a magnet in her ear and cured
her cancer, I'm not going to give that credence without some real data.

Most things on HN are not that, and a good story often compels us to think. So
FWIW, I'm not downvoting contrarians, and I'm not downvoting anecdotes for
being anecdotes.

------
D_Alex
I totally get where the author is coming from, but I am not _reeeally_ sure if
that is good advice. Anecdotal evidence has its place, eg: proof by
counterexample, challenging false, but widely held beliefs, or alerting people
to new circumstances.

A: "Murrumbidgee River is fun and safe for children to swim in! No one came to
any harm there for the last 100 years!" B: "Dunno... my dog was eaten by a
crocodile there last Saturday..."

B's dog is a sample of one, but maybe worth paying attention to.

~~~
jessedhillon
In your example, A is not relating a statistical or experimental finding, but
instead dispensing commonsense knowledge which happens to be objectively
false. It's not what the OP is talking about.

~~~
JoeAltmaier
Commonsense, like ulcers are caused by diet and anxiety, or that salt raises
blood pressure?

I think a good point is made: statistical evidence is also misleading - it
deliberately ignores (averages out) the extreme cases. The results are a
distribution; statistics folds that into one number. Anecdotes fill out the
distribution.

~~~
carbocation
Ulcers are not generally caused by diet and anxiety, but rather by bacterial
infection. Anecdote is a much more compelling refutation of "common sense"
than it is of empirical statistical evidence.

Statistical evidence is not misleading; it's simply the case that if you are
seeking outliers, as in your example, then looking at _measures of central
tendency_ won't contain what you're looking for. Anecdote has no role in
"fill[ing] out the distribution."

------
rlpb
A contrarian anecdote may help expose an article being false or misleading in
itself. For example: a review written by the manufacturer itself, or
astroturfing.

First we must determine the validity of the article existing in itself and any
motivations or biases in it. For this, contrarian anecdotes are useful. This
article seems to miss this point.

Once we can accept the article at face value, then we may hit the biases
described.

~~~
scalable
I would say it is far easier to astroturf a contrarian anecdote.

------
maigret
Or maybe just not upvote? Good argumentation, still can get in the way of why
HN is here. It's not just a place to make formal argumentation, but to
exchange experiences, or get some opinions from others. In some case the
opinion is wished from experts (how can I get funded etc), in others it is
about how a product is liked by general users. So please take a piece of salt
in applying that ;)

------
JohnHaugeland
This is an extremely toxic position, which will lead to novices (who outnumber
the experienced significantly) mathematically abusing voting systems to
promote whatever this month's fad tool golden hammer is.

The herd is rarely correct.

------
b1daly
Well I think Stalin summed up the perils of totally ignoring anecdotes pretty
well:

"One death is a tragedy; one million is a statistic."

This topic sort of came up the other day in the thread about the girl losing
the iPad software she needs to talk (Silencing Maya).
<http://news.ycombinator.com/item?id=4103344>

Some commenters thought the story should be ignored as a data point about the
societal value of patents. I disagree in that case because I don't believe
economics or social sciences have anywhere near the amount of rigor and theory
to make a good claim on whether the patent system is a net benefit to society.
Deciding such a thing is a very old, and classic problem in philosophy. The OP
seems to implicitly relying on a variant of utilitarianism, which IMO is
wholefully inadequate to rely on for moral decisions.

Since science is so hard, there is a lot of bad science out there. What gets
reported in the wider media has all sorts of weird selection bias, never mind
what gets picked for publication in journals. Anecdotes are human stories and
they are deeply connected to why we care. Statiscal tools can be used in ways
that justify harm people in the name of greater good. I agree that scientific
medicine provides tools for "real"'medicine that other methods don't. I just
think we should remember to real peoples stories, especially if they are true.
Anyone reading this probably gets that one can't exactly counter balance the
effect of anecdotes within themselves. But it can still be countered some, I
think it's worth it to hear stories (that are relaxant to the topic). Now what
is the basis of my beliefs? Mostly intuition, not rationality. But I don't
think it's possible to undermine my ideas with some kind of experimentally
bases argument. There's just way to many variables!

~~~
slurgfest
I don't understand the part about Stalin's advice. What was Stalin saying
would happen if you ignored anecdotes? How do you figure he was talking about
the value of anecdotes?

~~~
b1daly
He was talking about undertaking large activities (war for example) that would
kill hundreds of thousands of people could be tolerated politically by a
populace in a way that a single murder might cause an outrage. The large
numbers make the horror abstract. It's easier for people to relate to a story
about a specific individual.

The quote (and variations) are quite famous, it might be apocryphal, but I
think it is true. Hearing about hundreds of thousands of people massacred in a
foreign land doesn't hit home when you are reading about far away.

That's why reporters (New York Times style)try to weave in illustrations and
stories about individuals even when discussing a larger trend.

------
TimPC
As a caveat, you should only do this in response to data based articles.
Articles that are themselves anecdotal, should receive anecdotes that are
contrarian. I think the more general idea is comments should be held to
similar standards of evidence that the article presents.

------
the_bear
If a post is about a topic that can and has been thoroughly researched and a
reasonable conclusion has been drawn, then sure, a contrarian anecdote should
probably be disregarded. But honestly, how often does that apply?

Most of the conversations on HackerNews aren't about things that have right
and wrong answers. As a matter of fact, many of the most popular posts are
nothing more than anecdotes themselves. So why exactly should an anecdote in
the comments be downvoted?

------
nessus42
As someone who has had his ability to work saved due to anecdotal evidence, I
could not disagree more with the OP. Well, sure, if someone has anecdotal
evidence that the theory of gravity is wrong, or that 1 + 1 = 4, by all means
downvote them, but not all scientific evidence is on such solid ground.

E.g., I've been prescribed medication and ended up with a side effect not
listed on the drug literature. The doctor told me that the side-effect must be
psychosomatic. Later scientific evidence revealed that about 50% of patients
given the drug experienced the same side effect, but that it had been
previously underreported. Who's to say that researchers would have even
bothered to research the side effects more thoroughly if they hadn't paid
attention to the fact that the anecdotal evidence contradicted the scientific
evidence.

Or remember when the scientific evidence seemed to indicate that a high
carbohydrate, low fat diet was the healthiest choice? Should I have ignored my
anecdotal evidence that that diet made me feel like crap?

Science sometimes gets itself into harmful orthodoxies. See Thomas Kuhn for
more info on this if you are unfamiliar. One example of this is Behaviorism in
Psychology. In this field, it was scientific orthodoxy for many decades that
Behaviorism was scientific and Cognitive Psychology was not because
Behaviorism was based on only quantifiable, measurable data. It took Chomsky
to point out the idiocy of this orthodoxy, thereby breaking the orthodoxy,
allowing science to progress.

Re the anecdotal evidence that saved my career, I've read here that there is
no scientific evidence that ergonomic keyboards can help prevent or ameleroate
RSI. I am 100% sure, however, that the Kinesis Contour keyboard saved my
ability to type. All I have to do to know this for sure is listen to my body.
Furthermore, I personally know about a dozen programmers who feel similiarly.
I'm sure that someone will pipe in that this is almost certainly the placebo
effect. If that's the case, then Kinesis makes the world's very best placebo,
as placebo-like things have done precious little for me in any other area. In
any case, even if it were the placebo effect, what does it hurt anyone to
ignore the putative scientific evidence and try out a Kinesis keyboard for
themselves to see if it provides them with relief?

The idea that posts should be downvoted for recommending such ergonomic
keyboards is insane.

------
dennisgorelik
If best contrarian comments are weak - I assume that original article is
probably correct.

If I don't see any contrarian comments, then I suspect groupthink in
discussion.

------
lukev
It's true, anecdotes are not data and have no statistical weight whatsoever.

However, anecdotes can and do (and should) play a large role in influencing
how we think. They can humanize a problem and create food for thought in a way
that no amount of statistics ever could.

For example, it's one thing to cite statistics that children of gay parents do
just as well as those of straight parents (I have no idea if it's true, but
it's an interesting contemporary question). But that's not likely to change
the mind of a homophobe on the issue of gay adoption. On the flip side, a
lucid, heartfelt anecdote from a person who had gay parents might actually
help someone to _understand_ what it's like to grow up in that environment an
therefore become sympathetic.

Of course, it has no bearing on the actual statistics at all, but sometimes
statistics aren't the most important thing.

~~~
tylerhobbs
That is an excellent point, and well expressed. I do see how anecdotes and
stories can be used in a very positive way to shift opinion; there are
definitely some grey regions. (How do you know you're on the _right_ sight?)

------
alan_cx
Please forgive me if I have fundamentally misunderstood this, but is this
article suggesting that anecdotes are some how comparable to scientific
research data?

If one wants to "attack" an anecdote, then a contrarian anecdote is the
weapon.

If one wants to attack scientific data, you need contrarian scientific data.

At least, I hope that is right. There for, to mix the two is like attacking a
tank with a wooden stick.

Surely, anecdotes are used as the premise of scientific research. Lots of
people tell stories. There seems to be something interesting going on. Then
you do the research and produce the data. If the data is conclusive, then your
past the anecdote. If later contrarian anecdotes appear, and they seem
significant, off you go to scientific research again.

I know Im wrong somewhere. But where?

------
liquidcool
I've noticed that many of my biases, typically for or against some name brand
or technology, even if not anecdotal, are _old_. I now try to remember not
only how I came to this opinion, but _when_. I guess that's one of the
consequences of aging.

~~~
philwelch
My dad once received an inappropriate form letter from Texaco in the 1970's
and never went to a Texaco station since.

------
Shoomz
I'm curious what some of your (the community's) thoughts are on articles like
this. While the article itself has a large amount of personal bias with
unsubstantiated claims, the conversation it has broached has been rather rich
and enlightening. However, that is largely due to its controversial and
contrarian (ironically) nature. That said, while the conversation has been
thoughtful and looks to have largely refuted article, because so much of the
thoughts put up here in the comments have been to downplay the original
article, my question is: "Was this article worth our time or just an elaborate
waste of time since the premise was bunk?"

------
nathan_long
I downvoted some contrarian anecdotes once, and it ruined the thread.

~~~
hexagonal
I downvoted some contrarian anecdotes once, and my "downvote" button was
disabled.

------
sehugg
Anecdotes have their place, especially in complex domains. For example, I have
a friend who uses medical marijuana successfully to control MS symptoms. That
is a sample size of one, but it certainly colors my view of the debate. I have
a much deeper view into the decision making process of a MS sufferer now than
I would otherwise, one that goes beyond spasticity, pain, and cognitive
scores.

------
tokenadult
This is an interesting first submission by a Hacker News participant who
joined the community 551 days ago. The core idea in the submitted blog post
(by the submitter here) is

"Contrarian anecdotes like these are particularly common

<http://news.ycombinator.com/item?id=4076643>

<http://news.ycombinator.com/item?id=4076066>

in medical discussions, even in fairly rational communities like HN. I find
this particularly insidious (though the commenters mean no harm), because it
can ultimately sway readers from taking advantage of statistically backed
evidence for or against medical cures. Most topics aren’t as serious as
medicine, but the type of harm done is the same, only on a lesser scale."

The basic problem, as the interesting comments here illustrate, is that human
thinking has biases that ratchet discussions in certain directions even if
disagreement and debate is vigorous. The general issue of human cognitive
biases was well discussed in Keith R. Stanovich's book What Intelligence Tests
Miss: The Psychology of Rational Thought.

[http://yalepress.yale.edu/yupbooks/book.asp?isbn=97803001646...](http://yalepress.yale.edu/yupbooks/book.asp?isbn=9780300164626)

[http://www.amazon.com/What-Intelligence-Tests-Miss-
Psycholog...](http://www.amazon.com/What-Intelligence-Tests-Miss-
Psychology/dp/0300164629/)

The author is an experienced cognitive science researcher and author of a
previous book How to Think Straight about Psychology. He writes about aspects
of human cognition that are not tapped by IQ tests. He is part of the
mainstream of psychology in feeling comfortable with calling what is estimated
by IQ tests "intelligence," but he disagrees that there are no other important
aspects of human cognition. Rather, Stanovich says, there are many aspects of
human cognition that can be summed up as "rationality" that explain why high-
IQ people (he would say "intelligent people") do stupid things. Stanovich
names a new concept, "dysrationalia," and explores the boundaries of that
concept at the beginning of his book. His shows a welcome convergence in the
point of view of the best writers on IQ testing, as James R. Flynn's recent
book What Is Intelligence? supports these conclusions from a different
direction with different evidence.

Stanovich develops a theoretical framework, based on the latest cognitive
science, and illustrated by diagrams in his book, of the autonomous mind
(rapid problem-solving modules with simple procedures evolutionarily developed
or developed by practice), the algorithmic mind (roughly what IQ tests probe,
characterized by fluid intelligence), and the reflective mind (habits of
thinking and tools for rational cognition). He uses this framework to show how
cognition tapped by IQ tests ("intelligence") interacts with various cognitive
errors to produce dysrationalia. He describes several kinds of dysrationalia
in detailed chapters in his book, referring to cases of human thinkers
performing as cognitive misers, which is the default for all human beings, and
posing many interesting problems that have been used in research to
demonstrate cognitive errors.

For many kinds of errors in cognition, as Stanovich points out with multiple
citations to peer-reviewed published research, the performance of high-IQ
individuals is no better at all than the performance of low-IQ individuals.
The default behavior of being a cognitive miser applies to everyone, as it is
strongly selected for by evolution. In some cases, an experimenter can prompt
a test subject on effective strategies to minimize cognitive errors, and in
some of those cases prompted high-IQ individuals perform better than control
groups. Stanovich concludes with dismay in a sentence he writes in bold print:
"Intelligent people perform better only when you tell them what to do!"

Stanovich gives you the reader the chance to put your own cognition to the
test. Many famous cognitive tests that have been presented to thousands of
subjects in dozens of studies are included in the book. Read along, and try
those cognitive tests on yourself. Stanovich comments that if the many
cognitive tasks found in cognitive research were included in the item content
of IQ tests, we would change the rank-ordering of many test-takers, and some
persons now called intelligent would be called average, while some other
people who are now called average would be called highly intelligent.

Stanovich then goes on to discuss the term "mindware" coined by David Perkins
and illustrates two kinds of "mindware" problems. Some--most--people have
little knowledge of correct reasoning processes, which Stanovich calls having
"mindware gaps," and thus make many errors of reasoning. And most people have
quite a lot of "contaminated mindware," ideas and beliefs that lead to
repeated irrational behavior. High IQ does nothing to protect thinkers from
contaminated mindware. Indeed, some forms of contaminated mindware appeal to
high-IQ individuals by the complicated structure of the false belief system.
He includes information about a survey of a high-IQ society that found
widespread belief in false concepts from pseudoscience among the society
members.

Near the end of the book, Stanovich revises his diagram of a cognitive model
of the relationship between intelligence and rationality, and mentions the
problem of serial associative cognition with focal bias, a form of thinking
that requires fluid intelligence but that nonetheless is irrational. So there
are some errors of cognition that are not helped at all by higher IQ.

In his last chapter, Stanovich raises the question of how different college
admission procedures might be if they explicitly favored rationality, rather
than IQ proxies such as high SAT scores, and lists some of social costs of
widespread irrationality. He mentions some aspects of sound cognition that are
learnable, and I encouraged my teenage son to read that section. He also makes
the intriguing observation, "It is an interesting open question, for example,
whether race and social class differences on measures of rationality would be
found to be as large as those displayed on intelligence tests."

Applying these concepts to my observation of Hacker News discussions after
1309 days since joining the community, I notice that indeed most Hacker News
participants (I don't claim to be an exception) enter into discussions
supposing that their own comments are rational and based on sound evidence and
logic. Discussions of medical treatment issues, the main concern of the
submitted blog post, are highly emotional (many of us know of sad examples of
close relatives who have suffered from long illnesses or who have died young
despite heroic treatment) and thus personal anecdotes have strong saliency in
such discussions. The process of rationally evaluating medical treatments is
the subject on entire group blogs with daily posts

[http://www.sciencebasedmedicine.org/index.php/about-
science-...](http://www.sciencebasedmedicine.org/index.php/about-science-
based-medicine/)

and has huge implications for public policy. Not only is safe and effective
medical treatment and prevention a matter of life and death, it is a matter of
hundreds of billions of dollars of personal and tax-subsidized spending around
the world, so it is important to get right.

Blog post author and submitter here tylerhobbs suggests disregarding an
individual contrary anecdote, or a group of contrary anecdotes, as a response
to a general statement about effective treatment or risk reduction established
by a scientifically valid

<http://norvig.com/experiment-design.html>

study. With that suggestion I must agree. Even medical practitioners
themselves do have difficulty sticking to the evidence,

[http://www.sciencebasedmedicine.org/index.php/how-do-you-
fee...](http://www.sciencebasedmedicine.org/index.php/how-do-you-feel-about-
evidence-based-medicine/)

and it doesn't advance the discussion here to bring up a few heart-wrenching
personal stories if the weight of the evidence is contrary to the cognitive
miser's easy conclusion from such a story.

That said, I see that the submitter here has developed an empirical
understanding of what gets us going in a Hacker News discussion. Making a
definite statement about what ought to be downvoted works much better in
gaining comments and karma than asking an open-ended question about what
should be upvoted, and I'm still curious about what kinds of comments most
deserve to be upvoted. I'd like to learn from other people's advice on that
issue how to promote more rational thinking here and how all of us can learn
from one another about evaluating evidence for controversial claims.

------
vannevar
You shouldn't downvote to censor relevant information, period. You should only
downvote if the material violates site guidelines. Downvoting to disagree is
lazy, substituting easy censorship for thoughtful response. Instead, post your
counter-argument or upvote an existing opinion that already expresses your
position.

~~~
Symmetry
Censoring relevant information is, of course, never good but the authors
contention was that the information isn't actually relevant. Its always good
that people who are wrong get a cogent reply, but beyond that adding a "me
too" post doesn't have any more benefit than a downvote does. I used to just
upvote post I agreed with, but since those are no longer public the only way
to clue readers in to what the consensus is is to downvote.

~~~
vannevar
I'm not sure I'd agree he was arguing for the irrelevance of anecdotal
evidence, only that it has a disproportionate impact when considered with
statistical evidence. Rather than censoring the anecodotes, the better
solution is to simply point out the danger of overweighting them.

The purpose of a site like HN is not to arrive at some kind of imaginary
consensus, it's to inform and engage people in meaningful discussion. There is
no winning side to be on, and no ultimate arbiter of truth. By downvoting as
you do, you rob the site of content in exchange for an illusory sense of
victory.

------
hexagonal

      For example, if a close friend goes on and on about how the Ford he bought 
      was a piece of crap, detailing how the transmission failed at 30k miles 
      and the rear-view mirror fell off, you’ll be wary about buying a Ford in 
      the future, even if Consumer Reports rates them highly.
    

Personal anecdote: I've got a '93 Ford Explorer. It has 237,000 miles on it,
and has yet to have a single major component failure. I've been waiting for
the goddamn thing to die for the last 70,000 miles, so I can buy a car that
does better than 13 miles to the gallon, but no dice.

Don't buy Ford trucks. They're too reliable.

------
chris_wot
Have their been stories with seemingly solid statistical data, where anecdotal
evidence shows that the statistical evidence is wrong?

------
motters
Without contrary viewpoints there is always a danger of falling into
overconfidence and group-think.

------
dicroce
I disagree. In my experience, this article is wrong.

------
rsanchez1
The author didn't pick good examples of contrarian anecdotes found on HN. They
are contrary to a study on coffee consumption and dementia. The study,
however, is only a sample of 124 people from Tampa and Miami. The study is
just as likely to have found an isolated effect as the posters with family
members who consumed a lot of coffee and still developed alzheimers.

You would be better served by this:

[https://sites.google.com/site/mccormickphilosophy/home/criti...](https://sites.google.com/site/mccormickphilosophy/home/critical-
thinking-syllabus/critical-thinking-schedule/understanding-statistical-
arguments)

The author must be a frequent coffee drinker who didn't like that some people
had a different experience with coffee than some other people had, and felt
compelled to write that post. It's not the contrarian anecdotes that left me
with the sense that the research findings weren't conclusive. It's the fact
that a population of 124 people in two cities is not at all representative of
the target population which numbers over 40 million according to the census
bureau.

<http://quickfacts.census.gov/qfd/states/00000.html>

Maybe when the study researchers conduct a larger study I'll believe them.

~~~
tylerhobbs
Great comment. I am most certainly _not_ claiming that you shouldn't question
research findings on other grounds, such as sampling bias or statistical
significance, as you have done here; those types of points are extremely
valuable.

------
horsehead
But .... but .... anecdotes are what politicians use to get elected !!!! (e.g.
Joe the Plumber, et al).

If we downvote anecdotes, we won't have a justification to vote for scumbag
politician with a heartfelt story to tell!!

edit: here's a terrific example!!!
[http://www.washingtonpost.com/local/alabama-law-drives-
out-i...](http://www.washingtonpost.com/local/alabama-law-drives-out-illegal-
immigrants-but-also-has-unexpected-
consequences/2012/06/17/gJQA3Rm0jV_story.html)

------
kbronson
I'm going to downvote this article because contradicts everything said here.

------
its_so_on
Forget what you and your friends experience. If we manage to get a piece of
research sponsored, you'd better believe it!

Sorry, I'll go on the experiences of myself and people I trust over
research/article spinning.

~~~
its_so_on
if my friends and I have problems with your product, it doesn't matter how
reliable your research spins it as, I won't believe you. Forget research: make
something my friends and I, or people I hear about, don't have problems with.
That's all I pay attention to, and it works a lot better than the alternative.
(paying attention to whatever you "prove" at a statistically significant level
- nevermind how many commissioned studies you _don't_ publish, thereby
completely invalidating that statistical significance - and ignoring anecdotes
simply doesn't work, for me or anyone else.)

This article is literally asking for the right to lie (under the guises of
'research') and asking us to mod down anyone who calls them out on it. It
really takes some face to say "Ignore what you experience - and vote down the
experiences of others - and trust our data instead."

Next you'll sell me the most reliable cloud on the planet. All the responses
on the article say they've had nothing but problems and downtime. But, I
should just ignore these, right?

------
drivebyacct2
Is this not based in the assumption that the article to which the anecdotes
are offered as replies is omniscient? Which may in and of itself be
fallacious.

~~~
VMG
The assumption is only that the article is based on a scientific study, not
that it is omniscient.

~~~
r00fus
A study that rigidly follows the scientific principles, but is intellectually
dishonest (ie, sponsored by interested party of studied subject, etc) is often
not useful to the reader of the study results. Because you can follow the
letter of the principles, and still flout the spirit.

One should, when listening to a study, question the funding. Likewise,
dissenting opinions must also be examined in what interested parties have a
hand in their creation.

~~~
slurgfest
Sponsorship does not imply intellectual dishonesty. Intellectual dishonesty is
a matter of how the argument is made. Sponsorship is a heuristic you could use
to quickly filter papers but used as a counter-argument, it is simply ad
hominem.

You are describing a method for judging arguments without thinking critically
about the arguments themselves or examining their basis in evidence and that
is not any kind of science. One does not find any measure of objectivity by
averaging between opinions, only by holding arguments to the yardstick of
rationality and evidence.

