
Suggest HN: Studies based on small sample sizes shouldn't be on HN - syllogism
Reading about studies based on small sample sizes has in total misinformed me. I&#x27;ve been convinced of many surprising things based on a trust that the research was well conducted, and that published results are on average true. The building replication crisis has shown that this faith was misplaced, particularly in certain fields.<p>For instance, it&#x27;s now clear that I would have believed more true things and fewer false things if I had not heard about a single result from social psychology. I&#x27;m still rooting out beliefs that I came to based on bad research. (Unfortunately my mind doesn&#x27;t maintain an index of beliefs by source. I wish I could request this feature.)<p>I&#x27;m rightly bitter about this, and I see the problem continuing. For instance, this submission is worse than worthless: https:&#x2F;&#x2F;news.ycombinator.com&#x2F;item?id=13144483 . It&#x27;s impossible to tell what experiment was done without buying the paper, and on average experiments of this type have been shown to be of negative value: http:&#x2F;&#x2F;andrewgelman.com&#x2F;2016&#x2F;09&#x2F;21&#x2F;what-has-happened-down-here-is-the-winds-have-changed&#x2F; .<p>We&#x27;re all worse off for reading and talking about these articles. I&#x27;d like to see HN add a heavy link penalty on all reports of research based on small samples, with an exception for good replication studies. Off the top of my head, n&lt;10,000 might be a good threshold.
======
vivekd
I disagree with this idea. For starters, sample sizes alone don't give you
reliable results, even with large sample sizes you might have other issues
that affect the results such as poor design, it may not be reproducible or (as
is often the case) it may be that no one has tried to reproduce it yet.

Scientific research is often imperfect, as a practical matter due to lack of
resources and many other reasons. But over time, as scientists peer review and
reproduce each other's research, we come closer to scientific consensus
regarding the matters of study.

Scientific studies, by their very nature, are often designed to inquire into
matters of uncertainty. And no one study can completely claim to banish
uncertainty and provide concrete and certain answers. The path to certainty
requires the toil of reviewing and reproducing the results.

So I think anyone who puts forward a single scientific study to bolster a
claim or point of view is making a very weak and unconvincing argument to
anyone who understands science. Rather, we like scientific studies, not
because they give us concrete neat solutions but because they offer
interesting insights that may or not be conclusive.

I don't think it should be the job of moderators to determine the weight or
conclusivity of scientific studies. I think that should be done by the
community.

I also think the community would lose something if were to get rid of articles
on interesting research just because that research was inconclusive. Research
can be interesting and open up interesting possibilities even when it doesn't
give us concrete truths to hold to.

~~~
syllogism
It sounds to me like you reject the premise that an intelligent reader who
reads studies like this will have their beliefs diverge from the truth over
time. I read your comment as saying, "Sure you shouldn't be convinced by a
single study. But if you read enough of these, you'll converge towards the
truth". I used to believe that, but I don't now.

Let's say some study is an increment of evidence about some hypothesis H. We
start off with a prior on H, and we want to know how to update it after
reading about a study.

What I've taken away from the replication crisis is that my average update was
an order of magnitude too large. The appropriate evidentiary weighting of a
study in social psychology is so small that I can't usefully consider it as
any evidence at all. I'm better off ignoring it.

A little example for you. Let's say you read some study about the act of
physically smiling causing a positive change of mood, due to a feedback
effect. Before reading this study, you would've given this claim p=0.1.
Possible, and you can make sense of the explanation, but no particular reason
to believe it. After the study, you give it p=0.4. By no means settled, and
you don't even necessarily believe it. But you're giving it much more
credibility than before. Later it turns out that the study was actually on
very few people and was p-hacked. It therefore fails to replicate. The
appropriate post-reading belief level turns out to be around p=0.101. You were
tricked.

If you repeat this pattern, you end up carrying around a bunch of surprising
beliefs that have a quite random relationship to the truth. Some of the
studies will be correct, but on average you'll be more wrong.

Reading these articles might be "interesting", on an entertainment-based
definition, but they're the opposite of information. If you read enough of
these, you'll make yourself superstitious.

~~~
davelnewton
Information isn't the opposite of information.

You more or less sum up my reaction to your suggestion with this (rather
telling) phrase:

" _you 'll_ make _yourself_ superstitious"

(Emphasis mine.)

~~~
syllogism
I don't think I understand your comment.

What I'm saying is that if you read these studies, you'll become less informed
over time. You will believe fewer true things and more false things. I think
it's reasonable to call that the opposite of information

~~~
davelnewton
_I_ will? No, no I won't. And no, you don't become less-informed by taking in
information. You become _ill_ -informed if you choose to believe incomplete,
partial, or misleading information. But that's _your_ choice. Some people read
a fairy tale and believe it--not the fairy tale's fault.

To me it seems like you're absolving yourself of responsibility.

If I don't understand the nature of a study then I take it for what it is: a
report about results of a study I don't understand. _I_ choose whether or not
to believe it based on partial data. It doesn't _make_ me believe it. I _may_
believe it, or _parts_ of it, or _none_ of it. _I_ make that decision.

------
davelnewton
Disagree, somewhat loudly.

I don't blame other sources for my own beliefs. If I choose to believe
something based on a news article that's _squarely_ on me.

In any case, the paper you cite is available here:

[http://fulltext.study/preview/pdf/917856.pdf](http://fulltext.study/preview/pdf/917856.pdf)

------
ebbv
Seems to me like you haven't learned the right lesson. Small sample sizes can
still yield useful results. It depends on the nature of the study.

Instead of closing yourself (and the rest of us) based on a fairly arbitrary
criteria, you're going to have to use critical thinking. That should be
applied to everything you read not just studies.

------
sz4kerto
Good idea, but a hard limit like 10000 is not that good. Statistical methods
don't require specific sample sizes. Cherry picking, incorrectly chosen
methods are the problems, not the sample sizes, usually. A hard limit would
suggest that anything above this sample size is credible, while that's very
far from the truth. The reproducibility crisis is very much unrelated to
sample sizes.

~~~
syllogism
Sure, you can't design a study badly and make it up on volume. — but there's a
point at which the sample is so small we really don't need to look much
closer.

For instance, I don't want to spend time figuring out whether an n<100 study
is any good. It isn't.

~~~
aaron695
> For instance, I don't want to spend time figuring out whether an n<100 study
> is any good. It isn't.

So an AIDs cure that cured 80 out of 90 men wouldn't be worth reporting on HN?

Studies that have >10,000 people I'd think are close to non existent, these
are hugely expensive and I think most are well known.

I think it's a job for the scientific community to work out, I can't see
moderators on HN doing it.

~~~
syllogism
Personally I wouldn't click on a link that was reporting on a potential AIDS
cure that was tested on 80 or 90 people.

~~~
davelnewton
I'd click on a link on a report that cured 80 of 90 AIDS patients.

I think you've misunderstood the nature of preliminary clinical trials.

------
drbigglesworth
The focus on sample size is misplaced: small sample sizes frequently have
sufficient power to be statistically significant.

The error you are making is in placing too much faith in a single study, and
possibly in the journalists/bloggers who report on a single study. I have lost
count of the number of times I've read a press release or news article that
completely misrepresents the findings of a study (i.e., the error is in the
reporting, not the article). It's also frequently forgotten that journal
articles are written to communicate findings to peers, not the larger public.
There are nuances that are understood within a community that are unknown when
one is coming from a different background.

Until there is a body of work on a phenomenon, perhaps framed by a review
article or two, it's best to regard single reports (regardless of perceived
quality) as interesting curios but nothing more.

------
gus_massa
In an ideal world I agree, but it's very difficult to implement. I'd ask to
not upvote stories with a small sample size.

In many articles consider writing a small rant about the methodological
problems. Lucky most of the times I just found a comment by tokenadult that is
better than my draft and has link to support the claims. It's a pity that he
is not writing them recently.

Try writing a civil comment explaining that the study is dubious due to the
small sample size. Most press coverage skip this detail, so you have to look
at the research article to copy it. You'll get at least 1 upvote.

Don't worry too much. Just eat some chocolate
[https://news.ycombinator.com/item?id=9714985](https://news.ycombinator.com/item?id=9714985)

------
dbg31415
If we are promoting censorship can we start with:

1) Politics

2) Links to Reddit; any discussion about Reddit

3) Stories about the dinosaur tail found in amber
[https://hn.algolia.com/?query=dinosaur%20tail&sort=byDate&pr...](https://hn.algolia.com/?query=dinosaur%20tail&sort=byDate&prefix&page=0&dateRange=all&type=story)
(posted 26 times in the last week)

To be clear I'm not for censorship, but if I was the above is where I'd start.

------
brudgers
Generally, I think scientific studies are uninformative as news because:

1\. Most people are not close enough to the science to form an informed
technical opinion.

2\. The scientific conclusions of most studies are impractical to implement.

John Oliver's analysis is both sophisticated and entertaining:
[https://www.youtube.com/watch?v=0Rnq1NpHdmw](https://www.youtube.com/watch?v=0Rnq1NpHdmw)

------
golemotron
That's a great idea for a journal aggregator. This is a news site.

~~~
syllogism
It's a news site that has editorial policies about which stories are worth
paying attention to and which ones aren't.

------
Kenji
If you take the results of soft 'sciences' seriously you've already lost.
Recognize submissions like the one you linked as noise and move on. Common
sense goes a long way.

~~~
aaron695
I think this comment actually nails it.

Most of the current issues in published science are around the 'soft sciences'

They are notoriously hard to disprove I think because they are so wishy washy.

Maths, Physics, Chemistry can get you disproven in a hard way. There's a lot
more incentive to make sure your study is valid.

~~~
gus_massa
There is a lot of dubious Physics that is very popular here.

My antifavorite is the Em-drive:

Most popular article: "EmDrive study officially published"
[https://news.ycombinator.com/item?id=12995125](https://news.ycombinator.com/item?id=12995125)
(595 points, 21 days ago, 432 comments)

(My second choice is Cold fusion / LENR.)

This has slightly less votes that the most popular stories about colossal
experiments:

"Second Gravitational Wave Detected at LIGO"
[https://news.ycombinator.com/item?id=11910700](https://news.ycombinator.com/item?id=11910700)
(621 points, 178 days ago, 175 comments)

"Higgs Boson Explained by Cartoon"
[https://news.ycombinator.com/item?id=4193590](https://news.ycombinator.com/item?id=4193590)
(646 points, 1622 days ago, 128 comments)

The problem is that most people like nice stories, and a massless thruster
that can make space travel much faster is interesting. And it's much easier to
explain than why we need the Higgs's boson or gravitational waves.

The two later have no possible application in sight. They are interesting
results, but it's not clear how they will change your life. (I'm optimistic,
let's see what happens in 30 years.)

The Em-drive looks more promising. Cheap and fast space travel soon, for some
definition of soon. It's easier to explain, in particular because the
theoretical explanation is worse than the experimental setup. No one tries to
explain something about gauge invariance of curvature tensors.

~~~
Kenji
As far as I know, the EmDrive is still considered controversial and it is not
certain whether and how it works. It is interesting because it could cause a
revolution to how we view impulse, so it might be a colossal thing just as
much as it might be complete nonsense. So I think the comparison is
disingenous.

~~~
gus_massa
It's difficult to explain in a short comment, but the Em-drive is as
controversial as the yeti.

If it were true, then it breaks the current laws of physics. In particular the
conservation of momentum and energy. There are some bogus explanations that
try to make it compatible with the current laws of Physics, but they are
nonsense.

Breaking the current laws of Physics is not bad if you have a good experiment.
The problem is that they are measuring a very small force that is very close
to the experimental error. And there are many other forces that are difficult
to calculate and may explain the force they are measuring.

