
Laypeople can predict which social-science studies will replicate successfully - apsec112
https://journals.sagepub.com/doi/full/10.1177/2515245920919667
======
science4sail
According to the "method" section, 138 out of 257 candidate (233 final)
participants in the study were first-year psychology students - I can't help
but wonder if this might have biased the study's results. The investigators
also classified graduate students as laypeople, which doesn't quite feel right
to me.

~~~
RandoHolmes
Maybe this particular study, but I have to say I believe it in general.

I remember working for a manufacturing company where I drove a forklift
(during the summer).

One of my most vivid memories is walking into the breakroom and seeing on the
television a scrolling banner that declared homosexual couples were less
likely to have children than heterosexual couples.

And I remember everyone in that room laughing their asses off. And it's not as
if no one in that room realized the reason for the study was so they could
take into account things like adoption, etc. It's that the result was so
obvious, even taking that into account, that it was amazing that someone was
PAID to come to a conclusion that everyone knew without the money.

And this is the crux of the problem with "science". It wants to be
"interesting", so it will literally try to drum up something against what
"everyone knows".

So the idea that what everyone over generations "knows" is generally more
applicable than "science" is not surprising at all.

~~~
daveFNbuck
A lot of the things that "everyone knows" are wrong. The point of science is
to have a systematic way to tell the difference between things that seem
intuitively true and things that are actually true.

I did a quick search and found a study about percentages of couples raising
children [1]. They have a table by couple type and marital status. The
difference isn't as stark as you might think.

Looking at the smallest difference in the table, married female/female
partnerships have a 30.2% chance of currently raising children, while married
male/female partnerships have a 38.7% chance. Is it obvious to you that this
would be so close? Would you be surprised if there were countries where the
numbers are reversed?

[1] [https://williamsinstitute.law.ucla.edu/publications/same-
sex...](https://williamsinstitute.law.ucla.edu/publications/same-sex-parents-
us/)

~~~
heavenlyblue
The issue is that you don't really need a study to open up a table of
statistics in demographics department.

Your post just did a study on that topic. Should you get funded for that?

~~~
daveFNbuck
I spent a minute looking up the results of existing studies. If a professor at
a public university spent a minute to look up an answer for a reporter, I'm ok
with that professor having a salary.

------
r6r7rurufuru
I'm curious how much of this is lay people evaluating plausibility vs.
politics. I only know the mandatory minimum of social science I was required
to take in college but if you understand that gender and lgbtq+ matters are de
facto immune to real criticism then you can pretty easily guess which
hypothesis are likely to be right/wrong by whether or not them being correct
would validate a negative interpretation of any of those groups. Basically
anything that harshes the vibe is a no go in American social science unless
you can use it to criticize white people or cis people, if you know that then
you can predict most big stuff since everything else is just rhetorical
frameworks masquerading as science.

~~~
trabant00
Read the abstract of the studies involved. None of the ones evaluated
negatively involve anything like you mention. No gender studies, lgbt, things
like that. The most woke one was "Analytic Thinking Promotes Religious
Disbelief".

But the newest bad one was 2012 and you could say most of the bad ones had a
socialist lean.

------
BLanen
I'll believe this study when it's replicated successfully.

~~~
JshWright
I'm no expert, but I predict it will be.

~~~
wtvanhest
59% possible

~~~
kevin_thibedeau
±9%

------
gus_massa
They claim that "laypeople" can predict if the study is reproducible, but they
only got a 59% accuracy. It's technically true, but it's too close to 50%. I
can't find how many of the 27 studies there "reproducible" and how many no.
Specially because half of the subject were first year students of psychology
that should have a minimal idea of the subject.

And the 67% when the people is informed about the strength of evidence is even
less impressive. How much accuracy would have a parrot that just repeat the
information?

To be fair they translate the strength to the evidence from a numeric scale to
a simple words scale, and the participants should translate it back to a
numeric scale. The real question is how good is people doing this task with
random numbers, without additional information like the description of the
study.

------
jonplackett
Considering they only got it right 60% of the time, that still means there’s a
large proportion that are unintuitive, so I don’t think it makes sense to
trumpet this as if it’s super clear cut which are obvious and which aren’t.

~~~
rusk
Heh, pretty clever humour if intentional :-)

------
plaidfuji
Yeah, this checks out

Edit: but seriously, I would assume that this applies to hard sciences as
well, if you replace “laypeople” with “domain experts”. Plausible hypotheses
are cheap; designing a study and collecting data to prove them is hard. Many
studies confirm something that feels obvious to researchers in the field, but
being able to write that thing down as fact with a citation allows people to
move a step forward. I would say it’s the extreme outlier study where an
implausible hypothesis is proven true.

------
zepto
Why is this surprising at all?

Surely it’s the expected outcome.

The opposite would be something like ‘most social science results are
counterintuitive’.

------
cactus2093
59% of the time, that doesn’t seem very interesting at all. So ~10% of studies
where results are strong enough to be replicated also have predictable
outcomes (presumably these are just simple, obvious situations), the rest are
a coin toss.

~~~
computerphage
I actually would think it's the other way around: ~10% of studies that sound
like obvious nonsense and have p of ~0.049 are probably NOT going to
replicate.

------
jungletime
Isn't this why we have initiation? Since we don't ever have complete
knowledge, our brain has to predict/guess at some sort of probability curve
for an event happening all the time.

So much of our political rhetoric is just basic one dimensional math. "have
and have nots", "wage gap", and so on.

The World would be a better place if more people understood exponential
growth, statistics, game theory, and so on. It hampers our politics, when the
rhetoric needs to be reduced into basic algebra.

~~~
dash2
Just trying to imagine some rhetoric recast in the language of exponential
growth, statistics and game theory....

"In a sense we've come to our nation's capital to cash a check. When the
architects of our republic wrote the magnificent words of the Constitution and
the Declaration of Independence, they were making a costly signal of
commitment. But it's obvious today that America has failed to support the
separating equilibrium... We have also come to this hallowed spot to remind
America of the fierce urgency of hyperbolic discounting.... I have a dream
that my four little children will one day live in a nation where their skin
colour will not be a sufficient statistic of the content of their character!"

------
lazyeye
"Reproducibility crisis" doesnt really cover it. Would a layperson term like
"bullshit" be more accurate?

------
throwawaysea
See related recent discussion about what’s wrong with social science, which
also touches on prediction markets for replicability:
[https://news.ycombinator.com/item?id=24447724](https://news.ycombinator.com/item?id=24447724)

------
fifticon
I know hackernews is not the place for this, but my mobile browser truncated
studies to stud, and the headline about a successfully replicating social-
science stud had me stumped for a moment.

------
bsder
The problem with all this is that the interesting study is the one that
replicates that people _didn 't expect to_.

------
xenocyon
I find that I can usually predict which topics on HN are likeliest to spark
low-effort comments :)

------
ajuc
That would make social sciences half-bullshit and half-obvious :)

------
Stierlitz
“Are the social sciences merely dressed up pseudoscience?”

[https://www.quora.com/Are-the-social-sciences-merely-
dressed...](https://www.quora.com/Are-the-social-sciences-merely-dressed-up-
pseudoscience)

~~~
R0b0t1
The ones making conjectures divorced from a strong grounding in biological
purpose likely are.

