
An erroneous paper on religion and generosity is retracted - okfine
https://www.psychologytoday.com/us/blog/human-flourishing/201909/does-religious-upbringing-promote-generosity-or-not
======
mbell
I think it's situations like this that result in society having such a hard
time with 'science'. The term has been so heavily co-opted by fields that just
don't have sufficient rigger for the term to hold weight. Yet, on various
topics, we have this publicized attack of "your a science denier!". At the end
of the day, there are two 'types' of science, one where I can take the results
and make accurate predictions, and one where I can't. The later just amounts
to 'our best guess' where the accuracy is entirely unknown. If we want the
general populous to 'trust science', we need to stop calling the later
science. In short, if you can't repeat and predict, stop calling it science.

~~~
olooney
Scientists usually try to distance themselves by saying those are soft science
or even pseudo-science. This leads to the embarrassment of the demarcation
problem[1] which is that no one can give a bright-line rule[2] to distinguish
between the "real" science and pseudo-science. All of the demarcation criteria
that have been proposed (such as Popper's falsifiability[3]) are inadequate in
one way or another. In particular, they don't seem to capture the reasons a
scientist would give about why a particular nutrition or social science paper
is bad. The scientist would say things like, "Well, your sample size is small
and not representative of anything except psych undergrads, you didn't control
for age or gender, the participants and experimenters weren't properly
blinded, you tested 15 hypothesis and only reported the p-value for the one
that was under 0.05, and even that is wrong because you didn't apply Yate's
continuity correction on your chi-squared test, AND NONE OF THAT EVEN MATTERS
because the effect size you report is too small to be of practical
consequence!" Nothing in there about the hypothesis not being testable; yet
this is the kind of stuff that really separately the wheat from the chaff.

So we're left with a "No True Scotsman fallacy" where have to say that some
science is "good" and some is "bad" and the only way to tell is to ask someone
knowledgeable to evaluate each paper on a case by case basis. Not terrible
useful to the layman.

And why do we want any kind of science to automatically get respect anyway?
Good science is good because its already been subjected to an incredible
degree of scrutiny. It will hold up to a little more. The real problem is
disingenuous, bad faith arguments which are allowed to dominate the
conversation. The real problem is to teach the general public to distinguish
between sincere, good faith arguments and patent bullshit. This is much more
difficult than it sounds because bullshit can easily conform to any merely
superficial characteristics.

[1]:
[https://en.wikipedia.org/wiki/Demarcation_problem](https://en.wikipedia.org/wiki/Demarcation_problem)

[2]: [https://en.wikipedia.org/wiki/Bright-
line_rule](https://en.wikipedia.org/wiki/Bright-line_rule)

[3]:
[https://en.wikipedia.org/wiki/Falsifiability](https://en.wikipedia.org/wiki/Falsifiability)

~~~
tareqak
Why not have a general checklist with a minimum set of requirements for
scientific papers that are relevant across all branches of science? The people
putting their names on the paper would have to show that they followed
everything or give reasons for skipping a step. The receiving journals would
have their editors re-check the checklist. As part of reporting the results of
the paper, the level of completeness of the checklist would also be in the
report.

Yes, the checklist would not be all-encompassing or foolproof, and there would
likely be revisions to the checklist, and maybe even domain-specific variants,
but it would be an extra level caution that the media could report or choose
to ignore at their will. Over time, the apparent level of scientific rigour
would improve. No, it’s not bulletproof and there will be people who will try
to meet the checklist and still present erroneous conclusions as reliable
science, but it would be an improvement in the status quo for a layperson who
is aware and values said checklist.

~~~
olooney
There are things like CONSORT[1] which kind of do this. Statisticians like
Fisher[2] have a ton of good general advice of the design of experiments. (A
plug for The Lady Tasting Tea[3] and the 7 Pillars of Statistical Wisdom[4]
feels appropriate here.)

On the whole though, most of the things you _should_ and _should not_ do are
so domain specific its very hard to give much useful advice at the level of
"all science." Right now this seems to work because researchers are so eager
to anticipate objections and and avoid unnecessary arguments during peer
review they stick slavishly stick to the same methods used by seminal papers
in their field, and this has the same effect as running down a checklist.

There probably is a case to be made for using an actual checklist, though[5].

[1]: [http://www.consort-statement.org/](http://www.consort-statement.org/)

[2]:
[https://en.wikipedia.org/wiki/Design_of_experiments#Fisher's...](https://en.wikipedia.org/wiki/Design_of_experiments#Fisher's_principles)

[3]:
[https://en.wikipedia.org/wiki/The_Lady_Tasting_Tea](https://en.wikipedia.org/wiki/The_Lady_Tasting_Tea)

[4]: [https://www.goodreads.com/book/show/27311742-the-seven-
pilla...](https://www.goodreads.com/book/show/27311742-the-seven-pillars-of-
statistical-wisdom)

[5]: [http://atulgawande.com/book/the-checklist-
manifesto/](http://atulgawande.com/book/the-checklist-manifesto/)

------
neaden
"The paper received a great deal of attention, and was covered by over 80
media outlets including The Economist, the Boston Globe, the Los Angeles
Times, and Scientific American."

And how many of these will cover the retraction? A dozen at most? And all
those articles will be sitting out there, getting cited and read on occasion.

~~~
charliesharding
Really sad to see and feels like it's becoming more common (maybe just because
I'm paying closer attention). If it fits the narrative, accept first, retract
later. It would be interesting to see view statistics on the original article
vs the retraction.

~~~
astine
It's not just narrative fitting. There is also a strong bias towards
publishing results that seem surprising because that gets more readers. Of
course, that also biases toward wrong results because wrong results are likely
to be surprising.

~~~
gumby
"wrong results are _[more likely]_ to be surprising".

This case is interesting because there's a large population who would find
these (unproven as it turns out) results _confirmatory_ rather than
unexpected.

In the end they were neither.

------
ALittleLight
Releasing your data should be a requirement for publication. If the original
author had wanted to keep this a secret he could've withheld his data and
nobody would've been able to correct him, there simply would've been
discrepant studies.

~~~
0xb100db1ade
I see where you're coming from, but would the subjects be comfortable with all
their data becoming public?

~~~
ALittleLight
I think you could, and should anyways, make the data anonymous. Just give
every participant a GUID for a participant ID and add a step to purge
personally identifiable information. Then you can share records without
identity.

~~~
tobylane
That didn’t work for the AOL research several years ago.
[https://arstechnica.com/tech-policy/2009/09/your-secrets-
liv...](https://arstechnica.com/tech-policy/2009/09/your-secrets-live-online-
in-databases-of-ruin/)

------
makomk
Alas, these kinds of problems are not restricted to the social sciences. Case
in point, this retraction from a couple of days ago:
[https://retractionwatch.com/2019/09/25/nature-paper-on-
ocean...](https://retractionwatch.com/2019/09/25/nature-paper-on-ocean-
warming-retracted/) Very similar to this one really; the paper claimed to
overturn our existing knowledge in a way that fitted a narrative people were
inclined to believe (in that case: we're all doomed) and was immediately
seized on by all the news sites because of it, except the statistics were
mucked up and it couldn't show what it claimed to. The fact that it was so
surprising should've been even more of a massive warning sign in that case
though.

~~~
justinclift
Interesting. Hadn't seen the Retraction Watch website before.

Wonder if it'd be possible to automatically scan new papers added to (say)
arxiv.org, for retracted papers in their references?

eg to warn the authors, and maybe eventually automatically as part of the
upload process for arxiv.org (and similar)

------
pygy_
Conclusion of the new analysis:

 _In sum, Decety et al. [1] have amassed a large and valuable dataset, but our
reanalyses provide different interpretations of the authors’ initial
conclusions. Most of the associations they observed with religious affiliation
appear to be artifacts of between-country differences, driven primarily by low
levels of generosity in Turkey and South Africa. However, children from highly
religious households do appear slightly less generous than those from
moderately religious ones._

[https://www.sciencedirect.com/science/article/pii/S096098221...](https://www.sciencedirect.com/science/article/pii/S0960982216306704)

------
shriphani
From the article:

Although Decety’s paper had reported that they had controlled for country,
they had accidentally not controlled for each country, but just treated it as
a single continuous variable so that, for example “Canada” (coded as 2) was
twice the “United States” (coded as 1).

I mean I don't even understand how this seemed like a normal thing to do?

~~~
lmkg
The variable for Country should have been treated as a categorical variable,
but was instead processed as a numeric variable.

This mistake would be downright trivial to make in R. Just declare that
Country is a Factor (which is the built-in type for categorical variables),
and then throw the data into a library whose attitude towards errors is to
coerce everything to numbers until the warnings go away.

Background: Factors in R are the idiomatic way to work with categorical data,
and they work somewhat like C-style enums except the variants come from the
data rather than a declaration. So if you take a column of strings in a data
frame and cast it to a Factor, it will generate a mapping where the first
distinct value is coded as 1, the second distinct value is coded as 2, etc.
Then it replaces the strings with their integer equivalents, and saves the
mapping off to the side.

I forget the exact rules (if there are rules, R is a bit lawless), but it's
not very hard to peek under the hood at the underlying numeric representation.
Many built-in operations "know" that Factors are different (e.g. regressing
against a Factor will create dummy variables for each variant), but it's up to
each library author how 'clever' they want to be.

~~~
tempguy9999
Right then...

...strong typing: for or against?

(To be fair even strong typing won't save you if you don't use it. But
fuuuuuk, what an error. I noted that paper mentally and would have quoted from
it)

~~~
LorenPechtel
Yup, I'm all for extremely strong typing. In 40 years of writing code I can't
say I've ever had any real trouble with strong typing other than when dealing
with libraries that reinvent wheels. Weak typing, though--nuke it from orbit.

------
m0zg
Another, more consequential paper that popped up on HN recently has been
retracted as well. The one about ocean warming. The retraction notice is a
masterclass on weasel language, worth reading in its own right.

[https://www.nature.com/articles/s41586-019-1585-5](https://www.nature.com/articles/s41586-019-1585-5)

------
_edo
Sometimes I feel weird coding zip codes as strings but this is a great example
why. If my program ever treats a zip code like a number I would like it to
throw an error. At least in this case the error looks like an accident.

On topic, from yesterday:
[https://news.ycombinator.com/item?id=21067764](https://news.ycombinator.com/item?id=21067764)

It's another social sciences paper but in this case a co-author has requested
a retraction over his strong belief that the paper includes fabricated data.
The retraction request has been denied. It differs from this paper in that the
data anomalies look intentional.

~~~
ars
> If my program ever treats a zip code like a number I would like it to throw
> an error.

One interesting thing you can do though, is sort by zipcode. This sorts your
mail from East to West in the US. You can use that as a rough estimate of
shipping time.

~~~
_edo
That's really cool, even if I never find a place to use it.

Anyway you can still sort strings.

------
yesenadam
I wrote a webpage in 2015 about the Decety paper, what it was and how it was
presented in the media, which might be of interest. The paper seemed highly
suspect in various ways even at the time. I added an update in 2017 that "new
analysis shows that the original study failed to adequately control for the
children’s nationality". On the (unfinished) page I'm using the paper as an
excuse to teach myself basic statistics and research methods.

[http://www.adamponting.com/decety/](http://www.adamponting.com/decety/)

------
realradicalwash
so a categorical variable got mixed up as a numerical one and produced
misleading results.

to the credit of the authors, they released their data sets. -- however, i
suspect that proper data exploration and visualisation would have prevented
all this. visual inspection would have most likely revealed that there is no
visible effect, or even an effect in the opposite direction, and once you see
this, all alarm bells should go off if your model predicts otherwise. so i
suspect that the authors skipped some basic steps and got carried away by
results that promised a nice headline.

~~~
rossdavidh
Agreed. However, another factor may be that, we tend to scrutinize more
closely results that don't line up with what we expected. In fact, in this
case it was another researcher who had gotten results that pointed in the
opposite direction, who convinced the original researcher to release their
data (which, to their credit, they did).

Which, is one reason why having higher and higher percentages of academia and
science researchers be from the same part of the political spectrum, is
worrisome to me. If you have more diversity in ideology, there is more likely
to be someone in each field to have the instinct to scrutinize closely a
result which, when scrutinized, won't hold up.

------
whymsicalburito
There were even news sites that published articles about the original article
AFTER the retraction was announced! The state of science reporting is very
sad.

------
rahidz
"But when they included their categorically-coded country (1 = US, 2 = Canada,
and so on) in their models, it was entered not as fixed effects, with dummy
variables for all of the countries except one, but as a continuous measure.
This treats the variable as a measure of ‘country-ness’ (for example, Canada
is twice as much a country as the US) instead of providing the fixed effects
they explicitly intended"

How did this not get caught immediately? If I did a study and found out that
kids in Zambia are 47 more times as generous as American kids that'd make me
instantly suspicious.

Or maybe the reviewers were all Canadian /s

~~~
TheCoelacanth
I don't think it's quite as obvious of an error as you are suggesting.

They were trying to correct for scenarios like this: Hypothetically, Canadians
are twice as generous as Americans and twice as religious, but religious
Canadians are equally generous as non-religious Canadians and religious
Americans are equally generous as non-religious Americans. On the surface, it
appears that religious people are more generous, but really it's just that
Canadians are more generous.

Instead of treating the countries as discrete groupings, they treated them as
points on a spectrum with each country being assigned an arbitrary place on
the spectrum.

If #3 happened to be China, they would be assuming that people in China should
very similar to people in the US and Canada, because 1 vs. 3 on a scale that
goes to 200 is hardly any difference at all, but really the numbers are just
arbitrary identifiers.

~~~
tastygreenapple
I'm a data scientist and this is an incredibly embarrassing 'n00b' error to
make. If these researchers were using anything other than deep learning, it's
almost certain that each parameter of the model was manually selected. That
the author made the mistake is bad, that no one caught the error is a
disaster.

------
zdw
As much as I like spreadsheets and other general purpose numerical analysis
tools, I wish there was a restricted subset of their functionality that could
be used that was more formally verifiable, to prevent issues like this and
Reinhart–Rogoff (see:
[https://en.wikipedia.org/wiki/Growth_in_a_Time_of_Debt#Metho...](https://en.wikipedia.org/wiki/Growth_in_a_Time_of_Debt#Methodological_flaws)
) from being a common occurrence.

------
otakucode
If a research does not release the code used in their research, their papers
should not be trusted. We wouldn't trust a paper where the researchers hid
their experimental methods or analysis - it wouldn't even be allowed to be
published. But if their experimental methods or analysis are done in code,
those parts are allowed to be a black box and we're supposed to trust them. If
we were willing to just trust them, there wouldn't even be a peer review
process and it certainly wouldn't deserve the imprimatur of 'science'. This
has been a problem for years, and I will personally not be the least bit
surprised if we eventually see a death toll attached to it.

------
riffraff
I wonder if a religious education resulting in some positive effects might be
akin to the "any diet is effective" thing, i.e. it's not the specific
upbringing as much as being exposed to _any_ moral thinking in general.

I don't mean to imply that there aren't moral a-religious educators of course,
just that it seems likely to have less discussion of ethics for any kid once
the temple/church/mosque/whatever is removed.

------
krilly
Wait a second. If the issue with the analysis is as the article says, that is
that some countries were weighted far higher, that means that in some
countries the original conclusion does hold (probably for a small sample
size). Perhaps this is because some religions promote generosity and some do
not? Would be interesting to look in to.

------
bshoemaker
Fun to see all these software engineers criticizing scientists for a bug in
their code

------
EGreg
And even though it was corrected and the result retracted, it was already
cited by many papers, the media didn’t report the retraction etc. How much of
scientific reporting gets skewed by media interests (controversy and
sensationalism) and funding and political interests?

Furthermore, this whole field was called cargo cult science by Feynman, using
correlations, p-hacking, data dredging, and more

[https://en.wikipedia.org/wiki/Multiple_comparisons_problem](https://en.wikipedia.org/wiki/Multiple_comparisons_problem)

The best analysis I have seen of the corner we have painted ourselves into

[https://slatestarcodex.com/2014/04/28/the-control-group-
is-o...](https://slatestarcodex.com/2014/04/28/the-control-group-is-out-of-
control/)

------
imperio59
Can we go back to basic principles and all agree that psychology and social
studies are not true sciences anyways?

Science is to me something where you extract natural laws that predict
phenomena will occur 100% of the time given certain conditions. Physics,
chemistry, computer science, most branches of medicine operate this way.

A field like psychology that says "well sometimes people will..." or "we found
in 60% of cases that..." is not science. It's a comment upon society maybe,
but it does not produce broadly repeatable, predictable results.

Not true science.

~~~
ausbah
I think you're forgetting that the "replication crisis" also occurs in your so
called "100% deterministic fields". CS has the issue of ML models, medicine
drug studies, etc.

The bottom line is science is hard to do, and even harder to do right. I think
all fields do try to follow the scientific method to the best of their
ability, but for some fields it is simply harder to do because of the sheer
complexity of the subject matter being hard to model, experiment in, and
understand.

------
blackbrokkoli
This is fundamentally one of the many disastrous outcomes of scientific
publishing being controlled by greedy conglomerates.

For the uninitiated: If you want to publish a scientific paper today, you
basically sign up to sign over the rights to any publisher that's interested
(please someone publish me). That publisher will then review that paper in a
more often than not mostly undisclosed process and publish that paper.
Everyone knows that to be a high-regarded publisher one must have a very own
typographic formatting: Unreadable font, weird multi column layout to prevent
accessibility and tables disregarding standards are a good start. Afterwards,
the paper gets published on the publisher website, which again follows as
little agreed upon standard as possible. Data is of course excluded, study
itself is in PDF. Also put up a fat paywall, don't want those pesky poor
people be scientifically literate. Give a few cents of your $70 fee to the
authors, it is not an unethical business you're doing here!

THIS is the systemic error people seem to be so happy to ignore because
they're neck deep in social science not being real science memes.

This prevents interesting new startups for fact-checking or meta analysis,
which are e.g. happening in journalism because that field has a lot of the
things science is sadly lacking.

This creates a drift between extremely rich and rather poor
countries/unis/humans in scientific ability.

This generates a tar pit for scientific process.

This wastes billions in funds because of people unaware of each other doing
redundant studies (and not referencing/refuting/supporting each other in the
process neither, of course) because they are literally better search engines
to find Harry Potter fan fiction than there are for finding studies.

And finally, this of course allows anything from honest statistical mistakes
to snake oil sellers to slip through and doing generations worth of damage,
because correcting, fact checking, re-researching, comparing, meta research,
_anything_ is slowed to a crawl.

So please stop embracing scientific elitism and gatekeeping for this is
exactly what brought us here in first place...

------
d0100
> Decety

What an apt name.

I wonder about the damage to public this unintentional deceit will bring...

Go science! After all, a retraction is part of discovery.

------
jimmaswell
Interesting, waitstaff on reddit always say religious people tip worse or not
at all.

~~~
CDSlice
Probably because most religious people don't tend to flaunt it everywhere they
go, and in my experience the ones that do also tend to be jerks. It's also
possible that wait staff tend to remember the ones that do more than the ones
that don't because being religious and a jerk is more memorable than religious
and not a jerk.

Disclaimer: I am a religious person.

~~~
SauciestGNU
I worked in breakfast and brunch restaurants for a while, and my anecdotal
experience is that the post-church rushes (we were across the street from a
couple churches) had the worst overall tipping percentages that I would see
all week.

~~~
mikestew
One could argue that those are the folks that aren’t religious enough to feel
that they should not make others work on a sabbath as a result their personal
commerce. The religious folk that make it rain come in on other days of the
week.

------
peteretep
> In fact, Decety’s paper has continued to be cited in media articles on
> religion. Just last month two such articles appeared (one on Buzzworthy and
> one on TruthTheory) citing Decety’s paper that religious children were less
> generous.

“Media articles” is carrying a lot of weight there

------
quotemstr
My default presumption is that all results from the "social sciences" are
false if they contradict my intuition. Whatever these fields are producing,
it's not science in the Popperian or practical sense. A ton of policy has been
built on the bad research and wishful thinking of the past few decades, and
it's going to take a long time to unwind it all.

The problem, really, has been accelerating for a while. Something really went
off the rails after WWII.

~~~
sebastos
My theory is that it's engineering, not science, that confirms our knowledge.
You don't really know something about the world until you can use it to do
things. WWII-era science was being incorporated into tools (well, weapons), so
it had to work.

------
cjohansson
Psychology, sociology and theology (and more logys?) was never meant to be
sciences. We can blame the enlightenment for that idea, let’s revert them back
to renaissance activities

~~~
mr_overalls
Psychology and sociology can be made as rigorous as individuals & institutions
care to.

Theology, not so much.

~~~
high_5
Actually most of the (christian) theology (when discussed by theologists)
within its confines is very rigorous.

~~~
mr_overalls
Theology can be rigorous in the sense of its models possessing internal
validity (logical consistency, parsimony).

However, because it's basically speculation about unobservable entities,
theology completely lacks external validity.

Social sciences can have both.

