So we're left with a "No True Scotsman fallacy" where have to say that some science is "good" and some is "bad" and the only way to tell is to ask someone knowledgeable to evaluate each paper on a case by case basis. Not terrible useful to the layman.
And why do we want any kind of science to automatically get respect anyway? Good science is good because its already been subjected to an incredible degree of scrutiny. It will hold up to a little more. The real problem is disingenuous, bad faith arguments which are allowed to dominate the conversation. The real problem is to teach the general public to distinguish between sincere, good faith arguments and patent bullshit. This is much more difficult than it sounds because bullshit can easily conform to any merely superficial characteristics.
Yes, the checklist would not be all-encompassing or foolproof, and there would likely be revisions to the checklist, and maybe even domain-specific variants, but it would be an extra level caution that the media could report or choose to ignore at their will. Over time, the apparent level of scientific rigour would improve. No, it’s not bulletproof and there will be people who will try to meet the checklist and still present erroneous conclusions as reliable science, but it would be an improvement in the status quo for a layperson who is aware and values said checklist.
On the whole though, most of the things you should and should not do are so domain specific its very hard to give much useful advice at the level of "all science." Right now this seems to work because researchers are so eager to anticipate objections and
and avoid unnecessary arguments during peer review they stick slavishly stick to the same methods used by seminal papers in their field, and this has the same effect as running down a checklist.
There probably is a case to be made for using an actual checklist, though.
"if you can't repeat and predict, stop calling it science" seems like a nice bright line.
Peer-review obviously isn't enough, I'd like to see peer-replicated studies become a thing.
Some of these cases can be rescued by considering "retrodiction" as valid substitute for prediction in the right circumstances, but not all.
I personally think the analysis of the Mott problem points the way to the solution to some of these kinds of issues. That is, a prediction can take the form of a likelihood function which assigns high probabilities to certain combinations of events and low probabilities to others. Theories with low perplexity can be considered correct even if they can't make predictions, and the study of such theories can be scientific. But as far as I know I am the only one who thinks so.
For what it's worth, there is science meeting the "repeat and predict" definition that can be done in each of the fields you list.
(Also, most of the fields you listed do allow for replication.)
I don't see a "No True Scotsman fallacy" here because I think I define it quite clearly: Can you provide accurate predictions? I'll concede that there exists a bit of grey area in that question, but the answer is heavily bi-modal.
What I get really, really, really frustrated with is the attitude that some people have where retractions of stories or articles makes them trust the source less, rather than more. If a journal publishes a retraction, I have that much more faith in that journal, and that much more faith in the scientific process in general.
Similarly: news outlets retracting stories or issuing corrections makes me trust those outlets that much more, because it shows that they care about making a best effort to present truth, and they care more about their reputation as being a source of facts than they do about whatever short-term backlash there might be.
Any media outlet, journal, web site, or other publication that doesn't regularly issue retractions and corrections is not to be trusted. It'd be better if the erroneous information didn't get out there in the first place, but there's no scenario where everybody gets everything right the first time, all the time- it's going to happen.
You can really see the different perspectives highlighted from people posting to social media in response to things like changing estimates of the age of the universe. There's always a contingent of outraged morons screaming stuff like, "now they're saying it's 5 billion years older than they thought it was before, and people continue to trust them?!? All those scientists are such hypocrites for doubting my belief in anti-vax/Austrian economics/chemtrails/Noah's Ark/etc., how dare they!" They act like changing your mind based on new data, or admitting you were wrong about something is a sign that you shouldn't trust someone, whereas I would say the ability to constantly revise your beliefs is a fundamental requirement for trusting someone's judgement. The inability to do so is a reason to not trust anything someone says.
In this case, social science analyses are used to answer questions of major public importance. Governments are constantly trying to reduce suicide rates, and make their populaces happier. Papers making claims that religion makes children selfish and unhappy are used to make public policy. In this case, if this paper were used as justification for legislation, we now know that the policies it would tend to suggest would be bad for the population. Someone, somewhere probably ought to be hold accountable in the same way as any other professional. I do hope that journals take appropriate precautions with this researcher in the future, and that the peer reviewers assigned to this case are duly sanctioned. This is a complete failure as professionals.
There are situations in which scientists can and do get things wrong through no fault of their own. For example, during the highly publicized EM drive tests a while back, an initial NASA report indicated that thrust was observed after careful evaluation. This is fine... they reported what they saw. However, after some additional engineering and measurement tuning and stronger sensors, the thrust was attributed to another source, so the claims were retracted. This is science. At every step the scientists demonstrated competence and professionalism. Nowhere did anyone say 'oops we forgot to use the sensor the right way that's why it didn't work, and in the meantime our paper was used to engineer other solutions'. There is a fundamental difference between being wrong and misrepresenting what you saw, whether through mistake or ignorance.
Politics, religion, etc. don't have the same advantage.
The solution to this of course is for everyone to remain more skeptical when it comes to science, and -- for the times when you do have reason to believe particular studies -- to respond in good faith to other skeptics, rather than to resort to name calling. If you are unable to defend why a certain piece of research should be believed, you should probably either accept that you are not educated enough to be able to comment and thus are also not doing 'science', or that the guy you disagree with may have a good point and the researcher in question bears the burden of proof.
On it's own, I think scientists making bad research and then retracting it would not cause people to distrust science. The 24-hour pop science news cycle combined with the massive rise of scientism as a religion (and the skeptics the equivalent of a medieval atheist) has.
I think you missed the biggest segment: not paying attention. It's for these folks where terminology and clarity is important.
A statistical error was made, published, and then corrected.
That's possible in physics, chemistry, or biology just as well.
Yet it seems to happen far less often, or at the very least, in less publicly impacting ways. Part of this is the media, but part of it is the participants themselves. If you discover something new in physics for example, either the public doesn't care, or they don't really know, they just get a faster iPhone processor a couple years down the line when the predictions hold. The folks in your field are even skeptical at first glance, "do you have a 6 sigma result?". "Ok, well, lets talk then, but I still wanna see it reproduced". Psychology on the other hand, some person does a half assed 'study' and uses it to claim knowledge of some important aspect of humanity.
I'd really encourage looking at the Higgs discovery press conference as a perfect example. To my recollection, there was little to no mentioned of the Higgs, just cold hard facts, perhaps at the end there was a 'this is consistent with the Higgs'. Only months later were many of those involved even comfortable enough with their level of certainty to really say 'this is the Higgs'. They are searching out knowledge, and don't want to declare having found it unless they are certain. This is _good_, it's what we want science to be, the summation of our current, highly confident, view of the world. We don't want 'science' to encompass all untested and unproven hypothesis about the world.
The difficult point which I have to concede is that terminology is important. Certainly many folks in physics that maybe haven't hit on a predictive result would like to be recognized as scientists, and rightly so if they are on the path toward this endeavor. But the populous is simple, they want clearly defined words, they want 'science' to be known fact. Outside introducing new words, I don't know how to resolve this. If we don't define 'science' as denoting that which is rigorous, then we can't use "science denier" as a term, regardless of the topic.
> Yet it seems to happen far less often
Citation needed. And if so, that is a reason to draw a line where on one side is "not-science"? That is just absurd. Does a car A stops being a means of transport when it subjectively breaks down more often than car B?
> very least, in less publicly impacting ways
* A literal century ago someone failed to translate a German study so know about every child in the western civilization gets a good dose of distrust in science when they get indoctrinated that the tongue has separated regions for taste which is ridiculously easy to refute for yourself in about 15 seconds.
* The coup of the cereal industry to fund some studies telling everyone that breakfast is the most important meal of the day still misguiding health guides today.
* Schrödinger telling the world how stupid it would be to assume quantum principles in the visible world, still happily recited with the complete opposite meaning by about 500 media entities per day.
* Scientific entities failing to have any impact on people about the dangers of X-rays until people got impotent from having their shoes measured via X-rays in the local shopping mall
To be clear, this is not intended as some sort of smear campaign to science itself. I want to illustrate that all science is vulnerable to even dumb mistakes and that this dumb social sciences ain't real meme is only slowing down much overdue conversation!
Your definition of 'science' is 'the best we know', mine, and I think what is meaningful for public discourse is 'this is true'.
Derivable from first principles? General consensus? Observed once and seems to fit with the current model?
The idea that the sun and stars move around the Earth explained only why they move across the sky, but the idea that the Earth orbits the sun while rotating on its axis is more true, because it explains also why we have seasons. Strictly speaking, however, we will never know whether the Earth really revolves around the sun; another, even truer, theory could conceivably come along.
In support of his view, James pointed out that in practice all scientific theories are approximations. Rarely, if ever, does one theory explain all the facts of experience. Instead, one theory often does well with one set of phenomena while the other theory does well with another set.
A scientific theory that explains more is truer than one that explains less, and the truer theory is preferred. Kuhn might add that even a paradigm that explains no more phenomena than a rival but explains those phenomena better is preferred—as for example Copernicus’ heliocentric model of the solar system was preferred to Ptolemy’s geocentric model, because Copernicus’ model was simpler and more elegant that the cumbersome epicycles of Ptolemy’s model, even though at the time the two models fitted astronomical data about equally well. If scientists prefer theories that explain more phenomena and paradigms that make more sense of our experience more plausibly, then the progress of science no longer seems so unreasonable. It is the result of selection, the exercise of scientists’ preference for theories and paradigms that make better sense of our experience."
Taken from the book "Understanding Behaviorism" by William M. Baum.
A study friend of mine who studied an exact science ended up working in the social science department where he also was asked to teach some research methods class. Suffice it to say that even PhD students are not very good at it. Even at things where you would suppose they would be closer to their core competency, i.e., they were not that good at avoiding the pitfall of putting leading questions in a questionaire.
Does it? Do you have any data to indicate that, or even a good way to define your terms?
Science isn't just a matter of what people believe. It often gets considered when governments create policy. Bad science can kill millions. I've run into many 'bugs' in the algorithm descriptions published in computer science and mathematics papers, I shudder to think what the code backing nontechnical research is like.
1) replication failures
2) scientific errors both accidental (coding errors) and intentional (p-hacking)
I think part of it has to be physics envy, where many disciplines are attempting to adopt either the statistical techniques used by physicists or at least similar language and ideology in order to conduct "experiments" in fields as varied as sociology and psychology.
This seems like cargo cult science. Who says that, say, measuring the tendency of churchgoers to donate to a cause will reveal a truth similar to the mass of an electron -- something that can be accurately measured once and then you know a good measurement will return the same result? It's not at all clear that any real insight is gained of either philanthropy or church attendance from such an "experiment", nor is it clear to me that these fields contain statements with truth values at all, at least truth values as would be expected by mathematicians, physicists, chemists, etc.
But by adopting the methodology of science to fields which may not have yield scientific results to yield, a lot of people are creating a body of fake knowledge, or the appearance of knowledge.
You can't control labels, no one can control what is labeled science or not, we just need to educate people, demand more of our media, and breed a culture that take more pride in truth, accuracy, precision, and intellect.
Sure, I don't disagree, but do you have any path toward this 'enlightenment'? I mean, the average college attending student has a literacy level of about 7th grade. How would you propose we move from there to a point where even the average, much less most, voting age adults can not only digest a paper but also be capable of poking holes in its reasoning?
> You can't control labels
Of course you can, it's what politics and marketing are all about.
I think you're right that they are two different things, but I'm not sure that later isn't science.
And how many of these will cover the retraction? A dozen at most? And all those articles will be sitting out there, getting cited and read on occasion.
This case is interesting because there's a large population who would find these (unproven as it turns out) results confirmatory rather than unexpected.
In the end they were neither.
The history of science is full of drama where most issues took multiple generations to resolve. It's easy to forget, in those interim periods, people would build all kinds of castles on total BS all the time that cost society in so many ways.
Today stuff gets resolved faster and that's a good thing. People, qualified or not, who get carried away by hype or bias look foolish much much faster. And thanks to how hard it is to erase mistakes from the internet good luck rebuilding lost cred.
Study Shows Non-Religious Kids Are More Altruistic and Generous Than Non-Religious Ones (TIME)
Religious children more punitive, less likely to display altruism (Boing Boing)
Study: Religious children less altruistic, more mean than nonreligious kids (Chicago Sun-Times)
Religious Kids Aren’t as Good at Sharing, Study Finds (Yahoo Parenting)
Study: Religion Makes Children Less Generous (Newser)
Religion doesn’t make kids more generous or altruistic, study finds (LA Times)
Religious children are meaner than their secular counterparts, study finds (The Guardian)
Being religious makes you less generous, study finds (metro.co.uk)
Kinder Without God: Kids Who Grow Up In A Religious Home Less Altruistic Than Those Without Religion (Medical Daily)
Surprise! Science proves kids with religious upbringings are less generous — and so are adults (rawstory.com, the Medical Daily story reposted with a new headline)
Why don't reporters get angry about this and demand more and better retractions? This hurts their credibility more than anyone.
False reporting of any sort (including the failure to correct prior false reports, whether intended or not) always causes harm. Truth and principles matter.
Furthermore, reducing ideas to power struggles between those who speak them is to commit both the genetic fallacy and the ad-hominem fallacy.
Because ideas are the substrate of thought, belief, motivation, desire, and human life itself. Without ideas we lack the ability to understand anything at all.
Because it is true, or someone believes it to be so.
Because it carries explanatory power - something we all strongly desire.
You might call those reasons “purposes” - if so, I don’t disagree. But the truth or falsehood of an idea transcends any “purpose” someone may have for speaking it. This is the academic posture: to dispassionately evaluate truth claims without fearing the speaker.
That doesn't stop us from acknowledging the fact that ideas and worldviews have strong ties and are intermingled enough that they almost always warrant an underlying motivation whether that be a noble search for truth or a way of digging further into denial.
"Then Abraham said to him, 'If they do not listen to Moses and the prophets, they will not be persuaded even if someone rises from the dead.'" (Luke 16:31)
It may be true that reading about the study would not have convinced anyone to become a practicing Christian. But there were undoubtedly people who were on the fence about going back to church, either for themselves or for their kids, who decided not to based on the purported conclusion that it made kids less generous. So just because the study alone might not have convinced anyone to follow Christ, there still might be more practicing Christians if the study had been done correctly (and therefore garnered very little coverage) or if the correction had been covered as widely as the original flawed study.
Wonder if it'd be possible to automatically scan new papers added to (say) arxiv.org, for retracted papers in their references?
eg to warn the authors, and maybe eventually automatically as part of the upload process for arxiv.org (and similar)
In sum, Decety et al.  have amassed a large and valuable dataset, but our reanalyses provide different interpretations of the authors’ initial conclusions. Most of the associations they observed with religious affiliation appear to be artifacts of between-country differences, driven primarily by low levels of generosity in Turkey and South Africa. However, children from highly religious households do appear slightly less generous than those from moderately religious ones.
Although Decety’s paper had reported that they had controlled for country, they had accidentally not controlled for each country, but just treated it as a single continuous variable so that, for example “Canada” (coded as 2) was twice the “United States” (coded as 1).
I mean I don't even understand how this seemed like a normal thing to do?
This mistake would be downright trivial to make in R. Just declare that Country is a Factor (which is the built-in type for categorical variables), and then throw the data into a library whose attitude towards errors is to coerce everything to numbers until the warnings go away.
Background: Factors in R are the idiomatic way to work with categorical data, and they work somewhat like C-style enums except the variants come from the data rather than a declaration. So if you take a column of strings in a data frame and cast it to a Factor, it will generate a mapping where the first distinct value is coded as 1, the second distinct value is coded as 2, etc. Then it replaces the strings with their integer equivalents, and saves the mapping off to the side.
I forget the exact rules (if there are rules, R is a bit lawless), but it's not very hard to peek under the hood at the underlying numeric representation. Many built-in operations "know" that Factors are different (e.g. regressing against a Factor will create dummy variables for each variant), but it's up to each library author how 'clever' they want to be.
...strong typing: for or against?
(To be fair even strong typing won't save you if you don't use it. But fuuuuuk, what an error. I noted that paper mentally and would have quoted from it)
Many algorithms work on both categorical and continuous variables, with different results depending on the variable's type.
I.e., if you're controlling for country, that means you're bucketing by country, and looking at each subset, right? So if country is represented by a non-discrete value... what exactly happens?
Statistically, if you treat them as a continuous variable, the estimates you get will act like there's an ordering there, and give you the effect of a one unit increase in tree. So it will tell you the effect of Oak vs. Maple and Maple vs. Aspen, assuming those are proportional and that Oak vs. Aspen will be twice that.
This is...nonsense, for most categorical variables. They don't have a nice, ordinal stepping like that.
In practice, if you have n countries, you'll add n-1 binary variables to your regression equation. The first country is the reference level (all zeros), for the second country set the first new variable to one, the rest to zero, etc.
- people with no coding experience and, in some cases (especially in social sciences), a strong aversion to math
- code that isn't unit tested, so answering the question, "Did it run correctly?" is often softened into, "Does this look plausible to me?"
- a strong incentive to end up with certain results
I dated a quantitative geneticist for a while, and her coding education was almost zero. She was writing code in R and essentially just changing lines until the output "looked right". It was insanely complicated math, so there was no way to make sure the output was good. The code had to be an exact match for the algorithm she had written out in mathematical notation, and there was essentially no chance of that.
It got worse. She'd write the algorithm in R and then end up with batches that would take, in some cases, years to finish running. Obviously she ended up with even more dubious hacks.
(For anyone curious, she's had a fairly decorated academic career under an acclaimed advisor who reviewed all of this code to some extent, and she's worked with most of the top genetics programs in the US.)
I teach an introductory stats course and we hammer this in. Categorical data are often represented as numbers or other short indicators for storage purposes. Typically I fmultiple choice the encoding is by the order of the choice options.
I not infrequently see average of gender because male = 0 and female = 1 or vice versa and someone generates a table without thinking.
I think a major contributing factor to problems like this is people going into the soft/social sciences being more likely to be math/stats AND programming averse. Meanwhile, all sciences continue their long term trend towards applied math via programming. This leads to people using the math/stats via code without understanding very well what it is they are doing, and, naturally, the end result is lots and lots of mistakes.
On topic, from yesterday: https://news.ycombinator.com/item?id=21067764
It's another social sciences paper but in this case a co-author has requested a retraction over his strong belief that the paper includes fabricated data. The retraction request has been denied. It differs from this paper in that the data anomalies look intentional.
One interesting thing you can do though, is sort by zipcode. This sorts your mail from East to West in the US. You can use that as a rough estimate of shipping time.
Anyway you can still sort strings.
to the credit of the authors, they released their data sets. -- however, i suspect that proper data exploration and visualisation would have prevented all this. visual inspection would have most likely revealed that there is no visible effect, or even an effect in the opposite direction, and once you see this, all alarm bells should go off if your model predicts otherwise. so i suspect that the authors skipped some basic steps and got carried away by results that promised a nice headline.
Which, is one reason why having higher and higher percentages of academia and science researchers be from the same part of the political spectrum, is worrisome to me. If you have more diversity in ideology, there is more likely to be someone in each field to have the instinct to scrutinize closely a result which, when scrutinized, won't hold up.
How did this not get caught immediately? If I did a study and found out that kids in Zambia are 47 more times as generous as American kids that'd make me instantly suspicious.
Or maybe the reviewers were all Canadian /s
They were trying to correct for scenarios like this: Hypothetically, Canadians are twice as generous as Americans and twice as religious, but religious Canadians are equally generous as non-religious Canadians and religious Americans are equally generous as non-religious Americans. On the surface, it appears that religious people are more generous, but really it's just that Canadians are more generous.
Instead of treating the countries as discrete groupings, they treated them as points on a spectrum with each country being assigned an arbitrary place on the spectrum.
If #3 happened to be China, they would be assuming that people in China should very similar to people in the US and Canada, because 1 vs. 3 on a scale that goes to 200 is hardly any difference at all, but really the numbers are just arbitrary identifiers.
I don't mean to imply that there aren't moral a-religious educators of course, just that it seems likely to have less discussion of ethics for any kid once the temple/church/mosque/whatever is removed.
Furthermore, this whole field was called cargo cult science by Feynman, using correlations, p-hacking, data dredging, and more
The best analysis I have seen of the corner we have painted ourselves into
Science is to me something where you extract natural laws that predict phenomena will occur 100% of the time given certain conditions. Physics, chemistry, computer science, most branches of medicine operate this way.
A field like psychology that says "well sometimes people will..." or "we found in 60% of cases that..." is not science. It's a comment upon society maybe, but it does not produce broadly repeatable, predictable results.
Not true science.
The bottom line is science is hard to do, and even harder to do right. I think all fields do try to follow the scientific method to the best of their ability, but for some fields it is simply harder to do because of the sheer complexity of the subject matter being hard to model, experiment in, and understand.
No they don't.
For the uninitiated: If you want to publish a scientific paper today, you basically sign up to sign over the rights to any publisher that's interested (please someone publish me). That publisher will then review that paper in a more often than not mostly undisclosed process and publish that paper. Everyone knows that to be a high-regarded publisher one must have a very own typographic formatting: Unreadable font, weird multi column layout to prevent accessibility and tables disregarding standards are a good start. Afterwards, the paper gets published on the publisher website, which again follows as little agreed upon standard as possible. Data is of course excluded, study itself is in PDF. Also put up a fat paywall, don't want those pesky poor people be scientifically literate. Give a few cents of your $70 fee to the authors, it is not an unethical business you're doing here!
THIS is the systemic error people seem to be so happy to ignore because they're neck deep in social science not being real science memes.
This prevents interesting new startups for fact-checking or meta analysis, which are e.g. happening in journalism because that field has a lot of the things science is sadly lacking.
This creates a drift between extremely rich and rather poor countries/unis/humans in scientific ability.
This generates a tar pit for scientific process.
This wastes billions in funds because of people unaware of each other doing redundant studies (and not referencing/refuting/supporting each other in the process neither, of course) because they are literally better search engines to find Harry Potter fan fiction than there are for finding studies.
And finally, this of course allows anything from honest statistical mistakes to snake oil sellers to slip through and doing generations worth of damage, because correcting, fact checking, re-researching, comparing, meta research, anything is slowed to a crawl.
So please stop embracing scientific elitism and gatekeeping for this is exactly what brought us here in first place...
What an apt name.
I wonder about the damage to public this unintentional deceit will bring...
Go science! After all, a retraction is part of discovery.
Disclaimer: I am a religious person.
“Media articles” is carrying a lot of weight there
The problem, really, has been accelerating for a while. Something really went off the rails after WWII.
If you read the OP article, the research suggests religion has positive effects on children. Your response to the corrected findings?
What would I be classified as if I started worshipping, say, Thanos, and began preaching to everyone about him?
> the research suggests religion has positive effects on children. Your response to the corrected findings?
If it was safe for me to do so, my response would be to go out in my own religious country and film for you all the children throwing stones at animals (which they're encouraged to do so by their elders), all the beggars sleeping next to literal mounds of garbage, shunned by everyone, and the many other problems that religion is supposed to fix.
Liking deities from various fictions is not the same as seeking to open oneself to God.
Curious: why create a new account to post this comment? (unless it's your very first account on HN)
What would be your argument and which experience is it based upon?
> not something you'd say in mixed company in real life either.
Some people here criticize religion in private company, but would not dare to do so publicly, because of the very real threats to their very life they would face precisely because of religion.
But religion as a whole doesn’t have to be bad. Many religions preach tolerance for others—one of their primary missions is to eliminate the very hate and inequality you have spoken of. Just look at what some religions do to help people because they believe it’s something their God would like them to do: https://newsroom.churchofjesuschrist.org/article/humanitaria...
Again, I’m really sorry to hear that religion has been used to oppress rather than to bond in your country. But the right kind of religion (as this corrected study seems to show) actually improves communities. I hope you can find a solution. :-/
Theology, not so much.
However, because it's basically speculation about unobservable entities, theology completely lacks external validity.
Social sciences can have both.
Science is about constantly evolving via falsification of theories.
You might want to base your definition of what can be science on something different than an unsourced appeal to authority from 300 years ago?