You could publish it in the Journal of Trial and Error (https://journal.trialanderror.org), which I created with a number of colleagues a couple years ago!
While it’s easy pickings, it’s still always worth pointing out the hypocrisy of Nature publishing pieces like this, given that they are key drivers of this phenomenon by rarely publishing null results in their mainline journals. They are have extremely little incentive to change anything about the way scientific publishing works, as they are currently profiting the most from the existing structures, so them publishing something like this always leaves a bit of a sour taste.
Providing _places_ to publish the result is only part of the problem. The other part is incentivizing scientists to do so. And similarly, Nature itself is responding to incentives. The core problem is that scientists themselves, the individuals, do not mostly display much interest (in a revealed preference kind of way) for null results. If scientists were interested and wanted to read articles about null results, then either journals like Nature would do so, or the numerous examples of journals like yours that have come and gone over the years would have been more succesful and widespread.
Because of this revealed lack of interest, high tier journals don't tend to take of them (correctly responding to the lack of demand), and journals like your that specifically target these kinds of articles A) struggle to succeed and B) remain relatively "low impact", which means that the professional rewards to publishing in them are not very high, which means that the return on effort of publishing such a work is lower.
Don't get me wrong, the scientific community could do a lot more to combat this issue, but the core problem is that right now, the "market" is just following the incentives, and the incentives show that, despite what the stream of articles like this one over the past few decades is that most scientists don't seem to actually have an interest in reading null-result papers.
What if to each publication of a non-null result, academics are given the opportunity to publish their nulls as well, if only as a appendix or better, a counterpublication to their main conclusions? I don't buy the argument that papers need be of max-n length, now that documents and journals can be easily stored and distributed.
I would love something like Living Papers [0][1] to take off, where the null an non-non results could be compared interactively on similar footing.
A null result may be a dead end and so there is no related paper worth publishing it in.
A null result should be published right away in a searchable place, but probably isn't worth a lot of effort in general. I tried X, it didn't work, here is the raw data.
All of that - the setup, the instruments, dependencies, methods - should be pre-submitted to the journal before the experimental results arrive. The journal should be the one that uses the data from the experiment and runs your pre-submitted program over the data to produce a result.
I agree that in an idealized way, this would be much better. But what do you do about going through all this process and ending up with a bad reviewer?* In those cases, how would you handle re-submitting to a different journal without looking like you're creating those artifacts after-the-fact to suit your outcome? Would the pre-submittals need to be handled by some third party?
* the current process still has a lot of luck in terms of getting assigned referees. Sometimes you just plain get a bad reviewer who just can't be bothered to read the submission carefully and is quick to reject it. I would hate to see a system that only allows for a single shot at publication
The review happens at the experimental design stage, not the publishing stage. Very little actual work has been done at that stage. After such a review, the journal agrees to publish your results regardless of outcome.
Right now you don't even know who will publish you paper until all that is done. Your experiment might be try some promising molecule/drug in a petri dish, and see what happens, if the results are amazing you will get in a different journal than if the results are something happens but the control molecule/drug is better.
From the article: “A 2022 survey of scientists in France, for instance, found that 75% were willing to publish null results they had produced, but only 12.5% were able to do so.”
The question is how many of them are willing to review and read these publications. Of course as an individual scientist (not me, but someone who does experiments), I'd love to capitalize on my work, even if it is unsuccessful (in the sense of null result), by publishing it. But do I, and scientific community in general, care about null results? I'd say mostly no. Null results, if universally published, would overwhelm already overwhelmed publication system.
If you think it will be helpful to others to know about specific failure, put it in a blogpost or even on arxiv. Or talk about it at conference (for CS, workshop).
Also, if we use publications as a measure of scientists success, and we do, is a scientist with a lot of null results really successful?
Obviously most scientists are not going to be interested in null results from adjacent subfields, but when it comes to specific questions of interest it is absolutely useful to know what has been tried before and how it was done/what was observed. I know a lab that had documentation not only on their own historical null results but also various anecdotes from colleagues' labs about specific papers that were difficult to replicate, reagents that were often problematic, etc.
That is a non-ideal way for the scientific community at large to maintain such info. Trying to go through traditional peer review process is probably also non-ideal for this type of work though, for reasons you cited. We need to be willing to look at publication as something more broadly defined in order to incentivize the creation of and contribution to that sort of knowledge base. It shouldn't be implemented as a normal journal just meant for null results - there's really no need for this sort of thing to be peer reviewed specifically at the prepub stage. But it should still count as a meaningful type of scientific contribution.
In the old days, Science Weekly[1] used to print 4-5 paragraph summaries of published research in a three-column layout. The magazine was dense with information across a huge number of topics.
And in the very old days, newspapers used to publish in tabular form local election results and sports games.
I feel that Nature could dedicate one to two pages of one paragraph summaries of null results with links to the published papers.
It's amazingly easy to skim such pages to find interesting interesting things!
[1] I think that was the name; I canceled my subscription when they changed to a Scientific American wannabe. I was looking for breadth not depth! I could always get the original paper if I wanted more information.
I agree incentivization is definitely a big part of the problem, but I think in general a bigger issue is that as a society we tend to reward people who are the first to arrive at a non-null result. This is as true in science as much as in any other area of human endeavor.
Years ago, I came across SURE: Series of Unsurprising Results in Economics with the goal of publishing good, but statistically insignificant, research.
I thought "statistically insignificant" meant we couldn't conclude anything. So I was surprised.
[1] says:
> In statistical hypothesis testing,[1][2] a result has statistical significance when a result at least as "extreme" would be very infrequent if the null hypothesis were true
So I understand this journal publishes results for which a hypothesis was tested, found to give insignificant results, which would rule out the hypothesis assuming the research was correctly conducted, without biases in the methodology, with a big enough sample, etc. Which would be worthy to know but no journal usually takes this research because it doesn't make the headlines (which yes, I've always found was a shame).
Yes, statistical insignificance doesn't "prove" the null hypothesis, it just fails to reject it. It's a subtle, but sometime misunderstood distinction. It's a measure of how big the effect size is and how often you'd expect to see it just by chance rather than due to the variables you're measuring. If it's a really extreme difference, we expect it to happen less often just by chance alone than if it's a really miniscule difference.
>Which would be worthy to know but no journal usually takes this research because it doesn't make the headlines.
That's usually correct, which gives rise to all kinds of issues like the article talks about. It can result in a lot of wasted time (when you're conducting what you "think" is a new experiment, but it's been done many times but unpublished because it doesn't provide statistically significant results). It provides little incentive for replication, which can lead to stronger conclusions about the results than may be warranted, etc.
The flip side of this is that there is almost always a very small effect, even if you are testing a crazy hypothesis (there are very weak correlations between all sorts of things). So you can often get a ‘significant’ result just by using a huge sample, even though the effect size is too small to matter practically.
Omg I love this. For like 20 years I’ve joked about “The Journal of Null Results and Failed Experimenrs” and it looks like you and your friends are actually doing it.
Disagree, as we also publish “failed” research, where authors reflect on their experiment, such that others may learn from it, and that the others may still gain something useful academically from this (citations, a publication).
One of our goal is to change the perception of and culture around failure in academia. Research/science is not just a steady upward trend of progress, it comes with a lot of trial and error. Academics’s success and job prospects however depend mostly on them publishing in high impact journals, which in turn only publish “interesting” aka positive results, which creates this very toxic publish-or-perish culture. Having an experiment fail is a natural part of doing science, but academic institutions punish you for not producing positive results.
By providing a place to publish these failed experiments, at least it provides some relief for this problem. This is not real change however, that needs to happen at a much higher level, but that is one we do are not able to impact. Ideally our journal will not be necessary in the future, as we detail in our opening editorial: https://doi.org/10.36850/ed1
I see. I guess I see part of the problem you're trying to solve as reflected in the language of "failed" and "error" as opposed to framing say non-replication of a prior false positive as a "correction" or "additional evidence" (against a prior false positive). It may not matter to everyone but some funders might wince as research they've funded as being "failure" or "error." Just something to consider.
If you publish null results you accelerate the development of competing hypotheses by your competition. It's best to make sure they waste as much time as possible so you can maintain an edge and your reputation. /s
so trueeee! science is ofc a zero-sum game, and we should stifle competition and collaboration as much as possible. how else will i get that 3rd citation on my phd thesis? /s
The article hints at this, but not publishing null results (at least in a database - somewhere!) goes hand-in-hand with the replication crisis. An experimental outcome is always a single sample from a distribution of outcomes that you would obtain if you repeated the experiment many times.
Choosing to only publish the most extreme positive values means that when the experiment is replicated, "regression to the mean" makes it very likely that the measured effect will be weaker, and possibly not statistically significant. This is not an evidence of scientific fraud -- rather, it is a predictable outcome of a publishing incentive scheme that rewards hype and novelty over robust science.
I've said it before but it bears repeating - replicating published results, and adding the findings to a database, should be a standard part of PhD training programs.
There is a major issue of limited resources to replicate results, in terms of both time and funding. For example: I would assume that most important results are replicated. As a concrete example, if someone identifies a medication that (in one small trial) shows a statistically significant effect in curing some serious medical condition, then this will drive further replication attempts. On the other hand, if someone publishes a study showing that holding a pen in your mouth makes you 1% likelier to do well on the PSATs, this study will probably languish without replication for a decade — because honestly who cares? It’s basically a curiosity. I can’t help but notice that many of the headline results that characterize the “replication crisis” were small-effect-size social science experiments that fundamentally weren’t that important outside of popular science news.
I’m not saying that our current allocation of resources is optimal. I am pointing out that our resources are finite and “replicate everything” is not even a remotely practical allocation of those resources.
> I would assume that most important results are replicated.
GP is pointing out that the incentive structure makes this an invalid assumption. If publications reward hype and novelty when deciding what to publish, then there is no point spending your limited resources replicating other peoples' results, they won't get published anyway. And experiments that give a null result won't be published anyway. What's left are one-off results that showed something surprising simply by chance and don't replicate...but then, we generally will never know that they don't replicate, because the replication experiment is not novel, has a low chance of being published, and hence isn't worth spending limited resources on.
Basically the publication process introduces selection bias into the types of research that are even attempted, which then filters down into the conclusions we take from it. A cornerstone of the scientific method is random sampling, but as long as the results that get disseminated are chosen by a non-random process, it introduces bias.
>“replication crisis” were small-effect-size social science experiments that fundamentally weren’t that important outside of popular science news.
I don't know that this is accurate. Some of these make their way to large-scale public policy, or give bona fides to people who craft far-reaching policy. This includes changes to 401k allocations to car-insurance rates and other mundane, but consequential policies.
The truth is most of science is not important outside of popular science news. So we shouldn't be surprised that the bulk of replication crises are also in the same category. Claiming this means the replication crises is not really impactful may be a case of base rate neglect.
It's also important to note that your example of medical replication is a relatively highly regulated area, where most other science is much less so.
> There is a major issue of limited resources to replicate results, in terms of both time and funding.
Undergraduate students love being involved in research. It's one of the selling points of many top universities. Grad students replicate research all the time. Maybe funding is an issue in grant fields (some research is extraordinarily expensive) but that doesn't excuse the lack of replication across the board.
I’m a researcher at an R1 and I love working with undergraduates. But it’s not true that they’re a huge source of additional resources. Quite the opposite: like any new researchers they have to be carefully supervised, which means they can actually consume a lot of time from PIs and PhD students (you can send them off on their own without supervision, but that’s how you get low quality results.) Unlike PhD students, however, undergrads tend to graduate just as they’ve finally learned the ropes. It’s still worthwhile because a few of my undergrad research interns did great work, and at least a couple of them applied to PhD programs (at least two in my lab; they’re both Assistant Professors at top CS departments.) But the answer to a resource allocation problem is almost never “here’s a simple idea that makes the resource limits go away, why aren’t people doing that.” Unfortunately.
Do 20 experiments with a p<5% criterion, and it’s likely that one will be a false positive. Only publish positive results, and someone will eventually publish a false positive result without fraud.
Very few studies report a single statistical test as the sole conclusion. Most papers should assess some outcome in multiple ways using complementary data, multiple analyses etc. not always of course, but there are lots of ways of making sure your conclusions a robust without relying on a single analysis result.
That doesn't fix the problem at all. No matter how many statistical tests you run on a sample, you can't get around the fact that the sample may not be representative of the population or the underlying phenomenon.
You need different samples. There isn't a statistical trick that gets around this.
For example: let's say there's a cancer with 20% survival rate. You test a treatment with 25 experimental and 25 control patients, 40% in the experimental group survive[1].
You can analyze this with a bunch of statistical methods. You can ask different questions about the patients, focusing on well-being rather than simple carcinogenic remission. But ultimately, the thing that happened in this study is that 50% got better and no fiddling with numbers or changing the questions you ask is going to change that underlying phenomenon. You can check for blood markers of cancer: you get 50% have no blood markers. You ask them questions about how they feel: you get 50% feel better. You body scan the area where tumors were: you get 50% no longer have tumors.
You have only tested one phenomenon in one sample, and that essentially amounts to 5 people getting better.
[1] I know this is not how cancer treatment studies work exactly, this is a simplified hypothetical.
That’s not what I’m saying. Obviously you can’t run multiple tests on one set of data. If you have a hypothesis, test it on multiple data sets in multiple ways and find supporting evidence rather than finding a single p=.05 and writing a story around it.
It's a great xkcd but it's really wrong on one count, it's not some outside media/popular force compelling scientists to investigate something. Most of the time, researchers are looking to prove something they "know" to be true. They truly believe that jelly beans cause acne, they just need to prove it. When they get a negative result, they simply don't believe it. Something must have gone wrong, obviously, because jelly beans obviously cause acne, so maybe it's a color thing? Ah hah it's the green ones, now that we have our results we can construct other metrics to support this correct data!
Eventually if we (the public) are lucky someone else in the field will disagree and run the trial again, which is how you get the alt text.
Yes, I find that when reading a paper I think to myself "Do the authors really want this to be true?" and if the answer is yes as it often is, I boost my own acceptance criteria to p<.03
Particle physics still uses five sigma as the significance threshold.
If 20 different people all conduct the same experiment, and the 19 negative results are never published, there is no fraud involved when the 20th person publishes his positive result without realizing it is a 5% statistical anomaly. That person probably has no idea that 19 other people tried and failed at near-identical experiments.
This seems like such an obvious problem in the way science is currently done. Are people so focused on their own individual fields that they aren't thinking about and fixing such glaring meta problems?
As others already pointed out, there is no incentive to do so.
Consider: "Hey, look, I went on top of tower of Pisa and threw down two identically shaped balls, one iron and one wooden. They dropped at the same time!"
The above is the expected result and would only be interesting if the result is different from expectation. Now, if 1000 scientists did this and each published the confirmation of what we knew would happen then who would read that? But, if one scientist said: "I tried and they drop at different times!" that would be different. The 999 scientists would then try to replicate again and then the papers of the 999 would be interesting again.
>For example, if 1,000,000 tests are carried out, then 5% of them (that is, 50,000 tests) are expected to lead to p < 0.05 by chance when the null hypothesis is actually true for all these tests.
Even if you are right, how would this response be helpful? You're not giving the right answer, you're just saying the answer given is wrong. Nobody is coming out of this interaction with corrected knowledge in their head.
This is the sort of thing I used to do when I was younger, and looking back, the reason I did it was because I was basing my sense of self-worth in being smarter than other people. Ironically, this made me dumber, because I was less open to the possibility of being wrong and therefore was slower to learn. And, I found doing this made people dislike me.
> This is not an evidence of scientific fraud -- rather, it is a predictable outcome of a publishing incentive scheme that rewards hype and novelty over robust science.
And knowing all this, this behavy should be considered borderline scientific fraud at this point.
>> I've said it before but it bears repeating - replicating published results, and adding the findings to a database, should be a standard part of PhD training programs.
Wait, why should PhD students do that work? That just sounds like pushing more grunt work to the lower rung of the academic hierarchy.
Nope. If you want people to do that kind of work that is important to everyone but is not directly conducive to promoting one's research career then the solution is simple: pay them.
I think there is some reasonable argument that replicating research is the first step to learning how to do good research on your own. In an ideal world, PhD students should probably be trying to replicate similar work anyway and applying existing approaches own pet problem. In practice, many gloss over this because they are narrowly focused on doing something "new" so it can get published.
PhD students are in training, and replicating a published result is a great training exercise. PhD students ARE paid. But this work won't be prioritized by their PI unless it's also a requirement of the program.
>> PhD students are in training, and replicating a published result is a great training exercise.
I just got my PhD last July and that sounds like boring drudgery rather than "good exercise". Good exercise is to have a student write their own paper, dig up their own references, formulate their own experimental hypotheses, run their own experiments, write up their own results etc. Re-doing someone else's possibly badly-done work is grunt work that should not be forced upon anyone.
Snark aside, replication is a cornerstone of science. If someone doesn't want to be involved in science because they think it's soul-crushing, perhaps academia isn't the right place for them.
In its scriptures/philosophy, science describes extremely thorough and sound principles and guidelines...but in on the ground practice (by scientists, which are a part of "science"), they are often not achieved[1]. However, this distinction is not only not advertised broadly and without aversion, it is usually (in my experience) not mentioned at all, if not outright denied using persuasive rhetorical language (like, for example, when an object level instance of not achieving it is pointed to in the wild, such as in forum conversations). This may not be fraud (that requires intent I think?), but it achieves the same end: misinforming people.
I absolutely agree with your database idea, and if science would like me to take them seriously (something near how seriously they take themselves) they'd also have to go much further.
[1] Not unlike in religion, a competing metaphysical framework (model of reality) to science.
This sort of comment is why I think a lot of philosophy is just communicating poorly to make yourself sound smart.
In your footnote, for example, you translated your philosophy-speak into English (metaphysical framework -> model of reality). Why not just say that? Your entire comment goes into "philosophy mode" and communicates a few very simple ideas in overcomplicated language.
Science and religion are pretty poorly understood as competing models of reality. Religion originates when people make up answers to other people's questions to gain social standing, and religion continues due to (among other things) anchoring bias--the bias people have toward continuing to believe what they already believe. While religion does result in those people having a model of reality, there is no attempt being made at any point to relate the model to reality. When religious people and scientists disagree, it's not because the religious person is trying to model reality differently--the religious person isn't even trying to model reality--it's because the religious person is biased in favor of their existing belief.
You said:
> In its scriptures/philosophy, science describes extremely thorough and sound principles and guidelines...but in on the ground practice (by scientists, which are a part of "science"), they are often not achieved[1].
This is presented as some sort of gotcha, but it's not: few scientists will claim that science is being practiced perfectly or even well. Outside of a few areas such as particle physics, we're quite aware that our ability to practice scientific ideals is hampered by funding, publication incentives, availability of test subjects in human studies, data privacy, etc. And we're aware that this means that our conclusions need to be understood as probabilities rather than 100%-confidence facts.
There are certainly some people who treat scientific conclusions with religious absolute confidence, but doing that is fundamentally against scientific principles. The accusation you are leveling against science would be better targeted toward people: generally science journalists and the science-illiterate public rather than scientists themselves. The entire reproducibility crisis is scientists using science to show that our practice of science is too imperfect to result in high-confidence conclusions.
Religious people jumping on the replication crisis because they think it disproves science is rich. The replication crisis isn't a disproof of science, it's an application of science. The reason we know that there's a replication crisis is because scientists asked "How confident can we be in the conclusions of existing studies?" and applied science to answer that question. If you really think science is invalid, then you can't use science to prove that.
And the fact remains that any confidence in conclusions at all is more than religion has to offer, because again, religion isn't trying to model reality--the fact that religion produces a model of reality is merely an unfortunate side-effect.
> While religion does result in those people having a model of reality, there is no attempt being made at any point to relate the model to reality. When religious people and scientists disagree, it's not because the religious person is trying to model reality differently--the religious person isn't even trying to model reality--it's because the religious person is biased in favor of their existing belief.
A problem: you're talking to one right now, and you (your mind's model of reality, technically - you do not have access to the state of the things you claim to) could hardly be more wrong.
From large quantities of experience, I am confident I would have no success tackling your disagreements on a careful, strict, point by point basis. Instead, I will simply present two links (I have many others, but let's see what happens with these) and ask: do you believe these have some substantial relevance here, related to the truth value (and appearance of) of our respective claims?
I can appreciate that this approach may seem unworthy of anything more than a rhetorical response that dodges the question (and the importance of the phenomena these links discuss), so hopefully you can take the challenge seriously. I am more than happy to offer a more substantive reply later, but if you declare victory by fiat[1] it's a bit tough to have a serious conversation.
[1] Roughly: declaring that one's opinion of the unknowable is necessarily correct, and that it is(!) contrary to my (actual) stance.
Anything you're going to say to try to show that religion is trying to model the world is going to be an attempt to use logic and observation, because those are the tools that work, and you know those are the tools that work. You're just using them with an end goal in mind due to aforementioned anchoring bias, which isn't really the way to use them.
What you probably aren't going to do is tell me to pray and God will tell give me the answers, or tell me to read a book written by ancient people who believed stuff like "the sky is a dome", because you're well aware nobody takes that seriously as a way of modeling reality. I mean, maybe you are going to tell me to do those things: if so, that's embarrassing.
And the latter is what religion has to offer for modeling reality that science doesn't have. Sure, religion uses logic and observation because they so obviously work, but science has logic and observation too, and frankly, is better at it--we don't need religion (or philosophy) to tell us how to logic or observe.
So if you really want to present religion as a way to model reality, that's what you've got to prove has value for modeling reality: talking to invisible friends and consulting outdated writings. We're already convinced that logic and observation work: how do your weird additions compete with that?
> A problem: you're talking to one right now, and you (your mind's model of reality, technically - you do not have access to the state of the things you claim to) could hardly be more wrong.
A problem: you're talking to someone who used to be religious, so I have as much access to the internal thinking of a religious person as you do.
A second problem: people's self-perceptions of their own internal processes are quite often measurably wrong.
> From large quantities of experience, I am confident I would have no success tackling your disagreements on a careful, strict, point by point basis. Instead, I will simply present two links (I have many others, but let's see what happens with these) and ask: do you believe these have some substantial relevance here, related to the truth value (and appearance of) of our respective claims?
From large quantities of experience, I would guess that you're about to make a special pleading argument that based on convenience beliefs that you yourself don't believe in any other context, as evidenced by the fact that you don't practice them.
Those pages, particularly the latter, are another example of poor communication being presented as intelligence. If we translate to English instead of philosophy-speak, it boils down to an argument about whether perception is reality or not.
Let's cut to the chase with a relevant parable:
The Buddha and his disciple were walking down the road. Suddenly, the disciple drew his sword and cut the Buddha in half at the waist. The Buddha turned to his disciple and said, "Now you're beginning to understand!"
Would you be willing to reproduce this parable experimentally with you as the Buddha? After all, perception is reality, so if you're the Buddha and you perceive being cut in half with a sword as no big deal, that will be just fine, right?
The thing is, science is perfectly capable of answering this question--it's not unknowable. The experiment of cutting someone in half with a sword has sadly already been performed too many times in history: we don't need to perform it again. The scientific answer, which we already have, is that no amount of changing our perception prevents the person cut in half with a sword from dying in horrible agony. And when you're not speaking philosophese, you already believe the scientific answer just like every philosopher who believes perception is reality until faced with the prospect of being cut in half with a sword. So if you're about to make an argument about direct and indirect realism, I'd have to ask, why do you believe that reality is reality when it comes to swords (and everything else in your day-to-day life), but you suddenly you want me to believe that perception is reality when it comes to your invisible friend?
My only opinion of the unknowable relevant to this conversation is that by definition, neither of us knows it.
More parts of philosophy I think we can discard without losing anything of worth:
1. Arguing that perception=reality when it's convenient and refusing to practice it in any other context.
> A problem: you're talking to someone who used to be religious, so I have as much access to the internal thinking of a religious person as you do.
Just one way you've missed the mark: you are speaking as if your access to my mind is as good as it is to yours.
> Short answer: not in any interesting way.
To you, or to all people?
> From large quantities of experience, I would guess
At least here you realize you're guessing.
> that you yourself don't believe in any other context, as evidenced by the fact that you don't practice them.
How could you possibly know this?
> Those pages, particularly the latter, are another example of poor communication being presented as intelligence. If we translate to English instead of philosophy-speak, it boils down to an argument about whether perception is reality or not.
Here you are speaking as if you perceive reality as it is directly, no interpretation, no bias.
> Let's cut to the chase with a relevant parable:
> After all, perception is reality, so if you're the Buddha and you perceive being cut in half with a sword as no big deal...
It may be relevant, but it certainly isn't representative.
> ...that will be just fine, right?
Incorrect. Might you be thinking in binary?
> The thing is, science is perfectly capable of answering this question--it's not unknowable.
"Science" only has volition through scientists (or "scientific thinkers"), and Humans very often are not able to not know things (particularly educated Humans at times....a little knowledge is a dangerous thing as they say). Studying science does not turn one into a Perfectly Rational Human, after all.
> The experiment of cutting someone in half with a sword...
Look at you milking this incorrect strawman!
> you already believe
You can not read my mind. You are literally hallucinating.
> So if you're about to make an argument about direct and indirect realism, I'd have to ask, why do you believe that reality is reality when it comes to swords (and everything else in your day-to-day life), but you suddenly you want me to believe that perception is reality when it comes to your invisible friend?
Please, tell me who "my" "invisible friend" is, Human. Tell me in vivid detail, I would really like to know this about my actual, not indirect self.
> My only opinion of the unknowable relevant to this conversation is that by definition, neither of us knows it.
Note that this itself is an opinion (an opinion of an opinion).
It is also a tautology, which in this case I would classify as cheating (you are making claims to know the unknown, and backing it up with a tautology).
> More parts of philosophy I think we can discard without losing anything of worth
When you say "think", I'm curious what you mean. Could you (after the fact, giving you an advantage) write out a pseudocode representation of the chain(s) of logic (and the axioms & premises upon which it is based) you ran to arrive at that incorrect conclusion? Because I have a sneaking suspicion there might be some heuristics and bad axioms(!) in the cognitive pipeline.
> More parts of philosophy I think we can discard without losing anything of worth:
> 2. Talking about the unknowable as if we know it.
Ok, this is too much. Have you been putting me on this whole time?
I'll shoot the fish in a barrel in your followup later on (I am "posting too fast"...what on this site isn't at best a half truth?).
1. If you don't want me to speculate about your point, make your point. I'm not going to participate in you trying to "teach" with the Socratic method of a bunch of leading questions that try to make answers you don't like impossible.
2. I said I was guessing, and it's possible my guess war wrong. But frankly, nothing you've said actually indicates that my guess of your point was wrong--it simply indicates that you didn't understand my post. So I'll ask you point blank: do you believe perception is reality? A simple, "yes" or a "no" will suffice, no need to flower it up with philosophese.
3. You say you're a religious person. Which religion? I doubt you believe the vague, slippery, undefined religion that exists only to allow religious people to debate atheists without having to defend the absurd beliefs of their actual religion.
> Just one way you've missed the mark: you are speaking as if your access to my mind is as good as it is to yours.
No, I'm saying my access to the mind of a religious person is as good as yours.
> > The thing is, science is perfectly capable of answering this question--it's not unknowable.
> "Science" only has volition through scientists (or "scientific thinkers"), and Humans very often are not able to not know things (particularly educated Humans at times....a little knowledge is a dangerous thing as they say). Studying science does not turn one into a Perfectly Rational Human, after all.
Yep. Which is why I didn't say "science is perfectly capable of answering ALL QUESTIONS". What I said was "Science is perfectly capable of answering THIS QUESTION." Since you're intent on taking everything out of context, I will insist on clarifying that what I mean is, "Science is perfectly capable of answering the question of whether perception is reality or not--whether perception is reality or not is not unknowable."
I will go on to say, religion isn't capable of answering any questions. No criticism of science actually gives any validity to religion.
> Please, tell me who "my" "invisible friend" is, Human. Tell me in vivid detail, I would really like to know this about my actual, not indirect self.
Your invisible friend is whatever deity your religion believes in. You said you were religious.
> When you say "think", I'm curious what you mean. Could you (after the fact, giving you an advantage) write out a pseudocode representation of the chain(s) of logic (and the axioms & premises upon which it is based) you ran to arrive at that incorrect conclusion? Because I have a sneaking suspicion there might be some heuristics and bad axioms(!) in the cognitive pipeline.
Bro, what are you even asking? If you want to know why I think philosophy is often poor communication presented as intelligence, it's crap like the above quoted paragraph.
No, I'm not going to pseudo-code out the definition of the word "think", if that's what you're asking me to do.
The irony of this appearing on the Nature site… when their own editors routinely reject even remarkable results that go on to become highly cited seminal papers in “lesser” journals.
I wish we would disentangle publication from endorsement. Making the bits available and saying that they're useful are different things. Your null result could contain data which is relevant to some other inquiry.
All results should be published, some should be celebrated.
I just reread this and realized how much of a jerk I'm coming off as (too late to delete it). My point is just that if you're reading something because it was published then publication and endorsement must not be sufficiently decoupled.
In a world where they are decoupled, you'd only end up reading it because somebody you trust endorsed it (no reason for that to coincide with the publisher) or because you went searching for it despite its lack of endorsement--in which case you probably know what you're doing.
I was specifically talking about peer-reviewed papers and how bad some of them are. I'm working in the space of parallelization and optimization and a lot of papers out there just implement any given algorithm on any hardware of their choice. They describe vaguely what they have done, claim to be faster than any other papers and call it a day.
That's worthless in my opinion. For all we know they could have a bug in their code and that's why it's fast. Some may even lie, who knows without code or data.
But oh well, I guess it's a step towards graduation for them. So at least they got something for their time (and presumably they have that cool, fast implementation they need for a project and some useful experience, but the paper: worthless).
I took a bioinformatics class recently. The final project was taking wet lab data from a published paper and using it to do our own analysis and then comparing that analysis to the paper. I'm pretty sure I found an error in the paper that we were partially replicating: Their charts are swapped: mutant A's chart is labeled as mutant B and visa versa.
While the resemblance of my A chart with their B chart might be a bit subjective, I have a reproducible build of my chart and its labeling that I can trace all the way back to the source data (via nix). It might not be perfect, but at least there's an artifact available for scrutiny... at all. The published version though, it's just a PDF. I can't see their code so that I can point them at the bug that I suspect exists in it. Their data is available on NCBI, but not their code.
So yeah, I hear you re: being a bit disappointed with the reproducibility of the computational aspects of these papers. It's somewhat understandable for replication to be difficult in places like psychology or chemistry... but if the work is being done in silico, it should be a matter of two or three commands to re-do it in silico. And if that's not true, we need to fix the tooling until it is.
I'd prefer to have the raw data, intermediate artifacts, and their computational relationships, all made available as soon as the researcher has an inkling that they might be worth presenting to someone. Then, when it's time to celebrate those results, they can just reference things that have been publicly searchable and verifiable for a while now. That way computational verification can happen before the paper is even written, and we don't even have to bother writing prose around what turned out to be bogus results.
My thesis committee chair bristled when I said I hated the marketing aspect of academic science and I implied that he was a very good marketer because he played the game so well. After I kept making comparison after comparison, he didn't have much to say in response. I would say he begrudgingly accepted my view, but I don't think accepted it at all, he just couldn't refute any point I made.
This seems like a tough problem to solve, but given the article states 75% of researchers are _willing_ to publish the null results at least that's something to build on. Making publishing compulsory could lead to other, worse problems given humans are involved.
I realize as I'm writing this that I don't really understand what "publishing" means. It's more than just making the paper available, right? Is there a formal definition, or just a colloquial one in science?
“Publishing” in academia usually refers to being accepted in a “peer-reviewed” journal or conference. So your paper is not just uploaded somewhere – it is sent to 2-3 domain experts, who criticize your work and force you to jump through lots hoops before it’s either “accepted for publication” or “rejected”.
When this works well, it’s a good filter to prevent spam, fraud, methodological errors, etc. from being published, while improving the quality of the accepted research papers via feedback from other domain experts.
When it doesn’t work well, the referees can take it upon themselves to reject papers for subjective reasons, including that the work is “not novel enough”, that they don’t like the model you used, or that they are just not excited by the research field you work in. It also happens that they require you extend your work in a way that takes an order of magnitude more time before they’ll accept it. For the authors, it’s often difficult to defend themselves from this kind of attacks, since the referees in many journals don’t need to justify their claims much, and often feel free to be extra harsh since they tend to be anonymous.
Since going through the publication process can take months to years of work depending on your field, some researchers would not be willing to put in that effort for a negative result (which is unlikely to be cited and thus doesn’t help your career).
It is however possible to just upload a paper (e.g. to arXiv). These “manuscripts” are often useful and can be cited normally, but researchers tend to be a bit more wary of citing them unless the authors are well-respected due to the lack of peer review.
> I realize as I'm writing this that I don't really understand what "publishing" means.
1. Fully gather and analyse the data, no stopping early when you realise it isn't working
2. Write the paper, read those background papers you hadn't got to yet so you can cite them, chase down references for things you know from memory.
3. Realise there's a gap in your table because you tested A, B and D at three levels each but C you only tested at the low and high level, not at the medium level. Go set up your test equipment again to fill in the blank space.
4. Run the paper by your collaborators and your boss, all of whom will feel obliged to suggest at least some improvements, which you'll make.
5. Choose a journal, apply the journal's template and style, send it in.
6. Wait for as much as several months for peer review.
7. The first peer reviewer suggests you retest with a slightly different protocol for cleaning your equipment before the test. You do so.
8. The second peer reviewer replies suggesting you test combinations of A, B, C and D, not just one at a time....
From my experience, there's no definition as such but having your study "published" implies that it went through a peer-review process featuring at least two qualified referees and an editor. The implication being that the claims from the study are valid as reference for future studies, to varying extent depending on the quality of the journal etc.
It does not work as standard anyway. The publishing practices are different from field to field. Also journals have different policies and practices. So it is hard to get a real representative definition other than making paper available. In which case arxiv will be a publishing mechanism that does not provide editor, peer-reviewed and does not cost money.
The main issue with publishing research is the cost. Some time ago, I worked in a research lab studying Lupus. Our results were negative, and my initial inclination was not to publish them. However, my Principal Investigator (PI) emphasized that all results, whether positive or negative, should be published. Fortunately, we had the funds to do so. At that time, publishing in a reputable journal cost $2,300.
Not everyone is so fortunate. This lesson has stuck with me, as I have seen or heard from different labs where, unfortunately, they couldn't afford to publish their findings.
I think this problem is at least in part due to the hypothesis testing concept itself. Classical hypothesis testing is asymmetric: there is a "null" hypothesis, which is typically the uninteresting/useless case, and an "alternative" hypothesis, which is the one you would like to be true. Critically, you cannot determine if the data _supports_ the null hypothesis, only if the data _rejects_ it (and supports the alternative). A so-called "null result" occurs when data is not sufficient to reject the null hypothesis; then you can't tell if you actually have a useful finding (for example, that there is no major difference between species A and B) or a failed experiment (data was so bad/noisy that we cannot conclude anything). And so you end up with the unfortunate situation where you either succeed in proving your favorite hypothesis and get your degree / promotion / tenure, or you have nothing.
This happens because hypothesis testing conflates effect size (how big is the difference between A and B) with uncertainty about that effect size (significance/reproducibility). Confidence intervals are more useful IMHO, as they help untangle these two aspects, for example showing that the difference between A and B is small _and_ reproducible. Bayesian analysis is also a major improvement, as it allows examining both the "null" and "alternative" hypotheses on equal terms, as well as reasoning about our prior beliefs / biases. Unfortunately many areas of science are still stuck with statistical methods from the early 1900's.
As a researcher that does mostly numerics, I really hope not. This would be a huge bureaucratization of scientific exploration and would slow down progress. I understand why it’s necessary in some fields like medicine, but I don’t think it’s worth the trade off in say theoretical physics.
Imagine the corresponding concept for programmers: you are not allowed to sell or share any software you create unless you pre-register a detailed plan for what code you will write and how it will be used before you write the first line of code. Pretty sure that would reduce the innovation going on in public GitHub repos a lot :)
I think that not knowing the null results of others also slows down innovation, because you never know if an area of interest / whitespace in a field is 1) because there's something there or 2) because others tried, failed, and didn't publish.
Maybe pre-registering isn't the right answer; I'm sure there are practical hurdles, but the problem to be solved still remains the same (visibility into the graveyard of failed experiments to improve the rate of innovation).
Who would track this registration and would it require approvals now?. Then what if you changed your current research because you have personal reasons, change of plans, didn't see it fitting....etc. How would you handle these situations? And why are you introducing MITMs.
This will hardly achieve this goal as you basically make things harder, and now you are introducing more overhead. The main reason why people don't like publishing null results is that it hurts them in funding applications. The current system works with mentality, we shouldn't fund someone who don't get positive results. It is better to allocate this somewhere else. Most of the problems with research can be tracked down to funding issues and practices. But these are political issues, so people try to argue about other things because it is easy.
All good points- I was imagining that from a funder's perspective, knowing null results is actually important to guaranteeing future positive results / being strategic about what is worth funding in (versus what is destined to fail because others failed and they just didn't know). Not sure how it would work in practice; might be bumpy but certainly seems worthwhile / not impossible.
There’s a bit of irony in the second reference, as the author ended up with some controversial work related to COVID-19. He co-authored a study that was widely cited to downplay severity of the pandemic, but was also heavily criticized for poor methodology (and later I think firmly found to be very wrong). He also published a paper with personal attacks of a grad student that had disagreed with him, which is probably not in the spirit of encouraging constructive science.
The problem is that peer review is primarily focused on results. Peer review should be done up to but not including the results. Provide motivation, explain your methodology, explain how it will resolve issues in the literature, but don't say anything about your results. Papers should be conditionally accepted, subject to confirmation that the results you report are the results of the proposal that went through peer review.
True story: I wrote my ph.d. thesis on a special case of a general problem. After about one year of work, I realized that the approach would never work for the general problem for intractable reasons. But I also really wanted to finish my ph.d. within 5 years, so I spent the next two years refining the work enough to be able to write a dissertation on it and ignored the fact that it would never really work for what it was intended. I did do some interesting work and learned a lot, but I couldn't really bring myself to try and publish the results (beyond my thesis) because I very clearly had not made an advance in the field. Of course, I do think it would have been useful to publish why I thought that essentially the entire field of inquiry was a dead end, but that would not have made me very popular with my collaborators or others in the field and it wouldn't likely have ingratiated me with anyone else.
Great, so the economic {dis|in}centives are keeping a bunch of people fed but doing useless study of a dead end and even kept you from pointing out the emperor has no clothes (assuming your epiphany is valid). Is social conformity holding back science?
The introductory example is quite illuminating. No theory behind, just a random hypothesis tested (by comparing the preferences of 10 fishes from populations separated from as little as 50 meters, what effect size did the authors expect?), with claims of generalisation (climate change ok bad?? Or climate change bad bad??)
Perhaps my most prominent paper was a null result. If the question is important enough, the work to test it non-trivial, it will find an audience and a likely a reputable journal.
What is the value in reporting a null result other than reducing file drawer effects? Well one, even though the null does not tell us whether the effect is absent, it does help suggest bounds on how large an effect could be if it did exist. In my case, a particularly large effects were observed in cross-sectional comparisons (different people at different ages), but our research showed that longitudinal changes within individuals were generally negligible, suggesting systematic bias in the cross-sectional sampling.
Some others have mentioned this in their comments and I agree that once you succeed in getting a non-null result, publishing the null results (all the things you tried that didn't work) could be included as appendices or something.
Also, just because you get a null result doesn't mean that nothing was learned, that something new (and unexpected) wasn't stumbled on, or that some innovation didn't happen.
There are tiers of publications and journals. Even if you get a null result and you're not going to get it accepted in Nature, it's very possible that you can get a conference paper (sometimes peer reviewed) out of something that was learned.
I recall reading someone who proposed the need for what they dubbed "meta-science," and I think it's clear that this concept is becoming more needed as time goes on. Our publishing process, and the incentives therein, are obviously faulty and we are aware of it. We can do the math: I believe it's time we do away with playing speculative games with science.
Not all null results are created equal. To get your null results published, the null result must shed light on some phenomena. And of course, the study must be sound. E.g., everybody takes for granted that X causes Y, but in a well-designed study, X did not in fact cause Y, which reveals an error in what we assumed was true.
I run a journal where we publish both “real” null results and experiments were something practical went wrong, so i have a few thoughts:
1. Ideally peer review would catch this. A badly setup study should be critiqued in peer review. Forcing scientists to first publish their methods before doing the experiment also helps, as it validates the experimental setup before hand.
I also think it’s worth publishing studies where a null result was reached due to some error in experimental setup or other factors, as long as it’s presented as such and reflected upon. This can still be valuable information for future experiments. Offering scientists social capital for that (an “official” publication, citations) might also incentivize scientists to publish the results as is, rather than making it appear as a “true” null result, or even as a non-null one (eg through p hacking).
2. While obviously possible, given the amount of effort scientists have to go through to raise funding for an experiment nowadays, i find it highly unlikely that people would go through this effort.
3. This is already possible and a problem. This is a problem of academic misconduct, has very little to do with null results.
The current publishing system is of course already set up to be gamed, so I understand your worries. But null results should be published, as they are just science. Even if someone were to “game” the system by publishing a ton of null results, those publications should be held to the same level of scrutiny as any other publication. If someone is extremely prolific in replicating existing studies and comes up with a ton of null results, that should be lauded and those papers should be published, no?
I do believe the entire idea of a researchers output only being recognized by being allowed to be published in a journal is terrible and should be abolished, but baby steps I guess.
Currently, the system can be gamed exactly as you say by publishing sloppy studies that falsely find a "real" effect. But editors and readers of the article will look at and judge the methodology section. Sloppy experiments risk not being printed or cited.
>Would people run shoddy experiments to get null results to undermine their rivals?
In this case, the rival would be very much inclined to recreate the "null" experiment.
>Could somebody pump out dozens of null publications to pad their CV and screw up h-indexes?
Possibly, but would null publications be cited as often? Also, who's going to keep funding a researcher that mostly publishes null results?[0]
[0] Besides agenda-driven "think tanks". Which is worrying itself.
Is it an idea to publish null results in appendix when a sexy result will be published? Kinda like a Thomas Edison thing. How many ways are there to not make a lightbulb, included with the ways that do make it possible to create one
As someone whose early scientific career was destroyed by null results, no. No one will publish your negative results. Unless you win the lottery and stumble across a once-in-a-generation negative result (e.g. the Michelson–Morley experiment), any time you spend working on research that yields negative results is essentially wasted.
This article completely glosses over the fact that to publish a typical negative result, you need to have progressed your scientific career to the point where you are able to do so. To get there, you need piles of publications, and since publishing positive results is vastly easier than publishing negative ones, everyone is incentivized to not waste time on the negative ones. You either publish or you perish, after all.
Simply put, within the current framework of how people actually become scientists and do research, there is no way to solve the 'file drawer' problem. You might see an occasional graduate student find something unusual enough to publish, or an already-tenured professor with enough freedom to spend the time submitting their manuscript to 20 different journals, but the vast majority of scientists are going to drop any research avenue that doesn't immediately yield positive results.
We published a load of null results in particle physics. Simply go to arxiv.org and look for papers beginnig with “search for…” that would be a null result. Well technically a sigma<3 result.
I think the fundamental problem is that for every claim yielding a positive result there are many more, perhaps infinitely more, related claims yielding negative results.
Positive result claim: the sun comes up in the morning.
Negative result claims: the sun moves sideways in the morning. The sun was always there. The sun peeks up in the morning and immediately goes back down. And so on.
Positive result claim: aspirin is an effective pain reliever.
Negative result claims: eating sawdust is an effective pain reliever. Snorting water is an effective pain reliever. Crystal Healing is an effective pain reliever. Etc.
Because there are so many negative results, it's trivial to construct an experiment which produces one. So why should that be published?
Negative results should be published when people in the community are asking that question, or have a wrong belief in the answer (hence the replication crisis). But if nobody cares about the question, it's hard to argue for why a given negative result would be preferred over any other negative result for purposes of publication.
So let's say I claim that the sun goes in a circle in the sky in the morning. The null hypothesis is that it doesn't do that. Perform experiment. Null hypothesis wins. Write up paper! This is a negative result.
The point is that for every result where the alternative hypothesis wins, there are a massive, if not infinite, number of results where the null hypothesis will win. Are these publishable?
The idea is that some null hypotheses being true is actually interesting because it challenges an assumed belief. From the first paragraph of the article, the immediate feedback from the postdoc's supervisor was 'you did it wrong [because everyone knows that fish do like warmer water]'.
> It ain't what you don't know that gets you into trouble. It's what you know for sure that just ain't so.
> The idea is that some null hypotheses being true is actually interesting because it challenges an assumed belief.
??? As I had said originally, that's one of the primary situations where a negative result should be published.
But the huge, huge, huge majority of negative results are trivial and uninteresting. Thus the fundamental issue with negative results is that you have to provide rather more compelling justification for why such results should be published.
Yeah, I agree with your first point, but maybe misunderstood your reply? If there's nothing "surprising" about the result, it's not interesting, so not publishable. The article's first example, however, did seem to be surprising to the researcher's community, so it should have been published.
Sure. What I said, or had meant to say, was in reply to people complaining that there was some kind of cartel against negative results. Rather, what we're seeing is just the natural, if unfortunate, response to the basic problem with negative results as a whole. You can't just treat them as the same as positive results: because of their numerosity they require unusual justification for their publication.
Researchers are expected to publish papers at such a frequency today that spending time writing a paper for a null result would be considered a bad career move
A preprint will still give less value to the negative result. No peer review (being a broken system or not) and not being published in a "proper" journal will make it less likely that the results will be recognised / accepted.
The whole point is, that a negative result can have as much value as a positive one...
The whole point of publishing studies is to be able to brag about how many impressions you have, which is good for your career. Who is going to care that your blog got views?
Remember all the cries of "the science is settled"!
Yeah, that's not science - it's the exact opposite of science. This is the perfect example of why reasoned skepticism is more necessary than ever. Blind trust in any institution is a recipe for disaster.
Our editor-in-chief was interviewed for this related Nature article a couple months ago (https://www.nature.com/articles/d41586-024-01389-7).
While it’s easy pickings, it’s still always worth pointing out the hypocrisy of Nature publishing pieces like this, given that they are key drivers of this phenomenon by rarely publishing null results in their mainline journals. They are have extremely little incentive to change anything about the way scientific publishing works, as they are currently profiting the most from the existing structures, so them publishing something like this always leaves a bit of a sour taste.