
Reliable novelty: New should not trump true - nonbel
https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.3000117
======
jedberg
I'm surprise the solutions section didn't include the most obvious solution of
all: require the hypothesis, procedures, and expected results to be published
before beginning the experiment. This would force them to publish their
results if they fail, and also publish if they changed their methods and why.
It still wouldn't be 100% reliable but it would help.

The other solutions presented were also good though. Making the data cleaner
for other to analyze is always a good thing, especially in the "machine
learning" age.

~~~
denzil_correa
I believe this practice should be inculcated in most research projects.
However, if I put my adversarial hat on - I see one challenge. Scientists
could still perform their experiments beforehand with the hypothesis,
procedures and results. During submissions, they will _phase_ these two-stage
information and give an impression of not performing posthoc rationalization.
However, in reality they would. Such a system could be business as usual for
the scientists unless you control the power for scientists to perform their
experiments which seems impractical.

The idea is extremely great but I believe this could be difficult to implement
at a publication level.

~~~
jedberg
For short experiments you could game the system, but for an experiment that
requires two years of work, it’s unlikely that you’d hold your results for two
years to maintain the illusion.

Also you could tie the release of funds to publication of expected methods.

~~~
HeWhoLurksLate
Global. Warming. Exponential. Curve.

------
nonbel
In my experience quality vs prestige is a U-curve.

At one extreme you have the really crappy journals that publish nonsense, at
the other the really "prestigious" journals that are more like tabloids
reporting exciting sounding conclusions with totally inadequate methods
sections.

In the middle are the topic-specific journals that you need to read to guess
what they did to get the "exciting" results.

"Science" in the late 1990s and early 2000s is the absolute worst example of
this.

------
petschge
At least in my field there is the running joke of "it was published in Nature,
but it still might be correct."

------
cbkeller
Well yeah, obviously: high risk, high reward. Without the benefit of hindsight
(and subsequent tests), a paper that proposes something truly transformative
will look a lot like one that is just wrong (albeit, wrong in a creative/novel
way). If you want to publish the former, you're going to have to take a lot of
risks and publish a lot of duds. Don't we complain all the time that NSF
doesn't fund high-risk science?

~~~
nonbel
That isn't what is going on. The stuff that gets published in the "tabloids"
(Nature, Science, PNAS, etc) is sloppier.

~~~
cbkeller
I don't know your field, but that hasn't really been my experience at all.
Just about as much sloppiness all the way across the board.

My only systematic complaint here is that short format means much of the
substance has to go in an inconvenient supplement, but that's hardly unique to
Nature and Science.

~~~
nonbel
The supplements have vastly improved the situation. There is no denying that.

------
Animats
"Nature", outside the biology area, tends to be an example of this.

~~~
nonbel
Why do you make an exception for biology?

~~~
AlexCoventry
They might be familiar with a subfield where there's a strong culture of
accountability through reproduction of experimental results. Certain parts of
molecular biology at least used to be that way.

~~~
dekhn
most bio papers in nature are irreproducible, except by a few top competitive
labs, and even then, most don't bother unless they don't believe the results.

------
apathy
“Should” is an amazingly dangerous word (in the usage often embraced in the
US, at least).

I find that asking why something “should” be XYZ is much more enlightening,
both for the writer and the reader, in terms of exposing structural problems
that make it so.

“People seldom do what they know to be right. They do what is convenient, then
repent” —attributed to bob dylan

(Corollary: make the “right” thing the least inconvenient thing to do, and the
odds of people doing it will rise appreciably.)

------
austincheney
Sounds like a dangerous title since the word _true_ is subjective and requires
variable amounts of support, even no evidence, to qualify its use.

~~~
drdeca
What?

------
dekhn
it's better to think of the top journals as idea forges, designed to maximize
the flow of thought between the top labs, rather than the authoritative
location for technically correct research.

