
The pressure to publish pushes down quality - benbreen
http://www.nature.com/news/the-pressure-to-publish-pushes-down-quality-1.19887
======
Alex3917
Is it possible to model the rate at which human knowledge is increasing or
decreasing? Given that:

A) claims about reality are a lemon market, and it's often expensive or
impossible to evaluate the truth behind purported facts.

B) false claims undermine the value of true claims by decreasing trust in
them, and by causing people to stop believing things that are true so as to
realign their worldview to be logically consistent with misinformation.

It stands to reason that if the percentage of true claims made in papers is
less than X, the sum of human knowledge is actually decreasing even if good
papers are still being published.

~~~
sharkjacobs
I can't even think of a way to measure human knowledge which isn't problematic
in immediately obvious ways.

~~~
astazangasta
Yeah, what is a 'true claim'?

------
eganist
I really, _really_ hate to ask the obvious, but doesn't this apply to damned
near literally everything?

Did we genuinely need a column to remind us of this? Is this how bad it's
gotten? I'm not asking rhetorically; I'm dead serious. Has publishing gone so
far off the rails that we have to restate the blindingly obvious just to
reboot the conversation? I'm nowhere involved in the process and will readily
admit that I've got not a single paper to my name, so I'm clueless about the
current state of affairs when it comes to published research.

Edit: Wow. Yeah, things actually look pretty bad if we take the opinion at its
word. Particularly telling are the examples given, such as:

> For example, a breast-cancer cell line used in more than 1,000 published
> studies actually turned out to have been a melanoma cell line. The average
> biomedical research paper gets cited between 10 and 20 times in 5 years, and
> as many as one-third of all cell lines used in research are thought to be
> contaminated, so the arithmetic is easy enough to do: by one estimate,
> 10,000 published papers a year cite work based on contaminated cancer cell
> lines. Metastasis has spread to the cancer literature.

~~~
PepeGomez
It indeed seems to be bad. If you try reading some papers you can find
everything from conclusions obviously based on formal fallacies, confusing
statistical significance with practical significance, experiment designs that
couldn't provide any new information no matter how they turn out, to outright
bizarre flaws of logic such as dietary intakes calculated so that the body
isn't able to excrete more than it absorbs.

The medical science has decayed into a strange cargo cult where experiments
are made and papers are written, but the essence of science, the pursuit of
better understanding, is lacking. It's no wonder there has been no major
progress in the last few decades and we have an epidemic of obesity and other
metabolic disorders.

~~~
thingification
Where can I read about medical science having been more focused on
understanding in the past? I think the focus on small increments in
(relatively) unimportant medical knowledge is perhaps the greatest tragedy of
our time.

Also, who calculates dietary intake that way?

"Confusing statistical significance with practical significance" \-- that's a
great phrase!

~~~
nonbel
>"Where can I read about medical science having been more focused on
understanding in the past?"

As far as I know no one has really written this up, it is just something you
figure out by reading the literature. You see that despite the technological
limitations at the time they tried to include tables of the data in as much
detail as was practical and rule out competing explanations, etc. Over the
decades this degenerated into testing null hypotheses and reporting p-values
and averages. Now there is somewhat a recovery due to the sheer ease of
transmitting information, but it is still amazing how little this is taken
advantage of in general.

I've thought about a book but not sure people would buy it. How much would you
pay for something like that?

------
CM30
So in other words, publishing papers in journals has gone much the same route
as the media, where getting something out quickly is rewarded more than doing
a good job.

~~~
tremon
and consumer tech, and manufacturing in general, and politics, and policing,
and...

------
curiousgal
In other news; water is wet.

------
gexla
Pressure is a good thing. Pressure does affect quality. We never have enough
time to ship things. There are always improvements that we can make.

Pressure is always there. It's one of those things which expands to take up
whatever space which we are given. More time (shipping less) doesn't
necessarily mean better quality. There are days when I can get twice as much
done just because my task list has twice as much things. Then there are days
when simple things on a short task list seems to take me forever. I feel I
have more time so maybe I'm more susceptible to going down rabbit holes. Or
maybe I procrastinate until I have enough time that I know the task will
really take.

I wonder if people would really improve the quality if given the time to do
so. Maybe the issue is that they don't have the tools to improve the quality
in the first place. At a certain point your brain calls the task done and you
aren't going to do anything more to it. Give more time to a developer writing
spaghetti code and you'll probably just get more spaghetti code. To really
improve, the developer would have to go through some big personal change (a
steady diet of code reviews, reading good code and using other resources for
improvement).

Maybe it's best to improve the processes. In software we can ship something
with a beta label and expect problems. Then we can iterate on the software to
improve it.

And is publishing less a solution to drowning in noise? That seems to be a
separate problem to be solved using different tools. There will always be a
growing volume and with volume comes noise.

~~~
p4bl0
For the first part of your comment I will just say that research does not work
like this.

However, that is interesting:

> Maybe it's best to improve the processes. In software we can ship something
> with a beta label and expect problems. Then we can iterate on the software
> to improve it.

That's more or less an analogy of what people talking about open peer review
are saying, and I think it is quite right.

~~~
gexla
I don't live in the world of publishing. But pressure is what pushes me to pay
the rent. And that's probably the case for most people.

