
The mathematics of science’s broken reward system - seycombi
http://www.nature.com/news/the-mathematics-of-science-s-broken-reward-system-1.20987
======
infinity0
The challenge will be to fix this whilst keeping the precise, mechanical and
quantitative spirit of science alive.

The reason why scientists have a natural "disdain" for sociological studies,
is because they don't have these qualities. Many of the arguments aren't
convincing from a critical angle, and only convincing to people who have a
pre-existing bias towards certain conclusions. Yet the scientific culture of
today is falling into that trap itself.

The article criticises metric-driven incentives, but it is not metrics (the
general concept) that are at fault. It is the _choice_ of metric, and the
meta-analysis of this that is lacking. These choices are themselves often
backed up by non-scientific vague arguments, of the similar sort that
scientists often criticise other fields as depending upon.

We must certainly not conclude from these studies that quantitative analysis
is _itself_ what is at fault. I know the article doesn't explicitly say this,
but it hints at it - using suggestive phrasing like "disdained sociological
studies" and referring to all "metric incentives" as a single group - and it
is a point I have seen made by many non-scientists. That is, using these flaws
as a straw man to attack the very qualities of what has made science so
successful and useful.

To improve the situation, we must reject these straw-man arguments against
science, and develop better methods that are more quantitive, over a broader
spectrum of what is being analysed, and that are more self-critical.

~~~
hammock
And yet, the greatest scientists in history did NOT have the stereotypical
Scientist's Disdain for sociology or softer topics. For example Einstein said
things as "As far as [the laws of mathematics] are certain, they do not refer
to reality" and "All religions, arts and sciences are branches of the same
tree."

There are plenty of brilliant scientists who are held back by their excessive
rationality, or whatever you want to call it. Another great book on this topic
as it relates to econometrics (as opposed to physics) is "The Romantic
Economist" by Nicolson.

~~~
bzbarsky
There's a difference between "disdain for sociology" and "disdain for the way
sociology is drawing conclusions from the experiments it performs".

Sociology is very important to study. The tools we have for it are not great,
in various ways. For example, one can study people and societies by reading
Balzac's writing, but the number of people who can produce that sort of thing
is fairly limited. We can try to do controlled experiments, but the way we do
it in practice is not great. We really really need better tools here...

~~~
TeMPOraL
Indeed, psychology and sociology are _much_ more difficult fields of study
than hard sciences. Physical laws, however tricky they are, don't seem to
change at all, and tend to be the same everywhere you look. Psychology is
about studying a behaviour of an advanced computing system that's about as
smart as the researchers themselves. Sociology is about studying how those
advanced computing systems interact with each other at scale. It's _insanely
difficult_ , and that's why it's so hard to even come up with an experimental
setup that makes some sort of sense. Not to mention ethical issues (many
experiments would be _so_ much simpler if you could disregard the well-being
of the test subjects).

So yeah, personally, I have utmost respect for the complexities involved in
sociology - while at the same time I absolutely hate all the bullshit that's
being done because doing actual research feels too hard.

------
hedgew
Outside of fields with direct penalties for invalid science (think aviation,
nuclear reactors, sinking boats, collapsing bridges), we're just relying on
the integrity and honor of researchers. On an individual level it's much more
effective to cut corners and publish trendy research than it is to make valid
science.

The situation is only made worse by the fact that universities reward this;
because they too are rewarded more for high-profile research than they are for
scientific validity.

My guess is that most studies in softer sciences are simply erroneous. Many
social sciences have already been shown to be unreliable by the replication
crisis.

My personal experience comes from reviewing hundreds of published studies that
evaluated the applied effectiveness of machine learning models. Half had
significant statistical errors.

~~~
chestervonwinch
> My personal experience comes from reviewing hundreds of published studies
> that evaluated the applied effectiveness of machine learning models.

In the social sciences? Just curious.

> Half had significant statistical errors.

Such as? Have you enumerated the typical errors somewhere, so that other can
learn to avoid these common pitfalls?

~~~
Balgair
The stats errors are so common and easy to correct these days that there is a
statscheck (ala spellcheck) program (can't find the link, sorry) out of the
Belgium that trawls through journals and checks the stats. Mind you, these are
peer reviewed and published articles. Still, about half have erroneous stats,
sometimes ones that invalidate the paper. Yeah, we all screw up, but it seems
like a lot of the screw ups happen much too often.

As for a place that has common stats errors and how to fix them? Well, just
about every single intro to stats book, the entire R language, just about any
library for just about any language, google, etc. The issue is not that the
folks are making honest mistakes, that happens, it's that the system is
perverse and incentivizes them.

~~~
c06n
> statscheck

You mean this?
[https://cran.r-project.org/web/packages/statcheck/index.html](https://cran.r-project.org/web/packages/statcheck/index.html)

~~~
Balgair
Yep, thanks!

------
randomsearch
I agree that governments should stop using metrics to evaluate science in this
way, and instead qualitatively review research output.

It's obvious that citation counts are open to manipulation (e.g. through self-
citation and encouraging citations from others via various mechanisms) but
there's also a very large factor that can't be easily quantified: an enormous
amount of good will has been lost in the system.

Many scientists who are interested mainly in satisfying their curiosity and
contributing to society resent the top-down mismanagement of science in the UK
and US. They see a system where those who play political games do well, but
the smartest, most dedicated, and passionate researchers are often sidelined
or ignored because they do not spend the requisite time playing the political
system to artificially boost their reputations. Many times I've seen very
talented academics with huge potential leaving academia as they were passed up
for promotion, left on a temporary contract indefinitely, or neglected in
other ways.

Strangely enough, I think that if you don't have pressure to climb the ladder
(e.g. a family), then the current system offers great opportunities to do
outstanding science. If you choose to stay low on the ladder and spend the
vast majority of your time actually focusing on research problems, you can get
far more actual research done than those chasing promotion or esteem. And
because so many people are focused on their citation count, or some political
game, the competition isn't as strong as it should be - you can make real
advances if you quietly focus and leave the politicians to fight each other.

------
lbhnact
I've always wanted a good word for the 'science of studying science'.

Never really got past 'Scientology', and that probably won't stick.

What's a better term than the vapid 'Metascience', or the clunky
'scientometrics'.

How about 'Superscience'? That would be an awesome Doctorate to hang on the
wall.

~~~
Malarkey73
"Epistemology" is probably the best word.

~~~
lbhnact
Yeah, maybe 'Quantitative Epistemology'. You'd probably be thrown right out of
the Philosophy department though.

~~~
johncolanduoni
Not really, many people who do philosophy of science/math have science/math
PhDs as opposed to philosophy or other humanities.

------
cuantos
You can have science without academia and visa versa. Perhaps it should say
that academia has a broken reward system.

------
nonbel
This is presented as if it is a new problem. It stems back to the 1940s with
the adoption of NHST (with its arbitrary "statistical significance" metric)
first by educational researchers, then by psychologists, followed by spreading
like a cancer throughout the research community from there.

Things have only been getting worse and worse as the "old guard" in each field
retires/dies leaving behind only people trained to think rejecting a strawman
hypothesis according to an arbitrary metric, then concluding something about
your hypothesis, counts as science.

------
hannob
The irony is that this is published in Nature, which undoubtedly is a huge
contributor to questionable research practices by publishing spectacular
findings and ignoring negative results and replications.

~~~
tdaltonc
I think it's more helpful to blame the incentives and the individual actors.
Nature isn't malicious or powerful enough to break science all on it's own. It
didn't write the rules, it just plays by them.

ie Hate the game, not the player.

~~~
epistasis
In this case, Nature _is_ the game. Nature is far more powerful in science
than any individual scientist, or any individual research institution.

------
BurningFrog
Where does this broken system come from?

Did anybody construct it, or has it just emerged out of the surrounding world?

Is it the same across the planet?

~~~
danieltillett
Like most problems it comes from too many people chasing too few dollars. The
ultimate cause was the construction of a system that trains too many people
for the money avalible to employ them.

The current system was deliberately constructed by politicians after WWII when
we had more money (and demand for science) than scientists. They set out to
create more scientists by designing a system where each scientist trained 10
to 100s of additional scientists (PhDs). This worked great until around the
mid 1970s when the supply of scientists finally caught up with the money.
Since then the problem has been getting worse.

------
Balgair
Science has a lot going wrong with it.

Here are some I can list: P-values, the h-index (article discusses), the
funding crunches and (US specific) the cyclic nature of NIH funding, the
massively skewed incentives of publish or perish, the entire idea of trying to
assign value to discoveries (article discusses), the 2-body problem of family
life and science (at least the Germans expect you to have no life and are
clear on that), the work-life non-balance, the relatively very poor pay, the
hyper-competition and sabotage (in some fields, labs, or universities), the
total ignorance of doctors/politicians that take your work and apply it in
appalling ways, the entire racket of scientific publishing, etc.

As always, we should point out to newcomers to this discussion of one of the
_most read scientific paper EVER_ by John P. Ioannidis "Why Most Published
Research Findings Are False" [0]. If you don't have access and don't want to
use Sci-Hub or ICanHazPDF, here is a youtube of John discussing his work [1].
You can use the google to really jump into a rabbit hole here. Essentially,
flip a coin, heads the paper is right, tails the paper is wrong, it really is
that bad these days.

Ok, so lets just scrap the whole thing then, right?

Here is why not:

[https://www.youtube.com/watch?v=_Px5sZyxFYc](https://www.youtube.com/watch?v=_Px5sZyxFYc)

[https://www.youtube.com/watch?v=XcPuRaSEq1I](https://www.youtube.com/watch?v=XcPuRaSEq1I)

[https://www.youtube.com/watch?v=5EL_bLOK8_A](https://www.youtube.com/watch?v=5EL_bLOK8_A)

[https://www.youtube.com/watch?v=qXWYSdijCGU](https://www.youtube.com/watch?v=qXWYSdijCGU)

[https://www.youtube.com/watch?v=uBh2LxTW0s0](https://www.youtube.com/watch?v=uBh2LxTW0s0)
(a good showing of why science is right and helps folks)

[https://www.youtube.com/watch?v=gs0JQRT6TpY](https://www.youtube.com/watch?v=gs0JQRT6TpY)

So, yeah, science is a mess right now. But if we scientists don't step up and
make it _not_ a mess, those scummy money-grubbing scammer dirtbags and their
ilk are more than happy to take up the slack. These fucking asshats are going
to _kill_ people like you poor uninformed cousins and their kids and take all
their money because they think homeopathy is right and the moon landing was
faked.

So, you young scientist that looks at this pile of garbage that is modern
academia, do not despair! Fight the good fight! Yes, you may end up shirtless
and ridiculed by the older scientists. But you _have to_ do what you think is
right because the rest of this world is depending upon you! Even for that tiny
little bullcrap paper you are getting out just to graduate, that matters too.
Be in the mud, be in the arena, fight for truth!

[0][http://journals.plos.org/plosmedicine/article?id=10.1371/jou...](http://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.0020124)

[1][https://www.youtube.com/watch?v=KOZAV9AvIQE](https://www.youtube.com/watch?v=KOZAV9AvIQE)

~~~
nonbel
>"These fucking asshats are going to kill people like you poor uninformed
cousins and their kids and take all their money"

They have already taken over healthcare and passed laws to force you to pay
for their "help". To get you started: people are quitting cancer
reproducibility projects out of disgust for the low quality, and just trying
to figure out wtf was done to generate the data is draining all the funds
before they can even attempt replication:

"Early on, Begley, who had raised some of the initial objections about
irreproducible papers, became disenchanted. He says some of the papers chosen
have such serious flaws, such as a lack of appropriate controls, that
attempting to replicate them is “a complete waste of time.” He stepped down
from the project's advisory board last year.

Amassing all the information needed to replicate an experiment and even figure
out how many animals to use proved “more complex and time-consuming than we
ever imagined,” Iorns says. Principal investigators had to dig up notebooks
and raw data files and track down long-gone postdocs and graduate students,
and the project became mired in working out material transfer agreements with
universities to share plasmids, cell lines, and mice.

[...]

ALTHOUGH ERRINGTON SAYS many labs have been “excited” and happy to
participate, that is not what Science learned in interviews with about one-
fourth of the principal investigators on the 50 papers. Many say the project
has been a significant intrusion on their lab's time—typically 20, 30, or more
emails over many months and the equivalent of up to 2 weeks of full-time work
by a graduate student to fill in protocol details and get information from
collaborators. Errington concedes that a few groups have balked and stopped
communicating, at least temporarily."

[http://www.sciencemag.org/content/348/6242/1411](http://www.sciencemag.org/content/348/6242/1411)

Also, it seems 4/5 doctors will tell you there is 95% probability you have a
disease when it is actually 98% chance you _do not_ have it, 40 years of
education reform has not affected this at all (admittedly, this needs to be
repeated on a larger scale, but from personal experience I have no doubt it
will hold):

"Nearly 40 years ago the New England Journal of Medicine published a short
survey of doctors’ understanding of the results of diagnostic tests.1 The
participants, all doctors or medical students at Harvard teaching hospitals,
were asked, “If a test to detect a disease whose prevalence is 1/1000 has a
false positive rate of 5%, what is the chance that a person found to have a
positive result actually has the disease, assuming that you know nothing else
about the person’s symptoms or signs?” This wasn’t a very difficult question,
which made the results all the more shocking. Fewer than a fifth of
participants gave the correct answer, and most thought that the hypothetical
patient had a 95% chance of having the disease.

Of course, this was a long time ago, and medical curriculums now contain much
more in the way of statistics and probabilistic reasoning. You might expect
that if the exercise were repeated today almost everyone would give the right
answer. But you’d be wrong. Earlier this year a similar study was carried out,
also in hospitals in the Boston area of Massachusetts, and the results were no
better.2 Most doctors who were asked exactly the same question thought that
the patient had a 95% chance of having the disease."
[http://www.bmj.com/content/349/bmj.g5619](http://www.bmj.com/content/349/bmj.g5619)

Medical errors were "unintentionally" left out as a possible official cause of
death, then when people actually estimate this, it is a leading cause:

"In 1949, Makary says, the U.S. adopted an international form that used
International Classification of Diseases billing codes to tally causes of
death...medical errors were unintentionally excluded from national health
statistics...based on a total of 35,416,020 hospitalizations, 251,454 deaths
stemmed from a medical error, which the researchers say now translates to 9.5
percent of all deaths each year in the U.S...According to the CDC, in 2013,
611,105 people died of heart disease, 584,881 died of cancer, and 149,205 died
of chronic respiratory disease—the top three causes of death in the U.S. The
newly calculated figure for medical errors puts this cause of death behind
cancer but ahead of respiratory disease."

[https://hub.jhu.edu/2016/05/03/medical-errors-third-
leading-...](https://hub.jhu.edu/2016/05/03/medical-errors-third-leading-
cause-of-death/)

I could go on if you are interested.

~~~
apathy
ah, Begley, such a saint:

[http://www.nature.com/nature/journal/v497/n7450/full/497433a...](http://www.nature.com/nature/journal/v497/n7450/full/497433a.html)

This is the guy that published the irreproducible results paper whose results
were (of course) irreproducible (since it did not name any). I.e., "trust me
on this", the opposite of science.

Note above that you can pay for more of his sage wisdom, thanks to the
generous NatureMacSpringer megaconglomco.

Post everything as a preprint. Submit your trial results to OpenTrials or
OpenFDA. Quit killing patients. Maybe work in pediatrics so that, right or
wrong, you still won't make any money, so you might as well not kill any
excess kids.

I'm not bitter or anything, after participating in clinical trials and
analysis for over a decade...

~~~
nonbel
>"I'm not bitter or anything, after participating in clinical trials and
analysis for over a decade..."

I'd love to hear your story, up to you.

~~~
apathy
To be honest it's really only adult trials that are fucked. Well, those and
rare diseases with a noisy yet ignorant "advocacy" lobby ("We won't have to
pay if the drug doesn't work, the magical Insurance Fairy will pay for it!").
I understand the latter (some hope is better than none, if it's your kid
that's sick) but as for the former... ick.

Take away the profit motive and it's astounding how much more reliable the
trials get. Of course reviewers whine about how they're often equivocal or
negative, because the thought of the literature being horribly biased either

a) has never crossed their mind (too busy with protocols),

or

b) is precisely what they want, for "showing progress".

So in order to keep up the appearance of positive results and keep the gravy
train flowing, the trials that get published tend to be the ones that "show"
"progress".

Mind you, these are clinical trials with preregistered endpoints and
protocols. Pretty much the entirety of experimental science outside of physics
is much worse. So what I'm telling you is that this is the BEST-CASE SCENARIO
for an awful lot of science.

I'm not going to write up "my story" beyond this, because it's the same as
most everyone else's story in the field. Killing humans for profit is not my
idea of a good time, even if adult trials tend to pay better than the Army.

YMMV. I may be jaded, but I still believe there are populations that, through
no fault of their own, end up with few or no options. There is room in this
world for integrity -- but get (real) money involved and the bad money drives
out the good. The linked paper is dead on.

~~~
nonbel
Thanks.

------
foolrush
Science hasn't existed forever, and Foucault already smashed to bits of any
sort of "evolution with direction" account of science in The Order of Things.

The most disturbing thing appears to be that many folks aren't aware of
Foucault's work.

------
AlphaWeaver
Interesting, even if a bit meta.

------
jsprogrammer
Couldn't this model be used to detect and alert on bad actors?

------
known
[https://en.wikipedia.org/wiki/Economic_mobility](https://en.wikipedia.org/wiki/Economic_mobility)
!=
[https://en.wikipedia.org/wiki/Social_mobility](https://en.wikipedia.org/wiki/Social_mobility)

