
Stagnation and Scientific Incentives - hhs
https://www.nber.org/papers/w26752
======
knzhou
As an academic I will again make the dull but necessary point: figuring out
the fundamental laws of nature is hard, stagnation is the norm, and the
advances of the mid-20th century were the aftershocks of the great revolutions
of quantum mechanics and relativity. You can't just make that happen again on
demand. Fundamental laws are in short supply.

Whenever people tell me we're stagnating I ask them to name an alternative. So
far, from best to worst case, every answer has fallen into one of the
following buckets:

\- a subfield that already exists and already has plenty of people working on
it

\- an idea that was extensively investigated and carefully ruled out over 50
years ago

\- something that requires more money than the field will receive in total
over the next 50 years

\- a mathematical formalism which simply refactors existing laws in a way that
makes them unreadable to almost everyone, without a chance of leading to any
new predictions

\- a complicated, ad hoc model that isn't any more predictive than simpler
models, but gives the modeller hundreds of shiny knobs to tune

\- a metaphysical suggestion about how to view what nature "truly" is, which
amounts at best to rewriting existing laws with bigger, more "profound" words

I can't see how this is going to be fixed by changing how citations work. If
anything, in my experience the particularly bad stuff has been _correctly_
punished by low citations.

~~~
btrettel
I see that you work in particle physics. I don't think particle physics is
representative of science in general. My impression is that particle physics
is seen as sexy and attracts a larger proportion of better researchers than
many other fields for that reason.

The stagnation in fluid dynamics (my general field) seems fairly obviously
linked to incentives. Pumping out papers is seen as more important than doing
a good job as far as I can tell. In the past few years I think I've made a lot
of progress in my subfield by simply _compiling tons of data from the open
literature_ [1], which apparently no one thought to do on the scale that I've
been doing. (It's a lot of work up front, which would lead to a delay in
publishing results.) In the process I've found that a lot of what's written in
review articles and books on the subject is obviously wrong. This is not a
sign of a healthy field! And I don't think you can do this in particle
physics, but I think you can in many fields of engineering.

[1] [https://github.com/btrettel/pipe-jet-breakup-
data](https://github.com/btrettel/pipe-jet-breakup-data)

~~~
m-ee
Could you elaborate on some of the obviously wrong stuff that you've seen? I'm
curious. I have an ME background but strayed away from it into electronics,
then oddly enough ended up in a fluids heavy business.

~~~
btrettel
Here's a recent preprint of mine:

[https://engrxiv.org/nqhs5](https://engrxiv.org/nqhs5)

This paper is on the different varieties of liquid jet breakup, e.g., for
applications like fire hose streams and fuel sprays. At low speeds you have
more regular breakup but this changes to various types of less regular breakup
at higher speeds. (There are photos of each type in the preprint.) These
varieties are called "regimes" in the literature. Because models are typically
only valid in a particular regime, it's important to identify the correct
regime. This is often done with a "regime diagram".

There's a lot I could write about what was wrong before. The easiest thing to
do is compare figure 3 (old regime diagram, p. 5) and figure 4 (my new
diagram, p. 13). These are in exactly the same coordinates but don't resemble
each other much. If science were working correctly then the change would not
be anywhere near as dramatic as it was.

Now, with the small amount of data past researchers used to construct these
diagrams, they couldn't see that the diagram was wrong. The data was simply
too sparse to see the big picture. But once you start adding tons of data it
becomes dead obvious that the diagrams you see in textbooks and review
articles are wrong. So I can't blame previous researchers that much, but
compiling open data is something that should happen regularly. The current
academic system does not incentivize data compilation, at least in
engineering.

The most recent study mentioned in the paper used a grand total of 11 data
points and claimed to have enough resolution to move some of the established
boundaries slightly. This actually is a _regression_. The _first_ study to
construct a regime diagram had 63 data points. Mine has roughly 1200, and I
still want more data! 11 data points is not scientifically acceptable, but it
was enough to get a publication.

(I've since revised and extended this paper but I can't upload the new version
yet due to the publisher's policies. One example: I determined that "turbulent
dripping" is probably not possible so I removed it from the diagram. ;-)

I could list more if you want. There's no shortage of problems. But keep in
mind that these problems are usually only obvious after you get enough data.

------
city17
The most difficult thing of allowing for more novel science is accepting that
more 'useless' research will be done. If you read stories about the old days,
you'll see how often professors/researchers used the freedom they had to do
absolutely useless things, just based on their own personal preferences. The
same freedom of course allowed others to do great things.

I think nowadays there is much more awareness about what researchers are
doing, and they will be held accountable, not just by the people who fund
them, but also by the general public (Why are we funding this research about
levitating toads?!?!). Shielding the researchers from such outrage and
building acceptance for (seemingly) useless research will be just as important
as the article's suggested new metrics for novelty.

~~~
mjfl
People say you need to let scientists do useless exploration in order to
promote innovation. I think that is true to a degree, but definitely should
not be the spirit of any policy where innovation is the goal.

Many of the scientific innovations that pushed the 20th century forward were
in fact purposefully done. The Von Neumann architecture was not invented to
have fun, it was invented to aim artillery and design nukes.

Similarly, much of the tech infrastructure in Silicon Valley descends directly
from radio engineers coming out of WWII theater.

Bardeen, Brittain, and Shockley didn't invent the solid state transistor
because it would be interesting, they invented it because it would allow
miniaturization of vacuum tube computers.

The laser was invented for telecommunications purposes. There is now an entire
field of "quantum electronics" describing the theory behind them.

Even going back further, the invention of the steam engine preceded the
development of thermodynamics, which initially sought to describe the
limitations of these engines. IF "science leads technology", then one would
expect steam engines to be rationally designed from the results of
thermodynamics. The opposite is true.

I think there are enough examples of scientific fields emerging from
technological innovation that "uselessness" should not be considered
correlated with innovation.

~~~
nine_k
The von Neumann architecture became possible because Church and Turing wrote
their highly theoretical, mathematical works.

Lasers became possible because people like Fresnel studied the properties of a
highly impractical phenomenon, coherent light, while other people discovered
another short-lived curiosity, inverse energy levels population.

Many such works had to be done decades before any engineering applications, or
a prospect thereof.

Science is when you study the literally unknown, including no known practical
applications.

When you study small pockets of unknown in a generally understood and
practically fertile field, it's engineering.

~~~
mjfl
> The von Neumann architecture became possible because Church and Turing wrote
> their highly theoretical, mathematical works.

I don't know about others but Von Neumann architecture is much more described
by Church and Turings work than derived from it, like steam engines are
described by thermodynamics. Neither invention was actually dependent on the
theory.

Science often derives from engineering, not the other way around. Get over it.

~~~
Karrot_Kream
> I don't know about others but Von Neumann architecture is much more
> described by Church and Turings work than derived from it, like steam
> engines are described by thermodynamics. Neither invention was actually
> dependent on the theory.

Huh? Turing's paper on the halting problem and Universal Turing Machines was
released in 1936. Von Neumann, Eckert, and Mauchly's memo was not released
until Jan 1944. There is no way that Von Neumann's architecture is described
by Turing's work, given the 8 year difference between them!

> I don't know about others

I'm puzzled by how you can make such a strong claim about the nature of
innovation without precisely drilling down into the evidence backing your
theory.

~~~
catalogia
Not saying I agree with it, but I don't think 'describes' in the sense it's
being used above is meant to imply causality.

------
awinter-py
> This emphasis on citations in the measurement of scientific productivity
> shifted scientist rewards and behavior on the margin toward incremental
> science

this sounds like the science equivalent of how search engines lead to the
creation of content farms. citation scores are to spam science as pagerank is
to wikihow?

Imagine going through life with a high-tech spam filter from the future that
can filter out wikihow and boring science. Impossible to build, I suspect, but
that would be the life.

~~~
tryptophan
About the search aspect, I wonder if there is any way to create a custom
search engine or filter out crap from google.

I would love a browser addon that auto-filters out quora, wikihow, w3schools,
techcrunch, etc...

None of those garbage sites are what I'm looking for, ever. I think that,
amusingly, filtering out the top 5% of best SEO optimized sites would make
results much better.

~~~
awinter-py
w3schools loads faster than MDN but I'm mostly with you

reevaluating our social attitudes towards 'trust in experts' will be an
aftermath of the covid crisis, and it may bleed into search & social media as
well -- is there value in 'peer-reviewed everything'?

(for some definition of peer)

~~~
petra
What is the alternative for the "trust in experts" model, for most people ? Is
there any ?

~~~
op03
Maybe more multi-disciplined group based (given how much more complex most
problems are for individual experts to grapple with by themselves). Probably
rated like team sports.

------
laughinghan
This premise that novelty should be better-rewarded seems odd to me, because I
thought a common complaint about scientific journals was _overly_ rewarding
novelty instead of robustness? Isn't that considered one of the contributing
factors to the replication crisis?

For example:

"Novelty in science – real necessity or distracting obsession?"
[https://phys.org/news/2018-01-novelty-science-real-
necessity...](https://phys.org/news/2018-01-novelty-science-real-necessity-
distracting.html)

"Facts Are More Important Than Novelty: Replication in the Education Sciences"
[https://journals.sagepub.com/stoken/rbtfl/w5mrNxPVD8zSg/full](https://journals.sagepub.com/stoken/rbtfl/w5mrNxPVD8zSg/full)

In the text: "Funding bodies and academic journals that value “novelty” over
replication deserve blame too." [https://theconversation.com/science-is-in-a-
reproducibility-...](https://theconversation.com/science-is-in-a-
reproducibility-crisis-how-do-we-resolve-it-16998)

Etc.

I haven't made it through the entire 43-page paper yet, but a quick search for
"replic" and "repro" suggests they don't address this point at all.

------
nullc
I think it's interesting to think about the incentives in scientific
publication by comparison with the incentives in HN (or Reddit) comment
posting.

Someone could spend days of time writing a well researched HN comment that was
exceptionally informative and accurate. There are occasionally comments that
represent an hour or two of work on them... But multiday-effort comments are
non-existent: the incentives of the venue don't reward that effort. And if you
do take the time the discussion will have moved on before you get it
published. If you published them they would likely be lost in a see of other
low effort comments, or if they were acknowledged-- not likely much more than
comments that merely took an hour.

You don't just see fewer multiday comments, you see essentially none at all.
Nothing technical about HN or Reddit _prevents_ people from writing some epic
work of commentary or research in a comment, and there would often be value
from such works existing...

The same pattern exists in academic publishing but with the effort levels
shifted up one or two orders of magnitude: the maximum moves from an hour to
(say) weeks (exact threshold varies by field). Works taking more effort than
some field specific cutoff are extremely seldom done. Why make one 10x effort
paper when you will serve your interests much better and with lower risks
making 10 1x effort papers?

This would be fine if all of science in that field could be done in the window
of efforts allowed by those incentives, but that isn't the case... especially
since a lot of the low hanging fruit-- results that can be obtained below the
cutoff-- is already picked.

So this is part of why you see things like immunologists pointing out that
there has been relatively little academic investigation of virus seasonality--
though its an apparent, interesting, and seemingly important phenomena.
Studying it in any depth would require experiments spanning years and likely
dealing with human subjects on top of the possibility that your ideas don't
turn up anything new... a lot of risk for someone who actually needs to get
things published.

~~~
knzhou
The extent to which this is an issue depends on the field, though. In my
field, you can get massively rewarded, in both soft and hard criteria, if you
put effort into writing a good review paper. And many of the giants of the
field have sunk even more effort into writing textbooks, which yield even
greater rewards. People can do this precisely because the incentives are
_better_ than for internet comments; a good textbook can be celebrated for
decades.

~~~
zrkrlc
Care to share which field you’re in?

~~~
knzhou
Particle physics. I'm rereading a 30 year old textbook with 10,000 citations
right now.

------
bognition
There have been converastions for years about the slow down in scientific
discovery. Often times the explanation is that science is getting more
expensive, something i've often felt was a bit of a straw man argument.

Instead pointing at the current incentive structure (that everyone already
agrees is broken) makes a lot more sense.

So lets start a conversation about a better incentive structure. What do we
want to incentivize? How do we do that? How do we fix science?

~~~
wegs
I would start by cutting academic salaries at elite schools. These are
$150k-$200k for a typical professor, and $1 million at the top. Maybe $100k at
the bottom.

I'd place these at $60k-$100k: enough to live on, but not enough to go into it
for anything other than love-of-science. A university president might hit
$200k.

I'd hire many more academics, and given them much more freedom. Not as much
publish-or-perish, and more intellectual exploration. Anyone qualified to do
research should have the option to do so in their field of interest.

That's kind of how academia used to work before massive endowments.

I might also do something about tenure. It seems like an obsolete idea as
structured right now. It's not a horrible idea, but it's obsolete in a lot of
ways. It forces people to put in massive efforts early-career. That, for
example, it doesn't line up with biological clocks, and puts in many other
bizarre incentives. I don't mind someone gaining tenure if they've done
fantastic work, mind you, but it shouldn't be a 7-year clock. For example,
perhaps you're a professor with a 5-year renewable contract. If you do
fantastic work, you become a professor-with-tenure, whether that's 4 years in
or 40.

~~~
ramraj07
I will disagree with every single suggestion you make:

1\. Salaries should be -higher-, not lower! Why will any self respective smart
person want to throw their intelligence away for a pittance? You want them to
be smart with science and stupid with money is it? Live like Diogenes?

2\. There shouldn't be more researchers, there should be less. My decade-long
experience with academia has been that too many people who aren't exactly
scientifically smart (more smart at socializing and grant writing) are too
established. We need to recruit the type of minds that are truly capable of
innovation and make sure they don't have to compete with beuraucrats who're
there simply because they chose biology in undergrad and just kept making the
default career choice every time they were presented one. These new people
should also be REALLY smart, not just marginally better than public. Which
means that there can't be too many of them anyways. They should then be given
resources that don't inherently convert the entire system into a Ponzi scheme
(like the phd system now does). In the grand scheme of things they can be
given lower resources if they are given structures to manage things well.

3\. I'd argue that the tenure system worked quite well despite its flaws. If
anything tenure doesn't give the same guarantees it gave half a century ago,
so people are still incentivized to continue running the rat race. If you
still want to hold them accountable maybe a much longer cycle might be okay,
perhaps 15 years? 5 year contract sounds like hell for most fields. Some of
the most interesting work I did took more than that time to bear fruition and
that's not uncommon.

Only thing I'll agree with you is that we should make sure that whatever new
process is conceived must try to correct perverse incentives for women, given
how the current system plays against some common life choices they might want
to make (having kids).

~~~
amirkdv
I have an agreement and a disagreement with your take:

* I agree that reducing salaries for academics will only make the currently misplaced incentives worse. It will deprive _society_ from the valuable research that more competent and talented folks would have done. In my mind, the key point in all these budgetary conversations is the ballooning bureaucratic/administrative layer that is eating up more and more of the budget _and_ making it harder for universities, as collectives of teachers and researchers, to adapt to the evolving priorities of the real world.

* I disagree that there should be less researchers. Fewer academics, maybe, but we definitely need more people who can make a lifestyle out of entirely or partially doing research. My admittedly anecdotal impression is that for every "not exactly scientifically smart" person who secures tenure, there are a handful of would-have-been-great researchers who just get fed up and leave academia despite having the passion, competence, and the willingness to even make a few sacrifices.

IMHO we have cornered ourselves into a false dichotomy (broad strokes here,
there are exceptions of course): either (a) you are an academic, you have to
pull 60 hr weeks to do any meaningful research, and you have to deal the
ossified structural issues of academia, or (b) you are out in the wild, make a
lot more money, but you spend your 40 hr weeks towards maximizing short-term
profits of your employer. I would think were there a viable alternative to
this dichotomy that many passionate and competent people would happily make
some reasonable sacrifices (in pay, work hours) to engage meaningfully in
much-needed research.

~~~
ramraj07
As someone who has now experienced both sides of the dichotomy (80 hr weeks
underpaid and overworked in academia, 40hr weeks bored out of mind doing
coding), a compromise would be great.

Personally I'm hoping to keep my job to pay the bills and start a garage lab
and pursue my passions in science. It'll severely be underfunded, but I'm
hoping to conceive things that can be done with machines that hopefully costs
only as much as a boat would. Further, Richard hamming is on to something [1]
when he suggests that great science happens when resources are scarce.

[1]
[http://www.paulgraham.com/hamming.html](http://www.paulgraham.com/hamming.html)

------
DubiousPusher
There is so much about papers like this that make me skeptical.

1) It assumes we've been good at measuring growth. Which to me is dubious. Our
current system counts the production, deployment and detonation of a bomb all
as positive production meanwhile not counting unpaid domestic work. I know
economists don't like to out a value to different types of services and goods
because they feel they are putting a finger on the scale. But they are doing
just as much by using a blind metric which ends up counting some work and not
other.

2) It never cost adjusts growth. For example, all that wonderful growth in the
midst of the 20th cwntury incurred significant externalities. The system we
have now tries much harder to make producers realize their externalities. This
curbs growth which is not necessarily a bad thing.

3) It assumes we can atomize historic growth and single out what things
contributes to what. But this seems ridiculous especially how these factors
interact and are not necessarily seperable. For example, we happen to live in
a universe in which the movement of electrons can be used transmit energy and
appropriate it to many tasks. The story of the 20th cwntury's economy is very
much the story of how we mastered this one property of nature. Electrification
doesn't just provide heat and light. It enables the transportation of water.
It enables the construction of more and larger structures. It enables the
creation of aluminum and plated metals. Aluminum itself allows many features
of our world we take for granted. From airplanes to electronics. But does
abundant aluminum lay at the feet of scientific innovations? In its infancy
for sure but after some basic science it is widespread cheap electrification
which allows us to mass produce aluminum. How much of that economy do we count
to cutting edge science? It's not easily disentangled from popular politics
which allowed the mass construction of hydroelectric damns throughout the U.S.
and other countries. There is a possible future in which for centuries, growth
has slowed to something more than Renaissance levels but well below mid 20th
century levels. In such a future we might look back at this period as the time
we mastered the single most useful physical property of our universe and so of
course it was an era of unprecedented growth.

I'm not saying they're wrong. But every time someone from Bill Gates to NBER
talks about this issue of growth, whether it has slowed, why it as slowed, how
it can be increased, I just get this feeling that people are failing to
realized what a unique time we live in. How unique the 150 years preceding it
really are. And how little we know about why it happened.

------
dluan
Economists, policymakers, and scientists studying how to improve science
output have themselves been hamstrung for decades because economists have to
measure _something_ , and for so long it's been nothing but bibliometrics.

Until we can come up with a way to measure the other outputs of science
productivity, we're stuck with this citation-based machine that has all of
this institutional and cultural inertia behind it. Which is why I've come to
believe this kind of change or new introduction of value isn't going to come
from within academia. E.g. novelty is in some fields directly at odds with
what makes you a "productive" professional scientist.

Last year I was in the NBER's Science of Science Funding working paper
session, and most of the datasets discussed are still heavily focused on
patents, citations, and bibliometrics
([https://projects.nber.org/drupal/SOSF/data](https://projects.nber.org/drupal/SOSF/data)).

~~~
jackcosgrove
I think the answer is that trust should be part of the equation.

"If you can't measure it, you can't manage it" is a description of how to keep
track of a low-trust system. Scientists are generally conscientious people,
and bean counting is demoralizing.

If there is some organizational solution, I think it should be to keep
organization size small enough that it can be governed by personal
relationships and trust.

Otherwise social trust takes decades to build up, and I don't think there is a
quick fix.

------
alpineidyll3
Simple changes like ditching conference travel and ending grade inflation
would totally repair academia.

I gave up on it myself. I'm happier in finance. Life's too short.

------
parentheses
same has happened to software engineering:

things that have a chance to be quantum leaps for a software project get shot
down. instead we choose incremental improvements with less risk of failure.

incentives driving this include performance review cycle length, focus on
immediate impact, and the startup/vc fundraising norms.

~~~
m_j_g
> things that have a chance to be quantum leaps for a software project get
> shot down

Just genuinely cuorius, what examples do you have in mind ?

Personally i have hope that in 50years homotopy type theory (ane generally
formal methods) will find real applications in the industry

~~~
StandardFuture
"Real applications" start with someone applying them.

------
INGELRII
Same authors have interesting related papers:

* Neophilia ranking of scientific journals [https://www.nber.org/papers/w21579](https://www.nber.org/papers/w21579)

* Age and the Trying Out Of New Ideas, Journal of Human Capital (2019) [https://www.nber.org/papers/w20920](https://www.nber.org/papers/w20920) [http://www.nature.com/news/young-scientists-lead-the-way-on-...](http://www.nature.com/news/young-scientists-lead-the-way-on-fresh-ideas-1.16934)

------
netwanderer3
First of all, I believe the stagnation today in scientific discoveries is
resulted from the lack of big grand visions of a future that can draw
inspirations.

Over the years, society has shifted into favoring financial languages and
metrics in most of today communications instead of telling stories that are
often associated with lifetime generational experiences. Most things are
calculated based on precise risks and probabilities so naturally we would opt
for the least risky path. As a result, the system has evolved into favoring
incremental improvements rather than explorations of uncharted territories
that are much more riskier.

In scientific publishing, this metric is represented by an over-emphasize in
citations which has become the main criteria those publications are now being
evaluated based on. Novelty or a desire for new experience that can generate
large and meaningful impact, or even simply playful experimental ideas are no
longer valued as much. Citations quantity has become the main currency in
scientific publishing, and understandably has also led the community to
prioritize incremental improvements.

In the paper, it mentioned that many seemingly irrelevant or uninteresting new
scientific discoveries initially took a long time for the community to
understand its potential, but those very same discoveries would later lead to
much bigger and more meaningful inventions, such as the gene-editing tool
CRISPR we have today. It took 20 years for this to happen counting from the
initial discovery, so this is where the disconnection occurred.

In that sense, there is a great need to help propagating those initial
discoveries both in its magnitudes and speeds so that it can receive more
attentions from other scientists and community. The novelty should once again
be the main focus to drive motivations and inspirations. Scientific
publications shouldn't just prioritize on hard cold metrics like citations,
but instead attaching more metaphors and new visions of future possibilities
that can excite and propel both science community and public interests.

More than ever, people today are craving for that common naivety which used to
connect everyone together into believing making the impossible possible.
That's precisely what has made Elon Musk and his companies so successful.

------
maxfan8
Perhaps initially, citation indexes were useful (i.e. they were a good metric
to solve certain productivity issues), but lost their usefulness since people
started to optimize and target that specific metric (along the lines of
Goodhart's law).

------
H8crilA
There's a much much simpler explanation for the stagnation of real GDP. You
have to look at energy consumption (joules / year). The correlations are
uncanny and seem to explain just about everything.

[https://youtu.be/wGt4XwBbCvA](https://youtu.be/wGt4XwBbCvA)

[https://www.resilience.org/stories/2015-02-06/charts-
showing...](https://www.resilience.org/stories/2015-02-06/charts-showing-the-
long-term-gdp-energy-tie/)

[http://illusionofprosperity.blogspot.com/2012/04/total-
energ...](http://illusionofprosperity.blogspot.com/2012/04/total-energy-
consumption-vs-real-gdp.html)

It's so simple and accurate that I have no idea why doesn't everybody know
about this. It better than discovering central banking.

------
StandardFuture
Related:
[https://news.ycombinator.com/item?id=18493019](https://news.ycombinator.com/item?id=18493019)

------
Gatsky
Part of it is also the research workforce composition. Research students are
increasingly people without other good options or who feel like they need to
do it for status reasons. It is a form of low paid labour that they take in
preference to working low status jobs, or moving away from where they are
currently living. Of course it is hard for them to take on bold projects, they
are too busy trying to survive and ultimately aren’t that interested in the
material. The recent protests from UCSC students saying they can’t afford the
rent, and the constant blog posts from long suffering adjuncts are classic
examples of this - why would anyone put up with these ridiculous arrangements?
PIs don’t care about this stuff, they need warm bodies to churn out papers,
and the pay is so low it doesn’t matter. This transition is enabled by the
massive expansion of university footprint, staffing and student count.

The higher degree has become the current generation’s desperate attempt to
show they are making progress after the boomers.

