
No Paper Is That Good - zt
https://www.econlib.org/no-paper-is-that-good/
======
anonytrary
> If you want me to read the vast literature, cite me two papers that are
> exemplars of that literature. I will read them. If these two papers are full
> of mistakes and bad reasoning, I will feel free to skip the rest of the vast
> literature. Because if that’s the best you can do, I’ve seen enough.

This is one of worst ideas I've ever read. Wouldn't you just be cheating
yourself?

How is tying your own learning to someone else's ability to find the best
papers in any way a smart thing to do? It would be much better to doubt the
person who gave you the papers (perhaps he miscalculated which two papers were
the best), than to dismiss the entire field.

~~~
jldugger
It's not about dismissing the field, as much as the claim that you must have
read the literature to constructively participate in a debate.

~~~
skywhopper
I think it’s reasonable to expect familiarity with “the literature” beyond
“two exemplary papers” to be given any respect in an academic debate. If you
aren’t reading the lit reviews and broadly in the topic how can you possibly
contribute to the discussion productively? Almost certainly any points you
want to make will have been examined and discussed already. This rule is about
FOMO for wannabe rennaisssance-man know-it-alls who think their gut reactions
are God’s own truth and demand a response from actual experts. If you don’t
know the topic and don’t want to learn about it then please find something
else to do.

~~~
xamuel
Here's an example.

You stumble on a bunch of psychology papers in which psychologists noticed
that whenever 6+ kids get together, there are either three mutual friends, or
three mutual non-friends.

The psychologists think this has something to do with child psychology and
have written 10,000 pages on it.

You immediately recognize it has nothing to do with psychology and is just
basic Ramsey Theory [1]. Should you have to review thousands of pages before
being allowed to chip in?

[1]
[https://en.wikipedia.org/wiki/Ramsey%27s_theorem#Example:_R(...](https://en.wikipedia.org/wiki/Ramsey%27s_theorem#Example:_R\(3,_3\)_=_6)

A less hypothetical example: You're Bertrand Russel and you discover Russel's
paradox, a one-liner which negates thousands of pages of Frege's not-yet-
published logic textbook. Do you have to wait for him to publish it, and then
you read it, before raising your voice?

~~~
joe_the_user
Some scientific subfields are ridiculous on their face.

But there's a crucial difference between that and the way original rule is
phrased.

The fields one has to dismiss are characterized by things one knows is false
by physics or combinatorics or other common sense things. Theories of
telepathy, the flat earth, creation science or whatever.

The "you can't show me one good paper on the subject" is an appealing-seeming
pronouncement but it's really dumb way to do it. Just say, "that doesn't make
sense and you'd need huge evidence to prove it". Paper quality isn't really
the question.

~~~
uncle_d
Regarding telepathy...

[https://www.sheldrake.org/research/animal-powers/a-dog-
that-...](https://www.sheldrake.org/research/animal-powers/a-dog-that-seems-
to-know-when-his-owner-is-coming-home-videotaped-experiments-and-observations)

~~~
Elrac
Sheldrake is my go-to example of bad science.

In one of his experiments monitored by another scientist, when the dog didn't
respond in a way that met Sheldrake's criteria, Sheldrake changed the criteria
in mid-experiment.

That one incident alone should be enough to dismiss any results coming from
Sheldrake. He's not only dishonest but blithely unaware that he's being
blatantly so.

------
kabouseng
Claude Shannon's paper on information theory [1] is possibly an example of
such a paragon of work. No citations, 50 pages of pure awesome, spawned a
whole research field of its own and 70 years on just as valid.

[1] A Mathematical Theory of Communication -
[http://math.harvard.edu/~ctm/home/text/others/shannon/entrop...](http://math.harvard.edu/~ctm/home/text/others/shannon/entropy/entropy.pdf)

~~~
eboyjr
You said it. He proposed the model for digital AND analog communication still
in use today. Although being a digital software person I admit I've never been
motivated to read the part about continuous channels.

I was motivated however to install an archway for my house from the exact
shape in Figure 7 (binary entropy function)

------
traverseda
On the psychology replication crisis

>Overall, 36% of the replications yielded significant findings (p value below
0.05) compared to 97% of the original studies that had significant effects.
The mean effect size in the replications was approximately half the magnitude
of the effects reported in the original studies.

[https://en.wikipedia.org/wiki/Replication_crisis#Psychology_...](https://en.wikipedia.org/wiki/Replication_crisis#Psychology_replication_rates)

Of course if you point out that psychology research is strongly biased, the
immediate response is that you must be an anti-intellectual. As if the system
that produced such misleading results is a representation of all intellectual
pursuits, and not just a mistake. The same way you get called an anti-
intellectual for questioning the post-modern-analyses of whatever.

I'm really sick of people claiming that criticizing obviously bad science is
"anti-intellectual". When you produce that many useless papers you're
obviously, as a field, lacking an understanding of why science is good and
useful in the first place.

I'm presuming that economics has similar problems with replication, and that
you can only trust the most basic and obvious of their findings.

~~~
sarbaz
A lot of this is simply that p < 0.05 is a very low bar for research. If we
want to understand _how_ the world works, we need to be able to construct
mechanisms that explain why the results we measure in experiments come about.
Without this thing, it's not science.

A good mechanism that accurately explains some part of the world can be
validated with a much better result than p < 0.05, because it can be properly
isolated from other factors and its effect size can be strengthened.

Without a mechanism to explain _how_ things work, the experimental results
that get published are probably just noise. And because p < 0.05 is such a low
bar to pass, lots of noise gets published. Trying to reproduce an experiment
whose results were drawn from /dev/urandom is pointless.

In fact I'm surprised the replication was so high. I was expecting it to be
around 5% (i.e. results are totally random). So maybe there is some hidden
merit to all of this

~~~
nicoburns
> If we want to understand how the world works, we need to be able to
> construct mechanisms that explain why the results we measure in experiments
> come about.

This, 1000 times over. Unfortunately we seem to teach that statistical
analysis is sufficient to infer causation in a lot our undergraduate science
programs.

------
hzhou321
I would agree that "no paper is that good" \-- on the first try or even the
second try. Just as we need constantly refactor code, just as we need
constantly re-edit a book, a paper that explains a scientific idea also needs
constant re-write to become good. The sad reality is that scholars simply do
not do that.

It is understandable though. Once the paper is published, then it is no-longer
novel, and there is obviously lack of reward and motivation to write the same
idea again -- just to write it better. ... unless you are actually writing a
review paper, but then, it appears there is little value unless the review is
"complete". A good illustration of the idea need be able to high-light the key
idea while avoid having minor ideas obscure the key. Therefore, a "complete"
review paper rarely provide a good read.

Compared to papers, a good text-book often is a much better read than all the
original papers. It is not really because of the size, rather it is because a
good text-book is written from the reader's point of view and focuses on
conveying the idea itself (vs. selling the idea). And a good text-book takes
many rounds to develop.

There is no reason papers cannot be developed in a similar way as text books:
once a ground-breaking paper is published, it shall be constantly updated,
each new edition reflects what the author has newly learned and incorporating
new development including the entire community...

Alas, that is not the culture, and there is no motivation to do such.

~~~
allenz
In addition to the incentive problem, you would also need to get people to
peer review any changes to a paper. Most of the changes would be minor and
annoying to peer review.

I think that courses remain the best way of distilling and communicating
knowledge. The only problem is that they're not always available to a lay
audience, since there's little incentive to make them available.

The other alternative for the lay audience is science/economics journalism
(Economist, Scientific American, Discover, etc). This works to some extent but
mostly just scratches the surface, since even regular journalism is struggling
to be profitable these days. With deep technical topics there are too few
readers and too few qualified, willing writers.

------
dalbasal
I assume this relates mostly to economics or the humanities (social science,
if you insist) in general. These apply to sciences too, but are less
debilitating in the long run.

 _Fifth, most researchers’ priors are heavily influenced by some extremely
suspicious factors._

Academics in the humanities identify themselves "I am an X." X can be post
structuralist, rational materialist, classical liberal or some other broad,
hairy, intellectual identity. This is bad news for objectivity. Everyone has a
dog in the fight.

~~~
ci5er
It's interesting. As an engineering type (EE, primarily, but I had several
other undergrad degrees), everything was about what is true. How do electrons
move with their respective related fields across geometries.

This other stuff seems to be about "what is useful to us as humans in a group
matrix (culture/community". That seems like it should be studied - but I don't
think we all have the same objectives. Also - some people keep trying to make
it "true" vs. "useful".

What say ye all? Do you think that "True" has any place in the social sciences
vs. the observational "I observe this to be useful in this context" or the
almost psycho-analytical "I notice that when people believe X together,
society does better"?

I don't know - it just seems (to me) that we spend too much time on the truth
as opposed to (subjectively) communally useful. i.e. What should we decide to
believe in as a community to hang together (because otherwise, we shall surely
hang separately)?

~~~
dalbasal
I'm in. Not everything needs to be true in the electrons-in-a-circuit sense to
be useful. A lot of things mgs are subjective, both in the sense that it's
arguable, but also in the sense that how people see it is important that in
itself.

Man may not be the measure of *all" things, but he (she) is the measure of
some things and that's ok. I think a lot of arguments (politics and political
ideologies in particular) would get a lot nicer if we just accepted that we
aren't arguing about absolute truth.

~~~
ci5er
I've always wondered why most people are not passionately opinionated about
methods for heart surgery, and yet are highly opinionated about economics (for
example). The latter seems more complicated, in many ways, and yet people have
no problem having an opinion about it. Why do you think that is?

~~~
aaron_m04
It's probably because economics relates to every adult's life experiences.
These experiences give people a reason to form theories about how economics
works.

------
nerdponx
One conclusion you could draw from this: economics might actually be a giant
mud moat.

~~~
BurningFrog
Economics is several separate sciences, awkwardly placed in the same
department.

~~~
allenz
Most departments are heterogeneous, with mutually unintelligible subfields. CS
encompasses both information theory and robotics. Cultural anthropology has
little in common with medical anthropology.

------
dekhn
Well, Watson & Crick (Nature 1953) is that good. It does contain an error (the
actual DNA structure is slightly wrong) And Avery (JEM 1944)
[http://jem.rupress.org/content/79/2/137](http://jem.rupress.org/content/79/2/137))
which "proved" that DNA is the molecule of heredity, is also there.

But to be qualified to read these papers and appreciate why they are points of
quality within a sea of crap? That's hard.

------
spruciefic399
I was sort of on board with the two-paper rule until I got to the arguments
about the reviews. Maybe in his mind integrating literature into a cohesive
summary (ala meta-analysis) is a novel contribution, but if not, the attitude
behind the two-paper rule is part of why science is in such crisis. The author
is right, that the interpretation of the literature should be based on an
accumulated read, and not one or two papers (unless they're reviews or meta-
analyses).

------
will4274
Allow me to disagree. Chris Okasaki's thesis, "Purely Functional Data
Structures" is That Good for the field of functional programming. There are
other exemplary papers, like Godel's incompleteness theorem, Cantor's diagonal
argument, or Einstein's statement of special relativity. Or Satoshi's paper on
Bitcoin.

These papers are not exhaustive summaries of a field. But a reader comes away
understanding the type of problems a field is devoted to solving and many of
the existing ideas. And I believe that each is a paragon of their field.

~~~
xevb3k
The article limits itself to discussing “empirical paper”s not theoretical
papers.

In a theoretical paper, it’s possible to make statements that stand on their
own merits. In empirical science, a single paper is never really enough to
support an entire field. Empirical sciences generally rely on the development
of scientific consensus.

~~~
antt
That seems like an artificial distinction made up for economics. In physics
there are plenty of theory papers that make very clear and concise
predictions. If these papers hold then they are very much both. E.g. the
original special relativity paper goes out of its way to make predictions and
calculations at the state of the art of the time, which are quite jarring
today.

~~~
eesmith
Dirac's early papers are, from what I have heard, amazing pieces of work. One
commentary about the quality is given at
[https://michaelberryphysics.files.wordpress.com/2013/07/berr...](https://michaelberryphysics.files.wordpress.com/2013/07/berry291.pdf)
.

------
salty_biscuits
Papers aren't meant to be this crystallized nugget of truth, they are a
progress report on an ongoing piece of living research. They aren't meant to
be infallible.

------
6ak74rfy
On a slightly tangential note: how do you guys _read_ research papers? Do you
go through them word by word or you only skim them to get a general
understanding?

I try to do the former but the work seems so boring that I am hardly motivated
to do this more.

~~~
pmiller2
In mathematics:

Read the title and abstract. Then read the statements of the main theorems and
corollaries (skip the lemmas and interstitial commentary at first). At each
step, evaluate whether you want to continue or abandon the paper for whatever
reason (not interesting, not relevant, whatever).

Finally, if you made it this far, read the whole paper word for word with a
pen and paper (or chalk + chalkboard) handy.

For longer papers or books/theses (say, 20+ pages), do this whole process on
each individual section/part/chapter that you want to consume.

It also helps to have a reason to read the paper besides curiosity. I find
things that are relevant to current work to be easier to get through than more
peripheral sorts of things. Having a colleague to discuss things with also
helps.

~~~
sweezyjeezy
FWIW, I don't think this article really applies to areas like maths or
computer science, where the main output is theorems or algorithms that can be
verified line by line. It's more about empirical science where one is trying
to prove/disprove hypotheses through rigorous experimentation.

~~~
pmiller2
No, I don’t think so, either. In math, there are excellent papers that cover
.01% of what needs to be said about the subject, and ordinary or even bad
papers that say much more. Provided there are no serious logical errors, the
difference is simply how important is the question being answered.

Edit: Although it may go without saying, all else being equal, a better paper
is easier to read than a lesser one. But, really, content is king.

------
nazgulnarsil
Is there any cross disciplinary repository of excellent research reviews?

------
gjm11
It seems like a lot of commenters here are under the impression that the "two
paper rule" (if someone says you should read the literature in field X, ask
them for the two best papers in that field they can think of, and if those
aren't impressive then don't bother looking further) is a proposal of Bryan
Caplan, who wrote the OP here.

That is incorrect. The two-paper rule is Noah Smith's, and Bryan Caplan is
_disagreeing_ with it.

(It's scarcely possible to read any of Caplan's post without realising that; I
conclude that many commenters here have not bothered to read the OP before
commenting.)

------
nitwit005
Following the link through to the original proposal, this seems like a
misinterpretation of the suggestion. The problem this was supposed to "fix"
was people using the existence of their being "vast literature" to shut down
arguments.

From that perspective, the suggestion seems fine. You should be able to dig
out two examples that show your field isn't nonsense. I don't think it was
meant to be a high bar:

> There are actual examples of vast literatures that contain zero knowledge:
> Astrology, for instance. People have written so much about astrology that I
> bet you could spend decades reading what they've written and not even come
> close to the end. But at the end of the day, the only thing you'd know more
> about is the mindset of people who write about astrology. Because astrology
> is total and utter bunk.

------
coldtea
> _The best papers get up to around .20. Again, No Paper Is That Good. If you
> demur, consider this: In twenty years, will you still hold up the best
> papers of today as “paragons” or “exemplars” of compelling empirical work?_

If the field is not totally vague, then yes. We can still consider certain
papers in physics, or chemistry, or medicine, computer science etc. as
exemplary decades, or even centuries, later, even when they deal with
empirical work.

Soft sciences need not apply.

~~~
Jweb_Guru
Nonsense. Plenty of old computer science papers don't hold up well to scrutiny
or are largely irrelevant nowadays, such as those that were trying to optimize
for bottlenecks in hardware that no longer exist or have shifted to different
places. Even in pure mathematics, something that was considered a great
insight a long time ago might be considered fairly trivial now, and not just
because the other paper came out first. Just because a paper is correct
doesn't make it interesting or worthwhile--a lot of the best papers have
errors in them, but propose something that's a genuinely new contribution to
the field.

~~~
coldtea
> _Nonsense. Plenty of old computer science papers don 't hold up well to
> scrutiny or are largely irrelevant nowadays_

Nonsense, knee-jerk answer.

First, we don't need all of them to "hold up well to scrutiny" but just 2 (as
per TFA challenge). And we have way more than just 2 -- hundreds of great
papers.

Second, the papers "that were trying to optimize for bottlenecks in hardware
that no longer exist or have shifted to different places" can still be
perfectly valid as per the challenge we have, which was:

1) that they were not "full of mistakes and bad reasoning", 2) that they did
not "contain little or no original work" (and where thus just references and
meta-papers)

The question wasn't if we have "plenty of" papers that are bad, or plenty of
papers that were very good but have been super-ceded by changes in technology.

Just that we have at least 2 (and I argue we have way more than two) seminal
papers that have original work, and are not full of mistakes and bad
reasoning.

------
B1FF_PSUVM
'We are all in a mud moat, but some of us are looking at the stars', as
comrade Oscar Wilde almost said, then?

------
turc1656
I suspect this is limited to economics and the social sciences. There is
absolutely no way this holds up in the natural sciences like math, physics,
etc.

People forget the original name for what we call economics was "political
economy". That alone should tell you all you need to know about the dangers of
treating that field like a science. If you ever want to know why expert
economists can't seem to agree on things that happened 50-100 years ago or
make accurate predictions for the future, the original name is very telling.
Wouldn't it be ridiculous if we were still debating the validity of f=ma or
e=mc^2? Wouldn't it be crazy for someone to claim general relativity is just
flat out wrong, even though GPS systems would not work properly without humans
accepting it?

Why is nothing remotely approaching reasonable standards used in the social
sciences before acceptance?

------
acoye
Bitcoin's paper was that good.

Simple.

Easy to read.

Easy to replicate (a poc in python is easy)

It is a recent key innovation (the blockchain tech).

------
User23
What happens when you apply this author’s standard to this work?

~~~
bpchaps
Heh. From the article:

Does all of this hold for my papers, too? Of course. The most I can claim is
that I am hyper-aware of my own epistemic frailty, and have a litany of self-
imposed safeguards. But I totally understand why my critics would look at my
best papers and say, “Meh, doesn’t really prove anything.”

~~~
User23
I stopped reading before I got to that point thanks to the author’s argument
on account of I took his advice. Thanks for the datum.

