
Why Academic Papers Are A Terrible Discussion Forum - apsec112
http://rationalconspiracy.com/2012/06/20/why-academic-papers-are-a-terrible-discussion-forum/
======
kijin
Academic papers today are not meant to be discussion forums. To be fair,
undergrads and early-stage graduate students often use academic papers as a
way to figure out what kind of debates are taking place in their field. But
once you get into advanced levels, academic papers are little more than
_archives_ of discussions that you had many months and years ago. All the real
discussion happens in labs, offices, colloquiums, conferences, by e-mail, and
through countless other channels that everyone knows are much more efficient
than academic papers. Only those who are relatively isolated from the rest of
the academic community, like scientists working far away from major research
centers, rely on academic papers to catch up with what everyone else is up to.

Also, sometimes it doesn't matter that the paper gets published a few months
or even a couple of years late. Not every academic field progresses at the
rate of cutting-edge science. In my philosophy PhD dissertation I criticized
works that were 10-16 years old, and even older if you count how long they
took to be published. But those works are still considered a hot topic in
their field. Hardly surprising in a discipline that has been moving slowly and
steadily for thousands of years. The same might be said of many disciplines in
the humanities and social sciences, though perhaps to a slightly lesser
extent. So whether or not papers can be a suitable discussion forum depends on
the field and how people expect their discussion forum to work. Blanket
statements that cover everything from philosophy to mathematics to the latest
nanotechnology are not helpful.

Some of the other problems you mention, though, are real.

------
zmb_
This article seems terribly confused. Of course academic papers are a terrible
discussion forum. They're not supposed to be any kind of discussion forum.

Academic papers are supposed to be contributions to science. They take by
definition a long time to publish because they need to go through a peer
review process, and they are only interest to a narrow group of people because
they typically advance the understanding of a very narrowly defined problem
deep within their field.

Academics don't just sit in a dusty office communicating with the world
through journal articles. We meet in conferences and work in joint projects,
have informal discussion forums and correspond through a multitude of
different channels. Many academics also have non-academic publications about
their research.

Academic papers serve a very specific purpose and they do it well. Just
because they don't serve some other purpose that you'd like them to serve
doesn't make the process broken.

------
ArbitraryLimits
From "How To Do Artificial Intelligence at the MIT AI Lab"
([http://www.cs.indiana.edu/mit.research.how.to/section3.4.htm...](http://www.cs.indiana.edu/mit.research.how.to/section3.4.html)):

After the first year or two, you'll have some idea of what subfield you are
going to be working in. At this point-or even earlier-it's important to get
plugged into the Secret Paper Passing Network. This informal organization is
where all the action in AI really is. Trend-setting work eventually turns into
published papers-but not until at least a year after the cool people know all
about it. Which means that the cool people have a year's head start on working
with new ideas.

How do the cool people find out about a new idea? Maybe they hear about it at
a conference; but much more likely, they got it through the Secret Paper
Passing Network. Here's how it works. Jo Cool gets a good idea. She throws
together a half-assed implementation and it sort of works, so she writes a
draft paper about it. She wants to know whether the idea is any good, so she
sends copies to ten friends and asks them for comments on it. They think it's
cool, so as well as telling Jo what's wrong with it, they lend copies to their
friends to Xerox. Their friends lend copies to their friends, and so on. Jo
revises it a bunch a few months later and sends it to AAAI. Six months later,
it first appears in print in a cut-down five-page version (all that the AAAI
proceedings allow). Jo eventually gets around to cleaning up the program and
writes a longer revised version (based on the feedback on the AAAI version)
and sends it to the AI Journal. AIJ has almost two years turn-around time,
what with reviews and revisions and publication delay, so Jo's idea finally
appears in a journal form three years after she had it-and almost that long
after the cool people first found out about it. So cool people hardly ever
learn about their subfield from published journal articles; those come out too
late.

You, too, can be a cool people. Here are some heuristics for getting
connected: ...

TL;DR Academic papers are for presenting final results and assigning credit,
not discussion.

~~~
FrojoS
"A reference graph is a web of citations: paper A cites papers B and C, B
cites C and D, C cites D, and so on."

A bit off topic, but where is the computer generated version of this graph.
This was the inspiration to Googles Page Rank algorithm, right? We do have
Google Search and even Google Scholar now. But where can I access this
gigantic graph of published research?

~~~
ArbitraryLimits
That's what CiteSeer was supposed to do, unfortunately they quit halfway
there.

~~~
FrojoS
So, what you are saying is, that it doesn't exist? Why is no one building
this? I understand, that there is less demand than for web search. But there
still is a lot of demand from smart and skilled people, no?

~~~
ArbitraryLimits
It's a question of academic paywalls. If all journal papers were freely
available it would have been done by now.

~~~
FrojoS
Thx for your reply. Makes sense. Other than abstracts, the reference are
usually behind a pay wall.

But how does, e.g. Google Scholar estimate the number of citations? Do they
only look at papers which are not behind a pay wall or do they buy a
subscription to all journals?

It still buggles my mind, that the science loving google founders Larry and
Brian haven't tackled this yet.

------
eykanal
With respect to item #1 (huge time lag), it's worth mentioning that in some
fields—I'm familiar with psychology, neuroscience, and biomedical engineering,
but this likely applies to others as well—many if not most labs are in some
form of contact during the course of the year, whether it takes the form of
planned or serendipitous meetings at conferences, email conversations, second-
hand talks through mutual collaborators and similarly-minded faculty, or other
channels. In these fields, it's pretty unusual to be working in a total
vacuum, where your only method of communication is through academic
publications.

------
scott_s
I think the author is implicitly talking about academic philosophy papers.

Regarding point 8, the "bottom line" rule
(<http://lesswrong.com/lw/js/the_bottom_line/>) is about _process_ , not
actual placement of the bottom line. It is about asking how the conclusion and
evidence came about: is the conclusion derived from the evidence, or was the
conclusion stated, and the evidence filled in afterwards to support it?

Sure, I will state my conclusions in the abstract of my (computer science,
mostly systems) papers. But I _arrived_ at those conclusions after designing
and experimenting with the artifact I'm presenting in the paper. That my
conclusion comes first in the paper is not what matters.

I assume the author does not mean to apply point 8 to engineering and
scientific papers. But I want to point this out since he didn't say so.

~~~
voyou
But many of the author's points don't apply to philosophy papers, either -
when philosophers are deciding what to write they don't start with their
conclusion any more than scientists and engineers do. Likewise, the author's
claim not to have been convinced by a philosophical paper seems odd to me, and
suggests he doesn't have much familiarity with philosophy. If anything, I
think there are _more_ influential papers than books in philosophy,
particularly of the analytic sort; think of Quine's "Two Dogmas of
Empiricism," Gettier's "Is Justified True Belief Knowledge," or Putnam's "The
Meaning of Meaning," for instance, all of which are more influential than any
of the books written by these authors.

------
slurgfest
Academic papers fail as a Reddit-style discussion forum because they were
never meant to be a Reddit-style discussion forum.

If you need to discuss things with people, join a mailing list or go to a
conference or get to know them. The paper is not supposed to be the
transitional step where you are bouncing ideas off people.

I agree that academic journals should be open access. I don't agree that it is
unfair for subject matter experts not to publish every crank who comes down
the pike in every journal.

If you want to talk about enforcing conformity as a bad thing, a discussion
forum (particularly one with upvotes and downvotes) is the last thing you
should use as a model.

I guess you are more directly engaged with Less Wrong and Singularity
Institute and other online fora which lack any academic credibility, but sort
of pretend to have some.

------
pgbovine
Who claimed that the primary purpose of peer-reviewed academic papers was to
serve as an all-inclusive, real-time, 'democratic' discussion forum?

Next up: "4chan flame comments are a terrible way to incrementally advance the
state-of-the-art in a narrow academic field, presenting theorems, experimental
data, and citations to closely-related work."

~~~
planetguy
>Next up: "4chan flame comments are a terrible way to incrementally advance
the state-of-the-art in a narrow academic field, presenting theorems,
experimental data, and citations to closely-related work."

Now I'm tempted to test that, by submitting my next paper as a 4chan comment
and seeing what kind of response it gets.

~~~
pgbovine
semi-related ... some of my colleagues wrote a research paper about 4chan (but
I don't think they posted it ON 4chan!):

<http://projects.csail.mit.edu/chanthropology/4chan.pdf>

------
neutronicus
Say what you will about academic papers, but I needed to know how to solve a
tridiagonal matrix efficiently on a distributed computer, and damned if I
didn't find some that told me how.

As a survey of the field, maybe papers in journals aren't so great, but as a
way to find heavyweight answers to heavyweight questions? You bet your ass
they're good.

(Of course my University pays through the nose so that I can do that, but that
has no bearing on the format).

------
rmk2
There is a reason why _conferences_ are so hugely important. Often, what will
be published as an article is presented beforehand on conferences, among
knowledgable and interested peers. And while some conferences are invite-only
or have to be paid for, most universities have a lot of conferences either
completely open to the public or at least open to students. Often, just
_asking_ someone of one is allowed to just audit the talks without going to
all he social events aimed at invited people will be enough in order to be
"let it".

Overall, what bothers me most about this blog post is this:

>> Papers are, intentionally, written for an audience of specialists rather
than a general interest group, which reduces both the tendency and ability of
non-specialists to read them when asked

This repeats the same bias that _everyone can say something about everything_.
Nobody here would complain that mailing list discussions about bugs, features
or programming questions are hard to understand for non-programmers. However,
a scientific discussion in a paper is supposed to cater to everyone. In
reality, especially within the confined spaces of an article, one has to
assume some baseline of knowledge in the reader, because one can't afford to
explain all the basics one implicitly builds upon. To give an example: I do
literary studies, and often people assume that they know _just as much_
without having properly dealt with it, because _hey, everyone can read a book,
right? right?_ While that part might be true, every scientific discipline over
time builds up a plethora of instruments for analysis of their respective
topics, an understanding of the matter's and the subject's history, its
pitfalls, shortcomings and strengths.

Also, I do not understand his paragraph regarding the "bottom line", since I
know noone who writes the abstract first. The abstract is the very last bit to
be written, _after_ the paper's structure, arguments, theses, supporting
evidence and rebuttals are decided, fixed and usually written. What both the
cited Paul Graham essay and his text fail to see is the _process of writing a
paper_. It is true that the actual writing process usually doesn't start with
a thesis statement and a chain of arguments. But structuring a paper is just
as much part of writing as finding words for what one wants to express.

The author hints at being an undergrad in maths, having trouble with the
understanding of state of the art research. A professor I know once said this
about degrees: undergrad & grad work are a necessity, they teach how to start
a car, how to accelerate, break and also drive on basic roads (research
methods/skills). A PhD then allows (and teaches) you to drive on the autobahn
(research), where you can start to close the gap somewhat to all the people
already on it going faster than you do. The actual entry into novel research
doesn't happen until after the PhD. That's where journals and conferences,
workshops, groups, funding etc. become interesting and important. First then
are you within the actual target group for academic papers, since you are seen
as an _equal_ , equal also in terms of discussing on eye level.

Nobody expects complex technology etc. to be understandable by everyone, but
one could say: "Everyone thinks they are a writer, much like many people stand
in front of contemporary art saying they could produce the same thing." What
they fail to understand is the complexity involved in the writing _process_.
What sometimes looks easy or even "trivial" in a finished paper often has a
highly complex, dense and far-reaching background.

[edit] Mhm...I apologise for the length of this comment, it grew larger than I
originally thought it would... [/edit]

~~~
Perceval
I disagree about the value of conferences, especially large ones, for actual
discussion.

If you go to a large conference (e.g. the International Studies Association
annual conference), you are almost guaranteed to be fielding questions from an
audience that _has not_ read your paper.

You would get more valuable feedback on your actual work (rather than your
academic powerpoint/presentation notes) from a small colloquium than from a
large academic conference. The only person is guaranteed to have read your
work is the discussant, and if you're lucky the panel chair and other
presenters.

Conferences are a good way to network and get a feel for the overall state of
the field, but for having actual discussion of a problem or issue area they're
pretty ineffectual.

~~~
rmk2
Oh, I agree with you in terms of big conferences.

I suppose I should have defined "conferences" more closely, which for me
includes workshops, workgroups etc. Right now, my subject isn't the biggest,
so people usually know each other, each other's interests and what most other
people are currently working on.

So yes, different experiences with conferences here. However, you are
obviously right that the bigger the conference, the less constructive the
overall discussion. This in a way corresponds to the aforementioned amount of
readers targeted. If you aim to reach a broader audience, discussion will be
less in-depth, if you specialise and focus on a smaller amount, discussion is
more likely to be more relevant.

------
siavosh
The real problem is that 99% of research publications are, to put it bluntly,
worthless. This is the root problem of the entire system which leads to many
of the problems the author here is listing.

These 99% are often churned out because they help sustain someone's carrier or
lab. To be published, crap research thus needs to convince people that it is
smart and important. This is usually done by being written poorly.

The remaining 1% will progress human knowledge and help other scientists to
continue down the right path. If they are important enough for wider
dissemination, well, that's what science journalists are for.

------
6ren
> 8\. [...] one starts with the conclusion in the abstract, and then builds up
> an argument for it in the paper itself.

Yeah but you _write_ the abstract last (intro too). Having a conclusion at the
beginning is just UI.

~~~
thesteamboat
I cringed at this point. Especially in mathematical fields where the abstract
is in some sense an API -- it tells you what theorems you might use from this
paper and how to apply them.

------
paulsutter
This post might be more useful if it offered more specific alternatives,
rather than mostly criticizing the academic process. Imagine the same ideas
reworded as suggestions for improving the process. Then the posts here could
discuss the new ideas (useful!), rather than debating the criticism
(tedious!).

Perhaps peer review should be done in public, as a page-rank style rating
according to the authority of each reviewer in that domain. Would it be
completely open, single blind? double blind? Good questions. But it might
reduce delays and decrease the effect of politics on the process.

------
niels_olson
I think the central urge in there is to get scientific discussion more out in
the open. The single biggest issue by far, is not the pace of the publication
process or the level of writing. It is access.

I would argue that access would actually solve a number of his other
complaints (poor moderation, percieved time lag, people don't wanna read 'em,
academia selecting for conformity). If we raised the quality of content on the
open internet, the quality of discourse would go up.

------
sentinel
Could this be a niche for a software solution enabling scientists to
comment/share/contribute on each other's papers faster and more efficiently?
Perhaps in the end speeding up research times and in the grand scheme of
things furthering the scientific process?

------
mck-
HN outperforms on all those points

~~~
pgbovine
yes, except for the part about disseminating novel research findings

------
beloch
This article has some good points and some bad ones too.

e.g. Academic papers violate the "bottom line" rule.

If you were making stuff up as you write it down then it would be true that
starting with a conclusion might bias your arguments towards justifying that
conclusion. This might describe how a (bad) journalist writes, but not how
scientific papers are usually written.

The majority of the work for a scientific publication usually comes well
before writing the first word. Theorists will discard many theories before
they find one promising enough to do extensive work (primarily aimed at
proving that theory wrong) before they even consider publication.
Experimenters must work with theories that have been through the above,
conduct a complex (and usually expensive) experiment, and then analyze the
results extensively before they start writing. The writing process can trigger
some consistency checks and digging through references may reveal new spins
for or weaknesses in the theory or experiment. However, if the entire point of
the paper was wildly misguided to begin with, it's unlikely writing the paper
in a different order (so that the authors aren't beguiled by their conclusion)
will be what reveals it.

Now, the big benefit to writing scientific papers with the conclusion
presented right up front is that you don't have to read entire papers to
figure out what they're talking about. Scientists generally don't read papers
to be led down a grand story with a surprise payoff at the end. The first
thing they want to know is what this paper has to say so they can decide if
it's worth reading. Science is big. Really big. Go to arxiv.org (a preprint
repository for both articles that will be published and some that won't). Pick
one specific sub-field and look at how many papers were submitted _yesterday_.
The reason few people read material from outside their bailiwicks is because
they barely have enough hours in the day to keep up with their own field and
do their own work. If scientific papers followed the "bottom line" rule things
would get much, much worse.

Personally, my biggest complaint about journals is how expensive they are. The
only reason university researchers have "free" access to journals is because
the universities pay (frequently exorbitant) subscription fees to the
journals. It's better than having every researcher pay $30 every time they
want to look at a paper, but it's still far from cheap. The really weird thing
is that it costs several thousand dollars to _publish_ a paper in a decent
scientific journal. Yes, you do ground-breaking work, write a beautiful paper
about it, and then you have to _pay_ the journal to publish it if they accept
it, and color figures are almost always extra! Where does all the money go?
Journals have remarkably few expenses. In terms of producing content the only
people they really have to pay is their editors. The authors pay and the
referees work for free. Publishing the content does require some
infrastructure, like servers and code. The print version requires type-
setting, production, distribution, etc... However, I find it very difficult to
believe that journals need to charge what they do to either creators of the
content or the consumers of it. I'd love to see journals open up their books
and reveal where the money is going.

------
api
Upvoted!

Wish I could do that for academic papers.

~~~
agilebyte
This might work for you: <http://www.mendeley.com/research-papers/>

------
planetguy
Academic papers also make terrible pornography. That's not what they're
designed for either.

------
hastur
Well, there's at least one way in which academic papers are a great discussion
forum: People think a lot before posting and do thorough research on their
arguments - and those of their opponents.

