
Retire This Idea: Scientific Knowledge Structured as “Literature” - jordigh
http://edge.org/response-detail/25514
======
jgmjgm
I what is missing from the article is an understanding of the incentives of
academics. Quite often putting a piece of research "to bed" is the goal. Why?
Because any good academic has a long list of things they want to get to.

Very few people want to get mired in the (inevitable) problems that arise in a
paper for eternity. This is why there are review papers that summarize the
state of the knowledge at a given point. These articles are useful summaries
of what came before and what (at least appear to be) dead ends were found.

Maybe I'm missing some point with the author's article. But getting units of
production out and finished (i.e. papers) is a useful process. I'm not sure
what would be gained by keeping documents editable forever. Scientific
literature is not code.

~~~
xiaq
Exactly.

We often have the idea of "solved problem" in science, but very few of them in
software development (overgeneralizing a bit, any field of technology). This
is probably because science is often about "what" (what is the fastest
algorithm for matrix multiplication?), "whether" (does P = NP?), while
technology is often about "how" (how to implement matrix multiplication
efficiently?).

Once you solve a problem in science, you solve it for good. But in technology
there can always be "better" ways to solve it, and things keep evolving.

~~~
return0
can you give an example of _any_ problem in science (not engineering) that is
definitely solved?

~~~
tormeh
All that is known in the entire field of mathematics. But yeah, mostly just
mathematics. I think sorting algorithms is a solved problem, but you can say
that's just math as well.

~~~
seanmcdirmid
Even math systems are engineered somewhat: we can always invent a new and
better math. It has happened a few times in history.

------
dalke
While not essential to the argument, it uses software development as an
analogy for scientific publications. Unfortunately, that analogy isn't
correct.

> Just as the software industry has moved from a "waterfall" process to an
> "agile" process—from monolithic releases shipped from warehouses of mass-
> produced disks to over-the-air differential updates—so must academic
> publishing move from its current read-only model and embrace a process as
> dynamic, up-to-date, and collaborative as science itself.

"Agile" is one name for a long line of software processes, which includes XP,
RAD, and Unified Process. Iterative development stretches back to the dawn of
software engineering. In modern practice, "agile" seems used for anything
which is even a touch more flexible than the strictest of waterfall. Take your
process, add a standup meeting or story cards, and _poof_ , you're agile.

Nor is "agile" development tied so closely to 'over-the-air differential
updates'. The development methodology has very little to do with deployment.
Perhaps the easiest example is to look at the various language implementations
(Python, Ruby, gcc, etc.). They don't have auto-update. So I guess the
question is, are they agile?

> But academic literature makes no distinction between citations merely
> considered significant and ones additionally considered true. What academic
> literature needs goes deeper than the view of citations as kudos and shout-
> outs. It needs what software engineers have used for decades: dependency
> management.

> A dependency graph would tell us, at a click, which of the pillars of
> scientific theory are truly load-bearing.

I have no idea of how to make this work. I give as an example the Kabsch
algorithm algorithm for RMSD minimization of two molecular structures (see
[http://en.wikipedia.org/wiki/Kabsch_algorithm](http://en.wikipedia.org/wiki/Kabsch_algorithm)
). The original paper contained an error that sometimes resulted in a sign
error. This was fixed in a subsequent paper.

Most cite the first paper, because it introduced the core concepts. Some cite
the first paper but actually implement the correct solution from the second
paper without even realizing there was a problem, because they combined the
principles with their own mathematical understanding rather than duplicating
the exact steps in the paper. Still others cite both papers.

How does this hypothetical dependency graph incorporate those details?

That paper is a simple one, because it really is a single algorithm for a
single goal. Many papers introduce multiple concepts, solutions, or
observations. It can well be that one of them was wrong while the others were
correct, interesting, and influential. For example, in one paper I read the
authors did not include a test case because, they said, the source material
was invalid. I investigated, and discovered that it was due to a transcription
error by the authors of the paper, and not a problem in the source material.

The addition of that one data point would have no material impact on the
overall paper. But it's still wrong.

How does his hypothetical dependency graph capture the multitude of
disprovable points in the average paper? How does it capture what the
"pillars" are?

Bear in mind that there's been over 50 years of attempts to do this, so I
think there's no easy answer.

~~~
jordigh
> How does his hypothetical dependency graph capture the multitude of
> disprovable points in the average paper? How does it capture what the
> "pillars" are?

How about bug trackers? Papers that are continuously revised as they're
published? Scientific prestige based on how often your papers are cited and
how active their bug trackers are?

That's how we do it in programming. I think the analogies could be stretched
to scientific publishing.

~~~
dalke
Could you be a bit more concrete as I am having a difficult time trying to
understand how this would work?

To start with, if there is no activity in the bug tracker, doesn't that
indicate success in the paper, because it contained no problems? Or should I
put in a few silly errors (misplaced commas, typos in the citation list, etc.)
in order to increase the bug activity?

You suggest that programming prestige is based on the activity of the bug
tracker. Can you give some examples? As an admittedly extreme case, TeX has
has very few bugs, and has no bug tracker, but is a well known project.

Bug trackers only work for active projects. If the main author writes a paper
in solid state physics, is awarded a PhD, and gets a job working for Seagate
on high density magnetic compounds for storage tapes, then why should the
author care about maintaining the paper's bug tracker?

But getting back to the topic, the dependency graph for programming is
extremely coarse grained. For example, here is a recent bug for Python -
"integer overflow in itertools.permutations" \-
[http://bugs.python.org/issue23363](http://bugs.python.org/issue23363) .

Given your software dependency graph, can you tell me which programs that
depend on Python are affected by that bug?

Because that's the sort of thing that Brian Christian (the author of this
piece) wants for scientific publications:

> A dependency graph would tell us, at a click, which of the pillars of
> scientific theory are truly load-bearing. And it would tell us, at a click,
> which other ideas are likely to get swept away with the rubble of a
> particular theory. An academic publisher worth their salt would, for
> instance, not only be able to flag articles that have been retracted—that
> this is not currently standard practice is, again, inexcusable—but would be
> able to flag articles that depend in some meaningful way on the results of
> retracted work.

How do I indicate "in some meaningful" that my program depends on
itertools.permutations() not having an integer overflow?

~~~
jordigh
Every paper has problems. Yes, even typos should be mentioned. Why not? They
might trip up someone.

Perhaps instead of structuring the whole thing around papers, it should be
something like Wikipedia and editing articles in it.

~~~
dalke
I still don't see a concrete explanation of how this would work. Instead, I
see you tossing off possible solutions that don't pass even a basic test of
feasibility.

Is bug tracker activity a sign of a good paper? Or a poor paper?

The use-case I objected to, from the original article, is:

> A dependency graph would tell us, at a click, which of the pillars of
> scientific theory are truly load-bearing. And it would tell us, at a click,
> which other ideas are likely to get swept away with the rubble of a
> particular theory. An academic publisher worth their salt would, for
> instance, not only be able to flag articles that have been retracted—that
> this is not currently standard practice is, again, inexcusable—but would be
> able to flag articles that depend in some meaningful way on the results of
> retracted work.

How would structuring it "something like Wikipedia" make this use-case more
feasible? It seems instead like you are talking about a completely different
topic.

~~~
darkmighty
Why do we need all things to indicate paper quality? A lot of bug tracker
activity might be both a good or bad sign, but more important that's not
relevant.

What one should aim for is improving research quality and productivity.

------
kriro
The two things that are most ridiculous about science in this day and age
(imo, I work in the field)

1) Tons of papers get published without the underlying data. Why is there no
easy system in place where I have to upload all data from experiments etc. and
if the data isn't there to be verified -> questionable research at best (or in
the current framework of science autoreject)

2) Paywalls and the entire publishing system. Most science is state funded in
some way, shape or form and as such it should be available for free for all
citizens.

I think publish and improve is an ok model but there's some issues like the
author says it's pretty strange how the whole redacting a paper system works.
It's also nontrivial to say "well I was wrong in my paper X as shown in papers
Y and Z" (some sort of HTTP redirect to X/Z would be in order)

Maybe I'm just not aware of it but it would also be interesting to have a huge
Dung-style argumentation system of papers for fields (X attacks Y, Z defends Y
etc.). Might be a neat project if it doesn't exist.

~~~
cafebeen
These things take time and money to build and maintain though, and there's
very little of that to go around unfortunately.

------
anifow
It seems the problem is that the author is trying to create a representation
of knowledge, whereas research papers are more of a logging of work done.

His approach may be a perfect fit for an experiment in meta-research. You
could run periodic reviews of articles released in specific topical areas and
create summaries of the findings. Any time those findings change, its a git
commit, and you can track this change over time. It's something like
Wikipedia, but designed for research. I would not be surprised if this already
exists.

I see a challenge in figuring out the document structure which will be the
most conducive to distributed version control (pull requests, etc), and can
provide some insights on a historicalbasis. For example, you could run these
summaries retrospectively, say looking at DNA research in the 1960s, and do a
separate commit for every key finding through the years. It seems to me that
you would pretty much have to specify a specialized coding language for
scientific knowledge for this structure to work.

~~~
kijin
Exactly. The problem is that papers aren't worth what the rest of the world
think they are. Nobody who is working on the cutting edge of a given field
give a damn about the papers that are published in the field's major journals.
Papers are little more than permanent records of what people talked about at
some conference several months before, and through other informal channels
even earlier. By the time they're published, they're already old news. They
may be worth some archival value, but that's about it.

Unfortunately, the rest of the world thinks of papers as the primary method by
which scientists exchange ideas. This is a myth. Sure, there some scientists
(mostly in developing countries, or those coming from a different field) who
rely on papers to figure out what their colleagues are up to, but if so,
that's only a reason to improve real-time communication, not a reason to turn
papers into real-time communication tools.

~~~
christudor
This and the previous comment are /exactly/ right, especially: "... that's
only a reason to improve real-time communication, not a reason to turn papers
into real-time communication tools".

------
scythe
When all you have is a hammer, everything looks like a nail. The hubris on
display here is: "it works for me, it must be right for everyone". What the
author is missing is that the article-reference system is actually _more
flexible_ than a tree of dependencies. Consider the obvious example: citing
Wikipedia. It's a pain, because Wikipedia changes, so you have to link to a
certain revision and if you want to know the author you have to trawl the
contributions, which nobody does. And this is true of Wikipedia because
Wikipedia is a work in progress: it's not meant to be a _presentation_ of
knowledge but a _repository_ of knowledge.

Science works like that too, but on the blackboards and notebooks and today
blogs and comments of the investigators, which lack real start and end dates
and are regularly crossed out or replaced. The work is refined for publication
because publications are essentially learning materials, they're written and
edited to be as comprehensible as possible for any active researcher in the
field to read for education or leisure; likewise in order for citations to be
useful the cited works must include some background information which goes a
long way in keeping the sciences connected, because it's not rare for
analogous problems to pop up in unrelated fields, and it is crucial that
researchers from outside each others' fields can read each others' work in
order for interdisciplinary collaboration of this form to be possible. But all
of this background information is just background noise to the people actively
working on a problem and that's why it isn't on the blackboard or in the blog
post, and it's why terse notations like dummy indices became popular in
scratch work. The divide between the didactic and generative arrangements of
scientific work is better explained thus: papers are _binaries_ , not source
code. Binaries don't go in source control.

------
brchr
Author here. Delighted to see this near the top of HN this afternoon, and to
see the healthy discussion it's provoked. Happy to answer any questions as
best I can if you've got 'em.

~~~
tokai
Are you familiar with the field of bibliometrics?[0] It is a not-that smalish
research area that works on quantitative research studies. Qualifying types of
citations have been a pipe dream there for decades. There are a lot of
scientific literature on the subject out there.

[0] Or sciencemetrics, informetrics and other assorted subfields

~~~
brchr
I'm aware of portions of that literature, but haven't studied it explicitly.
What do you think are the most interesting things happening in that space at
the moment?

For me one of the most salient things in that space -- albeit a different
point than the one raised in the essay -- is the debate about what sorts of
metrics are most appropriate (if any) for informing decisions about hiring,
tenure, and the like. In some ways it's quite like the be-careful-what-you-
wish-for problems any business has in choosing which metrics to care about.
For instance, the h-index might shape a researcher's choice of whether to
publish something as one larger paper or two smaller ones.

As Sam Altman put it, "It really is true that the company will build whatever
the CEO decides to measure." I think a similar principle holds for research
and scholarship.

------
entee
Speaking from the perspective of life science, I think the idea of the
"article" and "review" as two separate complimentary parts of constructing
knowledge actually makes a lot of sense.

An individual article is an argument, with evidence for how some process
works. A review looks at all the papers, getting the more global view of
"knowledge" the OP seems to want.

Perhaps in the spirit of the article we should have reviews that are ongoing,
sort of like wiki pages for various research areas. These might capture a
living, complete state of the field better than current review articles.

It's really unclear how this would be done though because:

1.) How fields and areas interact is constantly changing, how do you manage
the merging and splitting of review articles? You end up with a confusing view
of "what we know" once again. 2.) As others have noted, who gets to write
these things? The cutting edge literature is full of big personalities in
conflict with one another. It takes years to figure out who is actually right.

That said, at a minimum retractions should be much easier to track. I'll bet
we'd save a whole bunch of wasted effort.

~~~
tormeh
Well, now Wikipedia is the review. Depending on the field, you may just do all
the real research on Wikipedia and then add in citations in your paper to look
professional.

How about a Wikipedia with original research? "Sciencepedia"? Each article
gets a section for each editor. If you want to contribute you can either
submit a pull request to an editor you view favourably or you can write your
own section. Sections can be voted up or down.

~~~
jszymborski
Wikipedia is not the review, not in the life sciences sense of the word.

A review, despite "reviewing" the literature, offers a unique perspective, and
_does_ have an argument, although one that is based solely on the aggregation
of existing evidence and literature.

Wikipedia just describes parts of the field, reviews offer insight and
perspective.

------
xiaq
Cool ideas. We will probably end up with a better world if someone implements
them, but I would argue that these "agile" things are much less important in
scientific publications than in software development. Science and technology
are different.

Software artifacts are huge and uncontrollable. Today it is very rare for a
software developer to know all the ins and outs of a software he/she develops,
_along with all its dependencies_. There is simply way too much code, and it
is impossible to control them manually.

On the other hand, scientific papers tend to contain very few core ideas. It
is still reasonable today for a researcher to understand everything he/she
publishes, along with every idea in all the papers he/she cites. Once you get
the core idea, you get the whole paper. They merely act as mediums for
conveying ideas and there are not _that_ many of them. Often you can just
"manage" them with your own brain.

Anyway, I resonate with the first few paragraphs about retraction a lot. It is
frustrating to read a paper for a whole day, find out that it is wrong, and
that someone already refuted it in another paper :-/

~~~
dalke
What about scientific papers that depend on software artifacts?

Either scientists somehow understands all the "ins and outs of a software
he/she develops, along with all its dependencies" in order to publish a paper,
where a programmer doesn't have the same understanding, or you are comparing
two entirely different concepts as if they were the same.

------
cyphunk
> A dependency graph would tell us, at a click, which of the pillars of
> scientific theory are truly load-bearing.

This is the problem with the academic form of knowledge transfer as a whole.
It's too dependent on feeding of ego. And the "improvements" discussed here-in
only reenforce that. If you boil knowledge down to meme, the necessity to
"weight" and define pillars becomes less relevant. Without the ego's you get a
remix culture where the Amen Brothers are more highly valued than say some one
hit wonder... but not because we've measured them, and because we know that
sample was from the Amen Brothers. It's more highly valued not because we know
the source. In-fact few would know the music they are listening to has the
Amen Brothers sample in it. It is more highly valued because it has taken so
many forms and created actual value.

We don't need tools to allow for better attribution and knowledge measurement.
We need to remove ownership from knowledge as ownership is a hindrance to the
transfer and consumption of it.

------
tikhonj
Personally, what really gets me with the current publishing regime is how
fashion-driven it is (at least in many parts of CS). Oftentimes, it feels like
the most challenging aspect of producing publishable research at first is not
rigor or novelty or writing a sufficiently readable complete paper but hitting
something sufficiently "interesting"—largely a function of what happens to be
popular at the moment.

Publishing has other issues, but they can be mitigated by releasing results in
forms besides simple papers. (Again, I'm mostly talking about CS.) A lot of
successful projects end up releasing blog posts, enhanced online resources,
libraries or complete, living open source projects.

At least in CS, this is happening enough to be useful even if it is by no
means universal. This even comes up in more theoretical fields—somebody
recently pointed me to a website[1] on tree edit distance, which is a great
example. (I really wish it had existed a few years ago when I needed the
algorithm originally!)

[1]: [http://www.inf.unibz.it/dis/projects/tree-edit-
distance/](http://www.inf.unibz.it/dis/projects/tree-edit-distance/)

~~~
jonlucc
In biology, there are a large number of _very_ specific niche journals such
that a paper on a topic out of vogue is likely to land in a more specific
journal. This is not without drawbacks, but it sort of works.

------
Trombone12
First of all, there exists "living review" journals, where authors are
mandated to regularly update their texts every now and then to reflect new
developments in the field covered. This seems like something the author ought
to really comment on.

Secondly, while research and software development have many similarities, the
idea that academic research is best conducted with methodologies created to
allow efficient development of CRUD applications which nice interfaces is not
very credible.

------
justbecause
I like the concept, however I'm afraid "Going Agile" is not a solution for
science, it's just more jargon.

Would be nice to see science testing several different methods though. Also,
it feels like some areas of discipline might work better with different
workflows. For instance theoretical physics may need a different process than
structural engineering or chemistry. I think it's also weird that we currently
use very similar processes across disciplines when there are surely better
tools that could be made specifically for each discipline with
interoperability between disciples handled on another level.

The right tools for the job would hopefully lead to more science faster.

------
nopinsight
One possibility is to keep the current publishing model but add an additional
"meta" layer in a central repository to indicate the relations between the
contributions of each paper.

The repository should be independent of an individual journal or academic
field so that multidisciplinary collaboration would be easier and more
visible. If designed well, cross-pollination of ideas between fields may
multiply.

Each node in the dependency graph should not represent an entire paper, but a
chunk of contribution (knowledge, information, or experimental data), with an
option to delve deeper to see its source code/raw data. A hierarchy of
compositional structures can represent broader units of knowledge. Citation
links should allow for labels to indicate the reason(s) for citation: prior
work, refutation, tool, methodology, etc.

This is a more practical transition than revamping the whole system and should
still add much valuable information for researchers and consumers of research.

If realized, an automated system may be able to analyze the graph for
additional insights into our knowledge structure and point to interesting
research directions as well.

~~~
tormeh
Sciencepedia, with hyperlinks.

------
ThomPete
Yes. Scientific knowledge structured as literature means scientific knowledge
as subject fields which basically means a lot of lacking knowledge between
these fields.

It's not that we should not have specific categories but rather that we should
have a lot more than we have right now.

"Removing literature" as a framework for knowledge will mean the flourishing
of knowledge that can only be created by combining several fields.

------
gojomo
Baby steps. Can we just ban using two-columns in PDFs?

That single change will speed the rate at which new results can be read and
understood.

~~~
IndianAstronaut
Forget PDFs. JOVE and other video scientific journals should become the norm.
Much less chance of fabrication and much better replication if you can
demonstrate on video.

------
mhuffman
There has also been a big push to only count "recent" scientific literature as
valid for reference in some scientific fields. I know that this helps keep any
new papers "fresh", but I have always had a bitter aftertaste about the way
that still-valid, but older, research is looked down upon.

------
tehchromic
I see his point. What's true of scientific literature is true of all
literature. But the solution doesn't match the ambition of the article's
title. Charts and graphs, and all the high tech tools available are certainly
potentially beneficial, but what's needed is a new product and center of
culture, and that's a function of ethics. Those are always defined in terms of
the challenge of their day. But tools with out ethics are dangerous indeed.

------
sam_lowry_
I had a similar article written once on lawmaking. In general, many fields
will benefit from the advances in software-defined collaboration.

[https://www.linkedin.com/pulse/20140814084517-815484-lawmake...](https://www.linkedin.com/pulse/20140814084517-815484-lawmakers-
should-learn-from-software-programmers-to-improve-transparency?trk=mp-author-
card)

------
ExpiredLink
The author clearly doesn't understand academia and scientific work.

> _It is time for science to go agile._

Goodness gracious!

------
stellographer
This solution is like trying to stir a pot of chili with an absinthe spoon.

I'm not sure why there's this holy grail of "the unified master version of"
whatever.

Let me give an example. Say I write a paper on the shape of the non-dark-
matter (stellar) density in the milky way by looking at y-type stars and I get
an answer x. Now Bob comes along and looks at y2-type stars and gets answer
x2. People have the idea that you just go to the first paper, add a footnote
to y and x showing alternate values for y2 and x2...

But what that doesn't take into account is the fact that I used telescope a (a
10 meter hawaiian behemoth) and pointed it in one beam of the sky for 8 hours
to get an ultra-deep pencil; but bob used telescope a2 (a modest 1.8 meter in
la palma) that took an all sky survey and only goes very shallow. Now we add
this in a footnote.

Next, there's a critical difference in the stars we studied. My y-type stars
take 8 Gyr just to form, but Bob was using y2-type stars which live anywhere
from 100 Myr to 15 Gyr. So I'm looking at the old stars and he's looking at
all the stars. Since we know that different age stars live in different parts
of the galaxy (old in the halo, young in the disk and bulge), our results are
starting to look not as comparable as we thought... but it's minor, we'll add
a footnote.

But then we realize that, since my old stars are giants and his all age stars
are dwarfs, my stars are way brighter than his. Since my telescope is
monstrous, and his is a small surveyor, my stars actually end up being
observed to a distance 10 times that of his sample. In fact at these
distances, the original model is a bad fit and we need to change from a power
law to an Einasto profile. Bob can do that too, so our answers are easily
comparable, but the Einasto law has more parameters so it would give a worse
fit per parameter value than the power law he wanted to use originally... We
add an appendix to the paper to explain this bit.

Then I notice that Bob's been using infrared data, and in the infrared there's
a well known problem separating stars and galaxies in the data on telescope
a2. In fact, Bob has to write a whole new section on some probabilistic tests
and models he uses to adequately remove these galaxies from his y2 star
sample. My telescope, observing in the optical at high resolution, has no such
problem, so that section doesn't exist in my paper. Bob looks around awkwardly
and stickies a hyperlink to his meta-analysis somewhere in my data section.

Then Jill comes along and says she doesn't agree at all with us; she got value
x3 using the distribution of dwarf galaxies and if you believe in theory z,
then _hers_ is the most accurate answer.

And we tell Jill to go write her own fucking paper.

------
peter303
The title made think this article was going to be about how to write science.
Writing of all kind benefits if you incorporate elements of a story.

------
fiatjaf
Stop writing filler text.

