> How does his hypothetical dependency graph capture the multitude of disprovable points in the average paper? How does it capture what the "pillars" are?
How about bug trackers? Papers that are continuously revised as they're published? Scientific prestige based on how often your papers are cited and how active their bug trackers are?
That's how we do it in programming. I think the analogies could be stretched to scientific publishing.
Could you be a bit more concrete as I am having a difficult time trying to understand how this would work?
To start with, if there is no activity in the bug tracker, doesn't that indicate success in the paper, because it contained no problems? Or should I put in a few silly errors (misplaced commas, typos in the citation list, etc.) in order to increase the bug activity?
You suggest that programming prestige is based on the activity of the bug tracker. Can you give some examples? As an admittedly extreme case, TeX has has very few bugs, and has no bug tracker, but is a well known project.
Bug trackers only work for active projects. If the main author writes a paper in solid state physics, is awarded a PhD, and gets a job working for Seagate on high density magnetic compounds for storage tapes, then why should the author care about maintaining the paper's bug tracker?
But getting back to the topic, the dependency graph for programming is extremely coarse grained. For example, here is a recent bug for Python - "integer overflow in itertools.permutations" - http://bugs.python.org/issue23363 .
Given your software dependency graph, can you tell me which programs that depend on Python are affected by that bug?
Because that's the sort of thing that Brian Christian (the author of this piece) wants for scientific publications:
> A dependency graph would tell us, at a click, which of the pillars of scientific theory are truly load-bearing. And it would tell us, at a click, which other ideas are likely to get swept away with the rubble of a particular theory. An academic publisher worth their salt would, for instance, not only be able to flag articles that have been retracted—that this is not currently standard practice is, again, inexcusable—but would be able to flag articles that depend in some meaningful way on the results of retracted work.
How do I indicate "in some meaningful" that my program depends on itertools.permutations() not having an integer overflow?
I still don't see a concrete explanation of how this would work. Instead, I see you tossing off possible solutions that don't pass even a basic test of feasibility.
Is bug tracker activity a sign of a good paper? Or a poor paper?
The use-case I objected to, from the original article, is:
> A dependency graph would tell us, at a click, which of the pillars of scientific theory are truly load-bearing. And it would tell us, at a click, which other ideas are likely to get swept away with the rubble of a particular theory. An academic publisher worth their salt would, for instance, not only be able to flag articles that have been retracted—that this is not currently standard practice is, again, inexcusable—but would be able to flag articles that depend in some meaningful way on the results of retracted work.
How would structuring it "something like Wikipedia" make this use-case more feasible? It seems instead like you are talking about a completely different topic.
Why do we need all things to indicate paper quality? A lot of bug tracker activity might be both a good or bad sign, but more important that's not relevant.
What one should aim for is improving research quality and productivity.
How about bug trackers? Papers that are continuously revised as they're published? Scientific prestige based on how often your papers are cited and how active their bug trackers are?
That's how we do it in programming. I think the analogies could be stretched to scientific publishing.