On: Challenge to scientists: does your ten-year-old code still run?

[OminousWeapons]

I am in 100% agreement and would like to point out that many papers based on code don't even come with code bases, and if they do those code bases are not going to contain or be accompanied by any documentation whatsoever. This is frequently by design as many labs consider code to be IP and they don't want to share it because it gives them a leg up on producing more papers and the shared code won't yield an authorship.

[acutesoftware → OminousWeapons]

If published research is based on a code base, then surely the documentation and working code is equally important than the carefully written paper.

[OminousWeapons → acutesoftware]

I completely agree, the problem is the journal editors and reviewers largely don't.

[freeone3000 → acutesoftware]

No, the paper is what matters. The code is a means to generate the paper.

[bumby → freeone3000]

I agree, but that’s similar to saying the data is what matters, not the methodology.

In the research germane to this conversation, software is the means by which the scientific data is generated. If the software is flawed, it undermines the confidence in the data and thus the conclusions.

[freeone3000 → bumby]

Most researchers would agree with the first statement without significant qualification. Methods are at the end for a reason.

[bumby → freeone3000]

Not disagreeing with your assertion on the opinion of “most researchers” but you’ll often find quite a few people advocating for using the methodology sans data as a means to determine publication worthiness to try and avoid the perverse incentives for novel or meaningful data.

I think it’s too easy to game the data (whether knowingly or not) with poor methodology. I advocate process before product, in other words.