Hacker News new | comments | ask | show | jobs | submit login

It's unfortunate that the suggestions at the end don't seem to offer a realistic attack vector.

> First, scientists would need to be incentivized to perform replication studies, through recognition and career advancement. Second, a database of replication studies would need to be curated by the scientific community. Third, mathematical derivations of replication-based metrics would need to be developed and tested. Fourth, the new metrics would need to be integrated into the scientific process without disrupting its flow.

Yes, absolutely those things need to happen, but the problem is how to get this funded, how to get people to not see reproducing results as career suicide, right? Items 2-4 will fall out as soon as item #1 happens.

How do we make item #1 happen? What things could be done to make reproducing results actually an attractive activity to scientists?




The problem is that, if you put mere reproduction as a goal, many scientists would see that as low hanging fruit to beef up the resume, so we'd get countless unnecessary "experiments".

I'd say the goal that gets credited should not be merely reproducing the results, but finding errors in the previous research. That would count as novel, and is something that is presently recognized as contribution. The only problem is that journals or conferences treat it as unattractive, so good luck publishing something of the kind...


> The problem is that, if you put mere reproduction as a goal, many scientists would see that as low hanging fruit to beef up the resume, so we'd get countless unnecessary "experiments".

Only if you assume the incentives for the 2nd, 3rd, 4th, etc. reproduction experiments remain the same, right? I wouldn't assume that, both because the first reproduction is the most valuable, and for the reasons Ahmed discussed in the article - that scientists are motivated by their perceived ability to do something novel. So first reproduction might be novel, but the fifth would certainly be less valuable, so I wouldn't personally assume we'd get a flood of useless experiments.

> I'd say the goal that gets credited should not be merely reproducing the results, but finding errors in the previous research

Reproducing an experiment is meant to, without prejudice, either confirm or deny the previous research. It's not meant to confirm the previous results, it is meant to ask whether there could be errors in the research, but without assuming there are errors.

It is novel to validate a result the first time, whether it's positive or negative, and for this incentive system to work, it has to appeal to people who might not find something dramatic or contradictory. It must be appealing to do the work, regardless of the outcome, or it's not an incentive at all.




Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact

Search: