TL;DR: any computer science paper that presents practical work without disclosing source code should not be accepted to any scientific conference or journal.
Agree on all. I've been few years in research in computer security before quitting for industry. I must report that so many papers that presents some kind of algorithms (I would say the majority) very rarely also provides the source code of the implementation. I have always thought and advocated for that any computer science paper that presents practical work without disclosing source code should not be accepted to any scientific conference or journal.
I know (because I did many times) that opening the source takes an incredible amount of time, but it is mandatory for being capable of 'standing on the shoulders of giants'. Writing code and keeping private in research is just a non sense.
Your TL;DR should read "...source code and a virtual environment that allows the process to be repeated..."
From the article:
"There has been significant pressure for scientists to make their code open, but this is not enough. Even if I hired the only postdoc who can get the code to work, she might have forgotten the exact details of how an experiment was run."
also
"The only way to ensure recomputability is to provide virtual machines"
To that end, the site http://recomputation.org/ is mentioned as a repository for recomputable experiments.
Point being: source code alone does not specify the process or workflow in which it was used.
sorry... TL;DR was intended for those who were not interested to read my whole comment.
actually the idea of providing VMs preconfigured to run the test is very good since it saves time both for who write the code and for those who want to test
> Point being: source code alone does not specify the process or workflow in which it was used.
I completely agree! when we released the code, we spent hours to clearly define the environment where the code had to run.
Agree on all. I've been few years in research in computer security before quitting for industry. I must report that so many papers that presents some kind of algorithms (I would say the majority) very rarely also provides the source code of the implementation. I have always thought and advocated for that any computer science paper that presents practical work without disclosing source code should not be accepted to any scientific conference or journal.
I know (because I did many times) that opening the source takes an incredible amount of time, but it is mandatory for being capable of 'standing on the shoulders of giants'. Writing code and keeping private in research is just a non sense.