I wonder whether a 'GitHub' for Science wouldn't help things along, just as it's helped the open source software community.
Scientist could have repos with full version control, so you could go back and look at their experiments from day one. All their data would be accessible, as well as their statistical analysis, and you could see the evolution of their written paper.
The repos could start off private, and then opened on the day of publishing, so others couldn't take credit for their work.
> Scientist could have repos with full version control, so you could go back and look at their experiments from day one. All their data would be accessible, as well as their statistical analysis, and you could see the evolution of their written paper.
> The repos could start off private, and then opened on the day of publishing, so others couldn't take credit for their work.
Funny you should say that...
I work at the Center for Open Science, and our founder was quoted in the article. One of the major use cases for our primary product (the Open Science Framework) is to facilitate preregistration.
With an analogy to open-source, my failed projects (like we're talking about here) fall largely into two distinct categories. Those that I never started, or were totally half-baked, or turned out to trivial - the uninteresting or incomplete. 90%. And 5% turn out to be illustrative learning moments.
One of my first OOP projects trying to model the real world in hierarchical classes. It was a good non-contrived example of a need for multiple orthogonal inheritance methods as any one can easily be forced into contortions in modeling even simple structures.
I tried to write a project with my own trig implementations, just because. That'd at least be funny...
But the rest are a few lines of drunk-coding, or something I found a library to do and never developed, or in some other way a waste of time to look at.
So I don't want to hold scientists to standards I wouldn't want. Nor do I want to miss the value in the noise.
So maybe we should have a honor-system somewhat like this - when you first describe to project to anyone looking for funding, or discussed it with more than a handful of peers, or ... then you make a note in your lab book. "Research direction: see if flatworms are really flat because Y - hope it'll give insight into Z". And your internal audit system submits all of these to a pool of peers who pick by interest in the area. If it's chosen, an auditor checks if the idea was ever really developed, dropping it if not, and summarizing whatever is there (or letting you write a paper) if there are results.
It sounds like a lot of work, but considering that everyone is asking for grants and being audited anyways, and this would replace vast swaths of less-pointed simple bookkeeping audit with peer-review of your methods, etc. In other words, likely higher quality results for the funding bodies in less time, and with the additional benefit of being able to publish interesting failure results.
Thus achieving a balance between researcher privacy and ability to try without public mockery on failure, funding efficiency, and scientific benefit.
Very little of what is actually needed to reproduce an experiment goes into academic papers. What you'd really want is a more formal, digital version of the "lab book". Experimentalists are taught, from their earliest days as an undergrad, to keep notes on what they're doing. The idea is to put everything useful in there. Got a new piece of gear with a long startup procedure? Put it in your book. Designing an experiment? Put the design in there. Ordered parts? Make a note of it. Running an experiment and finally found a way to do things that works? Write down the procedure. Got some results from your experiment? Print them out and paste them in your book. Saw something odd? Write it down. The result is a linear log that tracks a lot of stuff, both useful and useless. With experience, you learn to make use of tables of contents and margin notes, but searching through them for something dimly recollected can sometimes be time consuming. Lab books are where the gory details of how to actually do things is stored. The procedures, diagrams, and details in them are simply not conveyed in papers because there's no space for them, and many are viewed to be "common knowledge", although this kind of knowledge is probably only common to a small number of labs in the world.
The problem with these lab books is that they're often the personal property of the scientists and, as such, almost a sort of diary. They're written with yourself as the intended audience and can sometimes be impenetrable for anyone else. If you're working in a commercial lab, as opposed to research in academia, these books might be considered the property of your employer. In this situation you might try to make them more friendly to other human beings, but that often isn't a priority.
Putting things into a digital form would, in general, take a lot longer. Even with good proficiency in LaTeX, complex equations are quicker to just write on paper. Hand-drawn diagrams are also very common in lab books, and the software/hardware to draw easily in a digital form is not yet ubiquitous. Finally, digital storage occasionally dies. A lab book is a precious possession whose loss can cause months or years of difficulty. Physical hard-copy is easier to store long-term. Some people have undergraduate lab books from half a century ago sitting on their office bookshelves.
There is nothing that is technologically insurmountable to moving to a digital lab book format. Microsoft's surface tablets make digital drawing pretty painless. Cloud storage could be maintained long-term. Etc. The benefits of digital lab books could be considerable. They'd be easier to search and distribute. You could include copies in a digital appendix to an experiment (currently, most journals discourage large attachments because they don't like to pay for storage). To my knowledge, nobody has actually put everything that's needed together into a single cohesive program. You'd want a note-taking program with full LaTeX and freehand drawing support. You'd need to be able to paste in tables, graphs, and data files easily. It would need to have a long-term archiving solution and a really good interface. It would need to be as fast or faster than writing on paper for pretty much everything. It would also probably need to be open source, as scientists would probably not trust proprietary software that might make their notes difficult to access at some point in the future. Finally, the hardware to run it on would need to be ubiquitous.
> Very little of what is actually needed to reproduce an experiment goes into academic papers.
This is also an issue where investigators apply custom software to complex systems. Getting source code can be nontrivial, and documentation is often inadequate. And there may be bugs that were avoided rather than patched.
>The problem with these lab books is that they're often the personal property of the scientists and, as such, almost a sort of diary
Further complicating things, individual researchers don't actually own the books. Often, an institution or the lab owns the rights to the book. This can make individuals who are otherwise keen on "open science" hesitant to place copies of this information in a publicly accessible place.
See page 4 of this manual from the NIH (National Institutes of Health) about the ownership of notebooks.
Everything on that page is stuff you're going to find in a grad student's lab books, which are their property. This document is basically a primer for people transitioning into paid research where the lab-books become lab property. A huge proportion of research, especially in less commercializable areas of fundamental science research, is done by students (because they're cheap). This means the kind of lab-book described in your link is probably less common that you might expect!
I'd imagine you can find guides on university websites. e.g. Search for physics, chemistry, etc. lab book format guides, etc.. These tend to be pretty formulaic and oriented towards ease of marking. However, they do establish habits that persist when experiments become life-consuming affairs rather than something that start and finish in a three hour period.
Scientist could have repos with full version control, so you could go back and look at their experiments from day one. All their data would be accessible, as well as their statistical analysis, and you could see the evolution of their written paper.
The repos could start off private, and then opened on the day of publishing, so others couldn't take credit for their work.
Just a thought.