
Response to “Unexpected mutations after CRISPR-Cas9 editing in vivo” - lucapinello
http://www.biorxiv.org/content/early/2017/07/05/159707
======
leemailll
All begins with this comment
[https://www.ncbi.nlm.nih.gov/m/pubmed/28557981/](https://www.ncbi.nlm.nih.gov/m/pubmed/28557981/)

~~~
JohnJamesRambo
I love that. Xiaolin Wu is really going in on them. More science needs to be
like that.

~~~
j7ake
In most labs there are journal clubs that function as this, but a lot of the
discussion stays within the institute.

------
Scaevolus
They did whole-genome sequencing for mice treated with CRISPR-Cas9 and
compared it to a _different_ mouse to see if any mutations were caused?

Why didn't they just take samples from the mouse before and after?

~~~
88e282102ae2e5b
Some of my colleagues speculated that this was not the intended study.
Originally, they might have tried to delete a gene, but the expected phenotype
didn't show up, which would have put an end to that line of research. But
instead of completely abandoning the project they figured they could sequence
the mice that were still around to see what the off-target effects were, and
they thought they found something surprising and important.

~~~
rhizome
Does that qualify as "p-hacking?"

~~~
firebones
Of all the stupidity and hype around blockchain tech, the idea of committing
the hypothesis of your research to an immutable record _beforehand_ so as to
avoid p-hacking seems to be one of the best ideas I've heard, provided you
could avoid the problem of people simply committing a tournament of
alternative answers...

~~~
semi-extrinsic
Even more interesting is the proposal I've seen where you submit the intro,
theory and methods section for review at a journal _before you do the
expeiments_. Once what you're planning to do is deemed good and interesting by
the reviewers, you get the go-ahead, and the journal will publish your results
_no matter what they are_. Not only does it prevent p-hacking, it also removes
bias against negative results.

FWIW, anti-p-hacking measures are already implemented for clinical trials;
they have to register expected outcomes at the beginning.

~~~
jacquesm
Journals should not be given the power to implicitly decide what research gets
done and conversely they will never accept to be forced to publish all results
because of the vast amount of research that goes nowhere.

The whole value for a journal is to be able to select _once the results are
known_ because that's how you build prestige.

All you are doing in effect is to lay out a good case why journals should and
probably will simply die because this level of transparency would be a great
thing to have but is incompatible with the publishers model.

~~~
hackeyed
Many of the journals currently publishing these kind of Registered Reports
only publish special issues, which might help with your concern about journals
being given control over what work is done in a field, though it also limits
how much of the literature can benefit from the new approach.

Another way of thinking about a journal's incentives is that, by moving the
primary peer-review stage before a study has actually been conducted, you
greatly expand the number of studies that a journal will interact with, which
may provide a larger and more valuable role for entities like journals in
world where actual publishing is a trivial matter.

------
hayabusa
They have a response paper though :

"Here we provide additional confirmatory data and clarifying discussion,
including sequencing data showing extensive heterozygous mutations throughout
the genome in the CRISPR treated mice, which are all progeny of inbred mice
purchased from a commercial vendor (JAX). The heterozygosity in these cases
cannot be parentally inherited. The summary statements in our Correspondence
reflect observations of a secondary outcome following successful achievement
of the primary outcome using CRISPR to treat blindness in Pde6b/rd1 mice. As
the scientific community considers the role of WGS in off-target analysis,
future in vivo studies are needed where the design and primary outcome focuses
on CRISPR off-targeting. We agree that a range of WGS controls are needed that
include parents, different gRNAs, different versions of Cas9, and different in
vivo protocols. We look forward to the publication of such studies. Combined,
these results will be essential to fully understand off-targeting and can be
used to create better algorithms for off-target prediction. Overall, we are
optimistic that some form of CRISPR therapy will be successfully engineered to
treat blindness."

[http://www.biorxiv.org/content/early/2017/06/23/154450](http://www.biorxiv.org/content/early/2017/06/23/154450)

~~~
nonbel
Thanks, I was looking for this info earlier
([https://news.ycombinator.com/item?id=14448371](https://news.ycombinator.com/item?id=14448371)):

>"In our case, 56 zygotes were harvested from six pregnant females bred to six
stud males and injected with CRISPRCas9."
[http://www.biorxiv.org/content/early/2017/06/23/154450](http://www.biorxiv.org/content/early/2017/06/23/154450)

So they started with 56 zygotes, of which 11 survived injection of Crispr/etc.
Of these 11, five and two mice showed some evidence of containing NHEJ/HDR
cells, respectively.

In the case of NHEJ, they do not tell us what percent of cells were "edited"
per mouse (or I missed it when I read that paper earlier). However for HDR it
was only about 1/3 and 1/5 for the two mice they found. So this suggests the
"editing" is occurring not at the zygote stage, but somewhat later (based on
the percentages, perhaps at the 4 or 8 cell stages).

Assuming the edit/mouse rate is similar for NHEJ and HDR (somewhat dubious),
it looks like 4x56 = 224 to 8x56 = 448 cells were treated with crispr/etc and
~ 7 got "edited". This gives ~1-3% "edits" per treated cell.

------
cing
A few other responses:
[http://www.biorxiv.org/content/early/2017/06/21/153338](http://www.biorxiv.org/content/early/2017/06/21/153338)
[http://www.biorxiv.org/content/early/2017/06/30/157925](http://www.biorxiv.org/content/early/2017/06/30/157925)
[http://www.biorxiv.org/content/early/2017/07/05/159707](http://www.biorxiv.org/content/early/2017/07/05/159707)

------
jostmey
Great example of why the peer review system is broke.

I've seen first hand how the peer review system works in the United States.
Foreign Postdocs studying in the United States are desperate for a Green Card.
They peer review as many papers as they can as quickly as they can to prove
_something or other_ for the purposes of a securing a Green Card. Some of
these foreign postdocs do a good job, many do not. Also, overworked professors
don't always take the time to carefully read a paper and don't care. The
result is that a lot papers make it through peer review that shouldn't.

I'd rather have my paper peer reviewed live on the internet by allowing people
to comment on it beneath a PDF link than to suffer through another drawn out
peer review process. People who care have a text-box where they say what they
want, and if no one cares then it indicates I need to do research on something
else!

~~~
zzz95
I do not think this is correct, at least for most of the sub-fields of
computer science and physics. Prestigious conferences and journals invite only
well known researchers to review, who are mostly professors. Few post-docs get
invited. Each paper gets multiple reviews (3-5), and conflicting reviews get
discussed making the review process quite rigorous.

Reviewing a paper is mostly charity and gets you nothing in return, except the
fact that when you submit a paper, someone will do the same. It does not help
in the Green Card process at all, or boosts your academic credentials.

Regarding overworked professors, they are allowed to delegate reviews
internally to students to ease the time they have to spend.

~~~
wfunction
> I do not think this is correct, at least for most of the sub-fields of
> computer science and physics. Prestigious conferences and journals invite
> only well known researchers to review, who are mostly professors. Few post-
> docs get invited.

What about PhDs? You've never seen PhDs review papers in CS? It happens for
prestigious conferences too.

~~~
seanmcdirmid
CS is diverse in field cultures but ya. You don't even need a PhD to be on a
PC, but usually every field has a crowd of reviewers that defines it by
constantly being on PCs.

