
23andme replies to the GAO: GAO Studies Science Non-Scientifically - jamesbritt
http://spittoon.23andme.com/2010/07/23/gao-studies-science-non-scientifically/
======
jballanc
First, let me say that I'm very much in favor of the direction that 23andMe is
headed. Personalized genetic medicine _is_ the future. Generic small molecule
treatments (and by generic, I mean that they're given to patients based on the
disease and not based on the patient+disease profile) have pretty much hit
their limit of efficacy and biologics are a lot like supersonic airliners:
they'd be great in theory if there weren't so many problems with them in
practice.

I also sympathise with the way that the GAO, the FDA, and a number of other
groups that have been weighing in on this issue have been lumping together
23andMe with some of the less reputable players in this burgeoning industry.

That said, I still don't agree with what 23andMe is doing. I've nearly
completed a Ph.D. in computational biochemistry and have been helping my wife
with her thesis in cell signaling, and the only thing I can tell you about the
information contained in a genetic profile that 23andMe provides you is that
you can't really tell much, if anything, from the profile that 23andMe
provides you. Even supposedly straight-forward genetics like the presumably
Mendelian pattern of inheritance in CFTR SNPs can get complicated fast. For
example: <http://www.ncbi.nlm.nih.gov/pubmed/14966131>

Of course, the real problem with 23andMe isn't with 23andMe but with the FDA.
The FDA's job, first and foremost, is to keep people safe. _IT IS NOT THE
FDA's CONCERN TO SEE NEW TREATMENTS DEVELOPED!_ The result of this is that the
medical community and the pharmaceutical industry are caught in a bit of a
catch-22. Everyone that I've spoken too (including past CSO's of large-name
pharmaceutical companies) recognizes that including genetic profiling of
patients in drug trials might reveal that different people benefit from some
drugs more than others, and that this is linked to genetics. In fact, you can
probably recognize this yourself: When you get a headache, what do you reach
for? For me, Ibuprofen does the trick, but Acetaminophen (Paracetemol) does
nothing. For my wife, it's the reverse. This is almost definitely linked to
genetics.

The problem is that genetic profiling for drug trials is expensive and, get
this, as of yet unproven to make a difference! Therefore, even if a
pharmaceutical company went to the expense of including genetic profiling in
their drug trial, the FDA would reject the results based on the fact that
genetic profiling is unproven. So, nobody does genetic profiling. So you can't
prove the usefulness of genetic profiling. etc.

Now, you might think that 23andMe has a chance of solving this issue, by
allowing people to take control of their own genetic profiling, but I think
they're actually doing more harm than good. Unless you are highly trained in
genetics and keep up with recent literature, you're not going to know what to
do with that profile. If you bring that profile to a doctor, they will pretty
much have to disregard it without a second thought (see FDA argument above).
Honestly, with the current state of medical research and pharmaceutical
regulation, a 23andMe profile is worth the paper you printed it on. What's
worse, the more doctors get pestered with people bringing in their own
profiles, the less likely they are to pay heed to any of it, and the
distraction that this causes the FDA, worrying about whether they should
regulate 23andMe or not, prevents the root issue from being addressed.

I liken 23andMe to Quest Diagnostics. If you've visited a doctor recently,
you're probably familiar with Quest. But when was the last time you went to
Quest asking for a blood test without being prompted to do so by a physician?
Honestly, you'd probably learn more on your own looking at the results of a
cholesterol test or CBC than by perusing your own genetic profile. Until the
U.S. (or, more likely these days, the E.U.) makes a major push for research
into personalized genetic medicine, 23andMe will be little more than a
novelty.

~~~
moultano
>Now, you might think that 23andMe has a chance of solving this issue, by
allowing people to take control of their own genetic profiling, but I think
they're actually doing more harm than good.

23andMe told me that I'm a carrier for phenylketonuria. That's useful
information to me, and strictly factual. The other slight-increased-risk-of-
this slight-increased-risk-of-that isn't all that useful, but who cares? They
give you the odds. I can't imagine what more you'd expect from them than to
present current research conclusions as unbiasedly as they can.

I think you have a mistaken impression about what their product does.

~~~
jballanc
> 23andMe told me that I'm a carrier for phenylketonuria. That's useful
> information to me, and strictly factual.

Quick! What's your chance of passing this on to your child? If you said that
it depends on the genotype of your spouse, you win! If your spouse doesn't
have phenylketonuria, what's the probability? If you said 1/4, your right!

Now, let's say you've met someone and convince them to get a genetic test, and
they turn out to also be a carrier, what do you do? Do you risk the 1/4
chance? One of the really interesting things about the human genome project
was that the scientists involved knew that these sorts of hard questions would
come up, so they really emphasized the human and counseling aspect of the
research, in addition to the hard science. These are the types of people who
are upset that 23andMe has mostly undone what they were attempting to do by
emphasizing a holistic approach to people understanding their own genetics.

Also, I just have to point out how ironic your example is. In fact, this
information is rather useless. Phenylketonuria is a relatively easily managed
disease, and all children born in the U.S. (and many other countries) are
already tested at birth, paid for by the government who did the studies and
decided this was a good test to do. You actually didn't learn anything you
wouldn't have potentially found out anyway (and at no cost to you).

~~~
moultano
>One of the really interesting things about the human genome project was that
the scientists involved knew that these sorts of hard questions would come up,
so they really emphasized the human and counseling aspect of the research, in
addition to the hard science.

How paternalistic of them. These questions are hard because they are personal,
and not the sort of thing that should be regulated.

>Now, let's say you've met someone and convince them to get a genetic test,
and they turn out to also be a carrier, what do you do? Do you risk the 1/4
chance?

That's my choice. Otherwise, it wouldn't have been. Though it isn't that
important for phenylketonuria, it might have been a deal-breaker if I were a
carrier of sickle-cell.

You're dancing around my point here. Some of the information they provide is
iron-clad binary, and can be very useful. Most people may not get more out of
it than slight increases or decreases in their relative risk, but some will
find out things that are life-changing. The last time 23andMe came up, one HN
commentator said that he found out that he was likely to be lactose-intolerant
from it, so he changed his diet and it changed his life. He had lived with the
symptoms for so long that he just assumed that was how life was supposed to
be.

Here's what Sergei Brin got out of it:
<http://too.blogspot.com/2008/09/lrrk2.html>

I don't understand your motivation for wanting to forcibly withhold this
information from people.

------
roder
I'm a 23andme customer (post-DNA day) and after seeing the DNA mixup[1] and
watching the youtube video released by the subcommittee on hearings and
oversite[2], I am becoming increasingly weary of consumer genetic testing.

I am glad to read that 23andme support regulation, because ultimately that is
what should be required. Much like the regulation that HIPAA provides, DNA
information should be federally protected and regulated.

[1] [http://www.switched.com/2010/06/08/23andmes-dna-mixup-
leaves...](http://www.switched.com/2010/06/08/23andmes-dna-mixup-
leaves-96-customers-with-wrong-test-results/) [2] <http://bit.ly/9MrBpe>

~~~
jamesbritt
I'd prefer to see Consumer Reports handle this instead of anything like the
FDA.

I'm an adult; I can decide for myself what to make of the information I
obtain.

~~~
timr
_"I'd prefer to see Consumer Reports handle this instead of anything like the
FDA. I'm an adult; I can decide for myself what to make of the information I
obtain."_

I realize that this isn't going to be a popular opinion around these parts,
but no, you can't. Smart as you may be, you're utterly ignorant when it comes
to this stuff, and you haven't got a chance of beginning to understand the
intricacies that go into interpreting the data that these companies are giving
you. More importantly, you don't have _time_ to understand.

I have a Ph.D. in Biochemistry, and for the most part, I couldn't tell you
whether the data in a 23andMe report is meaningful or utter garbage. I perhaps
have enough knowledge to go out and _find_ the necessary papers, read and
interpret them in light of the literature on genomic analysis, and evaluate
risks...but I wouldn't begin to have the time to do it properly for all of the
data in a given report. No offense, but you haven't got a prayer. These
companies could be fabricating results whole-cloth, and you'd have no way of
knowing it.

Most importantly, because there's no regulation of the methods used by these
companies, neither you nor I have any way of knowing what methods they're
using, whether the techniques are precise or accurate, or even if the labwork
is done by skilled technicians in a sterile environment. Without these
assurances, even if you _could_ understand the data, you would have no
guarantee that the data was even gathered correctly.

There are some domains where expertise matters more than anything else, and no
amount of brainpower makes up for the instincts provided by years of training
and experience. This is one such area.

~~~
moultano
And for some reason you think this is different from any other field? I don't
have a prayer when I walk into a car dealership of figuring out whether a car
I'm thinking of buying will function properly. There are too many parts for me
to conceivably understand. It's very complicated.

Thankfully, there are many experts and expert organizations that I trust to
make this determination for me. I defer to their authority.

How is this any different? The market seems to handle this just fine.

~~~
jballanc
> And for some reason you think this is different from any other field?

Yes, actually. If we must really flog a tortured analogy: This is not like
going to your local mechanic friend and asking for car buying advice. It's
more like this guy named Karl Benz comes and tells you he's got a great new
invention called the internal combustion engine and you should definitely buy
it. So you go to your friend the steam engine mechanic and ask him for advice.
Sure, he understands the principles, maybe, but this is something completely
new!

If you must appeal to authority, consider you have two experts in this thread
telling you that even they think the topic is too complex to draw useful
conclusions. If you don't trust random people on the internet (not that I
blame you), then go to the source. From the article:

 _There are valid scientific reasons for different estimates from different
companies, such as: companies employ slightly different statistical models for
making risk estimates; companies establish different criteria for the
inclusion of associations in their reports; new associations are being
discovered at a faster rate than companies’ development cycles; companies may
test for an imperfectly overlapping set of genetic variants for reasons
including the ability of different genotyping technologies to assay certain
variants._

To be clear, this is not a case of Car and Driver favoring German engineering
but JD Powers always skewing towards American manufacturers (completely
contrived example, BTW). In this case the statistical models _are_ the
science, not just some interpretation of the science. This field is young. Too
young. This money and effort would be better spent on basic research.

> The market seems to handle this just fine.

No, the _market_ gives us homeopathy and snake-oil salesmen. Humans,
especially when confronted with areas in which they are not knowledgeable, can
be surprisingly irrational!

~~~
moultano
>To be clear, this is not a case of Car and Driver favoring German engineering
but JD Powers always skewing towards American manufacturers (completely
contrived example, BTW). In this case the statistical models are the science,
not just some interpretation of the science.

Statistical models that are _far far_ more suspect drive decisions in areas of
all our lives that have far more material effect than this.

>This field is young. Too young. This money and effort would be better spent
on basic research.

Whose money? Mine? They aren't a non-profit.

>Humans, especially when confronted with areas in which they are not
knowledgeable, can be surprisingly irrational!

Isn't that their right?

------
bkrausz
I definitely treated my 23andme profile as a piece of entertainment, and don't
think people should take it too seriously. I found someone who may be a
distant cousin, and a few interesting traits, so I'd consider it a slightly
more informed (and similarly priced) palm reading.

I'm really glad I approached it like that, because I was one of the people who
were in the DNA mixup. They told me I was a Tay-Sachs[1] carrier, which has a
potentially vital impact on my relationship decisions (and was also a bit
concerning given that neither of my parents are carriers). I'm really glad
they caught and fixed it, and that I didn't take it seriously enough to lose
sleep over the results.

[1] - <http://en.wikipedia.org/wiki/Tay%E2%80%93Sachs_disease>

------
carbocation
23andme uses GWAS SNPs, about 1 million of them. You have a 3 billion base
pair haploid genotype, so they sample ~1/3000 base pairs. What does this mean?

First, most of the SNPs are noncoding, found not in exons but in introns and
intergenic space. Still, you can't dismiss these; some are causal for major
gene expression changes (my lab has articles set to be published next week on
this topic; more details then).

Second, the fact that you're only covering 1/3000 nucleotides tells you that
there's no way you can fully specify someone's genetic risk with this data,
ever. There is a debate in the community right now about whether common
variants (which are on 23andme type chips) or rare variants contribute most of
the genetic variability. I think it's clear that common variants are winning
this game — but this is on a population level! On an individual level, I don't
think I'd ever be comfortable telling someone their risk of X with just GWAS
data. For the population, their rare/private mutation in Gene Y has little
impact, perhaps, but for try telling that to the 5 people with the Gene Y
mutation who will die by age 20. (Extreme hypothetical to try to drive home
the broader point.)

Let's also not forget that phenotype is a computed expression of
genotype+environment. If your genetic risk score from 23andme says you are at
high risk of heart disease, yet your grandparents on both sides are 100 and
healthy and your LDL-C is 60, should you really be concerned?

~~~
khafra
If you know somebody's medical history and 23andMe results, can you say more
things about them at any given level of confidence than you could with just
zis medical history?

~~~
carbocation
Not really, at least not for cholesterol; not right now. [1] Hopefully in the
next few years, yes. But genetic data, since our knowledge is limited, does
not increase risk discrimination beyond family history. It does, modestly,
increase risk classification.

 _Edit_ \- Link didn't work initially. I was not trying to just snarkily link
to pubmed - I had a specific paper in mind. Sorry!

[1] <http://www.ncbi.nlm.nih.gov/pubmed/18354102>

------
jacquesm
23andme engage in infotainment with an additional helping of candidates
disease thrown in for each and every one of their customers. GPs are already
amongst the most overworked people on the planet, the last thing they need is
a herd of people with waving print-outs they (the people) don't understand in
support of yet another round of imaginary diseases.

It's the perfect product for the hypochondriac.

