
Study of Theranos Medical Tests Finds Irregular Results - navikohli
http://www.nytimes.com/2016/03/29/business/study-of-theranos-medical-tests-finds-irregular-results.html
======
aresant
Key quote to me from the paper's author, not covered in the linked article:

"There was as much variability between LabCorp and Quest as there was between
Theranos and LabCorp or Theranos and Quest."(1)

Also a link to the actual paper in case anybody feels like reading it from the
horse's mouth (2)

(1) [http://www.techinsider.io/theranos-study-compares-blood-
test...](http://www.techinsider.io/theranos-study-compares-blood-tests-to-
quest-and-labcorp)

(2)
[http://www.jci.org/articles/view/86318](http://www.jci.org/articles/view/86318)

~~~
kevinskii
_“There’s actually pretty widespread variability in these clinical measures,
even between the reference labs,” said Eric Schadt, chairman of genetics and
genomic sciences at the Icahn School and an author of the report. “But
Theranos was more outside of range and outside of range in ways that would
impact clinical decision-making.”_

------
searine
>On Thursday, Theranos sent a letter to the journal, hoping to stop it from
publishing the paper.

Yeah, so these are the kind of people I want to trust with my personal health?

The problem isn't the tech. The tech is close enough to accurate to be
massively profitable. The problem is their reaction to criticism. It erodes
any and all trust in the product.

When you are dealing with a company built upon measurement of the invisible,
trust is the most valuable commodity. I need to trust that the numbers from
your machine are reported as true.

I need to trust that you aren't lying and covering up data to make a buck...

~~~
kmonad
>The problem isn't the tech

How do you know? It says that apparently the methodology could be flawed due
to things in the blood clustering together

>you aren't lying and covering up data

What? From reading, admittedly only, this article here, it sounds at least
reasonable to complain given the hugely negative press when it is also true
that "the study authors made no attempt to find out which measurements were
right". So if I understand this correctly, the new method showed different
results from the old method, but nothing was done to show which was right, and
it was clearly insinuated the new one was wrong. I would complain, too. But
perhaps you know more about this incident.

~~~
searine
>How do you know? It says that apparently the methodology could be flawed due
to things in the blood clustering together

From the reported independent results.

+-5% compared to Quest and LabCorp at a fraction of the cost is good enough to
make a profit.

>It sounds at least reasonable to complain

It makes sense to complain in Silicon Valley...

But it makes you look like a slimeball trying to cover your ass in the
medical/scientific industry.

Industrial science is different than tech. In tech, if it works, it works, and
nobody really cares how. In fact, it is in your best interest to hide how you
did it, because that is your IP.

In industrial science, if it works, but you fight people trying to open the
black box, it makes you look like you're faking data.

------
hackaflocka
Can anyone please enlighten us on Elizabeth Holmes's qualifications (formal or
informal) for inventing blood screening technology? E.g., is she known for
having won science competitions as a child? Or for tinkering around with blood
screening technology in a garage as a teenager? Etc.?

~~~
carterehsmith
I don't know why you are being downvoted, you asked a very valid and obvious
question.

------
chmaynard
According to the WSJ article, a former Theranos employee has claimed that the
commercial Theranos lab routinely diluted fingerstick blood samples and tested
them using conventional machines from other manufacturers.

If true, this raises two questions in my mind:

1\. Were the non-Theranos machines approved for testing diluted blood samples?
The technology may be approved only for blood samples with normal
concentration.

2\. Were the non-Theranos machines qualified to test blood samples collected
by any method? Some researchers think that capillary blood collected with a
fingerstick may be too contaminated to produce accurate results.

In other words, was Theranos negligent with respect to federal regulations for
use of these machines? Someone with clinical lab experience may want to weigh
in.

------
et2o
I am the only person who thinks the entire idea of Theranos is nuts? It's far
more painful to take blood from a finger than from the MCV in your cubital
fossa (arm). The fingertips are some of the most highly innervated structures
in our body (probably second to the sex organs). Then you have to go around
touching things all day with a band-aid on your fingertips, while you would
immediately forget about the band-aid on your arm.

~~~
mchahn
> I am the only person who thinks the entire idea of Theranos is nuts?

Were personal computers replacing mainframes nuts?

> It's far more painful to take blood from a finger than from the MCV in your
> cubital fossa (arm).

Why didn't you just call it an arm?

> Then you have to go around touching things all day with a band-aid on your
> fingertips,

I've never had a band-aid for a finger-prick and do them constantly for
checking blood sugar. Doctors never did it either.

~~~
cbd1984
> Were personal computers replacing mainframes nuts?

Personal computers never replaced mainframes. Personal computers replaced
minicomputers and workstations.

The difference is meaningful, even in this context: Minicomputers and
workstations were high-end systems dedicated to things like relatively small-
scale file storage, graphics processing, network tasks (servers, routers), and
other workloads which required neither high throughput nor long up-times. They
mainly required more RAM, CPU, and, sometimes, specialized hardware than it
was feasible to put on a home system of the era.

Mainframes, on the other hand, had become specialized as high-throughput high-
availability systems, running large databases and supporting dozens to
hundreds of users at a time, for years on end. They'd grown entire specialized
architectures dedicated to handling that kind of workload, and were only
getting moreso as time went on.

My point is, if Theranos is the PC, it's only going to replace some aspects of
the market it's moving into, and, from this article, it might not even do
that. It's just too difficult to play catch-up with the extremely specialized
realms, even if you have technology which is an 80% solution for 90% of the
problems.

------
mc808
If some of the tests consistently err towards a false positive but are less
invasive and cost 80% less, that sounds like an acceptable trade-off for those
tests. The patients with abnormal results would have to follow up with a
different test, but the majority would save a lot of money.

~~~
HeyLaughingBoy
One of the _worst_ results you can have from a medical device is a test result
that is wrong, but looks correct. A wrong result that is clearly wrong, or is
on the border between good/bad will be re-run either manually by the lab
technician reviewing the day's results, or automatically if the instrument
supports automatic retesting.

However a result that is wrong, but looks correct does not set off any alarm
bells. You'd better hope that the physician requested a battery of tests and
in correlating the results, sees the outlier. Otherwise you're potentially
screwed.

In running tests on actual people, saving money comes far behind the
overriding concerns around quality of results.

~~~
mc808
To clarify, by "false positive" I mean showing abnormal results when it
shouldn't. For example, if the cheap test is usually accurate, but when it's
wrong it erroneously shows dangerously high cholesterol, then a more expensive
but more accurate follow up test can compensate. But I agree that if the test
has a lot of false negatives (showing normal levels when they're actually
abnormal), it shouldn't be used at all.

What prompted my comment was, "While most of the results were similar for the
three labs, 12.2 percent of the measurements reported by Theranos were outside
the expected normal range for healthy people, compared with 7.5 percent for
Quest and 8.3 percent for LabCorp."

To me that sounds like at least some of the tests may be erring towards false
positives. More research needed.

~~~
URSpider94
For most blood assays, there is no such thing as a positive or negative. The
normal range for cholesterol, for instance, has both an upper or lower limit.
This leads me to believe that the Theranos tests are just less precise
overall, which makes it hard to justify their value even at an 80% discount.

------
kqr2
Letter from Theranos to JCI (The Journal of Clinical Investigation) which
published the study.

[http://www.scribd.com/doc/306187782/Theranos-Letter-to-
JCI](http://www.scribd.com/doc/306187782/Theranos-Letter-to-JCI)

~~~
tclmeelmo
Thanks for posting this!

So on that first paragraph, are they implying that the journal should have
provided Theranos with a copy of the manuscript for review and comment? Maybe
I'm not reading that charitably, but that's not how the system works, and I
don't think scolding the journal (page 3) over manuscript decisions is an
appropriate response.

The second paragraph is strikes me as a bit rich considering the reputation
Theranos has for secrecy. As best I can tell, they still haven't published the
more detailed information on their technology that they promised when the WSJ
first broke the story?

Were I the editor, I think my response might be along the lines of "you raise
some excellent points, and so we invite you to submit a research manuscript
for accelerated review".

~~~
mathattack
There is some irony in a company that doesn't put their research up for peer
review to ask for someone else's manuscript to review. :-)

------
Gatsky
Sorry to rant, but I'm sure I'm not the only person who is getting sick of
seeing these articles about Theranos. They essentially all say the same thing,
and have a picture of Holmes at the top standing next to an unidentified piece
of lab equipment that looks like it was just unpacked.

I suppose they are popular due to some combination of schadenfreude and the
morbid fascination that comes from watching a slow motion train wreck. But I
find these articles empty of anything remotely interesting let alone any
edifying content. They are just a monthly sign post for the accumulation of
disparaging comments about Theranos.

------
jamisteven
How much bad press is it gona take for this company to sink already?

~~~
mathattack
I suspect that it's momentum burning off their cash on hand. Who knows what
their burn rate is, but could anyone ever give them more venture funding or a
customer contract?

~~~
jacquesm
> but could anyone ever give them more venture funding or a customer contract?

Chances of that are slim to none, they are damaged goods at this point. A
number of existing customers has already cut ties to avoid further association
and any future would-be investors would be _very_ wary of putting more money
in without iron-clad proof the technology is solid.

------
chmaynard
I have been following the Theranos articles in the Wall Street Journal
closely. (I’m not a scientist and have no experience working in a clinical
lab, so my opinions are those of an interested observer.)

My opinion is that Theranos should still be a small biotech company operating
in R&D mode. They had no business going to market with a methodology and a
product that were unproven and entirely hidden from scientific peer review.
After the dust settles and they reorganize, that’s probably what will happen.
Theranos should also allow their investors to pull out without any
consequences.

~~~
mathattack
Is peer review needed for them to go to market? Or is the real issue the
shoddiness of their results?

I'm amazed that they (or the people who contracted them) are not getting sued.
If they mess up a blood test, someone could get mistreated and die.

~~~
chmaynard
Just to clarify, the testing device they claim to have invented is proprietary
and they have every right to protect their IP.

By calling out the lack of peer review, I'm referring to the _science_ behind
the use of fingerstick sampling. When you extract blood from a vein with a
sterile needle, the blood is uncontaminated. When you stab someone’s finger
with a blade or sharp pin, presumably you have lacerated hundreds or thousands
of capillaries. The blood that emerges is contaminated by tissues inside the
finger and by oils and bacteria on the surface of the finger.

It’s possible to do simple tests like iron level with this blood and get
accurate results. More complicated tests are another matter, and require
scientific investigation.

------
kevinskii
_Theranos executives also said that taking a big draw of blood from a vein
immediately before a finger prick...affected the finger prick samples in a way
that made it difficult to return a reading._

Is this so? I don't see how one could affect the other.

------
api
Meta question: to what extent is the present-day "tech crash" or tech funding
crunch scare a reaction against this particular high profile implosion vs. a
more general sentiment?

~~~
jacquesm
Not at all. It's just that the investors here did a piss-poor job of looking
at the company pre-investment. Lots of herd mentality and external validation
going on rather than common sense and some skepticism. The funding crash seems
to be mostly based on less than stellar returns for previous funds now at the
end of their life-cycles.

That makes it a lot harder to get LPs to pitch in. Still, even today there is
a glut of money chasing _far_ too little viable start-ups so as far as I can
see if you have a solid proposition it should not be hard to find funding. If
you are a bs artist then it may get a little harder, and that's a good thing
(especially for those entrepreneurs that _do_ have something to offer).

------
lettergram
Taking a vial of blood prior to a finger prick blood test could dramatically
alter results. Further, when the baseline (i.e. other labs) also have varying
results, its fair to call into question the whole study.

Although I have my doubts Theranos blood tests are as acurate, I feel the
study was pretty flawed.

~~~
0xffff2
>Taking a vial of blood prior to a finger prick blood test could dramatically
alter results.

Could you elaborate on why? My expertise is limited to an EMT certification
that expired half a decade ago, but isn't obvious to me why this would be the
case.

Blood is constantly circulating, and even a veinous blood drawn doesn't remove
an appreciable amount of blood compared to the total flow.

~~~
lettergram
Besides a plethora of things that can occur[1], the main issue is a
combination a pain response and a foreign object entering the skin.

When a you stick a needle in a persons arm you often trigger a flight-or-
flight repsponse. It's actually fairly similar if you prick their finger. The
problem with this study is that one was done after or before the other. It
depends what you look for, but for example white blood count (which was
tested) would/should change subtly after a blood draw.

The amount can vary (or not vary) depending on the person. For example, when I
get my blood drawn my arm swells, it's not actually that I am allergic, but my
body reacts in said way.

Regardless, my only comment was that the study seemed to be on somewhat shaky
footing.

[1] [http://www.frhg.org/documents/Lab_Manuals/Blood-
Collection-A...](http://www.frhg.org/documents/Lab_Manuals/Blood-Collection-
Adverse-Reactions-and-Patient-Blood-Volumes.pdf)

------
jackcn
Theranos doesn't have anything behind it - this company needs to go away and
return its money to its investors. A single drop of blood is far too little to
provide an accurate representation of your own blood components, the blood
that's gathered from a fingerprick might end up containing a much higher or
much lower concentration of 'x' than the rest of your blood. It's why
traditional blood tests gather much larger amounts of blood - to increase the
sample size.

~~~
jackcn
Again, this company needs to die off. It's one thing to attempt to innovate,
it's another to be outright fraudulent.

[http://techcrunch.com/2016/03/31/government-report-
details-t...](http://techcrunch.com/2016/03/31/government-report-details-
theranos-quality-control-issues/)

------
qaq
With so much $ at stake it's hard to rely on any reporting on the subject.

~~~
qaq
OK downvoters convinced me I am wrong having 10s of billions of $ at stake
obviously is unlikely to incentivise parties on either side to finance studies
that would be favorable to their position and to spend their PR budgets on
having those studies highlighted in the press.

~~~
x0x0
The article claims the researchers are independent and external. Do you
believe the nyt or the researchers are lying? Obviously Theranos has an
interest in the outcome of these tests, but I think medical researchers would
applaud less expensive and accurate tests.

~~~
qaq
They don't have to lie just spin. For the study in question number of test
subjects looks very low to draw any meaningful conclusion.

~~~
Fomite
So here's the thing - people like to say "This doesn't look like a big enough
study..." but you never see them say what _would_ be.

Basically: How do you justify your belief that their sample size is to small?

There are calculations to do this. 60 patients is, while small, pretty
standard for a study like this, and decently powered. The authors also used
the appropriate techniques to deal with small numbers in particular strata
when they were controlling for the variables they mention.

~~~
qaq
for 5% confidence interval with 80% confidence level you would need 164. For
1% confidence interval with 95% confidence level 9604. For 1% confidence
interval with 99% confidence level 16641.

