
Stanford computer diagnoses breast cancer more accurately than human doctor - mrsebastian
http://www.extremetech.com/uncategorized/104407-computer-more-accurate-than-human-doctor-at-breast-cancer-diagnosis
======
2mur
I'm a pathologist and an avocational programmer. This is pretty neat material
and is very relevant to me, as I have been trying to bone up my math chops
with Khan Academy videos so that I can tackle some computer vision related
work in pathology.

With regards to the study, I will just point out that the system is not
diagnosing breast carcinoma, but rather is producing a score which reflects
the prognosis as relates to overall survival of the patient (so it is not as
impressive as the somewhat hyperbolic HN title makes it out to be... a better
title would be 'Stanford computer analyzes breast cancer more accurately than
human doctor' which is not surprising at all given the well-documented
interobserver variability in breast cancer scoring, particularly in
moderately-differentiated tumors). That is to say that the computer already
knows the tissue that it is looking at is a tumor and not benign breast
tissue. Furthermore, it is really only providing a histologic (or morphologic)
score, which is to say, that it is attempting to predict how aggressive the
tumor will behave based upon how well or not well it is differentiated (how
ugly it is). These days, this score is actually less useful in clinical
practice than other information such as whether or not the tumor cells are
expressing estrogen/progesterone hormone receptors, or if it is over-
expressing Her2-neu protein, as these are possible paths for cancer therapy
(anti-estrogen drugs vs. Herceptin) in addition to being prognostic indicators
(tumors which express ER/PR generally behave better than tumors which are
ER/PR negative and express Her2), as well as the stage (or how far the cancer
has already spread) at the time the patient is diagnosed. There are a bunch of
companies which are getting FDA approval for computer vision related
algorithms for scoring immunohistochemical assays for ER/PR/Her2 [1].

So I am actually far less concerned about a computer doing my job very well,
which is actually looking at a piece of tissue on a slide and making a tumor
versus not-tumor distinction. This is very hard to do and I think will
continue to be even harder for computers/computer-vision/AI to do for a long
time to come. I am far more concerned about molecular diagnostics. That is the
true-future for making cancer determinations and may even eliminate the part
of my job where I tell you if something is benign vs. malignant.

[1] [http://www.aperio.com/pathology-services/analyze-slides-
auto...](http://www.aperio.com/pathology-services/analyze-slides-automated-
pathology.asp)

~~~
marshallp
I've seen a lot of doctors chime in various threads and say their jobs
couldn't possible be done by machine learning. The same thing was said about
self driving cars before the darpa challenges - when some profs actually put
their mind to it, it was done in a couple of years. . If the data was
available, there is probably quite a few people who can actually detect cancer
in slides.

~~~
tptacek
That's not at all what he said. He said that this particular research involved
a simpler (for CS) problem than either the title or his day to day job
tackles.

Do you have a background in bio or medicine or computer vision? It's very
interesting to see two informed people disagree about applied computer
science, so I'd love you to contribute something more specific to the thread.

~~~
marshallp
Actually, he also said that he is not concerned that machine learning could
initially detect cancer anytime soon. Also, if you've been machine learning
trends recently(past 5 years), you'll see that deep learning methods (hinton,
lacunn, ng, bengio) have actually made a huge leap over what came before, and
are believed to be that "final" in some sense algorithm that can allow to
tackle any learning problem. These just haven't spread widely anough yet.

~~~
apu
As a computer vision researcher, I'm not at all convinced that deep learning
methods will be "final" in any sense. I know that in the past, neural networks
were "final", and then graphical models were "final", and so on.

And while deep learning methods have indeed shown remarkable improvements
recently, they're not yet state-of-the-art on the most important/relevant
computer vision benchmarks.

~~~
marshallp
As a computer vision researcher it must be pain you to see that all your
learnings are for nought when faced with deep learning methods which can get
amazing performances from raw pixels (see mnist results for example). Also see
ronan collobert's natural language processing from scratch paper where handily
beats the past few decades of nlp research in parsing (in terms of efficiency,
and probably performances soon too). Or see the microsoft research speech
recognition swork which has beaten out all previous by a significant margin
using deep learning.

~~~
apu
Not at all! I'd love for vision to be solved, no matter what the method. I'm
more than happy to move onto another field if that's the case.

But I don't think it is. MNIST data is not particularly challenging. It's
great that deep learning methods work there -- they must be doing something
right.

Come back and taunt me when deep learning methods start getting state-of-the-
art results on, e.g., Pascal VOC:
<http://pascallin.ecs.soton.ac.uk/challenges/VOC/>

~~~
marshallp
getting best results on the harder vision challenges is simply a matter of let
the computers run long enough. Collobert's work for example took 3 months of
training. I don't see why vision challenges should any different. Perhaps the
vision researchers, of which there are many more people than the few deep
learning groups should try it.

------
pak
Please: there ought to be a rule on HN that instead of linking to a popsci re-
interpretation of a scientific result, you instead link directly to the paper.

<http://stm.sciencemag.org/content/3/108/108ra113.full>

If it is behind a paywall, as it probably is for this paper if you are not at
a university, perhaps look around for the _least_ hyperbolic re-interpretation
of the paper, and link that instead.

[http://www.genengnews.com/gen-news-highlights/image-
analysis...](http://www.genengnews.com/gen-news-highlights/image-analysis-
system-predicts-breast-cancer-survival-based-on-stromal-and-epithelial-
features/81245934/)

Even if you do not have access to the full paper because of the paywall, you
should be able to still read the abstract and pick the popsci article that is
fairest to what the paper _actually says_.

Otherwise, this happens: <http://www.phdcomics.com/comics.php?f=1174> and I
really don't like it when HN blindly follows the hivemind in furthering that
phenomenon.

Let's examine what the authors _actually_ say.

 _To directly compare the performance of the C-Path system to pathological
grading on the exact same set of images, we applied standard pathological
grading criteria to the TMA images used in the C-Path analysis. [...] the
pathologist grading the images was blinded from the survival data. Although
the C-Path predictions on the NKI data set were strongly associated with
survival, the pathologic grade derived from the same TMA images showed no
significant association with survival (log-rank P = 0.4), highlighting the
difficulty of obtaining accurate prognostic predictions from these small tumor
samples._

That's it. They make one remark about it, and do not focus on this at all
elsewhere in the paper, because it was not the point of their study and the
methodology for this little result is far from robust. Note they used _one
pathologist_ to run this little test. Also note that a high p-value is not
evidence that the null hypothesis is true--it is quite possible that there
_is_ a relationship but the study is underpowered; this is a frequent point of
confusion.

Let's please keep scientific statements in context. The original paper says
_nothing_ tantamount to computers diagnosing breast cancer more accurately
than doctors. It is, principally, about a new morphological feature that the
researchers believe is tied more strongly to survival according to their
computational model.

~~~
fgimenez
I'm a colleague of the author, and I have talked to him quite a bit about
C-Path since I work on something related. While I agree with much of what you
are saying, I disagree slightly with your comment that the article is
"principally, about a new morphological feature that the researchers believe
is tied more strongly to survival according to their computational model."

The significance here is that he extracted 6000 low-level morphological
features without any pre-conception about their usefulness. He then used
GLMNET (logistic regression with L1-regularization) to _automatically pick_
which of these features was important. Then, the craziest part is that the
most informative features were not even cancer cells, but rather, surrounding
stromal tissue. To quote from the paper, "Pathologists currently use only
epithelial features in the standard grading scheme for breast cancer and other
carcinomas. Our findings suggest that evaluation of morphologic features of
the tumor stroma may offer significant benefits for assessing prognosis." He
essentially took a completely blinded, machine learning technique to find
features that have been relatively ignored in pathology.

I think this is more indicative of a new paradigm in computer vision and
machine learning in general that finely-tuned, human-crafted features can be
beat with more automatic methods. Whereas before, we have tried to program
features that characterize what we see, now we are finally looking at image
features that can characterize what we're _missing_.

------
euroclydon
Reminds me of a comment by _delirium a few days ago, in the "AI is killing
jobs" thread:

 _dermatologists are fairly worried that "upload a photo and we'll diagnose
your mole", either via software or outsourcing, will cut out a significant
percentage of their business._

<http://news.ycombinator.com/item?id=3204466>

~~~
AndrewDucker
Also announced today: [http://news.techeye.net/science/ai-used-to-hunt-
dinosaur-bon...](http://news.techeye.net/science/ai-used-to-hunt-dinosaur-
bones)

those darn robots are everywhere!

~~~
artmageddon
"We may never have to dig again" -Dr. Grant, Jurassic Park

------
kenjackson
One thing the article doesn't point out is that AI methods in the past did a
poor job of detecting breast cancer. This is an important new result and big
step for the field.

~~~
_delirium
Unfortunately this still doesn't look like it's _detecting_ breast cancer;
it's scoring the severity of tumors, given prior knowledge that it's being
presented tumors to analyze (but tumors of unknown severity).

------
melling
You can learn from some of the experts in the field:

<http://ai-class.com> \- Peter Norvig and Sebastian Thrun

<http://ml-class.com> \- Andrew Ng

I signed up 4 weeks into Andrew's class. Both of these are excellent.

Btw, the technique in the article is a classification problem, right? :-)

~~~
barry-cotter
Is there any reason to watch the videos if one has AIMA? Could one do the
assignments without without ever watching a video?

------
danso
This kind of development is not that new, conceptually, at least (obviously,
implementing it is another thing). One of the most consistently verified
themes of study is that human experts can be very inconsistent, and in some
cases, expertise can be a detriment when it leads to overconfidence.
Radiologists, for example, have been found 20% of the time to render two
different verdicts when looking at the same X-ray at two different times.

And of course there's the recent study showing how judges were consistently
more likely to deny parole during hearings that happen in the afternoon:
[http://blogs.discovermagazine.com/notrocketscience/2011/04/1...](http://blogs.discovermagazine.com/notrocketscience/2011/04/11/justice-
is-served-but-more-so-after-lunch-how-food-breaks-sway-the-decisions-of-
judges/)

------
dave_sullivan
They kind of hint at it in the article, but sounds like automatic feature
detection really helped here. That's really the future (or present): learning
algos discovering better features automatically, significantly cutting down
the labor involved in feature engineering. Pretty neat.

Here's a video talking about current research into this at Stanford:
[http://www.youtube.com/watch?v=ZmNOAtZIgIk&feature=youtu...](http://www.youtube.com/watch?v=ZmNOAtZIgIk&feature=youtube_gdata_player)

------
bh42222
This should not be surprising for anyone who understand software. And software
identifying breast and skin and other cancers is going to continue getting
better.

Now the bad news is that human medicine moves super slowly due to very strict
regulations, and I expect doctors to resist this.

I hate to say this, but I think health _insurance_ companies are our best hope
to push this technology into greater use.

~~~
tryitnow
I agree. A lot of these technologies will lead to improved outcomes at lower
costs (for certain sets of basic problems).

Guild professions (like doctors) are inclined to keep doing even the basic
simple-minded aspects of their job because they get paid "economic rents" for
doing so thanks to the regulations that insist even basic tasks must be done
by someone with 10+ years of education.

Insurance companies on the other hand ultimately have to respond to employer
demands for lower premiums (unfortunately this process is slow and HR
departments are usually horrible at keeping costs under control). Insurers and
to an extent employers are going to be the impetus for a lot of improvements
in effectiveness and affordability.

Read Clayton Christensen "The Innovators Prescription" for more on how this
might play out.

------
pigs
Although he's not listed as an author, I find it interesting that Andrew Ng
has been using "malignant or benign tumor" as an example of a machine learning
classification problem at <http://www.ml-class.org>

------
allenp
"...but imagine if you had a C-Path machine in every town or city."

How near-sighted of the author. An article about computer vision, AI, machine
learning, and no thoughts of tele-pathology or remote reading?

~~~
mrsebastian
_hat tip_

My thought was that such such a machine would basically have a built-in,
automated biopsy thing. Like, the machine really would do everything from
taking a biopsy, looking at the sample, and then diagnosing.

Tele-pathology is cool, but you still need to get the tissue sample somehow --
which I imagine is hard in a town/village without a hospital.

You could have some kind of automated biopsy unit that then sends data back to
a central C-Path server, I guess.

~~~
ippisl
There's an indian company selling biopsy helper robots. it robots simplify the
process and enable technicians to take the biopsy instead of doctors, and the
whole process is faster.

see : <http://www.perfinthealthcare.com/Procedure_Videos.html>

Looking at the videos , it seems that the whole process can be done by robot,
But maybe patients and other stakeholders feel this way is safer/more
economical.

------
separated
The funny thing is, most rads physicians would probably welcome this, as
mammography is often the least preferred sub-specialty (my wife is a
radiologist and feels this way, and I've encountered many radiologists with
the same outlook).

~~~
roentgen
The system in this article is not interpreting mammograms, which is something
that radiologists do. This is looking at tissue samples on slides, the domain
of pathologists.

------
JabavuAdams
Wow! The medicalese description (abstract of the original paper) seems
completely unrelated to the computerese description.

I need to learn medicalese ...

or train some ML algos.

------
davidhansen
This does not address the issue of feature detection, but it's interesting
nonetheless, inasmuch as well-trained human doctors are quite bad at
probability, with the specific case of breast cancer diagnoses. The first
1/8th or so of Eliezer's layman's Bayes document:

<http://yudkowsky.net/rational/bayes>

------
maeon3
Looks like weak AI is making inroads all over the medical industry. Here is
predictive ai technology for heart failure that works better/faster than human
diagnosis.

<http://en.m.wikipedia.org/wiki/Multifunction_cardiogram>

