
What happens when patients find out how good their doctors really are? (2004) - danso
http://newyorker.com/archive/2004/12/06/041206fa_fact?currentPage=all
======
arn
Great article. Might be hard to get through, so I'll put out this quote which
I enjoyed and compelled me to read the rest of it.

" _Matthews had started a cystic-fibrosis treatment program as a young
pulmonary specialist at Babies and Children’s Hospital, in Cleveland, in 1957,
and within a few years was claiming to have an annual mortality rate that was
less than two per cent. To anyone treating CF at the time, it was a
preposterous assertion. National mortality rates for the disease were
estimated to be higher than twenty per cent a year, and the average patient
died by the age of three. Yet here was Matthews saying that he and his
colleagues could stop the disease from doing serious harm for years. “How long
[our patients] will live remains to be seen, but I expect most of them to come
to my funeral,” he told one conference of physicians._

 _In 1964, the Cystic Fibrosis Foundation gave a University of Minnesota
pediatrician named Warren Warwick a budget of ten thousand dollars to collect
reports on every patient treated at the thirty-one CF centers in the United
States that year—data that would test Matthews’s claim. Several months later,
he had the results: the median estimated age at death for patients in
Matthews’s center was twenty-one years, seven times the age of patients
treated elsewhere. He had not had a single death among patients younger than
six in at least five years._ "

~~~
yesiamyourdad
What grabbed me:

"In this short speech was the core of Warwick’s world view. He believed that
excellence came from seeing, on a daily basis, the difference between being
99.5-per-cent successful and being 99.95-per-cent successful."

~~~
hsitz
I took notice of that passage also. But the main reason the 99.5 to 99.95
percentages is meaningful is that they were measures of a daily risk. So it
quickly compounds into a meaningful percentages; the 99.95% rate meant an 83%
chance patient would stay well this year, the 99.5% rate meant only a 16%
chance of staying well for this year.

Many of the medical percentages that we see quoted in media are derived using
a lifetime risk basis, _not_ a daily one. On a lifetime basis the difference
between 99.95% and 99.5% _is_ negligible.

I wonder how many people reading this article are aware of the difference.

~~~
roma1n
It is well explained in the article, since the point is about daily treatment
compliance. I would expect the readers to grok that in context :)

------
bdcravens
I have Cystic Fibrosis. I don't think it's fair to assume that the bell curve
associated with treatment results is fair to attribute to the quality of care.
Factors such as climate and pollution can significantly contribute to health,
as can socio-economic and average distance patients travel, as well as access
to insurance.

For instance, I'm from North Texas; I went to college in Oklahoma. Back then
(1996) Texas had a type of insurance for children with CF (it actually
extended to age 21) unavailable in Oklahoma. I ended up losing care, going off
of my meds, and by the time I was back in Texas and had the income to support
my needs (I came from a very poor family) I had lost 30 pounds (down to 98lbs)
and had lost 25% of my lung capacity. Was this attributable to the clinic in
Dallas? Of course not.

(FWIW, I'm up to 155lbs these days and my lung capacity is probably in the
upper 5-10% of all CF patients)

~~~
rcthompson
Of course there are other effects that go into that bell curve (I'd be more
worried about doctor-to-patient ratios and time per patient), but I think the
article does an excellent job of highlighting the differences in care.

~~~
bdcravens
Very true. It's important to note that the majority of a (well maintained) CF
patient's health is the result of the the breathing treatments and enzyme
treatments; I could probably not see a doctor for years if I had my meds.
(Probably true of many chronic conditions I'm sure, but speaking to my
experience only)

~~~
abdulhaq
I'm sure you know this but a good physician will closely track your lung
function and on a dip will get a sputum test done and then get you on the
right antibiotics asap. That is a very important part of CF treatment that
does need regular trips to the clinic. BTW I'm glad you're doing so well :-)

~~~
bdcravens
I've had the fortune of being pretty consistent lung function-wise - the dips
tended to be when I was doing poorly with my treatment regimen. (Of course
there are many who have issues despite their best efforts)

Thanks :-)

------
graeham
Medicine is like any other profession - it shouldn't be surprising that some
doctors are better than others, like some lawyers/engineers/teachers are
better than others.

There are a lot of challenges with healthcare metrics. Medical problems are
more complex, arising from more factors than practically any other problem.
Most other sciences can be resolved by theory or relatively convergent
empirical data. Medicine is empirical, but the scatter is huge.

Data is not tracked well. When it does exist, it is in formats that are not
well conducive to analysis and sharing. This seems to be slowly (and
expensively) improving. Pressure for privacy adds to inertia for innovations
that could use medical data for improved outcomes.

With the title of the article, I was hoping there would be some more
discussion about implications of some 'rating system' for doctors. It would
probably superficially be a good idea, but such a system would obviously
result in some level of gaming. Does a doctor in a rated system only take easy
cases to keep a good track record? Are hard cases given more weighting? Who/
how is such a weighting decided? Such a system could offer a results-based
compensation scheme for doctors, which is lacking from most publicly funded
systems.

~~~
yodsanklai
"Medicine is like any other profession - it shouldn't be surprising that some
doctors are better than others"

Surely, some doctors are better than others, but (for the sake of discussion
:)) I don't think every professions or activities are equivalent with respect
to 'rating'. Take the extreme example of walking. All human beings walk
equally well. This is unlike playing music or solving math problems for
instance.

I'm certainly not saying that medicine is as easy as walking, but maybe for a
wide range of health problems, there are well known standard procedures or
solutions that all doctors are competent enough to follow. And in doubt, they
usually refer their patient to a specialist, who in turn can send them to a
more competent doctor.

~~~
robotresearcher
"All human beings walk equally well."

This is obviously not true.

~~~
yodsanklai
I think you got my point. Usually, you don't stop looking at people in the
street thinking how great is their walking. Walking is a simple enough task so
that in normal conditions you don't see a wide range of abilities. Unlike
swimming, playing music or cooking where even in normal condition you can see
that some people are incredibly better han others.

An other example that comes to mind are regular airline pilots: nobody wonders
whether they are good or bad. We never have to worry that the pilot is unable
to fly the plane.

~~~
dmd
> nobody wonders whether they are good or bad

Again, this is a _terrible_ example. I'd say that a good third of passengers
are wondering exactly that. (It's irrational to do so, but that's another
story.)

~~~
graeham
And further, skilled pilots can be recognised by their landings, particularly
in making soft ones in stormy conditions. Perhaps also in their customer
service with witty commentary over the PA.

Divergent, but any skill I can think of will have a distribution of abilities.
Some of these skills have a threshold beyond which incremental improvement
gives little benefit. Walking example: if you can walk 100m in one go, you are
better off than someone who can walk 10m. Likewise, 1000m > 100m. Does 500km
give any more benefit than 50km? Maybe, but probably only for extreme use-
cases (maybe competitive extreme-distance walking? I'd rather drive!).
Similarly with technique - if walking technique is horrible it matter for
mobility and injury risk. Beyond a certain level, it probably matters less.
And maybe walking extremely well is a visual-social signal, as a marker for
someone who is in good shape? I'd read somewhere research done on dancing as a
marker of fitness for attractiveness to the opposite sex.

------
rcthompson
What stuck out for me was the difference in attitude between the two example
centers. It's "67% is still pretty good, so keep it up" vs "90% is a big
decrease from 105%; this is a _failure_ and we need to make adjustments." In
articular, the former type of attitude is certainly the norm from my
experience (with medical care in general, I have no experience with CF).

~~~
timbre
This idea applies to more than medicine. A boss once told me, speaking about a
piece of work I thought was pretty good, "If it could be better, it's wrong."
It stuck with me, and I've found that everyone I meet who is exceptional at
some skill has this attitude.

~~~
DannoHung
Yeah, that's sorta stuck with me. I know a lot of people who say that the
perfect is the enemy of the good.

Well, buddy, that just means that if you have a good solution, it's stopping
you from getting a perfect solution.

~~~
6cxs2hd6
When I hear "the perfect is the enemy of the good" it's usually in response to
the sentiment, "I can't ship this yet; it's not 'perfect'".

In the context of this article: You have a patient who needs to be treated --
today. They can't wait for a hypothetical perfect treatment. Indeed, more
fundamentally, the only way to discover a "perfect" treatment would be to use
today's least-worst solution, with the determination to measure and improve it
continually.

Also the point of the article was that it's not so much the techniques --
which all the CF centers know about and use -- but the aggressiveness of their
consistent application.

This isn't about a perfect design. It's about doctors who are willing to
coach/goad/persuade patients into doing consistently what they already know
they need to do. (And who are willing to be sticklers for consistency to the
point of being a bit of a PITA to their colleagues.)

------
SeanDav
This reminds me of a true story I once read about the best car salesman in the
country at the time. Single handedly he was selling more than some not-so-
small entire dealerships. The difference was his incredible focus and approach
to even the tiniest detail that could help a sale and willingness to try new
things.

One thing that stands out in my memory about this was the fact the he made
sure he always had 3 (!) pens for the customer to sign the sales agreements.
Just in case 1 pen stopped working and the backup pen malfunctioned. He
brought this sort of laser focus to absolutely every aspect of the business.

How many salesmen would have a spare pen, already in place, let alone 2
spares. How many would even have thought about it and what the right number
might be?

Focus, attention to detail and continuous innovation.

~~~
Terr_
Unfortunately, there's also a cargo-cult aspect to this kind of thing, when
managers start saying: "Everyone is now required to have three pens at their
station at all times."

I'm in a company where the suit-wearing side says "we need innovation" to the
tech side, but they don't actually want a discussion about what they see as
opportunities or directions. They just want to invoke it by rote ritual, or
order it as if we kept some on shelves.

------
PaulJulius
_Patients with CF at Fairview got the same things that patients everywhere
did—some nebulized treatments to loosen secretions and unclog passageways (a
kind of mist tent in a mouth pipe), antibiotics, and a good thumping on their
chests every day. Yet, somehow, everything he did was different._

I found this quote especially interesting. I've often found this to be true,
that you look over at someone who's much better, and on one hand they're just
doing the same things you're doing, but somehow they're just on a different
level. It's very difficult to distinguish exactly how someone achieves better
results, that is, until it's been explained. Here, we see this clearly in how
the level of lung function is perceived. Both centers provide the same
treatment, but at the Cincinnati center lung levels around 70% are "okay", but
at Minnesota, any drop is unacceptable, even if the patient is already at
above-normal levels.

It sort of reminds me of the "Don't deal with it, fix it" article that was
also posted recently. You'll be coasting around at a certain stable level of
engagement until one day you see something you've never considered before and
all of a sudden you see a whole new world of possibilities.

------
droopyEyelids
Good article, but I feel Atul Gawande mischaracterized "evidence-based
practice" in a way that does our whole planet a disservice when he poo-poohed
it to lionize Warwick.

Evidence based practice isn't really new, it's been a thing since '92\. And
while it is about using studies to make treatment decisions, it's not about
forbidding doctors from thinking for themselves.

But what really got my goat was the fact that evidence-based _medicine_ is the
new thing, and it's an essential step in the evolution of medicine, and more
people need to understand it. Evidence based medicine is about applying
statistics to determine when treatments cause more harm than good. With modern
medicine's advances in imaging techniques, we're hitting all sorts of new and
dangerous problems where the benign abnormalities we all accumulate over our
lives are treated at great expense, inconvenience and loss of health.

~~~
ars
I don't think he was "poo-poohing" it exactly. If everyone only follows
evidence based medicine there will never be any improvement because there will
never be any changes.

So someone has to be the person who ignores it, who tries new things. And that
someone, in this field, is Warwick.

The evidence part comes in when that person also _tracks_ their results! i.e.
makes it a study, and Warsick certainly seems to do so.

~~~
6cxs2hd6
Yes I got the impression his attitude about evidence-based medicine is similar
to that saying about accounting -- that "accounting is the art of looking in
the rear view mirror and dragging your ass into the future".

i.e. Accounting isn't worthless. It's necessary, but not sufficient.

------
Terretta
Money quote, applicable to hacking in general:

 _“We are used to thinking that a doctor’s ability depends mainly on science
and skill. The lesson from Minneapolis is that these may be the easiest parts
of care. Even doctors with great knowledge and technical skill can have
mediocre results; more nebulous factors like aggressiveness and consistency
and ingenuity can matter enormously... What the best may have, above all, is a
capacity to learn and adapt—and to do so faster than everyone else.”_

------
aidenn0
Doesn't the central limit theorum imply that most sane distributions of
quality of care (including all doctors are equally good) will result in a bell
curve?

There are tests you can do to tell if the variance between care centers exceed
that you would expect from chance, but the mere observation of a bell curve
seems completely uninteresting.

~~~
hessenwolf
Say each doctor's patients had a common constant death rate for that doctor,
not a crazy assumption, and those constant death rates across doctors followed
a gamma distribution (not hard, because I could fit my ass with a gamma), then
the distribution of each doctor's patients' life expectancies would be
exponential and the distribution of life expectancies of the whole group would
be Pareto.

Exponential is completely positively skewed and thin-tailed, and Pareto is
roughly the same shape but very fat-tailed.

------
conorh
I hope that more and more data becomes available around healthcare and
outcomes!

Medicare recently produced this data
[http://www.nytimes.com/interactive/2014/04/09/health/medicar...](http://www.nytimes.com/interactive/2014/04/09/health/medicare-
doctor-database.html) and the center that my wife works at used it to to
produce this graph [http://blog.parathyroid.com/parathyroid-surgery-
medicare/](http://blog.parathyroid.com/parathyroid-surgery-medicare/). They
are quite a bit better at this particular operation than anywhere else in the
US, not surprising given how many more they do, however it is often difficult
for patients to know or understand that - who should they trust? More data
please! Interestingly this operation is one that Atul Gawande (who wrote this
article in 2004) specializes in too.

~~~
angersock
If you're interested in referral patterns:

[http://omni.docgraph.org/](http://omni.docgraph.org/)

------
altrus
Significantly, this article neglects to consider how sample size (and
demographics) might affect the measurement of hospital treatment quality.

tl;dr: If you don't adjust for the sample size, what may appear to be the
best, or worse, hospital, may look like that only because of how they have
(un)successfully treated a single patient, and thereby yield an unrealistic
estimate of patient quality (ie; 100% cure rate of a single patient, which may
not be reflective of the actual caliber of doctors).

Further reading, which is strongly recommended if you want to learn more;

[http://nsmn1.uh.edu/dgraur/niv/TheMostDangerousEquation.pdf‎](http://nsmn1.uh.edu/dgraur/niv/TheMostDangerousEquation.pdf‎)

~~~
danso
Sorry...but I don't get what you're postulating...that the doctors and
hospitals whose reputations are at stake...nevermind the lives of their
patients who are affected by policy decisions...would be totally ignorant of
that basic statistical caveat? That it's possible that the top-of-the-line
hospital just happened to cure a couple of really lucky patients and everyone
who has lauded that hospital has never heard of regression analysis?

I guess it's possible...but it's probably more likely that the New Yorker is
not trying to be a reference on statistical methods. In any case, the
statistical caveat you mention is arguably addressed in this catchall-
paragraph that briefly describes the problem of quality-of-care statistics:

> _In recent years, there have been numerous efforts to measure how various
> hospitals and doctors perform. No one has found the task easy. One
> difficulty has been figuring out what to measure. For six years, from 1986
> to 1992, the federal government released an annual report that came to be
> known as the Death List, which ranked all the hospitals in the country by
> their death rate for elderly and disabled patients on Medicare. The spread
> was alarmingly wide, and the Death List made headlines the first year it
> came out. But the rankings proved to be almost useless. Death among the
> elderly or disabled mostly has to do with how old or sick they are to begin
> with, and the statisticians could never quite work out how to apportion
> blame between nature and doctors. Volatility in the numbers was one sign of
> the trouble. Hospitals’ rankings varied widely from one year to the next
> based on a handful of random deaths. It was unclear what kind of changes
> would improve their performance (other than sending their sickest patients
> to other hospitals). Pretty soon the public simply ignored the rankings._

 _Even with younger patients, death rates are a poor metric for how doctors
do. After all, very few young patients die, and when they do it’s rarely a
surprise; most already have metastatic cancer or horrendous injuries or the
like. What one really wants to know is how we perform in typical
circumstances. After I’ve done an appendectomy, how long does it take for my
patients to fully recover? After I’ve taken out a thyroid cancer, how often do
my patients have serious avoidable complications? How do my results compare
with those of other surgeons?_

(the author himself is a surgeon and has written a lot about the problems of
reliably measuring quality of care performance)

~~~
altrus
To be clear, I'm not at all suggesting that doctors do this out of malice -
only that effective statistical analysis is really, really, hard, and it's not
immediately clear that surgeons (regardless of their international reputation)
would have the background required to properly analyze these studies in a
coherent framework, especially when it's not immediately relevant to a
patients outcome in an operating theatre.

I'm not suggesting that they're ignorant of basic statistical facts, but I am
definitely suggesting that they're not immediately aware of the subtle
assumptions implicit in many of the statistical models they use.

For example, given the large numbers we're talking about, not only is it
possible that the "number 1" hospital in any particular field is there because
of statistical fluke, it's actually likely that this is the case.

The (mis)use of statistics certainly isn't limited to medicine, but it is one
of the places where its misinterpretation has the biggest impact.

[https://www.sciencenews.org/article/odds-are-its-
wrong](https://www.sciencenews.org/article/odds-are-its-wrong)

------
trustfundbaby
Ostensibly an article about medicine but really more about how to achieve true
excellence at what you do ... definitely a long read but well worth the time

------
jsudhams
The same question came to my mind and I asked a doctor "Sir, what happens to
the students in MBBS who just pass taking a border mark?" The doctor said you
should still be ok because passing score itself is like 60% + lot of practice
involved. I did not bother to check the pass mark required for the course but
now-a-days I go to doctor who have at least 10+ years of experience.

~~~
lmm
> now-a-days I go to doctor who have at least 10+ years of experience.

That's not very Kantian. How are less-experienced doctors supposed to get
experience?

~~~
jsudhams
With some else :) Keeping the joke aside. I would want them to assist senior
doctor for some time and personally i always use second opinion. BTW, if you
do not you can get anti-biotic off the shelf and most of the doctors will give
you in first visit.

------
rsanek
Really good takeaway near the end of the article: "What the best may have,
above all, is a capacity to learn and adapt—and to do so faster than everyone
else."

------
grifpete
Q.What do you call the guy who passes out last in his class on his med finals?
A. Doctor

~~~
jtheory
Q. What do you call the guy who scored just a teensy bit lower than that?

A. Mister.

IOW, this joke/insight is only valuable if the standard for passing is set too
low.

If it's set correctly, then everyone who passed has shown they are prepared to
be a competent doctor.

~~~
herdrick
If competence is, as the article suggests, normally distributed, then there's
no natural level at which to draw the cutoff line. But anyway, iirc, studies
show no correlation between outcomes and class rank (nor selectiveness of
medical school), at least for surgeons.

