
What happens when patients find out how good their doctors are? (2004) - adenadel
https://www.newyorker.com/magazine/2004/12/06/the-bell-curve
======
neuro_imager
As a physician, I often think about how we lack truely objective assessment of
patient outcomes (either in the context of evaluating physician competence or,
probably more importantly, assessing and improving upon clinical practises).

I would really appreciate insight on how this could be achieved.

There are several issues which are particularly vexing:

\- The distinct lack of verifiable, objective markers of physician competence.

\- Each patient's case is unique and cases with the highest levels of
difficulty are often treated by the most experienced people. These cases, of
course, are likely to have worse outcomes than simple cases which may be
treated by less experienced (worse?) physicians.

\- Clinical outcomes are largely recorded by the same people treating the
patient so reported outcomes are often erroneous or frankly fraudulent.

\- This is made worse by the hierarchical nature of clinical medicine and
deference to seniority and title.

\- Medicine is parochial so clinical practises for the same disorder vary
tremendously. You might be treated a dozen different ways for the same
disorder and presentation depending on the facility and especially on the
specialty that ends up treating you.

\- Outcomes are not necessarily determined by clinician ability. There are
several other factors at play: the pre- and post-care (such as work-up by
ancillary staff or ICU care after a surgery), the cohesiveness of the facility
and its efficiencies (or lack thereof), availability and preferences for
resources such as medical devices, drugs and hospital equipment which may be
largely out of the hands of the physician.

~~~
Dwolb
I did some market research back in the day and there was a lot negative
opinions on even attempting to measure performance based on patient outcomes
because “each case is unique”.

To me there were two things at play: possibly the sample size of number of
patients doctors see with the same problem might be too small and there was
genuinely a lot of fear that the doctor’s skill could be boiled down to
protocols and statistics.

I think the biggest hurdle toward improving patient outcomes will be
alleviating the fear and then after that figuring out how to give better
context to the data.

~~~
phamilton
As someone who works in education, I find it interesting that "each case is
unique" hasn't prevented outcome based metrics in education.

~~~
Fordrus
A very simple hypothesis as to why doctors can argue this effectively but
teachers cannot is that teachers are not as powerful a lobbying group as
doctors.

Doctors have effectively been able to defend their "turf," from hostile
encroachment, while teachers have not, not because the situations do not
contain substantial parallels, but rather because doctors are politically
strong while teachers are politically weak.

~~~
freeflight
The term "Gods in white" exists for a reason, many people ascribe a lot of
authority and knowledge to physicians and doctors in general. Doubting your
doctor's diagnosis is usually considered a rather odd thing to do and if it's
a doctor of psychology it could even be interpreted as a symptom.

While with teachers it's kinda the opposite; Even tho their whole job is to
know and teach things, many people have a way easier time disagreeing with
them straight out of principle.

Wonder how much of that boils down to socialization aka in what contexts
children are introduced to these professions?

------
MarkMc
The article's author Atul Gawande more recently published a study showing that
you are 3 times more likely to die if you are treated in some hospitals
compared to others [1]

Both the article and study suggest what I believe is a major failing in modern
medicine: _We should be measuring risk-adjusted outcomes for hospital
treatments, then publishing a rating for each hospital on a bell curve_. With
such an approach, patients will naturally gravitate towards the better
hospitals and the poorer hospitals will have an incentive to improve their
procedures.

Here's something to consider: According to the article, "In 1964... the median
estimated age at death for patients in Matthews’s center was twenty-one years,
seven times the age of patients treated elsewhere...After Warwick’s report
came out, Matthews’s treatment quickly became the standard in this country."
If Matthews's treatment had been less spectacular - say only twice the life
expectancy of patients treated elsewhere - how much longer would it have taken
for his treatment to become the standard? Perhaps never?

[1] [https://www.nytimes.com/2016/12/14/business/hospitals-
death-...](https://www.nytimes.com/2016/12/14/business/hospitals-death-rates-
quality-vary-widely.html)

~~~
SapphireSun
I the interesting thing to consider is what happens to the hospitals that do
terribly, but not so poorly that they should be shut down. The publication of
their rating will start causing doctors and patients to avoid them. In our
fee-for-service model, this causes the hospital to starve and enter a vicious
feedback loop.

Perhaps we should have an independent commission that has the ability to
perform an outside investigation into these cases and attempt to understand
what's going wrong. For hospitals that are under-resourced, they would have to
power to increase allocations. On the flip side, it could shut down the ones
that are irreparably broken that can safely be removed from a region.

Also, we should do away with fee-for-service for good. It is the worst bag of
incentives this side of the line between healing and hurting.

~~~
JumpCrisscross
> _In our fee-for-service model, this causes the hospital to starve and enter
> a vicious feedback loop_

Sounds like an alignment of interests between hospital management and
patients.

~~~
SapphireSun
I don't understand how this helps hospital management once it enters the
feedback loop. It does incentivize them to not get there in the first place,
but I bet when you analyze the tradeoffs they make to forestall it you won't
be happy either.

This is especially a problem is when you have a lone hospital serving a poor
area. The patients don't have money, maybe they don't have insurance. If the
hospital starts cranking out bad results, it'll just deteriorate.

The market model only kinda works in theory for wealthy populations with
multiple hospitals nearby.

~~~
lmm
There's a market for cheaper, less safe cars. There would be a market for
cheaper, less safe hospitals - in fact there already is, we're just
embarrassed to talk about it openly.

~~~
SapphireSun
Think carefully about what you're saying. The rich get to live longer and
healthier and the poor get substandard healthcare? The poor are the ones that
need to work.

------
jknz
From the article, it seems that there are 2 opposite "evidence based
medicine":

(a) do research, publish papers, test thoroughly new treatments in well
defined trials and slowly build up a biological theory as well as official
guidelines and treatments. And

(b): measure the outcomes of different hospitals, declare the best performer's
methods as the state of the art and expand these methods elsewhere. If the
measurement is sound, one can argue that they are both evidence based. The
former is slow but may provide a deeper understanding of the inner workings of
the deaease. The latter is fast but it's hard to tell exactly why it works so
well.

The was a shift in machine learning research in recent years.

(a): write theoretical papers and study using math the generalization
performance of algorithms.

(b) release a new challenging dataset every year (except a test set) and
organize a prediction competition on this dataset. The winner algorithm is
declared the state of the art, and can be applied to other datasets, event
though Boone understands why it works so well.

The approach (b) was particularly fruitful and efficient in recent years.
Let's hope that applying this approach to medicine Will lead to great
outcomes!

~~~
make3
The obvious difference here is that no one dies if I start training my new
weird deep learning architecture on some challenging training set on an AWS
gpu instance and said architecture turns out to be crap.

Meanwhile, medicine is a lot less forgiving of wild goose chases, and
requiring that the taking of risks is soundly justified seems pretty
reasonable, as exciting as the idea of unrelenting research sounds.

~~~
Saaster
"One day when I was a junior medical student, a very important Boston surgeon
visited the school and delivered a great treatise on a large number of
patients who had undergone successful operations for vascular reconstruction.
At the end of the lecture, a young student at the back of the room timidly
asked, “Do you have any controls?” Well, the great surgeon drew himself up to
his full height, hit the desk, and said, “Do you mean did I not operate on
half of the patients?” The hall grew very quiet then. The voice at the back of
the room very hesitantly replied, “Yes, that’s what I had in mind.” Then the
visitor’s fist really came down as he thundered, “Of course not. That would
have doomed half of them to their death.” God, it was quiet then, and one
could scarcely hear the small voice ask, “Which half?”

—Dr. E. E. Peacock, Jr., University of Arizona College of Medicine;

~~~
Jwarder
I've seen this come up in EMS. There is the question for certain types of
patients if it better for the EMTs to provide care on scene or just get the
patient to the hospital soonest. To help determine what's better they pick
days of the week where EMTs provide minimum care. It's rough to do this to
patients and EMTs, but it's also important to know what truly affects
patients' outcomes.

------
ransom1538
Before you have a procedure performed, ask your doctor a simple question: "How
many times have you performed this procedure". What I have noticed doctors
absolutely hate his question - I have witnessed doctors become visibly angry.
I have never once had a doctor answer this question. I have heard answers
from: "We don't keep those records" to "We don't keep totals". Are these
records not kept? Why wouldn't this be public information? There must be
practices with 0 performed procedures completed, and are you patient 1? How
can doctors improve if they don't keep transparent numbers of procedures
performed, procedures failed, procedures which were successful? "How can you
define procedures failed?" \- I can hear some say. But that is a discussion
these professionals need to have to find incompetence hiding among their
ranks.

~~~
gambiting
The problem with this is that except for some really routine stuff, a lot of
operations are really custom-tailored to the patient. This might be a 100th
time the surgeon is removing a certain type of tumor, but first time doing it
in an obese person, or a child, or someone with rare blood condition, or
million other things. Once you get to the real life-saving stuff no two
operations are the same, even though on paper they might seem like they are.
That's why it's hard to put a number on it.

~~~
lightbyte
>This might be a 100th time the surgeon is removing a certain type of tumor

I think this is all OP was asking about. They want to know if you've ever
removed a tooth before, not if you've ever removed the right molar from a 27
year old very tall man who has a small jaw and bad breath.

------
aleyan
This article spent paragraphs describing a normal distribution where a couple
of descriptive statistics would do (mean and standard deviation). Ok, I kid,
that would be too technical. The old adage still stands though: a picture is
worth a thousand words. Instead of showing us what this curve looks like, they
wasted the one graphic of their column inches on a useless stethoscope Gordian
knot illustration.

One a plus side, this article appears to be famous and from 2004. Somebody
must have produced graphic of this bell curve of CF outcomes by hospitals.
Anybody got a link?

~~~
2bitencryption
...we know what the graph would look like. the author knows what the graph
would look like, the editor knows what the graph would look like.

The article isn't about the curve proves, the article is about what the curve
means.

(it also says in an early paragraph that you will see different distributions
for different diseases/operations; there isn't just one curve)

~~~
aleyan
I know what a normal distribution looks like, but I have seen too many flaws
in data or flaws in characterization of data revealed by graphs of data. If
this data and its graph is so earth shaking, show it.

------
Barrin92
>> _What makes the situation especially puzzling is that our system for CF
care is far more sophisticated than that for most diseases. The hundred and
seventeen CF centers across the country are all ultra-specialized, undergo a
rigorous certification process, and have lots of experience in caring for
people with CF. They all follow the same detailed guidelines for CF treatment.
They all participate in research trials to figure out new and better
treatments. You would think, therefore, that their results would be much the
same. Yet the differences are enormous. Patients have not known this. So what
happens when they find out?_

Maybe this is simply correct and the variance in care is small, but the
regional resources, environmental and other co-factors are distorting the
patients results. If income and insurance are lower patients might have to
take on more stressful jobs. If education is lower parents might not be able
to care as effectively because they don't understand the physician's
instructions. Other environmental cofactors in impoverished regions might
worsen certain diseases but not do so in others, and so forth.

This is a very important point because it might heavily distort how good the
care actually is, there was surprisingly little methodology in the article.

------
skrap
This one is from 2005.

But IIRC it launched Gawande's career, and I think it's a great read.

I saw him talk a few years after this. Very inspirational!

------
creep
This is interesting from another angle. I'm not sure if it has been mentioned
in the comments but I will go ahead and present it anyways.

I was diagnosed with anorexia when I was 14. The physical, medical care that I
received at my local children's hospital was alright, and I have no comparison
of course. I was refed, kept in bed, and slowly gained weight to a healthy
level. However, the psychological treatment was abhorrent. I was sent to an
outpatient eating disorder program, which treated both children and adults, at
the same hospital. I came from that place at a healthy weight, but with the
same ideas about food that I'd always had up to that point.

I no longer have an overt eating disorder, but of course they say with these
things anorexics are always in recovery. However, it took me four years to
learn healthy eating habits on my own. I did not have external help for this
at all. It took me four years to teach myself how to eat. I still look back on
the eating disorder program with distaste and distrust for the medical system.

My point is, it was very obvious to me that my psychological care was not up-
to-par, but I still don't know how my medical treatment was-- that is, fixing
the parts of my physical body that I'd damaged with malnutrition. Most people
think of psychologists, psychiatrists, and therapists in terms of their skill
and efficacy. But medical doctors are mostly assumed to be equally proficient.
I am not sure what can be done about this particular assumption, but it is
very real and can be life-threatening.

------
dennis_jeeves
When it comes to chronic cases I would think that the outcomes are more
measurable. For example treatment of diabetes or Lyme's disease. Word of
mouth, both from patients and other doctors, the internet etc. will work
relatively well depending on one's ability to separate wheat from chaff.

Any physicians reading this - feel free to contact me, I'm looking for
physicians for the long term for my own chronic, but relatively minor health
problems. I find the average physician some what mechanical on how they
approach a health condition and sometime (1)dismissive of my concerns/views.
Since I'm somewhat inclined to dig into details of a health condition, I read
up things on the internet and that means I may be more likely to question a
doctor's recommendations.

(1)= I fully understand why they would normally do it. One certainly cannot
entertain a patient who forms an opinion based on the first few articles that
they read up on the web, which is what I presume most people do.

------
shkkmo
That's a pretty interesting article.

While the measurement of performance is critical towards improving it, we need
to be careful about how we incentivize that performance. As performance on a
metric becomes more economically important, the less useful the metric is in
actually measuring performance and the less real improvement can be gained by
its measurement.

~~~
pjmorris
aka "When a measure becomes a target, it ceases to be a good measure." \-
Goodhart's Law [0]

[0]
[https://en.wikipedia.org/wiki/Goodhart%27s_law](https://en.wikipedia.org/wiki/Goodhart%27s_law)

~~~
JauntyHatAngle
I like this, a lot, but I feel like I'm going to start incorrectly applying
this in too many areas. It's dead true, but also a bit of a catch-all for so
many IT scenarios.

------
kwhitefoot
It isn't necessary to be the best. What is needed is that patient outcomes get
better. Those who are at the wrong end of the curve need to acknowledge that
they need to do better and those in charge of the system need to assist them
to improve.

Making payment conditional on results is not likely to have the desired
effect, especially if the decision to pay or not is not made by the person
actually affected. If anything it will simply drive medics away from difficult
areas of expertise.

Take the above with the usual scepticism, I'm a software developer after all,
not a medic. But it seems to me that you always need to have carrots as well
as sticks, or perhaps just carrots.

------
WalterBright
If health care was free, a problem arises. Everyone will think they're
entitled to the best doctor. Who gets the best doctor?

Under a market system, the best doctor goes to the patient willing to pay the
most.

~~~
nickpp
Funny thing about markets: they work even if you don't "believe" in them. So
the more you try to tightly control a market, the more alternatives show up.

In this case the alternatives being the usual power networks of the socialist
states: friends and relatives in government and administration. Who you know
becomes more important than what you know. Who you do becomes more important
than what you do. A small favor done once gets returned. Informal networks of
favors, relations and power turn rapidly into an impenetrable mafia.

Imagine your DMV visits applied to life-and-death situations. And it's not
only doctors. It's also access to latest medicine, newest medical devices,
best hospital beds and sections and especially doctor attention.

Source: I lived in these god-forsaken systems, unlike most of the down voters
here...

~~~
callmeal
>Imagine your DMV visits applied to life-and-death situations.

Kinda funny, but my DMV visits have been extremely smooth. If you get there
with everything they ask for (you did read the FAQ, didn't you?), there's no
reason why you can't have a similar experience.

Now my experiences trying to get a doctors appointment otoh...

~~~
SamReidHughes
My DMV experience was long lines and waiting around for an employee to
manually grade multiple choice tests. That was in CA. On the other hand things
were a lot better, way more efficiently run in PA. Why can't California give
me a similar experience as Pennsylvania?

------
agumonkey
Reminds me of Vince DeVita approach to cancer. Never give up, do not care much
about standard practice if they fail you.

Rant: the medical world is large and stretched, it's impossible to discuss
anything with a doctor (you'll have more answers than doctors asked, and they
won't tell you all). But whatever they say is godspel (or even more) so if
they decide to drop the ball, you will. And if a sibling goes against the
doctors advice, you'll only make the situation more tense.

------
dang
Discussed in 2014 at
[https://news.ycombinator.com/item?id=7664301](https://news.ycombinator.com/item?id=7664301).

------
Hasz
While Warwick and Matthews are both excellent physicians, what if there use of
non traditional/peer-reviewed methods ended up shortening someone's life?

How does a system as large an bureaucratic as health care allow for individual
discretion and experimentation at the individual physician level without
harming patients?

------
iliasku
As a patient, i feel that there is a need for the doctors themselves to
acknowledge that there is a need to measure their performance and give a way
to us patients to take informed decisions. That would be step n.1. once we
have established this need we can look on the how this can be done.

------
michaelbuckbee
An interesting corollary to this article is this:

[http://slatestarcodex.com/2017/08/29/my-irb-
nightmare/](http://slatestarcodex.com/2017/08/29/my-irb-nightmare/)

~~~
epmaybe
Everything in this story is so relevant to my current trials and tribulations
with medical IRB approval. I read this monthly just because I need to relate
with someone else's experiences.

------
brohee
The Don Berwick December 1999 speech mentioned in the article is here:
[https://www.youtube.com/watch?v=00aa6xcOXf4](https://www.youtube.com/watch?v=00aa6xcOXf4)

------
ams6110
> It belies the promise that we make to patients who become seriously ill:
> that they can count on the medical system to give them their very best
> chance at life.

Medicine makes no such promise. Doctors are going to handle any given
condition with the currently accepted "reasonable and prudent" treatment. This
is going to be appropriate for the vast majority of patients but it isn't
necessarily everyone's "very best chance."

~~~
killjoywashere
I predict that with rigorous study we will find that fully half of all
hospitals will be below average.

------
Timshel
Thank you for the repost, I read this article many years ago and was never
able to find it again.

------
bawana
So why is this 13 yr old article in hacker news? Or rather how? Is there a bot
among us?

~~~
buttcoinslol
It's an excellent article that illustrates the difference between 'good' and
'excellent' care and the attention to detail it requires to obtain 'excellent'
results. Dr. Warwick's line of questioning that revealed his patient Janelle's
reasons for not doing her treatments was illuminating and not every doctor
will dig for answers that allow their treatment plan to remain 99.95%
effective.

After finishing the article, I decided to take inventory of myself and see
where I can improve from 99.5 to 99.95 even though my job doesn't deal with
life and death patient outcomes.

Sometimes articles that aren't about computer touching can make you think
about computer touching in a different way, it's good to read a variety of
things.

------
EwanG
Not a bad article, but it is from 2004. Might be time for a follow-up.

------
MarkMc
Needs [2004] in the title

~~~
adenadel
Unfortunately, the title already was maxing out the character limit when I
submitted.

------
ataturk
I've been thinking about some of the same issues but with dentists--all the
sites to locate dentists are utter garbage and there are no reliable ratings
for dentists. Worse, dentists toe (and I argue, cross) the ethical line when
it comes to patient care, performing often unnecessary fillings in order to
make money. The whole dental industry is a cesspool of shady vendors and
salesmen pushing dentists to up sell "patients" and be able to charge bigger
bills to insurance.

------
avastmick
The depth of this article allows thinking beyond the medical domain. Warren
Warwick's success because of 'a capacity to learn and adapt—and to do so
faster than everyone else.' To do so, there needs to be a fierce consistency
of approach to enable easy measurement, yet at the edge a wild experimentalism
to ensure improvement.

Fascinating. Continuous improvement in action.

