
How a robot pharmacist wound up giving an overdose - steven
https://medium.com/backchannel/beware-of-the-robot-pharmacist-4015ebf13f6f
======
Terr_
I work developing an internal company app that's already a little trigger-
happy on the e-mail alerts (which are not at all life-and-death) and a lot of
it boils down to an organizational/political malaise.

The most common problem is when the people _setting_ alert policies are
divorced from the people _receiving_ them. The managers have no direct
incentive to keep the alerts sane or moderated. On an individual basis, every
alert that exists _shifts potential blame_ from the manager to the employee.
("It's all the employee's fault for not following the alerts.")

In this way, _over-_ alerting is often "automated micromanagement", but
without even the feedback-loop and mutual-accountability of actual
micromanagement. Until the people _acting_ on alerts have some feedback into
the alert-configuring process, the problem will continue.

Another dysfunction is when people ask for alerts _purely for an independent
paper trail_ , either because they're paranoid about assigning blame, or
because they want to use their e-mail client as a half-assed searchable
database.

P.S.: We've already had managers asking us if we could have e-mail alerts that
users must confirm receipt of by visiting the website... and for the site to
continuously e-mail followups until they do. Meanwhile, the IT staff have
their own horror-stories of ancient inboxes grown too large to reliably read
data from.

~~~
angersock
This is exactly it--nurses are usually not empowered to set alarm thresholds
or delays, but physicians are. So, docs ask for the most sensitive settings
they can (because they want that intimacy of care), but they're gone 15
minutes later and the rest of the unit has to deal with the fallout for the
rest of the day.

~~~
maddog99
Not sure what "intimacy of care" means, but most docs, like me, are frustrated
and annoyed by the barrage of wolf-crying that comes out of these systems, of
which Epic is one of the best and best-designed. (So far, all the others I've
seen are FAR worse.) I WANT to see the meaningful alerts, but it's difficult
to do with a sea of identical-appearing utter BS. Besides alert fatigue,
there's a related problem with cognitive load. For this patient, I want to
order A, B, C, and D. Now erect roadblocks in front of A, B, and C, and quite
often D will fall by the wayside. By the way, I've never been able to set or
even influence the alert policy. This was mandated by corporate risk
management, whose policy appears always to be more alerts = safer.

~~~
angersock
We've seen alarm policies range from "Dr. X wants this to be the threshold on
the patient" to "Unit policy is X, change during a PDSA cycle someday" to "Eh,
this seems alright for this patient" to "silence as many alarms as
reasonable".

I'm sympathetic to "it's corporate's fault", but instead of fixing _that_
problem I see a lot of folks lobbying for shiny new toys, meetings to stroke
egos, and basically spending their political capital on being prima donnas
instead of making their lives easier from an operations standpoint--and for
the rest of the docs such as yourself this obviously is a problem. :(

The roadblocks bit is what it is--if doctors were better at prescribing
medications and hadn't cowed their colleagues and nurses into a culture of
"oh, well, I shouldn't say anything" then automated systems wouldn't be so
compelling from a safety standpoint. As it is, they are undermined continually
(as explained in this article) by the humans who have to work with them and
who--as you've exemplified--don't trust them.

Let's face it: in the next 30 years, most physicians who are basically
knowledge workers are going to be doing work very differently than they are
today, most of it with heavy computerized assistance. You can choose which
side of history you want to be on.

------
coldcode
UX design, especially for medical software, is incredibly important and yet is
rarely done by people who actually understand how to do it.

I remember when my mom had radiation treatments for brain cancer, which were
so awful they killed her quicker than the disease would have (by a few weeks).
A few months later I read about how terrible the design for calibrating these
machines was leading to a large number of mis-treatments, my sister and I
wondered what dose she actually received.

During the Vietnam war pilots turned off all the alarms in some of their
planes due to constant alarms that always happened during intense combat
situations leading to an inability of focus on what mattered.

UX design is not easy but has to be treated as more important than just making
it pretty. Understanding how real people use these things is incredibly
important and not just a last minute bullet check.

~~~
sjg007
Sorry to hear about your mom. Do you have any references for the calibration
design article? Also do you know the linear accelerator that was used?

~~~
coldcode
No, it's been a few years.

------
comex
There are least two comments here already ascribing the error to incompetence;
I expect more will come.

In my opinion, this is a _dangerous_ misperception. Hindsight is 20/20\. You
saw one screenshot displaying the incorrect dosage; the doctors see "scores"
each day, day in, day out. The brain is hardwired to ignore repetitive
stimuli. Humans are smarter than computers overall but have a much higher
error rate; "competence" can decrease that rate, but not to zero. There is
_always_ a chance of a mistake.

The software could have been designed to make the probability of requesting a
38.5x overdose from the robot pharmacist _negligible_ , by reserving a special
warning for extreme circumstances. This requires the programmers to do a non-
shit (shall I say "competent"?) job once, but not constant vigilance by the
doctor, who I'm sure can use the saved mental cycles to check for many other
potential errors the machine can't handle _completely trivially_.

I expect a more thorough design overhaul would cause a non-negligible
reduction in the occurrence of a wide variety of mistakes, dangerous and not.

~~~
FireBeyond
There was a remark made by one of those involved, in the article, that EPICs
alerting didn't differentiate between a 'trivial' alert, say "Dose is 1.02x
maximum recommended dosage" and a much more critical "Dose is 38.5x..."

~~~
bentcorner
A scale would be useful:

    
    
         Recommended dosage
            |
        ----|-x------------
    

vs

    
    
        ----|------------x-
    

Also for mg vs mg/kg, that's a harder problem - off the top my head, get rid
of the modes, and show both mg/kg and mg textboxes, with a radio button to
select the one you want to edit. Edit it, and the non-selected text box is
grayed-out and uneditable, and shows you the corresponding calculation (enter
in mg/kg, and it shows you the mg amount of medicine; enter in mg, and it
shows you the mg/kg ratio).

~~~
maddog99
As a sanity check, what about showing the ordering physician a picture of 38.5
tablets? "Is this correct?" I think that would grab my attention.

------
discardorama
FTA: "There is no difference between a minuscule overdose — going 0.1
milligram over a recommended dose — and this very large overdose. They all
look exactly the same."

That here is the problem, right here. If we can have self-driving cars (which
can look ahead and figure out the result of their actions), so can a system
like EPIC. Why can't it calculate the probability of a Bad Thing(tm)
happening, and suitable tag the alerts?

~~~
vilhelm_s
Maybe this is just a case of comparing a tech demo (the car) against a
deployed system. Presumably hospital software will gradually improve in the
future.

------
Turbo_hedgehog
Confused about one thing, did the robot put 38.5 pills in a bottle, or did it
formulate a pill containing 38.5 pills worth of meds into one pill, which the
resident gave to the patient? I'm thinking sitting there and giving the
patient 38.5 pills for the patient to swallow would have been considered
unusual?

~~~
FireBeyond
It attached them in 39 little bags to a loop of plastic. You can see the
picture near the bottom. If I see 39 individual pills in 39 individual bags
like that and am being told that I am to administer them, at once, to a
patient, alarm bells in my head are ringing, as a paramedic. Swallowing 39
tablets like that is not fun for anyone, even aside from risks of OD.

~~~
bentcorner
I'm wondering about this too. I guess we'll have to wait until Part 3.

One other thing that went wrong here - through no fault of her own, the mom
was in a different room with the brother. She knows what the patient normally
takes, and would likely have started asking questions when she saw her son
taking way more than what he usually took.

------
angersock
A few different observations:

First, Epic does a lot of custom work--as noted in the article, each
installation is "customized" to only have particular alerts. This makes it
hard to count on a standard system behavior, _because there isn 't one_. This
continued coddling of clients makes it very hard to introduce new software
into the market, because you can't do it without deep pockets and sloooooow
sales cycles.

Second, the physicians are very much overloaded with alarms and alerts--and
instead of saying "Hey, let's figure out why we keep ending up with so many
alarms" they instead do the short-term thing and just ignore or turn them off.
The entire profession is rooted in split-second decisionmaking, for reasons I
cannot fathom. They really, really, really want to be the heros that come in
and save the day, and systematic approaches to operations optimization just
aren't a priority.

Third, the data coming off of these things (and at work we've seen this)
usually indicates large operational deficiencies. Unfortunately, you can't
just tell the clients "Hey, you all are doing things wrong, and here's the
proof", because they either won't care, or they will fire you and bring in
somebody who keeps them happy, or they'll care but be unable to make changes
because that introduces uncertainty into their line of work and that could
hurt someone--nevermind all the existing casualties.

~~~
ams6110
_The entire profession is rooted in split-second decisionmaking, for reasons I
cannot fathom_

Because insurance companies, Medicare, and Medicaid reimbursements barely
cover costs, so to be profitable they have to make it up on volume.

~~~
angersock
It's more fundamental than that, I think. It's core to the entire mindset of
the profession.

If you compare the operators of decades ago with the sysadmins of the 90s and
the devops folks of today you can see a rough path of evolution that hopefully
medicine will follow: a last-minute heroic effort to save a system is a
failure, not a victory.

------
robertfw
> (The computer symbol for swearing, #$@&%*!, was chosen because it’s the
> output when you’re trying to type numbers but you’ve inadvertently engaged
> the caps lock key.)

Did this used to be true? I have never encountered this. Holding down Shift,
perhaps, but not caps lock...

~~~
iamcurious
Symbol swearing goes to at least 1900:
[http://languagelog.ldc.upenn.edu/nll/?p=2483](http://languagelog.ldc.upenn.edu/nll/?p=2483)

Not sure about the Caps Lock thing though.

------
geoelectric
> Like many other physicians, pharmacists, and nurses, Jenny Lucca found
> alerts to be a constant nuisance. Even giving Tylenol to a feverish child
> every four hours triggered an alert that the dose was approaching the
> maximum allowed.

You know, given what we know about the liver toxicity of
acetaminophen/paracetamol at this point, not sure this is the best example of
a spurious alert. There's a good argument that this is one of the more
dangerous drugs OTC, in part due to the widespread impression that it's not
very dangerous.

It is interesting looking at the story in general as an example of alert
fatigue, though, and the need to differentiate different types of alerts.

Unlike some of the other comments I see here, I don't think this points to
incompetence so much as human nature. It takes a fair amount of cognitive
power to overcome the natural tendency to assume that when the alert was
spurious yesterday, it's not spurious today.

That power would be better spent elsewhere, and points to a failure of
automation to truly lighten the load.

~~~
nucleardog
Not "exceeding the maximum dose" just "approaching the maximum dose".

"Hey, you're almost taking _exactly_ the amount of this thing we say you can
take. Watch out!" is pretty much the definition of a spurious alert to me.
It's telling you that what you expect to happen is happening.

~~~
geoelectric
Well, yeah, except pills up the amount in steps--it's not like turning a dial.
So you'd want to warn before they can give another dose of medication that
would take you over the line. Really depends on how far off the line the
warning is.

But either way, in this case I'm pretty sure Tylenol was stressed in the
article because it was an example of an innocuous medication, which it really
kind of isn't.

------
matznerd
Serious cliff hanger on this article, with the second part of how it actually
made the mistake held off until tomorrow. Reminds me of one of those old tv
shows...

------
ccvannorman
> Computer experts call this type of problem — when the same action can result
> in two very different results — a “mode error,” and it is especially
> problematic when the user is not focused on the mode (in this case, mg
> versus mg/kg)

Terrifying example of one way AI or computer-augmented decisions can go
horribly wrong. A "Mode error" is at the core of the difference between
intelligence and common sense.

------
mcphage
I'm torn on this. On one hand: we can see the process, and clearly identify a
number of mistakes in the software that led to this problem. Serious mistakes,
that need to get fixed. And on the other: while this was an error that
wouldn't happen without computers and robots, it's clear that this process
will _reduce_ the number of overall errors, saving peoples lives. But it's a
different set of people.

This is similar to the "Trolley Problem" ethical thought experiment—we have a
computerized system, that will save a lot of peoples' lives—at the cost of
others, who wouldn't have otherwise died. It's a hard problem, with no easy
solution. See:
[http://en.wikipedia.org/wiki/Trolley_problem](http://en.wikipedia.org/wiki/Trolley_problem)

------
Someone1234
While I absolutely agree computers, UIs, and systems in general can stand
between a patient and an error, can we just stop and acknowledge how
incompetent the doctor was for prescribing 38 instead of 1 double strong dose?

The UI screenshot in the article makes that mistake VERY clear, and their "are
you 100% sure" question is obviously intended to illicit a particular response
(who is ever really 100% sure of anything?).

I know this article is about how systems can be altered to protect patients,
but much like with the NTSB's findings, sometimes it legitimately is pilot
error (or doctor error in this case).

Can things be better? Yes. Absolutely. But so can doctors. These types of
systems are stop-gaps, without them this doctor could have calculated the
wrong dose by hand and made just of large of a mistake.

~~~
DanBC
Yes, the doctor and pharmacist and administering nurse failed here. But that
is not useful. The useful bit is why they failed, and that seems to be a mix
of "busy" with "software sucks".

The doctor didn't order 38.5 tablets -- at least she didn't think she was. She
thought she was ordering a dose per kg, not a quantity of tablets.

You say the screen is very clear. It's clear enough when I read it. I had a
good night's sleep and I am alert and awake. I've been primed by the article
to look for the error. I imagine I would easily miss the mistake if I've been
working hospital wards for the last 10 hours on a rolling nightshift, looking
after some very ill patients.

The fact that it's possible to use two entirely different dosing systems to
order medication and have the screen not clearly indicate[1] which is being
used is so obviously sub-optimal I'm kind of surprised to see anyone saying
that it's "just pilot error".

~~~
33W
> She thought she was ordering a dose per kg, not a quantity of tablets.

Actually, she thought she was ordering a single dose at 160mg, and
accidentally ordered dose per kg. 160 was the size of the dose, so the
patient's weight became the number of pills.

------
seagreen
Pharmacists shouldn't be "clicking through" alerts. They should have a
separate, joystick style device with an actual trigger (bright red). That
trigger, and that trigger alone, can dismiss an alert.

Seriously, what would that cost, like $20? (Obviously there are a lot more
things wrong with this system than just using a mouse for too much, but this
would help). We should be willing to invest more in our machines:
[http://www.loper-os.org/?p=861](http://www.loper-os.org/?p=861)

~~~
chucksmash
And if they are beset with hundreds of mostly innocuous alerts a day, the low
signal/noise ratio of the alerts and their sheer quantity will train the
pharmacists to press that button just a bit more blithely each time. That
leads one the believe the correct course of action is to reduce the number of
alerts but then that was at least part of the cause of the incident related in
the article.

My wife is a Pharm.D. at a hospital (that coincidentally is currently
installing Epic). I know she worries (obsesses even) over the possibility of
making an error that hurts or kills someone. I'd like to think other people
doing that work feel the same way. Even if she can get three or four nines
worth of confidence in every decision she makes however, that leads to some
unfortunate probabilities when multiplied out by a career's worth of
decisions. We can improve the technology to reduce the chances of
significantly negative outcomes but they will always be non-zero.

I don't envy her or the programmers working on systems like Epic. It's messy
work with serious consequences.

~~~
seagreen
Reducing alerts is way more important than my idea, I totally agree with that.

------
maddog99
My proposal: usability experts should identify an acceptable range of alerts
on a per-order, per-encounter basis. End-users of all disciplines should have
the opportunity to provide feedback in real-time, in the live environment.
Just like Amazon reviews, "Was this alert helpful?" If you always vote "no",
your feedback will be down-weighted. With a few exceptions, like wrong patient
or 10x overdose, all the other alerts will be ranked in order of utility and
only the top x% will be shown.

------
brandonb
I wonder if somebody could build a bayesian filter for Epic alerts.

If we can filter out the Nigerian prince scams from your email, why not apply
that to save lives?

~~~
Kalium
I suspect a great deal of lurking complexity and privacy concerns there.
What's spurious in one case is critical in another, and the difference lies in
HIPAA-sensitive data.

------
sjg007
This is why you must always always be at your kids beside and if you are in
the hospital, always have a friend with you (if you are in the hospital and
otherwise not capable), who can double check and ask questions on your behalf
every time.

------
viciousambition
The title of this is basically false. The humans gave the overdose, the robot
merely processed the order.

------
revelation
Give me a break. The system has alerts and it spells out the exact dosage in
tablets in a dozen places, which is then apparently seen by at least three
people.

If these people in the course of their work choose to ignore alerts, are
distracted and not actually doing what they were hired to do (hint: theres a
reason there were three humans in the chain here), the fault lies on them, not
on the software.

You obviously can't have it both ways. We call it automation because things
will happen automatically. At the end of the day, this system probably causes
significantly less problems than the _old_ way of trying to read some doctors
handwriting only to pick up the wrong medicine regardless.

~~~
llamataboot
Really, you think the UX of those screens is in anyway clear or that the
alerts shouldn't be color coded or differentiated by thresholds from "hey this
looks like it might be an error" to "this is out of the realm of possibility,
do not pass go, do not collect 200, correct this immediately"

~~~
revelation
All I'm seeing is that the salient information is on the UI in at least three
places. It spells out the freaking calculation.

So clearly, these people simply didn't look. Not once. In their perpetual
multitasking frenzy that any hospital staff seems to be in, they didn't do the
actual task. I'm reasonably sure that you can color code or flash whatever yo
u want and they would still miss it if that's the way they work.

~~~
wyldfire
A graphical representation of the dose relative to the patient weight or the
dose prescribed relative to the recommended dose would be a great way to help
users from accidentally making this mode error.

