
Company was hosting cardiac patient monitoring on EC2 - jln
https://forums.aws.amazon.com/thread.jspa?threadID=65649&tstart=0
======
stick
This is a combination of posters not understanding what the technology is used
for and the tech guy exaggerating the urgency of the situation.

Remote cardiac monitoring isn't for people who are imminently going to die of
catastrophic heart attacks or who risk dropping into a fatal arrhythmia (wonky
heart rhythm). In fact, there isn't even solid evidence that it saves lives.
What it _is_ used for is investigating patients who have vague symptoms which
might be related to transient changes in their heart beat.

These are people who come to their doctor at the age of 54 and say "I felt my
heart beating really quickly and felt kind of faint" or "I felt kind of dizzy
and then passed out. It's happened twice in the last month." The more
traditional way to investigate these patients are with what's called a Holter
monitor, as alluded to by a previous poster. These are little belt packs you
carry around while wired-up and that record your ECG for 24-48 hours at a
time. The main weakness? You said it: the device only has a 24-48 hour windows
to capture the weird, often rare, rhythm.

There are different ways remote cardiac monitoring systems report their
results and I'm not sure which this particular company was using, but it
doesn't really matter. Some of them only report weird stretches, others only
report events when the patient says they're feeling symptom X, others are
reporting continuously.

Take-home message: this is not life-or-death data. When doctors (at least
those who are allowed to keep practicing) think a patient needs critical
cardiac monitoring, they admit them to hospital.

This was a tech guy, looking to jump the queue by trying to raise a red flag
because his servers were being used for--OMFG!--cardiac monitoring. A lot of
my doctor buddies use similar strategies when they're caught speeding by the
police. "I just got called in to the hospital!"

…to see a patient with a really nasty rash.

~~~
mamp
From the website I think this analysis is correct. Holters store the 24 hour
data on the machine as well. The 12 leads are not continuous but set pieces
which also store locally. If they were providing real-time monitoring with
alerts then this would be more serious. This seems to be a backup system,
ironically itself without backups.

------
_3u10
This reminds me of doing tech support back during the dot bomb. Everyday we'd
have day traders phone in complaining that their internet was down and they
were losing thousands of dollars. Then I'd advise them to switch to their
backup connection, then they'd say they didn't have one.

Finally, I'd ask why if they were running a business that could generate /
lose thousands of dollars in a few minutes why they had chosen a residential
internet service instead of opting for the business package and why they did
not have a backup connection as neither the business packages or residential
packages came with a SLA that was suitable for the type of risk inherent in
their business.

If their data is so important why are they hosting it on the absolute cheapest
and shittiest servers they can find without a backup solution in place? If
they read the TOS for EC2 it probably says not to run critical infrastructure
on it.

Perhaps for sensitive data upon which people's lives depend they might want to
look into something
like:[http://h20223.www2.hp.com/nonstopcomputing/cache/76385-0-0-0...](http://h20223.www2.hp.com/nonstopcomputing/cache/76385-0-0-0-121.html?jumpid=go/integritynonstop)

Yes, it costs more than EC2, but there's a reason for that. Wait until this
company finds out that if their values exceed what MySQL expects it will
silently truncate it.

------
byoung2
_We are a monitoring company and are monitoring hundreds of cardiac patients
at home._

If this is a serious post and there really is a service that monitors cardiac
patients from home, first you would have to deal with the unreliability of in-
home broadband connections. What if a router needs to be reset, or the DSL
goes out? Even if you ignore that, you would expect that any server setup will
go down at some time. It has happened to Google, Facebook, Twitter, and now
AWS and PlayStation Network. I remember when Rackspace went down a few years
back (a truck crashed into the transformer that powered the datacenter -
[http://www.datacenterknowledge.com/archives/2007/11/13/truck...](http://www.datacenterknowledge.com/archives/2007/11/13/truck-
crash-knocks-rackspace-offline/)). All of Armenia was knocked offline when a
woman cut a fiber optic line while digging for scrap metal. I'm not sure I'd
trust the internet with any kind of life or death monitoring.

~~~
esrauch
I could be way off base here since I only know what I saw on the site, but it
doesn't look like this is life-or-death monitoring. It looks like its long
term monitoring for later review by a doctor. It doesn't seem at all likely
that they have someone monitoring the incoming data in realtime for heart
attacks, that would be far better suited by hooking up the system to dial 911.

I think some IT person was freaking out that they had downtime and slightly
exaggerated the life endangering part, they likely just lost information that
may have been used in a future diagnostic manner.

~~~
dr_
Seems like it was real time monitoring. Otherwise why not just use the
traditional Holter monitor, which is later reviewed by a doctor?

~~~
weaksauce
Apparently it only has a 24-48 hour limit which lowers the likelihood of
finding the super rare event that they are looking for. Still they should have
a local cache of a few days that they can sync when amazon gets back up at the
very least. Typically though you would want to have a separate datacenter to
handle the emergency situations like this though and a smaller local cache for
local Internet outages.

------
recampbell
Interesting quote from a company providing such services:

"The Remote Health Monitoring System is the IT backbone that supports
HealthFrontier’s entire portfolio of solutions, including the ecg@home™ and
the microtel™/ecgAnywhere™. It captures data transmitted by the patient
through USB, Bluetooth, or trans-telephonically. The RHMS™ then stores the
information in a database, which can be accessed by the patient’s doctor
through a web-based interface. Benefits include:

Cloud computing: The RHMS is hosted on a network of servers across the US,
which increases reliability and eliminates the need for a large capital
investment in on-site hardware and maintenance."

<http://www.healthfrontier.com/rhms.php>

How many other companies provide home ECG monitoring which report over the
web? I'm looking but having a hard time finding more. I will update this post
as I do.

[UPDATE] Other Home-based ECG monitoring services:

Medtronic CareLink: [http://www.medtronic.com/your-health/heart-
failure/device/ca...](http://www.medtronic.com/your-health/heart-
failure/device/cardiac-resynchronization-therapy-
defibrillators/carelink/index.htm)

[UPDATE] A more complete listing:
[http://www.medcompare.com/matrix/1475/Outpatient-Cardiac-
Mon...](http://www.medcompare.com/matrix/1475/Outpatient-Cardiac-
Monitoring-\(Telemetry\)-Services.html)

~~~
jmm
My uncle owns a few elder care franchises and had me look into monitoring
services as a funemployment project. So I've spent a bit more time with this
than I'd like to admit...

The company I was most impressed with was <http://www.halomonitoring.com> .
The device seems to be designed well and the monitoring services (online
portal for caregivers or family members) are impressive, comparatively.

There's kind of a sea change happening with home health monitoring -- from the
"I've fallen and can't get up" devices to more robust and interactive
solutions. GE recently bought a company that provides institutional monitoring
equipment and services.

Lemme know if you want the full report :)

------
sliverstorm
Hmm... That would rather seem to be the monitoring company's own fault.
Admittedly I haven't used ec2 for about 3 years, but I seem to recall there
being an explicit disclaimer about how it wasn't supposed to be used for this
kind of uptime-critical application. I want to say it even specifically
mentioned medical stuff.

~~~
zaidf
Agree. I'd avoid cloud for half as critical apps where I cannot CLEARLY
picture where my servers are in a physical sense and how I can physically get
a hold of them if shit hits the fan.

~~~
jrockway
How do you physically get a hold of your servers if the gas main under them
blows up or a tornado sucks them up and drops them a hundred miles away?

Servers are not physical things that you can just go get when there are
problems. They are imaginary devices that can suddenly vaporize, leaving no
trace. Plan your deployment accordingly, and you won't need to be writing
emails saying that people are dying because your data center blew up. Your
systems will silently switch over to some other data center and you'll file a
claim with your insurance company to have the servers replaced on Monday
morning.

In other words, "the cloud" had nothing to do with this problem. It could
happen if the server was physically attached to you or in a abandoned missle
silo with an army of armed guards.

------
teilo
Elastic Load Balancer, anyone? I have no sympathy for this company. They
deserve to get sued for this.

Yes, for something life threatening like this, EC2 is a bad idea, but they
didn't even bother to take advantage of the geographic redundancy, much less
something so basic as having backup AMIs ready in another AZ in your current
region.

Of course, a whole lot of companies are learning this now.

~~~
kinofcain
ELB doesn't work across regions, the failure at EC2 affected multiple
Availability zones in the Virginia Region. ELB might have helped, but it's
likely that some people had all their "availabiliy" zones knocked out.

In fact, it's looking like all the automatic-failovers from people hosting on
those data centers compounded the problem when everyone's scripts tried to
recover at the same time – what was referred to in a previous article as a
"bank run".

But yeah, if you're doing heart monitoring, you need to have your servers
replicated across a wide geographic area.

------
zaidf
This is where you wish you had a data center address and a cabinet number
where you could pop in, grab your servers and move the heck out to another DC.

I wonder if this will get more companies to maintain a non-cloud, vanilla
setup on some dedicated or collocated boxes.

May be we should have an annual day to bring down our primary servers and see
how the backups do. The idea shouldn't be to confirm if everything works or
not. It should be to determine what did work and did not work.

~~~
sliverstorm
You know, that gets me thinking- wouldn't it be pretty damn slick if Amazon
offered a way to plug your own hardware into the greater EC2 cluster remotely,
so that if Big EC2 goes down, you still have copies of your machines running
locally?

Maybe you could even timeshare your machines out for some other instances for
a discount on your service, similar to how you can send solar power back up
the power grid for an electricity credit. Then ec2 really WOULD be a cloud!

~~~
ceejayoz
That sounds horrid. It'd mean my instances might be running on your ancient
crap computer in a closet hooked up to dialup.

~~~
sliverstorm
You could always add min spec requirements for machines to join, increase
redundancy of instances ec2-wide, add quality tiers that control what machines
your instances go on, etc.

Amazon could even simply benchmark your server and network connection, and
compensate you as a function of that, which would drastically reduce the
incentive to hook up old crappy hardware to the cloud.

It's really the idea of being able to run copies of your OWN instances locally
that intrigues me though.

~~~
ceejayoz
> It's really the idea of being able to run copies of your OWN instances
> locally that intrigues me though.

No reason you can't already do this.

------
cypherpunks
I'd like to see more back story first. Medicine is about risk management.
There are places where you need 1/100,000 reliability, and there are places
where you need 1/1000, and there are even places where 1/10 is good enough.
There are primary systems, and there are auxiliary systems.

Without any background, it is really tough to tell where this fits. You can't
blame the people for making the wrong decision until you know what the risk
profile for the failure is.

------
blantonl
Actual quote in the Amazon support forums from this company:

" _This not just some social network website issue, but a serious threat to
peoples lives!_ "

~~~
kierank
I have a feeling (and I really really hope) this is a joke because of that
sentence.

~~~
aheilbut
An event monitor isn't a life-critical system. Perhaps it messes up a study,
and ticks off some cardiologists, but it's not as if somebody's implantable
defibrillator is depending on EC2 being online.

------
ceejayoz
Later in the thread they say two instances are up but the third has most of
the patients. If any of their patients die, they should be convicted of
negligent manslaughter, not Amazon. Amazon explicitly warns that single
instances can be lost. Any EC2 system needs redundancy, let alone a patient
critical one!

------
renegadedev
I recently had the "good" fortune of being at the receiving end of a sales
pitch from one cloud hosting service (not Amazon). The pitch centered more
around flexibility (ability to add CPUs, RAM, storage, network nodes) rather
than reliability issues as the one EC2 is facing right now. All questions
about reliability were brushed aside with blanket statements like "it's in the
cloud" as if that was an end-all-be-all. After much prodding the sales rep
admitted that the hosting was all in one location (low lying hurricane prone
area) and if we desired redundancy we would have to mirror our apps at a
different geographical location. As with any other product, don't believe the
sales pitch. If it sounds too good to be true, it probably is. Always use
redundancy specially for critical apps.

------
protomyth
I really hope this is someone's idea of a joke, but the nature of this
person's other questions tends to make me believe no.

Sometime during the last couple of days, a lawyer was getting a lecture from
IT on how the system works and is going to have a very, very bad year.

------
jamesbkel
I wish they offered some more detail on how important/life-threatening this
is. Are we talking monitoring as in "detect signs of impending/occurring
cardiac event and notify emergency services"? Or is this monitoring in a
longitudinal sense, assisting physicians to look at trends in their patients?
If it is the later case, is data stored on the device until it's uploaded, or
is it simply lost if the connection fails?

Still, not really forgivable to be 100% AWS, but it's a very different story
if this is primarily a recording device vs. a monitoring device.

------
st0p
That's not even stupid, that is moronic.

~~~
lawnchair_larry
Why? Who else would you suggest for providing this type of service?

Given the information we had a week ago, I can see how Amazon may have looked
like the best solution for HA. _Their_ site never goes down, after all.

~~~
st0p
Well, first of all if they had used more geographical regions (I don't know
what the official amazon buzzword), they shouldn't have real bad problems. I
don't know about you, but I believe mission critical systems should be hosted
in at least 2 physically different data centers. When it's life critical, this
should be a no brainer.

Second of all, according to Amazon PR speak this failure should not be
possible. Every person in IT worth their salt should know PR speak is _not_ to
be trusted. Being reliant on AWS for your average business app is completely
okay though. However, hosting apps which might endanger peoples lives on black
box technology over which you have little influence is _moronic_ imo.

Last but not least I seriously question the privacy implications of this.
Hosting parts of your medical app in the cloud is fine and I don't know the
details of this app, but I wouldn't be happy to know that my medical history
is stored by some third party I don't entirely trust with such sensitive data.

~~~
lawnchair_larry
1) Amazon's outages took down multiple "availability zones"

2) Amazon is HIPAA certified, which means they adhere to the industry standard
for storing private medical data. In this sense, they are as good as any other
company that such data is being passed around.

~~~
st0p
Yeah, but availability zones are _not_ geographical zones. Amazons outages
were only in one datacenter.

Didn't know what HIPAA meant, but apparently they are certified for storing
medical stuff.

------
forgotAgain
This is too incredible to be true. But if it is true then I would expect the
company in question to be out of business quite soon. A company that put's a
life critical system on AWS shows a total disregard for their patients well
being.

~~~
jauer
I've seen things much worse than this with medical records and data.

Up to a few months ago we had a ASP.NET shared hosting customer that was doing
some kind of data relay web service for medical imaging. No encryption.
Patient data in full view on the server. No redundancy. Apparently it was used
for outsourcing imagery review or something. If it didn't work doctors would
have to drive in from home which slowed down the diagnostic process.

"Mission critical" on a $30 a month shared hosting plan. Very much not HIPPA
compliant.

------
csomar
Isn't this a good start-up idea? A start-up that backup your website and
deploy it to another hosting in case the one you are using is down.

~~~
troels
How about a host-abstraction layer, so that you can deal with different
providers through a uniform interface?

~~~
csomar
I'm actually thinking of that. You deploy to the service that take care of
deployment, backups and transfer in down times. It would make up of it another
kind of hosting company, though.

~~~
troels
Yes, it would essentially be a hosting company with multiple providers. That
in it self is a single point of failure, but at least it would give some
insurance against outage of the more physical kind.

------
yaix
Just wow! How difficult can it be to have a backup dedi server with the DB
mirrored running at some other provider to quickly switch to, just in case?
Even many simple web sites have a config like that. Clouds have been around
for decades, they are still new and may fail.

------
arkitaip
Surely this must be criminal? This is one area where atleast 99.9999% uptime
should be mandatory.

------
dasil003
Troll?

------
code_duck
This is just supplementary monitoring, not a true life-or-death situation.
Everyone (the company involved included) is welcome to stop blowing it out of
proportion any time.

------
gte910h
I see EC2 as a good piece of infrastructre to have as _part_ of a failover
plan. But should it be even 1/3 the providers for servers? No.

~~~
bluesnowmonkey
Triply redundant hosting providers is really setting the bar high. Lot of
20/20 hindsight being put into play the last few days!

In my experience, small companies typically have _one_ hosting provider, off-
site backups, and the probable but untested ability to bring up a fresh
installation in a few hours. That's both common in the real world and good
enough for most situations. It would get you through an outage like this.

~~~
gte910h
It's a medical device. It's not a imposition in that case.

------
FreshCode
Bullshit, linkbait post. After reading the whole thing, I would gladly spend
karma to downvote this.

~~~
lell
Are you referring to the link in the title? It is true that the company was
hosting cardiac patient monitoring on EC2. This was not life threatening
infrastructure, but that is not implied by the HN title. How is it linkbait?

The poster on Amazon's forum misrepresented the severity of the infrastructure
and tries to backpedal later on in the thread. This is an example of someone
using EC2 and fucking up, firstly for hosting production level infrastructure
without backups, and secondly for misrepresenting the importance of the
infrastructure in an attempt to get more technical support, making them look
liable for some sort of medical malpractice.

The poster probably panicked when they realised their first mistake, and
thought they were showing that they were trying to get back in control. I
don't think the poster was totally stupid, maybe she/he didn't understand the
risks of EC2, or maybe it wasn't even their decision to have no backups --- in
which case they should have pressed harder at meetings or refused to continue
to work on the project. And then they panicked. We can all learn something
from this.

------
rbanffy
So, they experienced a Schrödinger situation...

------
tybris
Those people should be prosecuted for utter incompetence.

------
chrisjsmith
Epic fail for the company which is hosting on their kit. Whoever threw it on
there is a total dick. There is nothing wrong with EC2 here. It's just not the
right tool for the job.

If they are monitoring cardiac patients, they should, at least for non-
critical people, have the instrumentation cache the data and send when
available over leased lines.

If it is critical, it should be PROPER HARDWARE attached to the POTS network
(like we have in the UK!) with multiple redundant networks over several
distinct carriers and multiple monitoring stations.

It sounds like their product is a) crap and b) dangerous if used for critical
care.

~~~
MichaelApproved
Ec2 is fine here. All systems have downtime. Its not having a backup system in
place that's the problem.

~~~
chrisjsmith
Cardiac monitoring systems DO NOT have downtime believe it or not.

~~~
ceejayoz
Bullshit. Leads come off, equipment malfunctions, patients fidget, and they
definitely don't have redundant stuff on each patient.

~~~
muyuu
I'm going to guess grandparent means that they don't include "downtime" in the
contract. Which would be outrageous, BTW.

------
logjam
If this story is true (and I have some doubts), you'd think this would have
been a no-brainer to not try to keep critical data like this in the cloud.

But even for non-critical medical data, non-tech savvy medical folks (and this
includes some physicians, although in general physicians are more tech-savvy
than most managed care decision makers) are going to get into increasingly
serious trouble if they don't think through these kinds of things very
carefully.

Driven by electronic health care mandates and incentives, companies are
pushing portable devices for medical data gathering (portable as in tablet, ie
without local data store), and some groups are going to be knee-jerk tempted
into buying into this without thinking through the ramifications of not owning
your own data -- and by that, I don't mean vague marketing-speak hand-waving
that "you own your data" through the simple mechanism of password-protected
access to your patient's data living on someone's server farm.

By actually owning your patient's data, I mean strongly considering
restricting your local practice to use of on-site media.

If you don't, you'd better have complete confidence that putting your
patient's data on the web reflects a reasonable standard of care/privacy, that
retention policies don't bite you in six months when your cloud provider goes
out of business (or changes hands or farms out data storage or backup to some
"economical" third-party), that you're going to have access to your data when
you need it, that your cloud provider is going to audit access and changes to
your patient's data and report the results of that audit to you accurately and
regularly, that legal requests for data are not acted upon without you or your
patient's consent, that backups are subjected to the same levels of security,
etc. etc.

Electronic medical records are a good idea. It can be done right, as systems
such as the Veteran's Administration's VISTA system have demonstrated.
However, there is a fine line to walk here. Implementing EHR for the sake of
government incentive, or for the convenience of insurance companies, or even
the convenience of walking into an exam room carrying a light-weight tablet
(which have their own disadvantages for data gathering) rather than a laptop
with a hard-drive isn't going to cut it when you put your patients and your
practice at risk of harm.

------
MSDjunior
who was the IT professional that decided to rely soley on the AWS cloud
without any kind of "just in case" solution....with people's lives at stake?
lol. I think I'm gonna subscribe to the hoax theory

------
mkramlich
"We'll gladly agree to take care of grandma as long as you understand we only
guarantee a 99% uptime. Good? Good. Sign here."

------
beck5
This is the sort of service which should have a back up, its not reddit.

If it can go wrong it will go wrong.

------
MichaelApproved
I hope no one dies because of this. Yet, the sad part is, it'll take someones
death to get serious attention and help prevent this stupidity in the future.

Of course, then things will swing wildly in the other direction with massive
legislation and regulation thats designed to get a lawmaker reelected more
than it is to save lives.

------
nblavoie
Wow, this is the worse idea I've ever seen since a freaking while. Poor him
(or them) though.

