
“I Want to Know What Code Is Running Inside My Body” - warp
https://backchannel.com/i-want-to-know-what-code-is-running-inside-my-body-ff9a159da34b#.gait3xuzi
======
yolesaber
I saw a talk about medical device security (or lack thereof) at the Eleventh
Hope a few weekends ago. Very scary. They started off with a story about
patients in a hospital who became horribly addicted to morphine because they
were able to hack the machine from resources found online
([http://www.massdevice.com/hospital-patient-hacks-his-own-
mor...](http://www.massdevice.com/hospital-patient-hacks-his-own-morphine-
pump-massdevicecom-call/)). Go on Shodan and search for medical devices and
terminology (e.g. "radiology") and you'll see the state of things. Sensitive
machinery exposed on the open internet. A lot of medical devices have
hardcoded passwords that are used for remote operations by technicians.

Open sourcing this code would do a lot to mitigate these issues.

~~~
cooper12
The sad part is that the companies will use this security by obscurity
argument against open sourcing.

~~~
TickleSteve
contrary to popular opinion....

Obscurity _is_ good practice as _one_ layer of a layered defence system.

See "Defence in Depth"
[https://en.wikipedia.org/wiki/Defense_in_depth_(computing)](https://en.wikipedia.org/wiki/Defense_in_depth_\(computing\))

"Defense in depth is originally a military strategy that seeks to delay rather
than prevent the advance of an attacker by yielding space to buy time".

We have to acknowledge that no system is perfect, there will always be holes,
therefore a good approach is to layer up the imperfect systems which delays
the attacker.

Obscurity is one of those layers, a system will always be more secure if you
have to find it first.

~~~
nitrogen
True, but history has demonstrated countless times that closed source code
doesn't provide near enough obscurity to deter hackers, and automated fuzzing
tools make it even easier.

~~~
TeMPOraL
It also demonstrates that obscurity can significantly reduce the number of
attack attempts that are made against you. See e.g. why people move SSH to
non-standard ports - raising the entry bar for the attackers has some value.

------
TickleSteve
You most certainly _don 't_ want people to be able to modify safety critical
code within a pacemaker.

What most developers don't realise is the level of engineering strictness that
goes into anything safety-related. The rules and regulations related to
anything that affects the human body is in a different league than what most
developers are familiar with.

What _is_ a problem here, is that the design (not the code) apparently did not
take into account any messaging security, relying on obscurity as its _only_
defence.

If the code was open-sourced, don't expect to find lots of buffer overflow
attack vectors, or simple things like that. Its the design of the system as a
whole at fault, and that is already open.

Medical devices such as these are _not_ black boxes to the people that certify
them, everything is open to them, source included. Having worked in that sort
of area, I trust the systems that are in place.

~~~
adrianN
Industrial control systems are also safety critical systems that have to
adhere to very similar regulations as medical systems. Yet, when you ask
hackers in that field you'll quickly learn that they have terrible code and
abysmal security. Rules and regulations do very little to improve code quality
and security, imho. Most things in these regulations are either best practices
that any software engineer does (write tests, for example, or do code
reviews), or just checkboxes for the QA to check.

I'd argue that over time safety critical systems have worse code than normal
software because refactoring is almost impossibly expensive due to all the
paperwork that each software change ensues.

~~~
TickleSteve
industrial control != safety critical.

"industrial control" covers both certified and non-certified code, you will
have to be more specific than that for the purposes of this conversation.

Quite often, cases such as the pacemaker are flaws in _system design_ , not
_code_. (e.g. no secure messaging, probably due to the lack of awareness when
it was designed).

That is not to say that safety-critical code is perfect... just that it has a
_lot_ more rigour and inspection involved than run-of-the-mill website code.

True, the expense and overhead does indeed affect the level of change that is
acceptable to a business, but given that change is allowed, the quality of
that change is what we're discussing here and all I'm saying is that there is
a _lot_ more rigour involved than most developers here are aware of.

~~~
munificent
> That is not to say that safety-critical code is perfect... just that it has
> a lot more rigour and inspection involved than run-of-the-mill website code.

I had assumed that as well until all of the horror stories around Toyota's
firmware came to light.

[https://en.wikipedia.org/wiki/2009%E2%80%9311_Toyota_vehicle...](https://en.wikipedia.org/wiki/2009%E2%80%9311_Toyota_vehicle_recalls)

~~~
TickleSteve
Unfortunately, we are human...

Yes, the toyota case is a well publicised case. Consider though, the number of
safety critical systems that are out there performing perfectly everyday. Of
course, that is not proof of much, but the fact that you can name the Toyota
case (and probably the Therac 25 case) means that the process generally works.

------
3chelon
We hear a lot about how digital obsolescence is a growing problem, and almost
all of it refers to not being able to access your old family photos and
movies, or maybe old documents and spreadsheets. But what happens when your
pacemaker is obsolete, the source code is long lost, and no-one knows how to
update it?

Is this problem being addressed in any real way? 50 years in the future some
of today's devices may still be operating in peoples' bodies, and it seems
hard to believe that anyone would still have the knowledge and/or tools to
upgrade them. And surely it's quite a big deal to open someone up to replace
the hardware every 5 years?

~~~
etendue
The pacemaker or ICD generator is replaced when the battery is exhausted,
typically 8–10 years. The procedure is not a big deal, it is commonly
outpatient and done under local. Outside of some durable orthopedic implants,
few implants will survive in the body for 50 years: it is a very hostile
environment.

~~~
abandonliberty
>it is a very hostile environment.

I've never thought of it this way, and you are right from both a technological
and biological perspective.

Biologically we are wonderful containers of nutrients, but we have an army
only the very sneaky or militant can overcome. Once that army stands down we
are rapidly colonized - which is why we must be so careful with food/meat
storage.

~~~
Apofis
You just made me realize why bacteria and other things want to kill us so
badly... we are precisely that, really extremely-high concentration stores of
nutrients.

------
pthreads
Does anyone know if at least the FDA is allowed to review the source code for
pacemakers? Or is it a complete blackbox? Personally I would be appalled if
even the FDA is not allowed to.

~~~
beambot
With or without a warrant...?

EDIT: Not sure what's up w/ the downvotes. There are well-established ways for
regulatory agencies (whether FDA, FCC, etc) to obtain firmware for devices --
and it almost always involves a warrant under extraneous circumstances -vs-
proactively receiving proprietary code.

~~~
ben_jones
What if they had to subpoena a tomato grown in a field next to the toxic waste
dump? What if they opted not to examine that tomato because they didn't have
the resources to issue, process, and support, the lengthy bureaucratic process
involved in such things?

~~~
beambot
Who owns the tomato? Who owns the land that the tomato is grown on? You can't
just willy nilly confiscate other's property...

~~~
ben_jones
But the government can and should have the right to inspect a tomato (not
necessarily a specific tomato) if all the tomatoes in that field are slated to
go direct to consumers, right?

Similarly, what about testing for drug quality? You could even extrapolate it
out to the SEC's right to examine a private financial transaction in order to
determine legality. Or the IRS's right to inspect one's taxes to determine
compliance.

Point is (IMO) there is a need put on government by society to bypass some of
our "Inalienable" rights. In most cases this societal decision is necessary
and makes sense, and I'm arguing that code inspection of life-critical systems
is a reasonable example of such a case.

------
jordigh
Another really good talk about the topic, "freedon in my heart" by Karen
Sandler:

[https://www.youtube.com/watch?v=5XDTQLa3NjE](https://www.youtube.com/watch?v=5XDTQLa3NjE)

It's mentioned in the article.

~~~
pserwylo
I can't recommend this talk enough. If you have the time, it is one the best
keynotes I've ever been privileged enough to watch.

Ever since then, I've been very interested in all the paperwork sent to my
grand father about his pacemaker + defib implant. Recently they mentioned to
him that they will send out a remote monitoring device and Karens talk
immediately jumped to mind.

Initially the documentation talked about how the remote monitoring device
needed to be held close to the heart for a few seconds in order to download
the relevant data. This gave me a modicum of hope that at the very least it
was some sort of NFC type communication. This would at least make it harder to
physically exploit.

However, they continued to say that if you want it to monitor you every night,
just put it within three metres of your bed while you sleep. I had a search
online and there does not seem to be a lot of people particularly interested
in this area, although repos such as
[https://github.com/openaps/decocare](https://github.com/openaps/decocare)
give me hope in the ability of the community to reverse the relevant protocols
and investigate these devices.

------
vog
Obligatory talk on that topic from the 32C3:

"Unpatchable - Living with a vulnerable implanted device"

[https://media.ccc.de/v/32c3-7273-unpatchable](https://media.ccc.de/v/32c3-7273-unpatchable)

------
TickleSteve
_Please people.... before you comment on this thread, please inform yourself
on what safety critical software is really like._

There is a lot of uninformed discussion in here currently.

[https://en.wikipedia.org/wiki/Life-
critical_system](https://en.wikipedia.org/wiki/Life-critical_system)

------
SubiculumCode
Sure this is about pacemakers, but cant we say something similar about the
rest of our body?

~~~
kiba
Unfortunately, it's a black box made from nature that we have to reverse
engineer.

~~~
JoshTriplett
It's also self-modifying code that bootstraps its own compiler.

~~~
gravypod
Yea it's got some cool genetic algorithms.

~~~
jsmthrowaway
One could frame the entire computing industry as a distributed genetic
algorithm, executed by the _real_ computers in order to understand themselves
and the environment around them. One could further posit that we don't really
have a good handle on the right fitness function yet.

(I realize this sounds like a low-effort joke, but think about it for a
second.)

~~~
TeMPOraL
Well, computing today seems like primordial soup flowing through the pipes set
up by Moloch[0] - we keep doing random shit, somewhat directed by economic
incentives.

[0] - [http://slatestarcodex.com/2014/07/30/meditations-on-
moloch/](http://slatestarcodex.com/2014/07/30/meditations-on-moloch/)

------
maerF0x0
I agree, I want to contribute to FOSS medical devices.

------
randyrand
By extension should every device I own require me to have access to the source
code and output data?

Not a rhetorical question.

~~~
jordigh
Y'know, back in the early 2000's and the days of Slashdot, it was quite common
to find people who advocated for free software everywhere.

Now we find people who like yourself have to specify that the radical position
that all software should be free is something worthy of serious consideration.
That they're not joking or trying to be deliberately provocative.

What happened to us? Why did we go from boasting about installing Linux on a
dead badger and talking about how all software will some day be free to being
afraid to seriously consider the proposition?

~~~
yostrovs
Because Linux was supposed to become a great thing, but instead it remained a
paradise for geeks to do what they think is best. Software built to make
money, on the other hand, was built to improve things like ease of use,
aesthetics, and buyer's happiness, because that's what buyers were looking
for. The open-source people never really cared about the dumb people and lay
folks, the ignoramuses that didn't care to learn how to compose commands and
figure out regular expressions. And that Linux was supposed to be the shining
example of open-source software, one that so many people installed and tried,
only to find out it blows and is effectively unusable for their needs.

~~~
visarga
I was under the impression that Mac OS, iOS, Android and recently Windows all
run "Linux" under the hood nowadays. Not to mention the presence of Linux in
the cloud, which is essential for the functioning of most websites, apps and
mobile devices. Am I wrong?

Basically, it's easier to tell where there is no Linux than the opposite. That
surely doesn't sound like such a failure. It's as if all the other OSes and
systems are front-ends for Linux subsystems.

~~~
ashitlerferad
Only Android actually uses Linux. macOS/iOS are proprietary Apple things with
bits of mach and FreeBSD. The Windows thing you are probably thinking of is
that Microsoft reimplemented various Linux APIs, so software built for Linux
can run under Windows without Linux being involved.

------
radicality
How about when I'm flying an airplane; I'm also putting my life in the hands
of people that wrote the code that controls it and I have to trust that the
plane won't shut itself down mid-flight because of faulty code. Should a
similar argument be made here?

~~~
korethr
Perhaps you are being sarcastic, but I shall attempt to answer the question
earnestly anyway.

I've been reading up on aviation regs due to a recent interest in getting a
pilot's license. By my understanding, airplanes certified by the FAA as
airworthy undergo some fairly heavy testing to exactly determine and prove
what their capabilities and limits are. Given that getting your prototype
wrong can cause the plane to crash and kill your test pilot, there's incentive
to get this stuff right. Furthermore, once a plane is type certified, it can't
be modified from that configuration without further testing to prove the
modified configuration. Thus you'd better be damned certain that your engine
control computers are correct, lest the FAA revoke the plane's airworthiness
certificate, making that model unsellable, nevermind the lawsuits of the
survivors of deceased passengers or insurance companies recouping their
losses.

Additionally, from what I've seen, most general aviation airplanes are stuck
in the 60s as far as engine tech goes, in part because of the strict FAA regs.
We're talking air-cooled engines with carburettors here, no engine computer to
speak of, or if you're running a fancy modern engine, mechanical fuel
injection. Unless you're flying a brand new Cirrus SR22 or Diamond DA20,
there's no code _to_ inspect. Even if there was, it'd be in the avionics. You
don't need a nav beaon, radio, or transponder to land an airplane safely --
though they can make it much easier, and the FAA is going to want to know what
happened once you're back on the ground.

As far is large passenger jets? I'd say as a passenger, no, you don't have
access to the code. It's not your airplane; it belongs to the carrier.

Personally, if I were to own a plane and fly it, I very much would want access
to any and all code that makes the plane go. Though if I'm going to be hacking
on said code, I suppose that's what the FAA's Experimental category is for.

~~~
amelius
> Furthermore, once a plane is type certified, it can't be modified from that
> configuration without further testing to prove the modified configuration.

Would this work similarly with autonomous cars? For example, what if google
wants to change one line of code in their car?

------
r3bl
Can the mods change the link to [https://backchannel.com/our-medical-data-
must-become-free-f6...](https://backchannel.com/our-medical-data-must-become-
free-f6d533db6bed#.o436sohwc), since this is the full version of the story?

------
tim333
Sounds like they could do with a law similar to the freedom of information
ones for software of this type. Without that device manufacturers are not
going to want to publish as competitors could copy it and some people may sue
over perceived errors.

------
drdeadringer
I am imagining that young folks have Library Anxiety, and old folks have
"Google Anxiety".

Ask any question and you will find an answer. Any question you have, no matter
how banal or left-field. What is the weather? Is my grandson a lesbian? How do
I eat pizza in Italy?

Where is the biography section? How do I understand the Dewey Decimal System?
What is in the Special Collections, and what are the hours -- and do I need an
appointment? The computers are down... is there a way I can search for books
offline without randomly roaming the stacks?

------
yostrovs
I want to inspect the blueprints of every building I walk into and know the
sourcing and composition of all the structural components as well. For my life
depends on these things to be true and properly constructed.

~~~
dtornabene
I'd like to read commments here without total strawmen derailing the
conversation to nonsense. Do you have anything constructive to add?

~~~
yostrovs
Do you?

------
tn13
Very soon we would have FBI and NSA requiring these pacemakers to have a kill
switch to kill whoever they don't like.

~~~
subway
They don't need to. Private industry has shown time and again that the device
will remain insecure until consumers care (that is, indefinitely).

~~~
tn13
Those are two different things. Private industry's safety assurance will grow
up steadily as the demand from consumers goes up. Currently we are just
thankful to be alive.

For example if someone invents an artificial device that helps me get rid of
diabetes I would be super happy. It is only a generation later we would demand
that device meet certain quality floor. That is the case with all innovations.

Government however can put a very hard nail into the head of an innovation.

------
meric
Philosophy -

She doesn't know the code that's running on machines inside her body.

I don't even know the code that's running my heart.

And yet, I trust it.

~~~
tyrust
The latter has undergone far more testing than the former.

~~~
mikestew
The bug fix turnaround makes waterfall look downright lean, though.

------
grondilu
> “You’re pulling data from my cardiac device that I paid for, implanted
> inside my body, the most intimate piece of technology anyone can have, and
> yet I’m devoid of access to the device? That moved me to my core,” he says.
> “That’s just not right.”

I'm sure she must have signed a user license agreement of some kind upon
buying the device. So she shouldn't have to complain.

------
mankash666
Reasons to NOT open up the code: 1> Loss of competitive advantage 2> Open
source is not necessarily any safer (heartbleed bug ... ) 3> If software for
the pacemaker is allowed to be updated like that on a computer, someone will
update it with buggy software that can cause adverse side effects. Who owns
the liability in that case?

~~~
woodruffw
Even if we assume that these arguments are sound (which they aren't), they do
not appear to provide a proper justification for _letting a human being die_.

Put another way: If you had to explain Marie Moe's death to one of her
relatives, which of your three points would make the relative understand your
position?

~~~
niftich
These are sound arguments. Medical devices have to be certified in all
applicable jurisdictions and companies want to protect their competitive
advantage. This is no different from, say, new drugs, which are similarly
proprietary and the maker retains exclusivity for several years. Further,
medications too are risky black-boxes towards end-users.

~~~
woodruffw
They are not sound in the context of human welfare.

We do not allow food companies to hide the ingredients of their products,
because we know that companies (in the interest of profit) will fail to inform
consumers about the dangers of eating unhealthily. Similarly, we do not allow
meat producers to leave their meat ungraded because it endangers their
"competitive advantage" \- we understand (empirically) that doing so leads to
food contamination and otherwise preventable disease.

Your choice of proprietary medicine as a counterexample is an interesting one.
The pharmaceutical industry in the United States has a _long_ history of
"protecting its competitive advantage" at the cost of individual welfare -
consider how frequently "reformulations" of the same base chemical are
patented to continue milking a lucrative product that could improve the lives
of thousands if genericized.

Perhaps even more pointedly, consider the fact that we deem it acceptable (and
_necessary_ ) that the FDA step in and regulate the release of new drugs. Is
there a valuable distinction to be drawn between the sort of regulation and
evaluation that the FDA does and the sort that would be possible if
programmers could openly evaluate medical devices?

