
Medical devices software testing overview - Totoradio
https://agilemeddev.com/2016/09/25/medical-devices-software-testing-overview/
======
rubidium
Note: this post is just the overview. There are a bunch more posts on the site
([https://agilemeddev.com/](https://agilemeddev.com/)) at the site that deep
dive into each of the parts.

It seems to be a helpful framework to work from.

------
kelvin0
I'm surprised by the 'shallow' nature of the article, and it's obvious points
(medical devices cannot fail). Almost seems self promotional.

I thought formal methods or at least advanced static analysis was going to be
mentioned before I clicked on the URL.

~~~
a3n
Are formal methods and advanced static analysis used a lot in medical device
software development?

~~~
joe_the_user
Well, a big question is "what is a medical device"?

Is that any object or software that comes in contact with patients?

I worked producing software that segmented images produced by a medical
imaging device. The software would be built-in but it was the kind of thing
one could have also have done exporting the images to Photoshop.

So the question of what the boundary between normal software and medical
software is going to be rather important if medical software winds-up
constrained by NASA level procedures.

(Obviously, implanted devices need very particular standards but other devices
raise questions).

~~~
seren
IEC62304 requires that for risk management you decompose your sw in modules or
units and assess the risk for each one.

Roughly a class A module or device even if malfunctioning can not cause harm.
For example it could be a logging or debugging module. It is annoying if it is
broken for you but should not harm the patient.

A class B malfunction can have minor or severe consequences but that should be
reversible like a minor injury, it can not have long lasting effect.

Finally class C means irreversible effects (amputing the wrong leg, death).
Misdiagnosis are often class C (unless your system is detecting common cold)

This analysis has to be done for SW you develop but also for SW you buy:
anything that goes into the product is concerned. In your case I assume you do
not sell the final product but this the responsibility of your customer to do
the risk management. And if you are class C you have to provide a whole lot of
evidence : requirements, tests, FMEA, tests results, detailed design and so
on.

This does not really answer what is or what is not a medical device. As far as
I know in most jurisdiction it is linked to "claim".

If you are claiming that you system is curing or doing the diagnosis of some
ailment, this is likely a medical device. On the other hand, if you do not
claim anything (like homeopathy or food supplement), you are not regulated.
That's why most of the time health mobile app do not claim to be able to
diagnose anything.

------
geoelectric
I know it was a throwaway, but be careful about the "untested code never
works" comment. It plays into the fallacy that if it works, it's adequately
tested.

You talk a lot about testing for specified requirements, but not so much about
looking for undesired behavior outside the requirements. That's where the
fallacy bites you. It's especially pernicious when you start to firm up a
working prototype or are inheriting a legacy production release with a history
of good behavior, but which is about to change execution context in some way.
Changing the context means hitting unexplored territory and there be dragons.

Of course, you're a professional tester (as am I), and I know this is obvious
for you. But for the average PHB or engineer-driven organization trying to
prioritize, it's important to be pretty clear about this:

Untested code works great in the 95% case--this is one of the big reasons QA
has a credibility problem sometimes, because they're spending $$$ to confirm
it already works. It's the 5% case you need to worry about. It can kill your
product or, in your case, the patient.

~~~
julienzaegel
(I'm the author of the aforementioned article)

Yes it was a throwaway :-) But I mean "untested" as not tested at all, by any
means; code that has went through automated testing (unit tests and the like)
has been tested quite a bit, so when QA receives it, the defect rate is not so
overwhelming (5% or so as you say). I maintain that the code I write almost
never works if I don't write automated tests for it (which is why I always
write tests now :-))

You're right about the importance of "looking for undesired behavior outside
the requirements". I'll think about an update of the article about manual
tests. Thanks for the input!

~~~
geoelectric
I was QA prior to the advent of mandatory unit test for commit (I'm actually
an SDET or tools/infra SWE nowadays, depending on hat) and, even then,
untested code was mostly fine, as evidenced by the fields of green "PASS"
spreadsheets.

Those spreadsheets either meant it was good when you got it or you did a
crappy job testing it, and either way it was fine before being tested.
Regression testing in particular is an exercise in generating "PASS" results,
and historically QA practice has been dominated by regression testing.

Luckily, SW practices have changed enough that (as you say) most code is at
least minimally tested by the developer and, maybe more importantly, some
minimal level of unit testing is expected now as a basic professional
standard. Hopefully we never move backwards again.

------
Beltiras
I just started work at a medical software company. This was a godsend.

------
thefastlane
medical software and agile development ... what could go wrong?

~~~
officialchicken
Not too much actually. You need to have a design history file (DHF) and a
Failure-Mode Engineering Assesment (FMEA) document as part of your "Quality"
system for FDA submission, usually at the time the project starts. Defining
and using agile methodologies forces those documents/proceedures to be more
in-sync with the software system and any unit/system/e2e testing and CI. It
also formalizes the testing process - bringing some structure to the agility
of your chosen methodology.

