> Conversely, it sucks to know that software you wrote is responsible for having killed even one person - to say nothing of doing so at scale.
But, how many people did your software improve the lives of?
This is the real problem with stuff like T1D automatic insulin control loops. You will likely kill some number of people through failure modes. However, how much better will the lives of those people whom you didn't kill be? T1D results in a lot of complications over time that, if they could be controlled, would result in a significant increase in quality of life for the survivors.
I actually had this discussion with a doctor who invented an asthma monitor for his daughter. It would automatically inform him and the school nurse as well as transmit the severity of event so the father could dispatch an ambulance immediately if it was really bad. I assure you it worked as well as it possibly could. However, nobody would touch turning it into a product because, at some point, the monitor was going to miss an event (bad detection, cellular system transmission failure, something) and some child was going to die and the liability was going to be enormous.
And that asthma monitor is a much easier case to discuss as it's all upside--the monitor failing doesn't kill anybody. Without the monitor, some number of children will die anyway. With the monitor, some number of those children will be saved.
Having this discussion about T1D is much more difficult.
But, how many people did your software improve the lives of?
This is the real problem with stuff like T1D automatic insulin control loops. You will likely kill some number of people through failure modes. However, how much better will the lives of those people whom you didn't kill be? T1D results in a lot of complications over time that, if they could be controlled, would result in a significant increase in quality of life for the survivors.
I actually had this discussion with a doctor who invented an asthma monitor for his daughter. It would automatically inform him and the school nurse as well as transmit the severity of event so the father could dispatch an ambulance immediately if it was really bad. I assure you it worked as well as it possibly could. However, nobody would touch turning it into a product because, at some point, the monitor was going to miss an event (bad detection, cellular system transmission failure, something) and some child was going to die and the liability was going to be enormous.
And that asthma monitor is a much easier case to discuss as it's all upside--the monitor failing doesn't kill anybody. Without the monitor, some number of children will die anyway. With the monitor, some number of those children will be saved.
Having this discussion about T1D is much more difficult.