CTO of a CV+AI/ML startup developing a radiology solution eh? Let me ask you a couple of quick questions: What was your liability insurance like? How much coverage per diagnosis did you carry?
Let me make it simpler: How much blame was your company willing to absorb if your algorithm made a faulty diagnosis?
Great question! We did our trials at two overseas locations in parallel with doctors. All uses cases were diagnostic for immigration purposes (e.g., detecting Tuberculosis and other chest infections at border points of entry). Given the non-medical use -- no liability insurance. No coverage diagnosis. Also given everything was run in parallel, double-blind with doctors also doing reads, no blame had to be absorbed. Once we got out of parallel, still we wouldn't need liability.
The importance here was demonstrating efficacy, which we did fantastically well.
Once we prove efficacy for multiple use cases, we can at least remove the "oh you computer scientists dont get it" argument and can have adult conversations about how to progress state of the art rather than continue to bleed patients dry.
I'll admit there are definitely barriers like what you mention. But those barriers are not some impenetrable force once we break down real issues and deal with them separately and start treating the problem as one we can solve as a society.
I can't help but think some of the barriers here involved proving the software in a situation decidedly different than a clinical setting. I would not be surprised if an immigration medical officer developed different views about diseases than a GP or ER doctor. They're not treating the person, they're not in a doctor-patient relationship with the person, they're not really even "diagnosing" the person, they're just deciding whether they're "too sick" to come into the country. Maybe if the person looks messed up in some other way, their chest x-ray gets interpreted a little more strictly.
>> I can't help but think some of the barriers here involved proving the software in a situation decidedly different than a clinical setting.
Totally agree. But science moves in baby steps and progress builds on progress. We started ML by doing linear regression. Then we moved onto recognizing digits. Then we moved onto recognizing cats. Suddenly, Google Photos can find a friend of mine from 1994 in images it appears to have automatically sucked up. That is amazing progress.
Similarly, our viewpoint as co-founders in the space was to solve a single use-case amazingly well and prove AUC and cost/value metrics. The field wont be moved by me or you, it will be moved by dozens of teams building upon each other.
But AI theater being good enough to replace no-stakes (because no one is liable to anyone for any errors, in either direction) medical theater is a step, just not as big a step or relevant to any use case of any importance as being sold upthread
> Once we prove efficacy for multiple use cases, we can at least remove the "oh you computer scientists dont get it"
No, you can't. Stating this is a clear proof that you don't understand what you're dealing with. In medical ML/AI, efficacy is not the issue. What you are detecting is not relevant. That's the issue. But I know I won't convince you.
They are detecting what they are testing for. But that's in most cases irrelevant regarding what happens to the patient afterwards, because it's lacking major connexions to the clinical situation that will have to be filled up by a human expert.
So it does in fact work. Unfortunately, only in trivial cases.
Maybe, but then the problem isn't an issue with AI/ML, it's that humans just suck at math.
We're terrible at bayesian logic. Especially when it comes to medical tests, and doctors are very guilty of this also, we ignore priors and take what should just be a Bayes factor as the final truth.
We're terrible at bayesian logic all right, but still better than machines lacking most of the data picture. That's why the priority is not to push lab model efficiency but to push for policy changes that encourage sensible gathering of data. And that's _far_ more difficult than theorizing about model efficiency vs. humans.
Why does it surprise you then that doctors and patients don't take the solution seriously? You don't have any skin in the game! Whereas the doctors are on the hook for malpractice, and as for the patient, well it's life or death for them.
This is largely where the art of "labeling/claims" comes into play regarding how explicitly worded a "diagnosis" can be. There is a lot of room to play on the spectrum from truly diagnosing a patient with a disease (which requires the most evidence and carries the most liability) all the way down to gently prompting a healthcare provider to look at one record earlier than another one while reviewing their reading queue.
Let me make it simpler: How much blame was your company willing to absorb if your algorithm made a faulty diagnosis?