It's possible to think of the process that airline safety regulators have gone through where they have observed crashes and guessed about the cause and then suggested fixes as a type of learning problem.
For goog's case it would be fun to try and build a supervised learning system whose sole purpose is to try and identify the queries which a human observer would consider a "catastrophic failure"
I've also heard of stories where decision trees supposedly outperformed human cardiologists in making diagnoses. (I'm skeptical of this claim but let's assume that it's true) If this type of advance is real then it could save a lot of lives. Unfortunately, if goog's engineering team has this kind of doubt about the machine, then I imagine that it would be easy to persuade a random jury that installing such a poorly understood black box is negligence.
"I've also heard of stories where decision trees supposedly outperformed human cardiologists in making diagnoses. (I'm skeptical of this claim but let's assume that it's true)"
If you substitute "Statistical Prediction Rules(SPRs)" for "decision trees" in your statement, then it's true in every tested field of medicine. Read the FuturePundit article (URL below) and it's links. The system _always_ outperforms the experts in it's field, without fail within the domain of expertise. Even the _best_ experts. Always.
This article is very interesting, thank you for posting it. I haven't read it yet, but it seems possible that this review suffers from a kind of selection bias, in that the only SPR's which have appeared in the literature are those which were more successful than human experts. This does not mean that a widespread switch to SPR's would outperform human experts, because the average implementation could be inferior to human experts. Consider how difficult it is for IT departments to follow best practices in security.
another aspect of these systems which people may find repugnant is that they allow one to weigh the costs and benefits of different treatment decisions in a consistent way. I personally feel, after watching people I loved suffer in an oncology ward that denying treatments which are very expensive but have small probability of success would be a good thing, but I know other people don't feel this way and there would be outrage.
For goog's case it would be fun to try and build a supervised learning system whose sole purpose is to try and identify the queries which a human observer would consider a "catastrophic failure"
I've also heard of stories where decision trees supposedly outperformed human cardiologists in making diagnoses. (I'm skeptical of this claim but let's assume that it's true) If this type of advance is real then it could save a lot of lives. Unfortunately, if goog's engineering team has this kind of doubt about the machine, then I imagine that it would be easy to persuade a random jury that installing such a poorly understood black box is negligence.