"The existence of an explanation system in adherence to the NIST's proposed guidelines will in itself signal that the underlying recommender can be trusted"
Not for long. This is trivially forgable with current AI tech. It's easy. Your AI tells you that X is in category Y because there's an 80% match on reason 1, a 65% match on reason 2, a 45% match for 3, etc. etc. Reason number 2 is, for the sake of argument, outright racist, and the person running this AI knows that, so they simply hand you an explanation with that reason removed. You have no way of knowing whether this has happened, neither does anyone else, and people continue to accuse your AI of bias. (Especially after I "helpfully" normalize the reason factors for you against the list I handed you, not what came out of the AI.)
Any human-comprehensible explanation produced by a program is certainly human-editable, and almost certainly practically editable by code.
If you were going to design an AI architecture to provide you parallel construction reasons systematically, it would be hard to produce something better than a neural net.
Wouldn’t a good explanation need to account for everything it’s purporting to account for and not contain any details that can be omitted or altered without changing what it accounts for? An explanation that omits that reason number 2 would not account for why the output changes when the input to reason number 2 changes in isolation.
You, the person expecting an explanation, won't have access to the model to twiddle with the parameters and see what happens. In this case I'm assuming something like "the bank explaining to you why your loan was accepted/rejected", not just the model explaining to the bank employees why it was accepted/rejected, because this is where the social friction is and why anybody cares at all about explanations.
Not for long. This is trivially forgable with current AI tech. It's easy. Your AI tells you that X is in category Y because there's an 80% match on reason 1, a 65% match on reason 2, a 45% match for 3, etc. etc. Reason number 2 is, for the sake of argument, outright racist, and the person running this AI knows that, so they simply hand you an explanation with that reason removed. You have no way of knowing whether this has happened, neither does anyone else, and people continue to accuse your AI of bias. (Especially after I "helpfully" normalize the reason factors for you against the list I handed you, not what came out of the AI.)
Any human-comprehensible explanation produced by a program is certainly human-editable, and almost certainly practically editable by code.
If you were going to design an AI architecture to provide you parallel construction reasons systematically, it would be hard to produce something better than a neural net.