
Trust in LIME: Yes, No, Maybe So? - gk1
https://blog.dominodatalab.com/trust-in-lime-local-interpretable-model-agnostic-explanations/
======
sgt101
Trust is not just about understanding the model's behaviour - although that is
clearly helpful.

1) The model may be applied to millions or billions of instances, humans can't
check them all, or even a small percentage. Explanations appear extensive and
general (it's using the right features) but it's very hard to prove or
convincingly demonstrate that this is so - it could be using the right
features or good features in a good way in this case - and then not in other
cases.

2) Experts / owners are generally doing the checking; users are not victims,
they should not have to simply accept what they are told. They need a means of
protection and representation in the process to have real trust. If this
doesn't happen adoption will be impeded, and I believe that once lost trust
can be very hard to regain especially in systems with stabilisation mechanisms
(memory)

3) Some users are less privileged than others, they don't have the freedom to
make decisions about trust. Models are imposed on them, there needs to be a
mechanism of wider accountability that prevents them from being abused.

I think that in addition to explanation three things are required :

\- mechanisms for managing model development accountably and transparently;
why were the decisions to use or discard data or techniques taken? What were
the tests that were applied and when?

\- mechanisms to manage models in production - what models are being used,
where? Are they demonstrated to be being used appropriately? Can any harm that
they do be discovered and repaired?

\- Market, legislative and social structures that impose these requirements
and penalise actors who fail to use them.

If not then expect AI to go the way of nuclear fission, airships and GMO.

