Kudos to authors for such detailed work. I am yet to go through in detail but the way they have approached the problem certainly is refreshing. Any thoughts on how to extend this to non-vision architectures?
The parameter saliency approach can be naturally generalized to any architecture, not just vision. The changes that need to be made:
1. Aggregation. Currently, our code aggregates saliency of the model parameters by averaging on the conv filter level. We used that because in the literature filters have been shown to be interpretable. However, no aggregation can be used and the saliency profile can be computed on individual parameter level allowing for any architecture.
2. Loss. The loss is also not limited to classification losses, any other loss function can be used, e.g. metric learning or regression.
These should be fairly simple modifications of the code. Happy to help if needed!
I have benn working so so much with explainability. I think the main thing to know when trying to extend this to non vision (which I assume to be standard feature engineering based ml, let's ignore texts for a second) is that all the explainability is in the features. You have to design features that make sense. It really doesn't matter if you have the best explainability method in the world if that perfect explainability method retrasces the models prediction to features nobody understands.
Well, I agree to your point that the explainability is only as good as the features used in the model. Yet it is important to attribute the correct set of features (along with quantification) for a given instance. The SHAP framework does that to a good extent but it's focus is not towards helping with identification of model parameter related issues. This work here seems to focus on the later and hence I found it a bit different and a refreshing perspective towards the explainability aspect.
