Integrated Gradients
Python implementation of integrated gradients [1]. The algorithm "explains" a prediction of a Keras-based deep learning model by approximating AumannโShapley values for the input features. These values allocate the difference between the model prediction for a reference value (all zeros by default) and the prediction for the current sample among the input features. TensorFlow version is implemented now!
Usage
Using Integrated_Gradients is very easy. There is no need to modify your Keras model.
Here is a minimal working example on UCI Iris data.
- Build your own Keras model and train it. Make sure to complie it!
from IntegratedGradients import *
from keras.layers import Dense
from keras.layers.core import Activation
X = np.array([[float(j) for j in i.rstrip().split(",")[:-1]] for i in open("iris.data").readlines()][:-1])
Y = np.array([0 for i in range(100)] + [1 for i in range(50)])
model = Sequential([
Dense(1, input_dim=4),
Activation('sigmoid'),
])
model.compile(optimizer='sgd', loss='binary_crossentropy')
model.fit(X, Y, epochs=300, batch_size=10, validation_split=0.2, verbose=0)- Wrap it with an integrated_gradients instance.
ig = integrated_gradients(model)- Call explain() with a sample to explain.
ig.explain(X[0])
==> array([-0.25757075, -0.24014562, 0.12732635, 0.00960122])Features
- supports both Sequential() and Model() instances.
- supports both TensorFlow and Theano backends.
- works on models with multiple outputs.
- works on models with mulitple input branches.
Example notebooks
- More thorough example can be found here.
- There is also an example of running this on VGG16 model.
- If your network has multiple input sources (branches), you can take a look at this.
MNIST example
We trained a simple CNN model (1 conv layer and 1 dense layer) on the MNIST imagesets. Here are some results of running integrated_gradients on the trained model and explaining some samples.
References
- Sundararajan, Mukund, Ankur Taly, and Qiqi Yan. "Axiomatic Attribution for Deep Networks." arXiv preprint arXiv:1703.01365 (2017).
Email me at hiranumn at cs dot washington dot edu for questions.