How the LIME algorithm fails

You maybe know the LIME algorithm from some of my earlier blog posts. It can be quite useful to “debug” data sets and understand machine learning models better. But LIME is fooled very easily. We use the eli5 TextExplainer which is based on LIME and the 20newsgroup dataset to make my point.

import numpy as np
from sklearn.datasets import fetch_20newsgroups
from eli5.lime import TextExplainer

# we only look at the case atheism vs. copmuter graphics
data = fetch_20newsgroups(categories=['alt.atheism', 'comp.graphics'],
                          remove=['footers'])
X, y = data.data, data.target

te = TextExplainer(random_state=42, n_samples=10000)

Now we code a fairly stupid blackbox model, that just uses the length of the longest token in the document as a feature and splits the data based on this feature.

class BlackBoxModel():
    def fit(self, X=None, y=None):
        pass
    
    def predict(self, X, y=None):
        return np.array([int(max([len(x_ii) for x_ii in x_i.split(" ")]) >= 27) for x_i in X])
    
    def predict_proba(self, X, y=None):
        return np.array([
            1.0-self.predict(X),
            self.predict(X)-0.0
        ]).T

model = BlackBoxModel()
model.fit(X, y)

Let’s explain the predictions of our black-box model:

print(X[2])

From: af774@cleveland.Freenet.Edu (Chad Cipiti)
Subject: Good shareware paint and/or animation software for SGI?
Organization: Case Western Reserve University, Cleveland, OH (USA)
Lines: 15
Reply-To: af774@cleveland.Freenet.Edu (Chad Cipiti)
NNTP-Posting-Host: hela.ins.cwru.edu


Does anyone know of any good shareware animation or paint software for an SGI
 machine?  I've exhausted everyplace on the net I can find and still don't hava
 a nice piece of software.

Thanks alot!

Chad

This is clearly talking about computer graphics here. So let’s see what our blackbox model predicts and how LIME would explain it.

y_pred = model.predict([X[2]])
print(data.target_names[y_pred[0]])

comp.graphics

So the prediction is corrent, but let’s see if we can understand why.

te.fit(X[2], model.predict_proba)

te.show_prediction(target_names=data.target_names)

y=comp.graphics (probability 0.999, score 6.993) top features

from: af774@cleveland.freenet.edu (chad cipiti) subject: good shareware paint and/or animation software for sgi? organization: case western reserve university, cleveland, oh (usa) lines: 15 reply-to: af774@cleveland.freenet.edu (chad cipiti) nntp-posting-host: hela.ins.cwru.edu does anyone know of any good shareware animation or paint software for an sgi machine? i've exhausted everyplace on the net i can find and still don't hava a nice piece of software. thanks alot! chad

So we would conclude, that email addresses and words like software and university are important, but we know they are not. More on this issue can be found on the eli5 website. Note, that it is fairly easy to rationalize this explanation as a human if you are note really careful. This is what makes Explainable AI so dangerous.