This repository provides code needed to reproduce the experiments in my 'Understanding Measures of Uncertainty for Adversarial Example Detection' paper. I've tried to clean it up a little to make it clearer and easier to understand, and to remove code not relevant to this particular paper. It has a couple of requirements; you will need to install cleverhans. Other than that the requirements are basically the standard scipy stack, plus keras and tensorflow.

Reproducing ROC curves

Prepare dataset

The ASSIRA cats and dogs dataset used in the paper can be downloaded here. Save the zip file in repository's root directory and unzip it:

unzip [DATASET FILE].zip

Train the classifier

To train the ResNet-based cats and dogs classifier on the dataset execute:

python cats_and_dogs.py

Evaluate classifier on synthetic dataset

To evalute the classfier on a synthetic dataset as described in the paper execute:

python ROC_curves_cats.py

The script calculates ROC curves and their AUCs based on entropy and MI for the different models and saves that data as *.h5 files.

Plot results

To actually plot the ROC curves and some adversarial examples execute:

python plot_roc_cats.py [FILENAME].h5 # e.g. my_roc_curves_fgm.h5