PIL-Test-Implementation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a33b74e2",
   "metadata": {},
   "source": [
    "## PIL TEST - IMPLEMENTATION"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9008bd5",
   "metadata": {},
   "source": [
    "Here we will see the implementation of the [PIL Test](Fairness-New-Definitions.ipynb) on the [Adult Dataset](https://archive.ics.uci.edu/ml/datasets/adult)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "668e6b80",
   "metadata": {},
   "source": [
    "### IMPORTNG THE NECESSARY LIBRARIES"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60e32dd8",
   "metadata": {},
   "source": [
    "We will be using the open-source [AIF360](https://github.com/Trusted-AI/AIF360) package to use several fairness based metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "75fb0494",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "np.set_printoptions(suppress = True)\n",
    "import pandas as pd\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "# Importing the Dataset\n",
    "from aif360.datasets import AdultDataset\n",
    "from aif360.algorithms.preprocessing.optim_preproc_helpers.data_preproc_functions import load_preproc_data_adult\n",
    "\n",
    "from aif360.datasets import BinaryLabelDataset\n",
    "from aif360.metrics import BinaryLabelDatasetMetric, ClassificationMetric\n",
    "\n",
    "from aif360.metrics.utils import compute_boolean_conditioning_vector\n",
    "from common_utils import compute_metrics\n",
    "\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "\n",
    "import pickle"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fc3bf9b8",
   "metadata": {},
   "source": [
    "### DATASET"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "c1787831",
   "metadata": {},
   "outputs": [],
   "source": [
    "priv_group = [{'sex':1}]\n",
    "unpriv_group = [{'sex':0}]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "bd865afb",
   "metadata": {},
   "outputs": [],
   "source": [
    "data_adult = load_preproc_data_adult(['sex'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c02179a6",
   "metadata": {},
   "source": [
    "### LOADING PRE-TRAINED MODELS"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7059a37",
   "metadata": {},
   "source": [
    "We trained two models simple logistic regression and equalized odds regularized logistic regression [here](Fairness-Regularization.ipynb). So, we load these models for testing our new definition of fairness."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "id": "99ce99d7",
   "metadata": {},
   "outputs": [],
   "source": [
    "class Log_Reg(nn.Module):\n",
    "    def __init__(self, size_in):\n",
    "        super().__init__()\n",
    "        self.linear = nn.Linear(size_in, 1)\n",
    "    def forward(self, x):\n",
    "        prob_pred = torch.sigmoid(self.linear(x))\n",
    "        return prob_pred"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a90d14e",
   "metadata": {},
   "source": [
    "We load the test data on which the models are not trained so as to test out our new definition."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "id": "c2e01327",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"raw-test-data.bin\", \"rb\") as input:\n",
    "    dset_raw_tst = pickle.load(input)\n",
    "    \n",
    "with open(\"trained-std-scaler.bin\", \"rb\") as input:\n",
    "    scaler = pickle.load(input)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "id": "eb598a2e",
   "metadata": {},
   "outputs": [],
   "source": [
    "M = Log_Reg(len(dset_raw_tst.feature_names)) # Non-Fairness Based Regularized Model\n",
    "M_F = Log_Reg(len(dset_raw_tst.feature_names)) # Fairness Based Regularization Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "id": "e3deee71",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Log_Reg(\n",
       "  (linear): Linear(in_features=18, out_features=1, bias=True)\n",
       ")"
      ]
     },
     "execution_count": 126,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "M.load_state_dict(torch.load(\"simple-logistic-regression.pt\"))\n",
    "M_F.load_state_dict(torch.load(\"equalized-odd-regualarized-logistic-regression.pt\"))\n",
    "\n",
    "M.eval()\n",
    "M_F.eval()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e174f90",
   "metadata": {},
   "source": [
    "### PREDICTIONS OF THE MODELS"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a43e8a63",
   "metadata": {},
   "source": [
    "Here we get the predicted values of both the models. Now, we will proceed to use these predicted values as the feature for training the model $M'$ in both cases, as suggested by our new definition."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "id": "b564df36",
   "metadata": {},
   "outputs": [],
   "source": [
    "dset_tst_pred_M = dset_raw_tst.copy(deepcopy=True)\n",
    "dset_tst_pred_M_F = dset_raw_tst.copy(deepcopy=True)\n",
    "dset_tst = scaler.transform(dset_tst_pred.features)\n",
    "dset_tst_pred_M.labels = (M(torch.from_numpy(dset_tst).float()) > 0.5).numpy().astype(float)\n",
    "dset_tst_pred_M_F.labels = (M_F(torch.from_numpy(dset_tst).float()) > 0.5).numpy().astype(float)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32136f56",
   "metadata": {},
   "source": [
    "We now prepare the data for training the **adversarial** models for predicting the protected group membership from the remaining features of the test data (and the predicted labels of the original model), as per the [PIL Test](Fairness-New-Definitions.ipynb)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "id": "f6db6e8d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# y is the target feature in this case i.e. protected group- \"sex\"\n",
    "y = dset_tst_pred_M.features[:,1].reshape((-1, 1))\n",
    "# replacing the protected \"sex\" column with model M's predicted values\n",
    "dset_tst_pred_M.features[:,1] = dset_tst_pred_M.labels.ravel()\n",
    "# replacing the protected \"sex\" column with model M_F's predicted values\n",
    "dset_tst_pred_M_F.features[:,1] = dset_tst_pred_M_F.labels.ravel()\n",
    "# Features except the predicted outcome and \"sex\"\n",
    "dset_other_features = dset_raw_tst.features[:,[0]+list(range(2,len(dset_tst_pred_M.feature_names)))]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "id": "23968e4e",
   "metadata": {},
   "outputs": [],
   "source": [
    "# scaling the new matrix formed by replacing the \"sex\" column by the outcome\n",
    "# predicted by the respective models\n",
    "scaler_M = StandardScaler()\n",
    "dset_tst_M = scaler_M.fit_transform(dset_tst_pred_M.features)\n",
    "\n",
    "scaler_M_F = StandardScaler()\n",
    "dset_tst_M_F = scaler_M_F.fit_transform(dset_tst_pred_M_F.features)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "id": "b86e18f7",
   "metadata": {},
   "outputs": [],
   "source": [
    "dset_tst_M = torch.from_numpy(dset_tst_M).float()\n",
    "dset_tst_M_F = torch.from_numpy(dset_tst_M_F).float()\n",
    "dset_other_features = torch.from_numpy(dset_other_features).float()\n",
    "y = torch.from_numpy(y).float()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6e90af3",
   "metadata": {},
   "source": [
    "Here we will train our model $M'$ for the case of $M$'s predicted value containing dataset and $M_F$'s predicted value containing dataset and $M''$."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "id": "083139c6",
   "metadata": {},
   "outputs": [],
   "source": [
    "adv_mod_pred_M = Log_Reg(len(dset_tst_pred_M.feature_names))\n",
    "adv_mod_wo_pred = Log_Reg((len(dset_tst_pred_M.feature_names)-1))\n",
    "adv_mod_pred_M_F = Log_Reg(len(dset_tst_pred_M_F.feature_names))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "id": "9c3f5a35",
   "metadata": {},
   "outputs": [],
   "source": [
    "num_epochs = 5000 # Number of Epochs\n",
    "learning_rate = 0.01 # Learning Rate\n",
    "\n",
    "# Stochastic Gradient Descent Optimizers\n",
    "optimizer_M = torch.optim.SGD(adv_mod_pred_M.parameters(), lr= learning_rate)\n",
    "optimizer_M_F = torch.optim.SGD(adv_mod_pred_M_F.parameters(), lr= learning_rate)\n",
    "optimizer_wo_pred = torch.optim.SGD(adv_mod_wo_pred.parameters(), lr= learning_rate)\n",
    "\n",
    "# Binary Cross Entropy Loss Functions\n",
    "criterion = nn.BCELoss()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "id": "dd9ef5ae",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training adv_mod_wo_pred:\n",
      "epoch: 500, loss = 0.3765\n",
      "epoch: 1000, loss = 0.3181\n",
      "epoch: 1500, loss = 0.2800\n",
      "epoch: 2000, loss = 0.2534\n",
      "epoch: 2500, loss = 0.2341\n",
      "epoch: 3000, loss = 0.2198\n",
      "epoch: 3500, loss = 0.2089\n",
      "epoch: 4000, loss = 0.2002\n",
      "epoch: 4500, loss = 0.1932\n",
      "epoch: 5000, loss = 0.1873\n",
      "Trained adv_mod_wo_pred's Performance on Data:\n",
      "Accuracy:  90.95065856821128\n"
     ]
    }
   ],
   "source": [
    "# Training adv_mod_wo_pred\n",
    "print(\"Training adv_mod_wo_pred:\")\n",
    "adv_mod_wo_pred.train()\n",
    "for epoch in range(num_epochs):\n",
    "    p_pred = adv_mod_wo_pred(dset_other_features)\n",
    "    loss= criterion(p_pred, y)\n",
    "    \n",
    "    loss.backward()\n",
    "    optimizer_wo_pred.step()\n",
    "    \n",
    "    optimizer_wo_pred.zero_grad()\n",
    "    \n",
    "    if (epoch+1) % 500== 0:\n",
    "        print(f'epoch: {epoch+1}, loss = {loss.item():.4f}')\n",
    "        \n",
    "print(\"Trained adv_mod_wo_pred's Performance on Data:\")        \n",
    "with torch.no_grad():\n",
    "    y_pred = (p_pred > 0.5).numpy().astype(float)\n",
    "    accuracy_wo_pred = (np.sum(np.sum(np.array(y_pred.ravel()) == np.array(y.ravel())))/len(y.ravel()))*100\n",
    "    print(\"Accuracy: \", accuracy_wo_pred)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9c05007",
   "metadata": {},
   "source": [
    "We see above that the model $M''$ which is trained on all the other features of the test set except the predictions of $M$ or $M_F$ is achieving $90.5\\%$ accuracy. The accuracy is high indicating the presence of some proxy variables for the protected group."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "id": "8be8ebe8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training adv_mod_pred_M:\n",
      "epoch: 500, loss = 0.1874\n",
      "epoch: 1000, loss = 0.1030\n",
      "epoch: 1500, loss = 0.0702\n",
      "epoch: 2000, loss = 0.0529\n",
      "epoch: 2500, loss = 0.0424\n",
      "epoch: 3000, loss = 0.0353\n",
      "epoch: 3500, loss = 0.0302\n",
      "epoch: 4000, loss = 0.0263\n",
      "epoch: 4500, loss = 0.0234\n",
      "epoch: 5000, loss = 0.0210\n",
      "Trained adv_mod_pred_M's Performance on Data:\n",
      "Accuracy:  100.0\n",
      "PIL value:  100.0\n"
     ]
    }
   ],
   "source": [
    "# Training adv_mod_pred_M\n",
    "print(\"Training adv_mod_pred_M:\")\n",
    "adv_mod_pred_M.train()\n",
    "for epoch in range(num_epochs):\n",
    "    p_pred = adv_mod_pred_M(dset_tst_M)\n",
    "    loss= criterion(p_pred, y)\n",
    "    \n",
    "    loss.backward()\n",
    "    optimizer_M.step()\n",
    "    \n",
    "    optimizer_M.zero_grad()\n",
    "    \n",
    "    if (epoch+1) % 500== 0:\n",
    "        print(f'epoch: {epoch+1}, loss = {loss.item():.4f}')\n",
    "        \n",
    "print(\"Trained adv_mod_pred_M's Performance on Data:\")        \n",
    "with torch.no_grad():\n",
    "    y_pred = (p_pred > 0.5).numpy().astype(float)\n",
    "    accuracy_m = (np.sum(np.sum(np.array(y_pred.ravel()) == np.array(y.ravel())))/len(y.ravel()))*100\n",
    "    print(\"Accuracy: \", accuracy_m)\n",
    "    print(\"PIL value: \", (accuracy_m - accuracy_wo_pred)/(100 - accuracy_wo_pred)*100)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5c61a44",
   "metadata": {},
   "source": [
    "We see above the trained $M'$ model for model $M$ and it is starking that we predicted back the gender with $100\\%$ accuracy, whereas our initial accuracy was $90.5\\%$. This suggests that the model's unfair behavior. The PIL value of perfect $100\\%$ is a clear indication that our model is very unfair."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "id": "da83a3a4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training adv_mod_pred_M_F:\n",
      "epoch: 500, loss = 0.1885\n",
      "epoch: 1000, loss = 0.1885\n",
      "epoch: 1500, loss = 0.1885\n",
      "epoch: 2000, loss = 0.1885\n",
      "epoch: 2500, loss = 0.1885\n",
      "epoch: 3000, loss = 0.1885\n",
      "epoch: 3500, loss = 0.1885\n",
      "epoch: 4000, loss = 0.1885\n",
      "epoch: 4500, loss = 0.1885\n",
      "epoch: 5000, loss = 0.1885\n",
      "Trained adv_mod_pred_M_F's Performance on Data:\n",
      "Accuracy:  94.91571691803726\n",
      "PIL value:  43.81598793363503\n"
     ]
    }
   ],
   "source": [
    "# Training adv_mod_pred_M_F\n",
    "print(\"Training adv_mod_pred_M_F:\")\n",
    "adv_mod_pred_M_F.train()\n",
    "for epoch in range(num_epochs):\n",
    "    p_pred = adv_mod_pred_M(dset_tst_M_F)\n",
    "    loss= criterion(p_pred, y)\n",
    "    \n",
    "    loss.backward()\n",
    "    optimizer_M_F.step()\n",
    "    \n",
    "    optimizer_M_F.zero_grad()\n",
    "    \n",
    "    if (epoch+1) % 500== 0:\n",
    "        print(f'epoch: {epoch+1}, loss = {loss.item():.4f}')\n",
    "        \n",
    "print(\"Trained adv_mod_pred_M_F's Performance on Data:\")        \n",
    "with torch.no_grad():\n",
    "    y_pred = (p_pred > 0.5).numpy().astype(float)\n",
    "    accuracy_m_f = (np.sum(np.sum(np.array(y_pred.ravel()) == np.array(y.ravel())))/len(y.ravel()))*100\n",
    "    print(\"Accuracy: \", accuracy_m_f)\n",
    "    print(\"PIL value: \", (accuracy_m_f - accuracy_wo_pred)/(100 - accuracy_wo_pred)*100)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2811def6",
   "metadata": {},
   "source": [
    "We see above the trained $M'$ model for model $M_F$ and we predicted back the gender with $94.9\\%$ accuracy, whereas our initial accuracy was $90.5\\%$. This suggests that the model is much fairer. One interesting thing to notice here that regularization deliberately increases the loss a bit and prevents it from becoming very close to $0$ when predicting back protected variable, thereby it holds out from revealing or letting the protected variable from influencing the outcome way too much such that it does not becomes unfair. The PIL value of $\\sim 43\\%$ indicates that our model $M_F$ is fair as per our new definition."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "07d8f9e1",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "authors": [
   {
    "name": "Rishi Dey Chowdhury"
   }
  ],
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  },
  "title": "PIL Test Implementation"
 },
 "nbformat": 4,
 "nbformat_minor": 5
}