Dataset_Preprocess.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# PreProcess APS dataset\n",
    "\n",
    "#### In this notebook, we first download the data from UCI and preprocess it so we can build a Machine Learning model. \n",
    "#### We then store this data in a training and testing folder."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset Description:\n",
    "\n",
    "The dataset we use here for predictive maintenance comes from UCI Data Repository and consists of Air Pressure System failures recorded on Scania Trucks. Read more about the dataset here: https://archive.ics.uci.edu/ml/datasets/APS+Failure+at+Scania+Trucks\n",
    "\n",
    "The positive class consists of failures attributed to APS and negative class consists of failures in some other system. The goal is to identify APS failures correctly so a downstream predictive maintenance action can be taken on this system, once the origin of the failure has been identified.\n",
    "\n",
    "This is a typical use case in Predictive maintenance (PDM): a first model identifies the root cause of the failure. Once this is identified, a second system identifies how much time one has until a failure might occur which then informs the actions that need to be taken to avoid it. Predictive maintenance, like most machine learning problems can be multifaceted."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Import Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Download the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n",
      "                                 Dload  Upload   Total   Spent    Left  Speed\n",
      "100 42.5M  100 42.5M    0     0  14.7M      0  0:00:02  0:00:02 --:--:-- 14.7M\n"
     ]
    }
   ],
   "source": [
    "! curl --insecure https://archive.ics.uci.edu/ml/machine-learning-databases/00421/aps_failure_training_set.csv --output aps_failure_training_set.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('aps_failure_training_set.csv', sep=' ', encoding = 'utf-8', header=None)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>8</th>\n",
       "      <th>9</th>\n",
       "      <th>10</th>\n",
       "      <th>11</th>\n",
       "      <th>12</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>This</td>\n",
       "      <td>file</td>\n",
       "      <td>is</td>\n",
       "      <td>part</td>\n",
       "      <td>of</td>\n",
       "      <td>APS</td>\n",
       "      <td>Failure</td>\n",
       "      <td>and</td>\n",
       "      <td>Operational</td>\n",
       "      <td>Data</td>\n",
       "      <td>for</td>\n",
       "      <td>Scania</td>\n",
       "      <td>Trucks.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Copyright</td>\n",
       "      <td>(c)</td>\n",
       "      <td>&lt;2016&gt;</td>\n",
       "      <td>&lt;Scania</td>\n",
       "      <td>CV</td>\n",
       "      <td>AB&gt;</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>This</td>\n",
       "      <td>program</td>\n",
       "      <td>(APS</td>\n",
       "      <td>Failure</td>\n",
       "      <td>and</td>\n",
       "      <td>Operational</td>\n",
       "      <td>Data</td>\n",
       "      <td>for</td>\n",
       "      <td>Scania</td>\n",
       "      <td>Trucks)</td>\n",
       "      <td>is</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>free</td>\n",
       "      <td>software:</td>\n",
       "      <td>you</td>\n",
       "      <td>can</td>\n",
       "      <td>redistribute</td>\n",
       "      <td>it</td>\n",
       "      <td>and/or</td>\n",
       "      <td>modify</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>it</td>\n",
       "      <td>under</td>\n",
       "      <td>the</td>\n",
       "      <td>terms</td>\n",
       "      <td>of</td>\n",
       "      <td>the</td>\n",
       "      <td>GNU</td>\n",
       "      <td>General</td>\n",
       "      <td>Public</td>\n",
       "      <td>License</td>\n",
       "      <td>as</td>\n",
       "      <td>published</td>\n",
       "      <td>by</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>the</td>\n",
       "      <td>Free</td>\n",
       "      <td>Software</td>\n",
       "      <td>Foundation,</td>\n",
       "      <td>either</td>\n",
       "      <td>version</td>\n",
       "      <td>3</td>\n",
       "      <td>of</td>\n",
       "      <td>the</td>\n",
       "      <td>License,</td>\n",
       "      <td>or</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>(at</td>\n",
       "      <td>your</td>\n",
       "      <td>option)</td>\n",
       "      <td>any</td>\n",
       "      <td>later</td>\n",
       "      <td>version.</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>This</td>\n",
       "      <td>program</td>\n",
       "      <td>is</td>\n",
       "      <td>distributed</td>\n",
       "      <td>in</td>\n",
       "      <td>the</td>\n",
       "      <td>hope</td>\n",
       "      <td>that</td>\n",
       "      <td>it</td>\n",
       "      <td>will</td>\n",
       "      <td>be</td>\n",
       "      <td>useful,</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>but</td>\n",
       "      <td>WITHOUT</td>\n",
       "      <td>ANY</td>\n",
       "      <td>WARRANTY;</td>\n",
       "      <td>without</td>\n",
       "      <td>even</td>\n",
       "      <td>the</td>\n",
       "      <td>implied</td>\n",
       "      <td>warranty</td>\n",
       "      <td>of</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>MERCHANTABILITY</td>\n",
       "      <td>or</td>\n",
       "      <td>FITNESS</td>\n",
       "      <td>FOR</td>\n",
       "      <td>A</td>\n",
       "      <td>PARTICULAR</td>\n",
       "      <td>PURPOSE.</td>\n",
       "      <td>NaN</td>\n",
       "      <td>See</td>\n",
       "      <td>the</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>GNU</td>\n",
       "      <td>General</td>\n",
       "      <td>Public</td>\n",
       "      <td>License</td>\n",
       "      <td>for</td>\n",
       "      <td>more</td>\n",
       "      <td>details.</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>You</td>\n",
       "      <td>should</td>\n",
       "      <td>have</td>\n",
       "      <td>received</td>\n",
       "      <td>a</td>\n",
       "      <td>copy</td>\n",
       "      <td>of</td>\n",
       "      <td>the</td>\n",
       "      <td>GNU</td>\n",
       "      <td>General</td>\n",
       "      <td>Public</td>\n",
       "      <td>License</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>along</td>\n",
       "      <td>with</td>\n",
       "      <td>this</td>\n",
       "      <td>program.</td>\n",
       "      <td>NaN</td>\n",
       "      <td>If</td>\n",
       "      <td>not,</td>\n",
       "      <td>see</td>\n",
       "      <td>&lt;http://www.gnu.org/licenses/&gt;.</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>----------------------------------------------...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>class,aa_000,ab_000,ac_000,ad_000,ae_000,af_00...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                   0          1         2   \\\n",
       "0                                                This       file        is   \n",
       "1                                           Copyright        (c)    <2016>   \n",
       "2                                                This    program      (APS   \n",
       "3                                                free  software:       you   \n",
       "4                                                  it      under       the   \n",
       "5                                                 the       Free  Software   \n",
       "6                                                 (at       your   option)   \n",
       "7                                                This    program        is   \n",
       "8                                                 but    WITHOUT       ANY   \n",
       "9                                     MERCHANTABILITY         or   FITNESS   \n",
       "10                                                GNU    General    Public   \n",
       "11                                                You     should      have   \n",
       "12                                              along       with      this   \n",
       "13  ----------------------------------------------...        NaN       NaN   \n",
       "14  class,aa_000,ab_000,ac_000,ad_000,ae_000,af_00...        NaN       NaN   \n",
       "\n",
       "             3             4            5         6        7   \\\n",
       "0          part            of          APS   Failure      and   \n",
       "1       <Scania            CV          AB>       NaN      NaN   \n",
       "2       Failure           and  Operational      Data      for   \n",
       "3           can  redistribute           it    and/or   modify   \n",
       "4         terms            of          the       GNU  General   \n",
       "5   Foundation,        either      version         3       of   \n",
       "6           any         later     version.       NaN      NaN   \n",
       "7   distributed            in          the      hope     that   \n",
       "8     WARRANTY;       without         even       the  implied   \n",
       "9           FOR             A   PARTICULAR  PURPOSE.      NaN   \n",
       "10      License           for         more  details.      NaN   \n",
       "11     received             a         copy        of      the   \n",
       "12     program.           NaN           If      not,      see   \n",
       "13          NaN           NaN          NaN       NaN      NaN   \n",
       "14          NaN           NaN          NaN       NaN      NaN   \n",
       "\n",
       "                                 8         9       10         11       12  \n",
       "0                       Operational      Data     for     Scania  Trucks.  \n",
       "1                               NaN       NaN     NaN        NaN      NaN  \n",
       "2                            Scania   Trucks)      is        NaN      NaN  \n",
       "3                               NaN       NaN     NaN        NaN      NaN  \n",
       "4                            Public   License      as  published       by  \n",
       "5                               the  License,      or        NaN      NaN  \n",
       "6                               NaN       NaN     NaN        NaN      NaN  \n",
       "7                                it      will      be    useful,      NaN  \n",
       "8                          warranty        of     NaN        NaN      NaN  \n",
       "9                               See       the     NaN        NaN      NaN  \n",
       "10                              NaN       NaN     NaN        NaN      NaN  \n",
       "11                              GNU   General  Public    License      NaN  \n",
       "12  <http://www.gnu.org/licenses/>.       NaN     NaN        NaN      NaN  \n",
       "13                              NaN       NaN     NaN        NaN      NaN  \n",
       "14                              NaN       NaN     NaN        NaN      NaN  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head(15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that this original dataset requires some preprocessing to get it in a suitable format for Machine learning. Run the function below to get a pre-processed dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def preprocessdataset(df):\n",
    "    ''' Preprocess the input dataset for Machine learning training'''\n",
    "    \n",
    "    import os\n",
    "    try:\n",
    "        os.makedirs('training_data')\n",
    "    except Exception as e:\n",
    "        print(\"directory already exists\")\n",
    "        \n",
    "    try:\n",
    "        os.makedirs('test_data')\n",
    "    except Exception as e:\n",
    "        print(\"directory already exists\")\n",
    "    \n",
    "    print(\"Start Preprocessing ...\")\n",
    "    wholedf = pd.DataFrame(np.zeros(shape=(60000,171)), columns=np.arange(171))\n",
    "    wholedf.columns = df[0][14].split(',')\n",
    "    newdf = [df[0][row].split(',') for row in range(15 ,60015)]\n",
    "    newdf = pd.DataFrame.from_records(newdf)\n",
    "    newdf.columns = df[0][14].split(',')\n",
    "    \n",
    "    print(\"Dropping last 2 columns...\")\n",
    "    newdf = newdf.drop(columns = ['ef_000', 'eg_000'])\n",
    "    \n",
    "    print(\"Shape of the entire dataset ={}\".format(newdf.shape))\n",
    "    \n",
    "    print(\"Convert the class categorical label to numerical values for prediction\")\n",
    "    newdf = newdf.replace({'class': {'neg': 0, 'pos':1}})\n",
    "    newdf=newdf.replace('na',0)\n",
    "\n",
    "    print(\"Changing data types to numeric...\")\n",
    "    newdf = newdf.apply(pd.to_numeric)\n",
    "    \n",
    "    print(\"Splitting the data into train and test...\")\n",
    "    \n",
    "    from sklearn.model_selection import train_test_split\n",
    "    X_train, X_test = train_test_split(newdf, test_size=0.2, random_state = 1234)\n",
    "    \n",
    "    print(\"Saving the data locally in train/test folders...\")\n",
    "    X_train.to_csv('training_data/train.csv', index = False, header = None)\n",
    "    X_test.to_csv('test_data/test.csv', index=False, header=None)\n",
    "    newdf.to_csv('rawdataset.csv', index=False, header=None)\n",
    "    print(\"Shape of Training data = {}\".format(X_train.shape))\n",
    "    print(\"Shape of Test data = {}\".format(X_test.shape))\n",
    "    print(\"Success!\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: user 2 µs, sys: 0 ns, total: 2 µs\n",
      "Wall time: 4.53 µs\n",
      "directory already exists\n",
      "directory already exists\n",
      "Start Preprocessing ...\n",
      "Dropping last 2 columns...\n",
      "Shape of the entire dataset =(60000, 169)\n",
      "Convert the class categorical label to numerical values for prediction\n",
      "Changing data types to numeric...\n",
      "Splitting the data into train and test...\n",
      "Saving the data locally in train/test folders...\n",
      "Shape of Training data = (48000, 169)\n",
      "Shape of Test data = (12000, 169)\n",
      "Success!\n"
     ]
    }
   ],
   "source": [
    "%time\n",
    "preprocessdataset(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now go to \"predictive-maintenance-xgboost.ipynb\" and run the code cells to train your custom XGBoost model using SageMaker built in algorithms for predictive maintenance"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}