lesson1-rf.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Important: This notebook will only work with fastai-0.7.x. Do not try to run any fastai-1.x code from this path in the repository because it will load fastai-0.7.x**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Intro to Random Forests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## About this course"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Teaching approach"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "This course is being taught by Jeremy Howard, and was developed by Jeremy along with Rachel Thomas. Rachel has been dealing with a life-threatening illness so will not be teaching as originally planned this year.\n",
    "\n",
    "Jeremy has worked in a number of different areas - feel free to ask about anything that he might be able to help you with at any time, even if not directly related to the current topic:\n",
    "\n",
    "- Management consultant (McKinsey; AT Kearney)\n",
    "- Self-funded startup entrepreneur (Fastmail: first consumer synchronized email; Optimal Decisions: first optimized insurance pricing)\n",
    "- VC-funded startup entrepreneur: (Kaggle; Enlitic: first deep-learning medical company)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "I'll be using a *top-down* teaching method, which is different from how most math courses operate.  Typically, in a *bottom-up* approach, you first learn all the separate components you will be using, and then you gradually build them up into more complex structures.  The problems with this are that students often lose motivation, don't have a sense of the \"big picture\", and don't know what they'll need.\n",
    "\n",
    "If you took the fast.ai deep learning course, that is what we used.  You can hear more about my teaching philosophy [in this blog post](http://www.fast.ai/2016/10/08/teaching-philosophy/) or [in this talk](https://vimeo.com/214233053).\n",
    "\n",
    "Harvard Professor David Perkins has a book, [Making Learning Whole](https://www.amazon.com/Making-Learning-Whole-Principles-Transform/dp/0470633719) in which he uses baseball as an analogy.  We don't require kids to memorize all the rules of baseball and understand all the technical details before we let them play the game.  Rather, they start playing with a just general sense of it, and then gradually learn more rules/details as time goes on.\n",
    "\n",
    "All that to say, don't worry if you don't understand everything at first!  You're not supposed to.  We will start using some \"black boxes\" such as random forests that haven't yet been explained in detail, and then we'll dig into the lower level details later.\n",
    "\n",
    "To start, focus on what things DO, not what they ARE."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Your practice"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "People learn by:\n",
    "1. **doing** (coding and building)\n",
    "2. **explaining** what they've learned (by writing or helping others)\n",
    "\n",
    "Therefore, we suggest that you practice these skills on Kaggle by:\n",
    "1. Entering competitions (*doing*)\n",
    "2. Creating Kaggle kernels (*explaining*)\n",
    "\n",
    "It's OK if you don't get good competition ranks or any kernel votes at first - that's totally normal! Just try to keep improving every day, and you'll see the results over time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "To get better at technical writing, study the top ranked Kaggle kernels from past competitions, and read posts from well-regarded technical bloggers. Some good role models include:\n",
    "\n",
    "- [Peter Norvig](http://nbviewer.jupyter.org/url/norvig.com/ipython/ProbabilityParadox.ipynb) (more [here](http://norvig.com/ipython/))\n",
    "- [Stephen Merity](https://smerity.com/articles/2017/deepcoder_and_ai_hype.html)\n",
    "- [Julia Evans](https://codewords.recurse.com/issues/five/why-do-neural-networks-think-a-panda-is-a-vulture) (more [here](https://jvns.ca/blog/2014/08/12/what-happens-if-you-write-a-tcp-stack-in-python/))\n",
    "- [Julia Ferraioli](http://blog.juliaferraioli.com/2016/02/exploring-world-using-vision-twilio.html)\n",
    "- [Edwin Chen](http://blog.echen.me/2014/10/07/moving-beyond-ctr-better-recommendations-through-human-evaluation/)\n",
    "- [Slav Ivanov](https://blog.slavv.com/picking-an-optimizer-for-style-transfer-86e7b8cba84b) (fast.ai student)\n",
    "- [Brad Kenstler](https://hackernoon.com/non-artistic-style-transfer-or-how-to-draw-kanye-using-captain-picards-face-c4a50256b814) (fast.ai and USF MSAN student)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Books"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "The more familiarity you have with numeric programming in Python, the better. If you're looking to improve in this area, we strongly suggest Wes McKinney's [Python for Data Analysis, 2nd ed](https://www.amazon.com/Python-Data-Analysis-Wrangling-IPython/dp/1491957662/ref=asap_bc?ie=UTF8).\n",
    "\n",
    "For machine learning with Python, we recommend:\n",
    "\n",
    "- [Introduction to Machine Learning with Python](https://www.amazon.com/Introduction-Machine-Learning-Andreas-Mueller/dp/1449369413): From one of the scikit-learn authors, which is the main library we'll be using\n",
    "- [Python Machine Learning: Machine Learning and Deep Learning with Python, scikit-learn, and TensorFlow, 2nd Edition](https://www.amazon.com/Python-Machine-Learning-scikit-learn-TensorFlow/dp/1787125939/ref=dp_ob_title_bk): New version of a very successful book. A lot of the new material however covers deep learning in Tensorflow, which isn't relevant to this course\n",
    "- [Hands-On Machine Learning with Scikit-Learn and TensorFlow](https://www.amazon.com/Hands-Machine-Learning-Scikit-Learn-TensorFlow/dp/1491962291/ref=pd_lpo_sbs_14_t_0?_encoding=UTF8&psc=1&refRID=MBV2QMFH3EZ6B3YBY40K)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Syllabus in brief"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Depending on time and class interests, we'll cover something like (not necessarily in this order):\n",
    "\n",
    "- Train vs test\n",
    "  - Effective validation set construction\n",
    "- Trees and ensembles\n",
    "  - Creating random forests\n",
    "  - Interpreting random forests\n",
    "- What is ML?  Why do we use it?\n",
    "  - What makes a good ML project?\n",
    "  - Structured vs unstructured data\n",
    "  - Examples of failures/mistakes\n",
    "- Feature engineering\n",
    "  - Domain specific - dates, URLs, text\n",
    "  - Embeddings / latent factors\n",
    "- Regularized models trained with SGD\n",
    "  - GLMs, Elasticnet, etc (NB: see what James covered)\n",
    "- Basic neural nets\n",
    "  - PyTorch\n",
    "  - Broadcasting, Matrix Multiplication\n",
    "  - Training loop, backpropagation\n",
    "- KNN\n",
    "- CV / bootstrap (Diabetes data set?)\n",
    "- Ethical considerations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Skip:\n",
    "\n",
    "- Dimensionality reduction\n",
    "- Interactions\n",
    "- Monitoring training\n",
    "- Collaborative filtering\n",
    "- Momentum and LR annealing\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "%load_ext autoreload # reloads all python modules automatically\n",
    "%autoreload 2 # Reload all modules (except those excluded by %aimport) every time before executing the Python code typed.\n",
    "\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "from fastai.imports import *\n",
    "from fastai.structured import *\n",
    "\n",
    "from pandas_summary import DataFrameSummary\n",
    "from sklearn.ensemble import RandomForestRegressor, RandomForestClassifier\n",
    "from IPython.display import display\n",
    "\n",
    "from sklearn import metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "PATH = \"E:/Fastai/Data/Bluebook/\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Invalid switch - \"Users\".\n"
     ]
    }
   ],
   "source": [
    "! ls {PATH}\n",
    "'''\n",
    "technically this should display all the contents of the directory that the \"PATH\" variable is assigned to\n",
    "but will work only on a linux system\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Train.csv']"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "os.listdir(PATH) # another way to list all the contents of the directory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Introduction to *Blue Book for Bulldozers*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## About..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### ...our teaching"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "At fast.ai we have a distinctive [teaching philosophy](http://www.fast.ai/2016/10/08/teaching-philosophy/) of [\"the whole game\"](https://www.amazon.com/Making-Learning-Whole-Principles-Transform/dp/0470633719/ref=sr_1_1?ie=UTF8&qid=1505094653).  This is different from how most traditional math & technical courses are taught, where you have to learn all the individual elements before you can combine them (Harvard professor David Perkins call this *elementitis*), but it is similar to how topics like *driving* and *baseball* are taught.  That is, you can start driving without [knowing how an internal combustion engine works](https://medium.com/towards-data-science/thoughts-after-taking-the-deeplearning-ai-courses-8568f132153), and children begin playing baseball before they learn all the formal rules."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### ...our approach to machine learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Most machine learning courses will throw at you dozens of different algorithms, with a brief technical description of the math behind them, and maybe a toy example. You're left confused by the enormous range of techniques shown and have little practical understanding of how to apply them.\n",
    "\n",
    "The good news is that modern machine learning can be distilled down to a couple of key techniques that are of very wide applicability. Recent studies have shown that the vast majority of datasets can be best modeled with just two methods:\n",
    "\n",
    "- *Ensembles of decision trees* (i.e. Random Forests and Gradient Boosting Machines), mainly for structured data (such as you might find in a database table at most companies)\n",
    "- *Multi-layered neural networks learnt with SGD* (i.e. shallow and/or deep learning), mainly for unstructured data (such as audio, vision, and natural language)\n",
    "\n",
    "In this course we'll be doing a deep dive into random forests, and simple models learnt with SGD. You'll be learning about gradient boosting and deep learning in part 2."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### ...this dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "We will be looking at the Blue Book for Bulldozers Kaggle Competition: \"The goal of the contest is to predict the sale price of a particular piece of heavy equiment at auction based on it's usage, equipment type, and configuration.  The data is sourced from auction result postings and includes information on usage and equipment configurations.\"\n",
    "\n",
    "This is a very common type of dataset and prediciton problem, and similar to what you may see in your project or workplace."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### ...Kaggle Competitions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Kaggle is an awesome resource for aspiring data scientists or anyone looking to improve their machine learning skills.  There is nothing like being able to get hands-on practice and receiving real-time feedback to help you improve your skills.\n",
    "\n",
    "Kaggle provides:\n",
    "\n",
    "1. Interesting data sets\n",
    "2. Feedback on how you're doing\n",
    "3. A leader board to see what's good, what's possible, and what's state-of-art.\n",
    "4. Blog posts by winning contestants share useful tips and techniques."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## The data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Look at the data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Kaggle provides info about some of the fields of our dataset; on the [Kaggle Data info](https://www.kaggle.com/c/bluebook-for-bulldozers/data) page they say the following:\n",
    "\n",
    "For this competition, you are predicting the sale price of bulldozers sold at auctions. The data for this competition is split into three parts:\n",
    "\n",
    "- **Train.csv** is the training set, which contains data through the end of 2011.\n",
    "- **Valid.csv** is the validation set, which contains data from January 1, 2012 - April 30, 2012. You make predictions on this set throughout the majority of the competition. Your score on this set is used to create the public leaderboard.\n",
    "- **Test.csv** is the test set, which won't be released until the last week of the competition. It contains data from May 1, 2012 - November 2012. Your score on the test set determines your final rank for the competition.\n",
    "\n",
    "The key fields are in train.csv are:\n",
    "\n",
    "- SalesID: the unique identifier of the sale\n",
    "- MachineID: the unique identifier of a machine.  A machine can be sold multiple times\n",
    "- saleprice: what the machine sold for at auction (only provided in train.csv)\n",
    "- saledate: the date of the sale"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "*Question*\n",
    "\n",
    "What stands out to you from the above description?  What needs to be true of our training and validation sets?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "df_raw = pd.read_csv(f'{PATH}Train.csv', low_memory=False, \n",
    "                     parse_dates=[\"saledate\"])\n",
    "# low_memory makes sure that python reads enough data to ascertain the column dtype; f'{} tells python to treat the \n",
    "# content inside the curly bracket as python variable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "In any sort of data science work, it's **important to look at your data**, to make sure you understand the format, how it's stored, what type of values it holds, etc. Even if you've read descriptions about your data, the actual data may not be what you expect."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "def display_all(df):\n",
    "    with pd.option_context(\"display.max_rows\", 1000, \"display.max_columns\", 1000): \n",
    "        display(df)\n",
    "        # the pd.option_context allows the user to change the default display of pandas; here we are asking python \n",
    "        # to display 1000 rows without truncating them"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "hidden": true,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>401120</th>\n",
       "      <th>401121</th>\n",
       "      <th>401122</th>\n",
       "      <th>401123</th>\n",
       "      <th>401124</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>SalesID</th>\n",
       "      <td>6333336</td>\n",
       "      <td>6333337</td>\n",
       "      <td>6333338</td>\n",
       "      <td>6333341</td>\n",
       "      <td>6333342</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SalePrice</th>\n",
       "      <td>10500</td>\n",
       "      <td>11000</td>\n",
       "      <td>11500</td>\n",
       "      <td>9000</td>\n",
       "      <td>7750</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MachineID</th>\n",
       "      <td>1840702</td>\n",
       "      <td>1830472</td>\n",
       "      <td>1887659</td>\n",
       "      <td>1903570</td>\n",
       "      <td>1926965</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ModelID</th>\n",
       "      <td>21439</td>\n",
       "      <td>21439</td>\n",
       "      <td>21439</td>\n",
       "      <td>21435</td>\n",
       "      <td>21435</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>datasource</th>\n",
       "      <td>149</td>\n",
       "      <td>149</td>\n",
       "      <td>149</td>\n",
       "      <td>149</td>\n",
       "      <td>149</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>auctioneerID</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>YearMade</th>\n",
       "      <td>2005</td>\n",
       "      <td>2005</td>\n",
       "      <td>2005</td>\n",
       "      <td>2005</td>\n",
       "      <td>2005</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MachineHoursCurrentMeter</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>UsageBand</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>saledate</th>\n",
       "      <td>2011-11-02 00:00:00</td>\n",
       "      <td>2011-11-02 00:00:00</td>\n",
       "      <td>2011-11-02 00:00:00</td>\n",
       "      <td>2011-10-25 00:00:00</td>\n",
       "      <td>2011-10-25 00:00:00</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fiModelDesc</th>\n",
       "      <td>35NX2</td>\n",
       "      <td>35NX2</td>\n",
       "      <td>35NX2</td>\n",
       "      <td>30NX</td>\n",
       "      <td>30NX</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fiBaseModel</th>\n",
       "      <td>35</td>\n",
       "      <td>35</td>\n",
       "      <td>35</td>\n",
       "      <td>30</td>\n",
       "      <td>30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fiSecondaryDesc</th>\n",
       "      <td>NX</td>\n",
       "      <td>NX</td>\n",
       "      <td>NX</td>\n",
       "      <td>NX</td>\n",
       "      <td>NX</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fiModelSeries</th>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fiModelDescriptor</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ProductSize</th>\n",
       "      <td>Mini</td>\n",
       "      <td>Mini</td>\n",
       "      <td>Mini</td>\n",
       "      <td>Mini</td>\n",
       "      <td>Mini</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fiProductClassDesc</th>\n",
       "      <td>Hydraulic Excavator, Track - 3.0 to 4.0 Metric...</td>\n",
       "      <td>Hydraulic Excavator, Track - 3.0 to 4.0 Metric...</td>\n",
       "      <td>Hydraulic Excavator, Track - 3.0 to 4.0 Metric...</td>\n",
       "      <td>Hydraulic Excavator, Track - 2.0 to 3.0 Metric...</td>\n",
       "      <td>Hydraulic Excavator, Track - 2.0 to 3.0 Metric...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>state</th>\n",
       "      <td>Maryland</td>\n",
       "      <td>Maryland</td>\n",
       "      <td>Maryland</td>\n",
       "      <td>Florida</td>\n",
       "      <td>Florida</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ProductGroup</th>\n",
       "      <td>TEX</td>\n",
       "      <td>TEX</td>\n",
       "      <td>TEX</td>\n",
       "      <td>TEX</td>\n",
       "      <td>TEX</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ProductGroupDesc</th>\n",
       "      <td>Track Excavators</td>\n",
       "      <td>Track Excavators</td>\n",
       "      <td>Track Excavators</td>\n",
       "      <td>Track Excavators</td>\n",
       "      <td>Track Excavators</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Drive_System</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Enclosure</th>\n",
       "      <td>EROPS</td>\n",
       "      <td>EROPS</td>\n",
       "      <td>EROPS</td>\n",
       "      <td>EROPS</td>\n",
       "      <td>EROPS</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Forks</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Pad_Type</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ride_Control</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Stick</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Transmission</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Turbocharged</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Blade_Extension</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Blade_Width</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Enclosure_Type</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Engine_Horsepower</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Hydraulics</th>\n",
       "      <td>Auxiliary</td>\n",
       "      <td>Standard</td>\n",
       "      <td>Auxiliary</td>\n",
       "      <td>Standard</td>\n",
       "      <td>Standard</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Pushblock</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ripper</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Scarifier</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Tip_Control</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Tire_Size</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Coupler</th>\n",
       "      <td>None or Unspecified</td>\n",
       "      <td>None or Unspecified</td>\n",
       "      <td>None or Unspecified</td>\n",
       "      <td>None or Unspecified</td>\n",
       "      <td>None or Unspecified</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Coupler_System</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Grouser_Tracks</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Hydraulics_Flow</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Track_Type</th>\n",
       "      <td>Steel</td>\n",
       "      <td>Steel</td>\n",
       "      <td>Steel</td>\n",
       "      <td>Steel</td>\n",
       "      <td>Steel</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Undercarriage_Pad_Width</th>\n",
       "      <td>None or Unspecified</td>\n",
       "      <td>None or Unspecified</td>\n",
       "      <td>None or Unspecified</td>\n",
       "      <td>None or Unspecified</td>\n",
       "      <td>None or Unspecified</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Stick_Length</th>\n",
       "      <td>None or Unspecified</td>\n",
       "      <td>None or Unspecified</td>\n",
       "      <td>None or Unspecified</td>\n",
       "      <td>None or Unspecified</td>\n",
       "      <td>None or Unspecified</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Thumb</th>\n",
       "      <td>None or Unspecified</td>\n",
       "      <td>None or Unspecified</td>\n",
       "      <td>None or Unspecified</td>\n",
       "      <td>None or Unspecified</td>\n",
       "      <td>None or Unspecified</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Pattern_Changer</th>\n",
       "      <td>None or Unspecified</td>\n",
       "      <td>None or Unspecified</td>\n",
       "      <td>None or Unspecified</td>\n",
       "      <td>None or Unspecified</td>\n",
       "      <td>None or Unspecified</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Grouser_Type</th>\n",
       "      <td>Double</td>\n",
       "      <td>Double</td>\n",
       "      <td>Double</td>\n",
       "      <td>Double</td>\n",
       "      <td>Double</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Backhoe_Mounting</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Blade_Type</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Travel_Controls</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Differential_Type</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Steering_Controls</th>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                     401120  \\\n",
       "SalesID                                                             6333336   \n",
       "SalePrice                                                             10500   \n",
       "MachineID                                                           1840702   \n",
       "ModelID                                                               21439   \n",
       "datasource                                                              149   \n",
       "auctioneerID                                                              1   \n",
       "YearMade                                                               2005   \n",
       "MachineHoursCurrentMeter                                                NaN   \n",
       "UsageBand                                                               NaN   \n",
       "saledate                                                2011-11-02 00:00:00   \n",
       "fiModelDesc                                                           35NX2   \n",
       "fiBaseModel                                                              35   \n",
       "fiSecondaryDesc                                                          NX   \n",
       "fiModelSeries                                                             2   \n",
       "fiModelDescriptor                                                       NaN   \n",
       "ProductSize                                                            Mini   \n",
       "fiProductClassDesc        Hydraulic Excavator, Track - 3.0 to 4.0 Metric...   \n",
       "state                                                              Maryland   \n",
       "ProductGroup                                                            TEX   \n",
       "ProductGroupDesc                                           Track Excavators   \n",
       "Drive_System                                                            NaN   \n",
       "Enclosure                                                             EROPS   \n",
       "Forks                                                                   NaN   \n",
       "Pad_Type                                                                NaN   \n",
       "Ride_Control                                                            NaN   \n",
       "Stick                                                                   NaN   \n",
       "Transmission                                                            NaN   \n",
       "Turbocharged                                                            NaN   \n",
       "Blade_Extension                                                         NaN   \n",
       "Blade_Width                                                             NaN   \n",
       "Enclosure_Type                                                          NaN   \n",
       "Engine_Horsepower                                                       NaN   \n",
       "Hydraulics                                                        Auxiliary   \n",
       "Pushblock                                                               NaN   \n",
       "Ripper                                                                  NaN   \n",
       "Scarifier                                                               NaN   \n",
       "Tip_Control                                                             NaN   \n",
       "Tire_Size                                                               NaN   \n",
       "Coupler                                                 None or Unspecified   \n",
       "Coupler_System                                                          NaN   \n",
       "Grouser_Tracks                                                          NaN   \n",
       "Hydraulics_Flow                                                         NaN   \n",
       "Track_Type                                                            Steel   \n",
       "Undercarriage_Pad_Width                                 None or Unspecified   \n",
       "Stick_Length                                            None or Unspecified   \n",
       "Thumb                                                   None or Unspecified   \n",
       "Pattern_Changer                                         None or Unspecified   \n",
       "Grouser_Type                                                         Double   \n",
       "Backhoe_Mounting                                                        NaN   \n",
       "Blade_Type                                                              NaN   \n",
       "Travel_Controls                                                         NaN   \n",
       "Differential_Type                                                       NaN   \n",
       "Steering_Controls                                                       NaN   \n",
       "\n",
       "                                                                     401121  \\\n",
       "SalesID                                                             6333337   \n",
       "SalePrice                                                             11000   \n",
       "MachineID                                                           1830472   \n",
       "ModelID                                                               21439   \n",
       "datasource                                                              149   \n",
       "auctioneerID                                                              1   \n",
       "YearMade                                                               2005   \n",
       "MachineHoursCurrentMeter                                                NaN   \n",
       "UsageBand                                                               NaN   \n",
       "saledate                                                2011-11-02 00:00:00   \n",
       "fiModelDesc                                                           35NX2   \n",
       "fiBaseModel                                                              35   \n",
       "fiSecondaryDesc                                                          NX   \n",
       "fiModelSeries                                                             2   \n",
       "fiModelDescriptor                                                       NaN   \n",
       "ProductSize                                                            Mini   \n",
       "fiProductClassDesc        Hydraulic Excavator, Track - 3.0 to 4.0 Metric...   \n",
       "state                                                              Maryland   \n",
       "ProductGroup                                                            TEX   \n",
       "ProductGroupDesc                                           Track Excavators   \n",
       "Drive_System                                                            NaN   \n",
       "Enclosure                                                             EROPS   \n",
       "Forks                                                                   NaN   \n",
       "Pad_Type                                                                NaN   \n",
       "Ride_Control                                                            NaN   \n",
       "Stick                                                                   NaN   \n",
       "Transmission                                                            NaN   \n",
       "Turbocharged                                                            NaN   \n",
       "Blade_Extension                                                         NaN   \n",
       "Blade_Width                                                             NaN   \n",
       "Enclosure_Type                                                          NaN   \n",
       "Engine_Horsepower                                                       NaN   \n",
       "Hydraulics                                                         Standard   \n",
       "Pushblock                                                               NaN   \n",
       "Ripper                                                                  NaN   \n",
       "Scarifier                                                               NaN   \n",
       "Tip_Control                                                             NaN   \n",
       "Tire_Size                                                               NaN   \n",
       "Coupler                                                 None or Unspecified   \n",
       "Coupler_System                                                          NaN   \n",
       "Grouser_Tracks                                                          NaN   \n",
       "Hydraulics_Flow                                                         NaN   \n",
       "Track_Type                                                            Steel   \n",
       "Undercarriage_Pad_Width                                 None or Unspecified   \n",
       "Stick_Length                                            None or Unspecified   \n",
       "Thumb                                                   None or Unspecified   \n",
       "Pattern_Changer                                         None or Unspecified   \n",
       "Grouser_Type                                                         Double   \n",
       "Backhoe_Mounting                                                        NaN   \n",
       "Blade_Type                                                              NaN   \n",
       "Travel_Controls                                                         NaN   \n",
       "Differential_Type                                                       NaN   \n",
       "Steering_Controls                                                       NaN   \n",
       "\n",
       "                                                                     401122  \\\n",
       "SalesID                                                             6333338   \n",
       "SalePrice                                                             11500   \n",
       "MachineID                                                           1887659   \n",
       "ModelID                                                               21439   \n",
       "datasource                                                              149   \n",
       "auctioneerID                                                              1   \n",
       "YearMade                                                               2005   \n",
       "MachineHoursCurrentMeter                                                NaN   \n",
       "UsageBand                                                               NaN   \n",
       "saledate                                                2011-11-02 00:00:00   \n",
       "fiModelDesc                                                           35NX2   \n",
       "fiBaseModel                                                              35   \n",
       "fiSecondaryDesc                                                          NX   \n",
       "fiModelSeries                                                             2   \n",
       "fiModelDescriptor                                                       NaN   \n",
       "ProductSize                                                            Mini   \n",
       "fiProductClassDesc        Hydraulic Excavator, Track - 3.0 to 4.0 Metric...   \n",
       "state                                                              Maryland   \n",
       "ProductGroup                                                            TEX   \n",
       "ProductGroupDesc                                           Track Excavators   \n",
       "Drive_System                                                            NaN   \n",
       "Enclosure                                                             EROPS   \n",
       "Forks                                                                   NaN   \n",
       "Pad_Type                                                                NaN   \n",
       "Ride_Control                                                            NaN   \n",
       "Stick                                                                   NaN   \n",
       "Transmission                                                            NaN   \n",
       "Turbocharged                                                            NaN   \n",
       "Blade_Extension                                                         NaN   \n",
       "Blade_Width                                                             NaN   \n",
       "Enclosure_Type                                                          NaN   \n",
       "Engine_Horsepower                                                       NaN   \n",
       "Hydraulics                                                        Auxiliary   \n",
       "Pushblock                                                               NaN   \n",
       "Ripper                                                                  NaN   \n",
       "Scarifier                                                               NaN   \n",
       "Tip_Control                                                             NaN   \n",
       "Tire_Size                                                               NaN   \n",
       "Coupler                                                 None or Unspecified   \n",
       "Coupler_System                                                          NaN   \n",
       "Grouser_Tracks                                                          NaN   \n",
       "Hydraulics_Flow                                                         NaN   \n",
       "Track_Type                                                            Steel   \n",
       "Undercarriage_Pad_Width                                 None or Unspecified   \n",
       "Stick_Length                                            None or Unspecified   \n",
       "Thumb                                                   None or Unspecified   \n",
       "Pattern_Changer                                         None or Unspecified   \n",
       "Grouser_Type                                                         Double   \n",
       "Backhoe_Mounting                                                        NaN   \n",
       "Blade_Type                                                              NaN   \n",
       "Travel_Controls                                                         NaN   \n",
       "Differential_Type                                                       NaN   \n",
       "Steering_Controls                                                       NaN   \n",
       "\n",
       "                                                                     401123  \\\n",
       "SalesID                                                             6333341   \n",
       "SalePrice                                                              9000   \n",
       "MachineID                                                           1903570   \n",
       "ModelID                                                               21435   \n",
       "datasource                                                              149   \n",
       "auctioneerID                                                              2   \n",
       "YearMade                                                               2005   \n",
       "MachineHoursCurrentMeter                                                NaN   \n",
       "UsageBand                                                               NaN   \n",
       "saledate                                                2011-10-25 00:00:00   \n",
       "fiModelDesc                                                            30NX   \n",
       "fiBaseModel                                                              30   \n",
       "fiSecondaryDesc                                                          NX   \n",
       "fiModelSeries                                                           NaN   \n",
       "fiModelDescriptor                                                       NaN   \n",
       "ProductSize                                                            Mini   \n",
       "fiProductClassDesc        Hydraulic Excavator, Track - 2.0 to 3.0 Metric...   \n",
       "state                                                               Florida   \n",
       "ProductGroup                                                            TEX   \n",
       "ProductGroupDesc                                           Track Excavators   \n",
       "Drive_System                                                            NaN   \n",
       "Enclosure                                                             EROPS   \n",
       "Forks                                                                   NaN   \n",
       "Pad_Type                                                                NaN   \n",
       "Ride_Control                                                            NaN   \n",
       "Stick                                                                   NaN   \n",
       "Transmission                                                            NaN   \n",
       "Turbocharged                                                            NaN   \n",
       "Blade_Extension                                                         NaN   \n",
       "Blade_Width                                                             NaN   \n",
       "Enclosure_Type                                                          NaN   \n",
       "Engine_Horsepower                                                       NaN   \n",
       "Hydraulics                                                         Standard   \n",
       "Pushblock                                                               NaN   \n",
       "Ripper                                                                  NaN   \n",
       "Scarifier                                                               NaN   \n",
       "Tip_Control                                                             NaN   \n",
       "Tire_Size                                                               NaN   \n",
       "Coupler                                                 None or Unspecified   \n",
       "Coupler_System                                                          NaN   \n",
       "Grouser_Tracks                                                          NaN   \n",
       "Hydraulics_Flow                                                         NaN   \n",
       "Track_Type                                                            Steel   \n",
       "Undercarriage_Pad_Width                                 None or Unspecified   \n",
       "Stick_Length                                            None or Unspecified   \n",
       "Thumb                                                   None or Unspecified   \n",
       "Pattern_Changer                                         None or Unspecified   \n",
       "Grouser_Type                                                         Double   \n",
       "Backhoe_Mounting                                                        NaN   \n",
       "Blade_Type                                                              NaN   \n",
       "Travel_Controls                                                         NaN   \n",
       "Differential_Type                                                       NaN   \n",
       "Steering_Controls                                                       NaN   \n",
       "\n",
       "                                                                     401124  \n",
       "SalesID                                                             6333342  \n",
       "SalePrice                                                              7750  \n",
       "MachineID                                                           1926965  \n",
       "ModelID                                                               21435  \n",
       "datasource                                                              149  \n",
       "auctioneerID                                                              2  \n",
       "YearMade                                                               2005  \n",
       "MachineHoursCurrentMeter                                                NaN  \n",
       "UsageBand                                                               NaN  \n",
       "saledate                                                2011-10-25 00:00:00  \n",
       "fiModelDesc                                                            30NX  \n",
       "fiBaseModel                                                              30  \n",
       "fiSecondaryDesc                                                          NX  \n",
       "fiModelSeries                                                           NaN  \n",
       "fiModelDescriptor                                                       NaN  \n",
       "ProductSize                                                            Mini  \n",
       "fiProductClassDesc        Hydraulic Excavator, Track - 2.0 to 3.0 Metric...  \n",
       "state                                                               Florida  \n",
       "ProductGroup                                                            TEX  \n",
       "ProductGroupDesc                                           Track Excavators  \n",
       "Drive_System                                                            NaN  \n",
       "Enclosure                                                             EROPS  \n",
       "Forks                                                                   NaN  \n",
       "Pad_Type                                                                NaN  \n",
       "Ride_Control                                                            NaN  \n",
       "Stick                                                                   NaN  \n",
       "Transmission                                                            NaN  \n",
       "Turbocharged                                                            NaN  \n",
       "Blade_Extension                                                         NaN  \n",
       "Blade_Width                                                             NaN  \n",
       "Enclosure_Type                                                          NaN  \n",
       "Engine_Horsepower                                                       NaN  \n",
       "Hydraulics                                                         Standard  \n",
       "Pushblock                                                               NaN  \n",
       "Ripper                                                                  NaN  \n",
       "Scarifier                                                               NaN  \n",
       "Tip_Control                                                             NaN  \n",
       "Tire_Size                                                               NaN  \n",
       "Coupler                                                 None or Unspecified  \n",
       "Coupler_System                                                          NaN  \n",
       "Grouser_Tracks                                                          NaN  \n",
       "Hydraulics_Flow                                                         NaN  \n",
       "Track_Type                                                            Steel  \n",
       "Undercarriage_Pad_Width                                 None or Unspecified  \n",
       "Stick_Length                                            None or Unspecified  \n",
       "Thumb                                                   None or Unspecified  \n",
       "Pattern_Changer                                         None or Unspecified  \n",
       "Grouser_Type                                                         Double  \n",
       "Backhoe_Mounting                                                        NaN  \n",
       "Blade_Type                                                              NaN  \n",
       "Travel_Controls                                                         NaN  \n",
       "Differential_Type                                                       NaN  \n",
       "Steering_Controls                                                       NaN  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display_all(df_raw.tail().T)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "      <th>unique</th>\n",
       "      <th>top</th>\n",
       "      <th>freq</th>\n",
       "      <th>first</th>\n",
       "      <th>last</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>min</th>\n",
       "      <th>25%</th>\n",
       "      <th>50%</th>\n",
       "      <th>75%</th>\n",
       "      <th>max</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>SalesID</th>\n",
       "      <td>401125</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.91971e+06</td>\n",
       "      <td>909021</td>\n",
       "      <td>1.13925e+06</td>\n",
       "      <td>1.41837e+06</td>\n",
       "      <td>1.63942e+06</td>\n",
       "      <td>2.24271e+06</td>\n",
       "      <td>6.33334e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>SalePrice</th>\n",
       "      <td>401125</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>31099.7</td>\n",
       "      <td>23036.9</td>\n",
       "      <td>4750</td>\n",
       "      <td>14500</td>\n",
       "      <td>24000</td>\n",
       "      <td>40000</td>\n",
       "      <td>142000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MachineID</th>\n",
       "      <td>401125</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.2179e+06</td>\n",
       "      <td>440992</td>\n",
       "      <td>0</td>\n",
       "      <td>1.0887e+06</td>\n",
       "      <td>1.27949e+06</td>\n",
       "      <td>1.46807e+06</td>\n",
       "      <td>2.48633e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ModelID</th>\n",
       "      <td>401125</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6889.7</td>\n",
       "      <td>6221.78</td>\n",
       "      <td>28</td>\n",
       "      <td>3259</td>\n",
       "      <td>4604</td>\n",
       "      <td>8724</td>\n",
       "      <td>37198</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>datasource</th>\n",
       "      <td>401125</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>134.666</td>\n",
       "      <td>8.96224</td>\n",
       "      <td>121</td>\n",
       "      <td>132</td>\n",
       "      <td>132</td>\n",
       "      <td>136</td>\n",
       "      <td>172</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>auctioneerID</th>\n",
       "      <td>380989</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>6.55604</td>\n",
       "      <td>16.9768</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>99</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>YearMade</th>\n",
       "      <td>401125</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1899.16</td>\n",
       "      <td>291.797</td>\n",
       "      <td>1000</td>\n",
       "      <td>1985</td>\n",
       "      <td>1995</td>\n",
       "      <td>2000</td>\n",
       "      <td>2013</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MachineHoursCurrentMeter</th>\n",
       "      <td>142765</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>3457.96</td>\n",
       "      <td>27590.3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3025</td>\n",
       "      <td>2.4833e+06</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>UsageBand</th>\n",
       "      <td>69639</td>\n",
       "      <td>3</td>\n",
       "      <td>Medium</td>\n",
       "      <td>33985</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>saledate</th>\n",
       "      <td>401125</td>\n",
       "      <td>3919</td>\n",
       "      <td>2009-02-16 00:00:00</td>\n",
       "      <td>1932</td>\n",
       "      <td>1989-01-17 00:00:00</td>\n",
       "      <td>2011-12-30 00:00:00</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fiModelDesc</th>\n",
       "      <td>401125</td>\n",
       "      <td>4999</td>\n",
       "      <td>310G</td>\n",
       "      <td>5039</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fiBaseModel</th>\n",
       "      <td>401125</td>\n",
       "      <td>1950</td>\n",
       "      <td>580</td>\n",
       "      <td>19798</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fiSecondaryDesc</th>\n",
       "      <td>263934</td>\n",
       "      <td>175</td>\n",
       "      <td>C</td>\n",
       "      <td>43235</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fiModelSeries</th>\n",
       "      <td>56908</td>\n",
       "      <td>122</td>\n",
       "      <td>II</td>\n",
       "      <td>13202</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fiModelDescriptor</th>\n",
       "      <td>71919</td>\n",
       "      <td>139</td>\n",
       "      <td>L</td>\n",
       "      <td>15875</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ProductSize</th>\n",
       "      <td>190350</td>\n",
       "      <td>6</td>\n",
       "      <td>Medium</td>\n",
       "      <td>62274</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fiProductClassDesc</th>\n",
       "      <td>401125</td>\n",
       "      <td>74</td>\n",
       "      <td>Backhoe Loader - 14.0 to 15.0 Ft Standard Digg...</td>\n",
       "      <td>56166</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>state</th>\n",
       "      <td>401125</td>\n",
       "      <td>53</td>\n",
       "      <td>Florida</td>\n",
       "      <td>63944</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ProductGroup</th>\n",
       "      <td>401125</td>\n",
       "      <td>6</td>\n",
       "      <td>TEX</td>\n",
       "      <td>101167</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>ProductGroupDesc</th>\n",
       "      <td>401125</td>\n",
       "      <td>6</td>\n",
       "      <td>Track Excavators</td>\n",
       "      <td>101167</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Drive_System</th>\n",
       "      <td>104361</td>\n",
       "      <td>4</td>\n",
       "      <td>Two Wheel Drive</td>\n",
       "      <td>46139</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Enclosure</th>\n",
       "      <td>400800</td>\n",
       "      <td>6</td>\n",
       "      <td>OROPS</td>\n",
       "      <td>173932</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Forks</th>\n",
       "      <td>192077</td>\n",
       "      <td>2</td>\n",
       "      <td>None or Unspecified</td>\n",
       "      <td>178300</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Pad_Type</th>\n",
       "      <td>79134</td>\n",
       "      <td>4</td>\n",
       "      <td>None or Unspecified</td>\n",
       "      <td>70614</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ride_Control</th>\n",
       "      <td>148606</td>\n",
       "      <td>3</td>\n",
       "      <td>No</td>\n",
       "      <td>77685</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Stick</th>\n",
       "      <td>79134</td>\n",
       "      <td>2</td>\n",
       "      <td>Standard</td>\n",
       "      <td>48829</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Transmission</th>\n",
       "      <td>183230</td>\n",
       "      <td>8</td>\n",
       "      <td>Standard</td>\n",
       "      <td>140328</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Turbocharged</th>\n",
       "      <td>79134</td>\n",
       "      <td>2</td>\n",
       "      <td>None or Unspecified</td>\n",
       "      <td>75211</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Blade_Extension</th>\n",
       "      <td>25219</td>\n",
       "      <td>2</td>\n",
       "      <td>None or Unspecified</td>\n",
       "      <td>24692</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Blade_Width</th>\n",
       "      <td>25219</td>\n",
       "      <td>6</td>\n",
       "      <td>14'</td>\n",
       "      <td>9615</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Enclosure_Type</th>\n",
       "      <td>25219</td>\n",
       "      <td>3</td>\n",
       "      <td>None or Unspecified</td>\n",
       "      <td>21923</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Engine_Horsepower</th>\n",
       "      <td>25219</td>\n",
       "      <td>2</td>\n",
       "      <td>No</td>\n",
       "      <td>23937</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Hydraulics</th>\n",
       "      <td>320570</td>\n",
       "      <td>12</td>\n",
       "      <td>2 Valve</td>\n",
       "      <td>141404</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Pushblock</th>\n",
       "      <td>25219</td>\n",
       "      <td>2</td>\n",
       "      <td>None or Unspecified</td>\n",
       "      <td>19463</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Ripper</th>\n",
       "      <td>104137</td>\n",
       "      <td>4</td>\n",
       "      <td>None or Unspecified</td>\n",
       "      <td>83452</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Scarifier</th>\n",
       "      <td>25230</td>\n",
       "      <td>2</td>\n",
       "      <td>None or Unspecified</td>\n",
       "      <td>12719</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Tip_Control</th>\n",
       "      <td>25219</td>\n",
       "      <td>3</td>\n",
       "      <td>None or Unspecified</td>\n",
       "      <td>16207</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Tire_Size</th>\n",
       "      <td>94718</td>\n",
       "      <td>17</td>\n",
       "      <td>None or Unspecified</td>\n",
       "      <td>46339</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Coupler</th>\n",
       "      <td>213952</td>\n",
       "      <td>3</td>\n",
       "      <td>None or Unspecified</td>\n",
       "      <td>184582</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Coupler_System</th>\n",
       "      <td>43458</td>\n",
       "      <td>2</td>\n",
       "      <td>None or Unspecified</td>\n",
       "      <td>40430</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Grouser_Tracks</th>\n",
       "      <td>43362</td>\n",
       "      <td>2</td>\n",
       "      <td>None or Unspecified</td>\n",
       "      <td>40515</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Hydraulics_Flow</th>\n",
       "      <td>43362</td>\n",
       "      <td>3</td>\n",
       "      <td>Standard</td>\n",
       "      <td>42784</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Track_Type</th>\n",
       "      <td>99153</td>\n",
       "      <td>2</td>\n",
       "      <td>Steel</td>\n",
       "      <td>84880</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Undercarriage_Pad_Width</th>\n",
       "      <td>99872</td>\n",
       "      <td>19</td>\n",
       "      <td>None or Unspecified</td>\n",
       "      <td>79651</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Stick_Length</th>\n",
       "      <td>99218</td>\n",
       "      <td>29</td>\n",
       "      <td>None or Unspecified</td>\n",
       "      <td>78820</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Thumb</th>\n",
       "      <td>99288</td>\n",
       "      <td>3</td>\n",
       "      <td>None or Unspecified</td>\n",
       "      <td>83093</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Pattern_Changer</th>\n",
       "      <td>99218</td>\n",
       "      <td>3</td>\n",
       "      <td>None or Unspecified</td>\n",
       "      <td>90255</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Grouser_Type</th>\n",
       "      <td>99153</td>\n",
       "      <td>3</td>\n",
       "      <td>Double</td>\n",
       "      <td>84653</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Backhoe_Mounting</th>\n",
       "      <td>78672</td>\n",
       "      <td>2</td>\n",
       "      <td>None or Unspecified</td>\n",
       "      <td>78652</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Blade_Type</th>\n",
       "      <td>79833</td>\n",
       "      <td>10</td>\n",
       "      <td>PAT</td>\n",
       "      <td>38612</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Travel_Controls</th>\n",
       "      <td>79834</td>\n",
       "      <td>7</td>\n",
       "      <td>None or Unspecified</td>\n",
       "      <td>69923</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Differential_Type</th>\n",
       "      <td>69411</td>\n",
       "      <td>4</td>\n",
       "      <td>Standard</td>\n",
       "      <td>68073</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Steering_Controls</th>\n",
       "      <td>69369</td>\n",
       "      <td>5</td>\n",
       "      <td>Conventional</td>\n",
       "      <td>68679</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                           count unique  \\\n",
       "SalesID                   401125    NaN   \n",
       "SalePrice                 401125    NaN   \n",
       "MachineID                 401125    NaN   \n",
       "ModelID                   401125    NaN   \n",
       "datasource                401125    NaN   \n",
       "auctioneerID              380989    NaN   \n",
       "YearMade                  401125    NaN   \n",
       "MachineHoursCurrentMeter  142765    NaN   \n",
       "UsageBand                  69639      3   \n",
       "saledate                  401125   3919   \n",
       "fiModelDesc               401125   4999   \n",
       "fiBaseModel               401125   1950   \n",
       "fiSecondaryDesc           263934    175   \n",
       "fiModelSeries              56908    122   \n",
       "fiModelDescriptor          71919    139   \n",
       "ProductSize               190350      6   \n",
       "fiProductClassDesc        401125     74   \n",
       "state                     401125     53   \n",
       "ProductGroup              401125      6   \n",
       "ProductGroupDesc          401125      6   \n",
       "Drive_System              104361      4   \n",
       "Enclosure                 400800      6   \n",
       "Forks                     192077      2   \n",
       "Pad_Type                   79134      4   \n",
       "Ride_Control              148606      3   \n",
       "Stick                      79134      2   \n",
       "Transmission              183230      8   \n",
       "Turbocharged               79134      2   \n",
       "Blade_Extension            25219      2   \n",
       "Blade_Width                25219      6   \n",
       "Enclosure_Type             25219      3   \n",
       "Engine_Horsepower          25219      2   \n",
       "Hydraulics                320570     12   \n",
       "Pushblock                  25219      2   \n",
       "Ripper                    104137      4   \n",
       "Scarifier                  25230      2   \n",
       "Tip_Control                25219      3   \n",
       "Tire_Size                  94718     17   \n",
       "Coupler                   213952      3   \n",
       "Coupler_System             43458      2   \n",
       "Grouser_Tracks             43362      2   \n",
       "Hydraulics_Flow            43362      3   \n",
       "Track_Type                 99153      2   \n",
       "Undercarriage_Pad_Width    99872     19   \n",
       "Stick_Length               99218     29   \n",
       "Thumb                      99288      3   \n",
       "Pattern_Changer            99218      3   \n",
       "Grouser_Type               99153      3   \n",
       "Backhoe_Mounting           78672      2   \n",
       "Blade_Type                 79833     10   \n",
       "Travel_Controls            79834      7   \n",
       "Differential_Type          69411      4   \n",
       "Steering_Controls          69369      5   \n",
       "\n",
       "                                                                        top  \\\n",
       "SalesID                                                                 NaN   \n",
       "SalePrice                                                               NaN   \n",
       "MachineID                                                               NaN   \n",
       "ModelID                                                                 NaN   \n",
       "datasource                                                              NaN   \n",
       "auctioneerID                                                            NaN   \n",
       "YearMade                                                                NaN   \n",
       "MachineHoursCurrentMeter                                                NaN   \n",
       "UsageBand                                                            Medium   \n",
       "saledate                                                2009-02-16 00:00:00   \n",
       "fiModelDesc                                                            310G   \n",
       "fiBaseModel                                                             580   \n",
       "fiSecondaryDesc                                                           C   \n",
       "fiModelSeries                                                            II   \n",
       "fiModelDescriptor                                                         L   \n",
       "ProductSize                                                          Medium   \n",
       "fiProductClassDesc        Backhoe Loader - 14.0 to 15.0 Ft Standard Digg...   \n",
       "state                                                               Florida   \n",
       "ProductGroup                                                            TEX   \n",
       "ProductGroupDesc                                           Track Excavators   \n",
       "Drive_System                                                Two Wheel Drive   \n",
       "Enclosure                                                             OROPS   \n",
       "Forks                                                   None or Unspecified   \n",
       "Pad_Type                                                None or Unspecified   \n",
       "Ride_Control                                                             No   \n",
       "Stick                                                              Standard   \n",
       "Transmission                                                       Standard   \n",
       "Turbocharged                                            None or Unspecified   \n",
       "Blade_Extension                                         None or Unspecified   \n",
       "Blade_Width                                                             14'   \n",
       "Enclosure_Type                                          None or Unspecified   \n",
       "Engine_Horsepower                                                        No   \n",
       "Hydraulics                                                          2 Valve   \n",
       "Pushblock                                               None or Unspecified   \n",
       "Ripper                                                  None or Unspecified   \n",
       "Scarifier                                               None or Unspecified   \n",
       "Tip_Control                                             None or Unspecified   \n",
       "Tire_Size                                               None or Unspecified   \n",
       "Coupler                                                 None or Unspecified   \n",
       "Coupler_System                                          None or Unspecified   \n",
       "Grouser_Tracks                                          None or Unspecified   \n",
       "Hydraulics_Flow                                                    Standard   \n",
       "Track_Type                                                            Steel   \n",
       "Undercarriage_Pad_Width                                 None or Unspecified   \n",
       "Stick_Length                                            None or Unspecified   \n",
       "Thumb                                                   None or Unspecified   \n",
       "Pattern_Changer                                         None or Unspecified   \n",
       "Grouser_Type                                                         Double   \n",
       "Backhoe_Mounting                                        None or Unspecified   \n",
       "Blade_Type                                                              PAT   \n",
       "Travel_Controls                                         None or Unspecified   \n",
       "Differential_Type                                                  Standard   \n",
       "Steering_Controls                                              Conventional   \n",
       "\n",
       "                            freq                first                 last  \\\n",
       "SalesID                      NaN                  NaN                  NaN   \n",
       "SalePrice                    NaN                  NaN                  NaN   \n",
       "MachineID                    NaN                  NaN                  NaN   \n",
       "ModelID                      NaN                  NaN                  NaN   \n",
       "datasource                   NaN                  NaN                  NaN   \n",
       "auctioneerID                 NaN                  NaN                  NaN   \n",
       "YearMade                     NaN                  NaN                  NaN   \n",
       "MachineHoursCurrentMeter     NaN                  NaN                  NaN   \n",
       "UsageBand                  33985                  NaN                  NaN   \n",
       "saledate                    1932  1989-01-17 00:00:00  2011-12-30 00:00:00   \n",
       "fiModelDesc                 5039                  NaN                  NaN   \n",
       "fiBaseModel                19798                  NaN                  NaN   \n",
       "fiSecondaryDesc            43235                  NaN                  NaN   \n",
       "fiModelSeries              13202                  NaN                  NaN   \n",
       "fiModelDescriptor          15875                  NaN                  NaN   \n",
       "ProductSize                62274                  NaN                  NaN   \n",
       "fiProductClassDesc         56166                  NaN                  NaN   \n",
       "state                      63944                  NaN                  NaN   \n",
       "ProductGroup              101167                  NaN                  NaN   \n",
       "ProductGroupDesc          101167                  NaN                  NaN   \n",
       "Drive_System               46139                  NaN                  NaN   \n",
       "Enclosure                 173932                  NaN                  NaN   \n",
       "Forks                     178300                  NaN                  NaN   \n",
       "Pad_Type                   70614                  NaN                  NaN   \n",
       "Ride_Control               77685                  NaN                  NaN   \n",
       "Stick                      48829                  NaN                  NaN   \n",
       "Transmission              140328                  NaN                  NaN   \n",
       "Turbocharged               75211                  NaN                  NaN   \n",
       "Blade_Extension            24692                  NaN                  NaN   \n",
       "Blade_Width                 9615                  NaN                  NaN   \n",
       "Enclosure_Type             21923                  NaN                  NaN   \n",
       "Engine_Horsepower          23937                  NaN                  NaN   \n",
       "Hydraulics                141404                  NaN                  NaN   \n",
       "Pushblock                  19463                  NaN                  NaN   \n",
       "Ripper                     83452                  NaN                  NaN   \n",
       "Scarifier                  12719                  NaN                  NaN   \n",
       "Tip_Control                16207                  NaN                  NaN   \n",
       "Tire_Size                  46339                  NaN                  NaN   \n",
       "Coupler                   184582                  NaN                  NaN   \n",
       "Coupler_System             40430                  NaN                  NaN   \n",
       "Grouser_Tracks             40515                  NaN                  NaN   \n",
       "Hydraulics_Flow            42784                  NaN                  NaN   \n",
       "Track_Type                 84880                  NaN                  NaN   \n",
       "Undercarriage_Pad_Width    79651                  NaN                  NaN   \n",
       "Stick_Length               78820                  NaN                  NaN   \n",
       "Thumb                      83093                  NaN                  NaN   \n",
       "Pattern_Changer            90255                  NaN                  NaN   \n",
       "Grouser_Type               84653                  NaN                  NaN   \n",
       "Backhoe_Mounting           78652                  NaN                  NaN   \n",
       "Blade_Type                 38612                  NaN                  NaN   \n",
       "Travel_Controls            69923                  NaN                  NaN   \n",
       "Differential_Type          68073                  NaN                  NaN   \n",
       "Steering_Controls          68679                  NaN                  NaN   \n",
       "\n",
       "                                 mean      std          min          25%  \\\n",
       "SalesID                   1.91971e+06   909021  1.13925e+06  1.41837e+06   \n",
       "SalePrice                     31099.7  23036.9         4750        14500   \n",
       "MachineID                  1.2179e+06   440992            0   1.0887e+06   \n",
       "ModelID                        6889.7  6221.78           28         3259   \n",
       "datasource                    134.666  8.96224          121          132   \n",
       "auctioneerID                  6.55604  16.9768            0            1   \n",
       "YearMade                      1899.16  291.797         1000         1985   \n",
       "MachineHoursCurrentMeter      3457.96  27590.3            0            0   \n",
       "UsageBand                         NaN      NaN          NaN          NaN   \n",
       "saledate                          NaN      NaN          NaN          NaN   \n",
       "fiModelDesc                       NaN      NaN          NaN          NaN   \n",
       "fiBaseModel                       NaN      NaN          NaN          NaN   \n",
       "fiSecondaryDesc                   NaN      NaN          NaN          NaN   \n",
       "fiModelSeries                     NaN      NaN          NaN          NaN   \n",
       "fiModelDescriptor                 NaN      NaN          NaN          NaN   \n",
       "ProductSize                       NaN      NaN          NaN          NaN   \n",
       "fiProductClassDesc                NaN      NaN          NaN          NaN   \n",
       "state                             NaN      NaN          NaN          NaN   \n",
       "ProductGroup                      NaN      NaN          NaN          NaN   \n",
       "ProductGroupDesc                  NaN      NaN          NaN          NaN   \n",
       "Drive_System                      NaN      NaN          NaN          NaN   \n",
       "Enclosure                         NaN      NaN          NaN          NaN   \n",
       "Forks                             NaN      NaN          NaN          NaN   \n",
       "Pad_Type                          NaN      NaN          NaN          NaN   \n",
       "Ride_Control                      NaN      NaN          NaN          NaN   \n",
       "Stick                             NaN      NaN          NaN          NaN   \n",
       "Transmission                      NaN      NaN          NaN          NaN   \n",
       "Turbocharged                      NaN      NaN          NaN          NaN   \n",
       "Blade_Extension                   NaN      NaN          NaN          NaN   \n",
       "Blade_Width                       NaN      NaN          NaN          NaN   \n",
       "Enclosure_Type                    NaN      NaN          NaN          NaN   \n",
       "Engine_Horsepower                 NaN      NaN          NaN          NaN   \n",
       "Hydraulics                        NaN      NaN          NaN          NaN   \n",
       "Pushblock                         NaN      NaN          NaN          NaN   \n",
       "Ripper                            NaN      NaN          NaN          NaN   \n",
       "Scarifier                         NaN      NaN          NaN          NaN   \n",
       "Tip_Control                       NaN      NaN          NaN          NaN   \n",
       "Tire_Size                         NaN      NaN          NaN          NaN   \n",
       "Coupler                           NaN      NaN          NaN          NaN   \n",
       "Coupler_System                    NaN      NaN          NaN          NaN   \n",
       "Grouser_Tracks                    NaN      NaN          NaN          NaN   \n",
       "Hydraulics_Flow                   NaN      NaN          NaN          NaN   \n",
       "Track_Type                        NaN      NaN          NaN          NaN   \n",
       "Undercarriage_Pad_Width           NaN      NaN          NaN          NaN   \n",
       "Stick_Length                      NaN      NaN          NaN          NaN   \n",
       "Thumb                             NaN      NaN          NaN          NaN   \n",
       "Pattern_Changer                   NaN      NaN          NaN          NaN   \n",
       "Grouser_Type                      NaN      NaN          NaN          NaN   \n",
       "Backhoe_Mounting                  NaN      NaN          NaN          NaN   \n",
       "Blade_Type                        NaN      NaN          NaN          NaN   \n",
       "Travel_Controls                   NaN      NaN          NaN          NaN   \n",
       "Differential_Type                 NaN      NaN          NaN          NaN   \n",
       "Steering_Controls                 NaN      NaN          NaN          NaN   \n",
       "\n",
       "                                  50%          75%          max  \n",
       "SalesID                   1.63942e+06  2.24271e+06  6.33334e+06  \n",
       "SalePrice                       24000        40000       142000  \n",
       "MachineID                 1.27949e+06  1.46807e+06  2.48633e+06  \n",
       "ModelID                          4604         8724        37198  \n",
       "datasource                        132          136          172  \n",
       "auctioneerID                        2            4           99  \n",
       "YearMade                         1995         2000         2013  \n",
       "MachineHoursCurrentMeter            0         3025   2.4833e+06  \n",
       "UsageBand                         NaN          NaN          NaN  \n",
       "saledate                          NaN          NaN          NaN  \n",
       "fiModelDesc                       NaN          NaN          NaN  \n",
       "fiBaseModel                       NaN          NaN          NaN  \n",
       "fiSecondaryDesc                   NaN          NaN          NaN  \n",
       "fiModelSeries                     NaN          NaN          NaN  \n",
       "fiModelDescriptor                 NaN          NaN          NaN  \n",
       "ProductSize                       NaN          NaN          NaN  \n",
       "fiProductClassDesc                NaN          NaN          NaN  \n",
       "state                             NaN          NaN          NaN  \n",
       "ProductGroup                      NaN          NaN          NaN  \n",
       "ProductGroupDesc                  NaN          NaN          NaN  \n",
       "Drive_System                      NaN          NaN          NaN  \n",
       "Enclosure                         NaN          NaN          NaN  \n",
       "Forks                             NaN          NaN          NaN  \n",
       "Pad_Type                          NaN          NaN          NaN  \n",
       "Ride_Control                      NaN          NaN          NaN  \n",
       "Stick                             NaN          NaN          NaN  \n",
       "Transmission                      NaN          NaN          NaN  \n",
       "Turbocharged                      NaN          NaN          NaN  \n",
       "Blade_Extension                   NaN          NaN          NaN  \n",
       "Blade_Width                       NaN          NaN          NaN  \n",
       "Enclosure_Type                    NaN          NaN          NaN  \n",
       "Engine_Horsepower                 NaN          NaN          NaN  \n",
       "Hydraulics                        NaN          NaN          NaN  \n",
       "Pushblock                         NaN          NaN          NaN  \n",
       "Ripper                            NaN          NaN          NaN  \n",
       "Scarifier                         NaN          NaN          NaN  \n",
       "Tip_Control                       NaN          NaN          NaN  \n",
       "Tire_Size                         NaN          NaN          NaN  \n",
       "Coupler                           NaN          NaN          NaN  \n",
       "Coupler_System                    NaN          NaN          NaN  \n",
       "Grouser_Tracks                    NaN          NaN          NaN  \n",
       "Hydraulics_Flow                   NaN          NaN          NaN  \n",
       "Track_Type                        NaN          NaN          NaN  \n",
       "Undercarriage_Pad_Width           NaN          NaN          NaN  \n",
       "Stick_Length                      NaN          NaN          NaN  \n",
       "Thumb                             NaN          NaN          NaN  \n",
       "Pattern_Changer                   NaN          NaN          NaN  \n",
       "Grouser_Type                      NaN          NaN          NaN  \n",
       "Backhoe_Mounting                  NaN          NaN          NaN  \n",
       "Blade_Type                        NaN          NaN          NaN  \n",
       "Travel_Controls                   NaN          NaN          NaN  \n",
       "Differential_Type                 NaN          NaN          NaN  \n",
       "Steering_Controls                 NaN          NaN          NaN  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display_all(df_raw.describe(include='all').T) # returns all the summary statistics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "It's important to note what metric is being used for a project. Generally, selecting the metric(s) is an important part of the project setup. However, in this case Kaggle tells us what metric to use: RMSLE (root mean squared log error) between the actual and predicted auction prices. Therefore we take the log of the prices, so that RMSE will give us what we need."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "df_raw.SalePrice = np.log(df_raw.SalePrice)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Initial processing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "ename": "ValueError",
     "evalue": "could not convert string to float: 'Conventional'",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mValueError\u001b[0m                                Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-10-6e70335c9573>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0mm\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mRandomForestRegressor\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mn_jobs\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0;31m# The following code is supposed to fail due to string values in the input data\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 3\u001b[0;31m \u001b[0mm\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdf_raw\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdrop\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'SalePrice'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0maxis\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdf_raw\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mSalePrice\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/anaconda3/lib/python3.6/site-packages/sklearn/ensemble/forest.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, X, y, sample_weight)\u001b[0m\n\u001b[1;32m    245\u001b[0m         \"\"\"\n\u001b[1;32m    246\u001b[0m         \u001b[0;31m# Validate or convert input data\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 247\u001b[0;31m         \u001b[0mX\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcheck_array\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mX\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0maccept_sparse\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m\"csc\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mDTYPE\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    248\u001b[0m         \u001b[0my\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mcheck_array\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0my\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0maccept_sparse\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'csc'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mensure_2d\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    249\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0msample_weight\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/anaconda3/lib/python3.6/site-packages/sklearn/utils/validation.py\u001b[0m in \u001b[0;36mcheck_array\u001b[0;34m(array, accept_sparse, dtype, order, copy, force_all_finite, ensure_2d, allow_nd, ensure_min_samples, ensure_min_features, warn_on_dtype, estimator)\u001b[0m\n\u001b[1;32m    431\u001b[0m                                       force_all_finite)\n\u001b[1;32m    432\u001b[0m     \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 433\u001b[0;31m         \u001b[0marray\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mnp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0marray\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0marray\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdtype\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mdtype\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0morder\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0morder\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcopy\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mcopy\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    434\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    435\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mensure_2d\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mValueError\u001b[0m: could not convert string to float: 'Conventional'"
     ]
    }
   ],
   "source": [
    "m = RandomForestRegressor(n_jobs=-1)\n",
    "# The following code is supposed to fail due to string values in the input data\n",
    "m.fit(df_raw.drop('SalePrice', axis=1), df_raw.SalePrice)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "This dataset contains a mix of **continuous** and **categorical** variables.\n",
    "\n",
    "The following method extracts particular date fields from a complete datetime for the purpose of constructing categoricals.  You should always consider this feature extraction step when working with date-time. Without expanding your date-time into these additional fields, you can't capture any trend/cyclical behavior as a function of time at any of these granularities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    2006\n",
       "1    2004\n",
       "2    2004\n",
       "3    2011\n",
       "4    2009\n",
       "Name: saleYear, dtype: int64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "add_datepart(df_raw, 'saledate') # getting separate attributes of the date columns and add each new attribute as a new \n",
    "# column\n",
    "df_raw.saleYear.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "The categorical variables are currently stored as strings, which is inefficient, and doesn't provide the numeric coding required for a random forest. Therefore we call `train_cats` to convert strings to pandas categories."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "train_cats(df_raw) # converting all the string variables to categorical variables so that python can autogenerate the \n",
    "# numeric codes for all all; converting a column from object to categorical variables will have several advantages such \n",
    "# as having all the categorical variable attributes i.e. categories, codes etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "We can specify the order to use for categorical variables if we wish:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['High', 'Low', 'Medium'], dtype='object')"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_raw.UsageBand.cat.categories"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "df_raw.UsageBand.cat.set_categories(['High', 'Medium', 'Low'], ordered=True, inplace=True) # Random Forest can process \n",
    "# categories as is but it will take longer to arrive at an optimum solution if the categories aren't hierarchically ordered\n",
    "# to help the model arrive at an optimum solution we are ordering the categories of this variable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Normally, pandas will continue displaying the text categories, while treating them as numerical data internally. Optionally, we can replace the text categories with numbers, which will make this variable non-categorical, like so:."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "df_raw.UsageBand = df_raw.UsageBand.cat.codes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "We're still not quite done - for instance we have lots of missing values, which we can't pass directly to a random forest."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "hidden": true,
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Backhoe_Mounting            0.803872\n",
       "Blade_Extension             0.937129\n",
       "Blade_Type                  0.800977\n",
       "Blade_Width                 0.937129\n",
       "Coupler                     0.466620\n",
       "Coupler_System              0.891660\n",
       "Differential_Type           0.826959\n",
       "Drive_System                0.739829\n",
       "Enclosure                   0.000810\n",
       "Enclosure_Type              0.937129\n",
       "Engine_Horsepower           0.937129\n",
       "Forks                       0.521154\n",
       "Grouser_Tracks              0.891899\n",
       "Grouser_Type                0.752813\n",
       "Hydraulics                  0.200823\n",
       "Hydraulics_Flow             0.891899\n",
       "MachineHoursCurrentMeter    0.644089\n",
       "MachineID                   0.000000\n",
       "ModelID                     0.000000\n",
       "Pad_Type                    0.802720\n",
       "Pattern_Changer             0.752651\n",
       "ProductGroup                0.000000\n",
       "ProductGroupDesc            0.000000\n",
       "ProductSize                 0.525460\n",
       "Pushblock                   0.937129\n",
       "Ride_Control                0.629527\n",
       "Ripper                      0.740388\n",
       "SalePrice                   0.000000\n",
       "SalesID                     0.000000\n",
       "Scarifier                   0.937102\n",
       "Steering_Controls           0.827064\n",
       "Stick                       0.802720\n",
       "Stick_Length                0.752651\n",
       "Thumb                       0.752476\n",
       "Tip_Control                 0.937129\n",
       "Tire_Size                   0.763869\n",
       "Track_Type                  0.752813\n",
       "Transmission                0.543210\n",
       "Travel_Controls             0.800975\n",
       "Turbocharged                0.802720\n",
       "Undercarriage_Pad_Width     0.751020\n",
       "UsageBand                   0.000000\n",
       "YearMade                    0.000000\n",
       "auctioneerID                0.050199\n",
       "datasource                  0.000000\n",
       "fiBaseModel                 0.000000\n",
       "fiModelDesc                 0.000000\n",
       "fiModelDescriptor           0.820707\n",
       "fiModelSeries               0.858129\n",
       "fiProductClassDesc          0.000000\n",
       "fiSecondaryDesc             0.342016\n",
       "saleDay                     0.000000\n",
       "saleDayofweek               0.000000\n",
       "saleDayofyear               0.000000\n",
       "saleElapsed                 0.000000\n",
       "saleIs_month_end            0.000000\n",
       "saleIs_month_start          0.000000\n",
       "saleIs_quarter_end          0.000000\n",
       "saleIs_quarter_start        0.000000\n",
       "saleIs_year_end             0.000000\n",
       "saleIs_year_start           0.000000\n",
       "saleMonth                   0.000000\n",
       "saleWeek                    0.000000\n",
       "saleYear                    0.000000\n",
       "state                       0.000000\n",
       "dtype: float64"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "display_all(df_raw.isnull().sum().sort_index()/len(df_raw))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "But let's save this file for now, since it's already in format can we be stored and accessed efficiently."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "hidden": true,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "os.makedirs(f'{PATH}/Converted Data', exist_ok=True) # creates a new directory to store the \n",
    "# raw file\n",
    "df_raw.to_feather(f'{PATH}/Converted Data/bulldozers-raw') # saving the df in a different \n",
    "# format so that we can read in large datasets within seconds; this format perserves all the processing on the dataset "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Pre-processing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "In the future we can simply read it from this fast format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "df_raw = pd.read_feather(f'{PATH}/Converted Data/bulldozers-raw') "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "We'll replace categories with their numeric codes, handle missing continuous values, and split the dependent variable into a separate variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "df, y, nas = proc_df(df_raw, 'SalePrice') # proc_df does all the pre-processing needed to run the model i.e. replacing \n",
    "# all the categorical variables with their numeric codes etc. ; nas returns a dictionary with all the variable names \n",
    "# along with the median value that's used to replace the missing values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "We now have something we can pass to a random forest!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "e:\\Anaconda\\envs\\fastai-cpu\\lib\\site-packages\\sklearn\\ensemble\\forest.py:245: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "0.9829543709606978"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "m = RandomForestRegressor(n_jobs=-1)\n",
    "m.fit(df, y)\n",
    "m.score(df,y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "In statistics, the coefficient of determination, denoted R2 or r2 and pronounced \"R squared\", is the proportion of the variance in the dependent variable that is predictable from the independent variable(s). https://en.wikipedia.org/wiki/Coefficient_of_determination"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Wow, an r^2 of 0.98 - that's great, right? Well, perhaps not...\n",
    "\n",
    "Possibly **the most important idea** in machine learning is that of having separate training & validation data sets. As motivation, suppose you don't divide up your data, but instead use all of it.  And suppose you have lots of parameters:\n",
    "\n",
    "<img src=\"images/overfitting2.png\" alt=\"\" style=\"width: 70%\"/>\n",
    "<center>\n",
    "[Underfitting and Overfitting](https://datascience.stackexchange.com/questions/361/when-is-a-model-underfitted)\n",
    "</center>\n",
    "\n",
    "The error for the pictured data points is lowest for the model on the far right (the blue curve passes through the red points almost perfectly), yet it's not the best choice.  Why is that?  If you were to gather some new data points, they most likely would not be on that curve in the graph on the right, but would be closer to the curve in the middle graph.\n",
    "\n",
    "This illustrates how using all our data can lead to **overfitting**. A validation set helps diagnose this problem."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((389125, 66), (389125,), (12000, 66))"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def split_vals(a,n): return a[:n].copy(), a[n:].copy()\n",
    "\n",
    "n_valid = 12000  # same as Kaggle's test set size\n",
    "n_trn = len(df)-n_valid\n",
    "raw_train, raw_valid = split_vals(df_raw, n_trn)\n",
    "X_train, X_valid = split_vals(df, n_trn)\n",
    "y_train, y_valid = split_vals(y, n_trn)\n",
    "\n",
    "X_train.shape, y_train.shape, X_valid.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true
   },
   "source": [
    "# Random Forests"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## Base model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Let's try our model again, this time with separate training and validation sets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "def rmse(x,y): return math.sqrt(((x-y)**2).mean())\n",
    "\n",
    "def print_score(m):\n",
    "    res = [rmse(m.predict(X_train), y_train), rmse(m.predict(X_valid), y_valid),\n",
    "                m.score(X_train, y_train), m.score(X_valid, y_valid)]\n",
    "    if hasattr(m, 'oob_score_'): res.append(m.oob_score_)\n",
    "    print(res)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "e:\\Anaconda\\envs\\fastai-cpu\\lib\\site-packages\\sklearn\\ensemble\\forest.py:245: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 57.3 s\n",
      "[0.09016927368168517, 0.24921806567413737, 0.983007707748041, 0.8890807266562013]\n"
     ]
    }
   ],
   "source": [
    "m = RandomForestRegressor(n_jobs=-1)\n",
    "%time m.fit(X_train, y_train) # the wall time is the time the entire program takes to complete\n",
    "print_score(m)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "An r^2 in the high-80's isn't bad at all (and the RMSLE puts us around rank 100 of 470 on the Kaggle leaderboard), but we can see from the validation set score that we're over-fitting badly. To understand this issue, let's simplify things down to a single small tree."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## Speeding things up"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "df_trn, y_trn, nas = proc_df(df_raw, 'SalePrice', subset=30000, na_dict=nas)\n",
    "X_train, _ = split_vals(df_trn, 20000)\n",
    "y_train, _ = split_vals(y_trn, 20000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "e:\\Anaconda\\envs\\fastai-cpu\\lib\\site-packages\\sklearn\\ensemble\\forest.py:245: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 2.36 s\n",
      "[0.10992584854191216, 0.34044699906247267, 0.9739348240099561, 0.7930111294913496]\n"
     ]
    }
   ],
   "source": [
    "m = RandomForestRegressor(n_jobs=-1)\n",
    "%time m.fit(X_train, y_train)\n",
    "print_score(m)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## Single tree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.5291790046376069, 0.5784714792087692, 0.39595927758135874, 0.4023982274694978]\n"
     ]
    }
   ],
   "source": [
    "m = RandomForestRegressor(n_estimators=1, max_depth=3, bootstrap=False, n_jobs=-1)\n",
    "m.fit(X_train, y_train)\n",
    "print_score(m)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\r\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\r\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\r\n",
       "<!-- Generated by graphviz version 2.38.0 (20140413.2041)\r\n",
       " -->\r\n",
       "<!-- Title: Tree Pages: 1 -->\r\n",
       "<svg width=\"720pt\" height=\"434pt\"\r\n",
       " viewBox=\"0.00 0.00 720.00 434.49\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\r\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(0.778659 0.778659) rotate(0) translate(4 554)\">\r\n",
       "<title>Tree</title>\r\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-554 920.667,-554 920.667,4 -4,4\"/>\r\n",
       "<!-- 0 -->\r\n",
       "<g id=\"node1\" class=\"node\"><title>0</title>\r\n",
       "<polygon fill=\"#eca470\" stroke=\"black\" points=\"170.167,-308.5 29.1667,-308.5 29.1667,-240.5 170.167,-240.5 170.167,-308.5\"/>\r\n",
       "<text text-anchor=\"start\" x=\"37.1667\" y=\"-293.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">Grouser_Tracks ≤ 0.5</text>\r\n",
       "<text text-anchor=\"start\" x=\"64.6667\" y=\"-278.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">mse = 0.464</text>\r\n",
       "<text text-anchor=\"start\" x=\"52.1667\" y=\"-263.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">samples = 20000</text>\r\n",
       "<text text-anchor=\"start\" x=\"58.6667\" y=\"-248.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">value = 10.115</text>\r\n",
       "</g>\r\n",
       "<!-- 1 -->\r\n",
       "<g id=\"node2\" class=\"node\"><title>1</title>\r\n",
       "<polygon fill=\"#eb9d65\" stroke=\"black\" points=\"404.167,-361.5 277.167,-361.5 277.167,-293.5 404.167,-293.5 404.167,-361.5\"/>\r\n",
       "<text text-anchor=\"start\" x=\"285.167\" y=\"-346.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">YearMade ≤ 1990.5</text>\r\n",
       "<text text-anchor=\"start\" x=\"305.667\" y=\"-331.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">mse = 0.405</text>\r\n",
       "<text text-anchor=\"start\" x=\"293.167\" y=\"-316.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">samples = 18213</text>\r\n",
       "<text text-anchor=\"start\" x=\"299.667\" y=\"-301.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">value = 10.206</text>\r\n",
       "</g>\r\n",
       "<!-- 0&#45;&gt;1 -->\r\n",
       "<g id=\"edge1\" class=\"edge\"><title>0&#45;&gt;1</title>\r\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M170.438,-289.972C200.898,-296.727 236.604,-304.645 267.348,-311.463\"/>\r\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"266.621,-314.887 277.142,-313.635 268.137,-308.053 266.621,-314.887\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"256.056\" y=\"-323.366\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">True</text>\r\n",
       "</g>\r\n",
       "<!-- 8 -->\r\n",
       "<g id=\"node9\" class=\"node\"><title>8</title>\r\n",
       "<polygon fill=\"#fbede2\" stroke=\"black\" points=\"404.167,-255.5 277.167,-255.5 277.167,-187.5 404.167,-187.5 404.167,-255.5\"/>\r\n",
       "<text text-anchor=\"start\" x=\"285.167\" y=\"-240.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">YearMade ≤ 1997.5</text>\r\n",
       "<text text-anchor=\"start\" x=\"305.667\" y=\"-225.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">mse = 0.108</text>\r\n",
       "<text text-anchor=\"start\" x=\"296.667\" y=\"-210.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">samples = 1787</text>\r\n",
       "<text text-anchor=\"start\" x=\"302.667\" y=\"-195.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">value = 9.183</text>\r\n",
       "</g>\r\n",
       "<!-- 0&#45;&gt;8 -->\r\n",
       "<g id=\"edge8\" class=\"edge\"><title>0&#45;&gt;8</title>\r\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M170.438,-259.028C200.898,-252.273 236.604,-244.355 267.348,-237.537\"/>\r\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"268.137,-240.947 277.142,-235.365 266.621,-234.113 268.137,-240.947\"/>\r\n",
       "<text text-anchor=\"middle\" x=\"256.056\" y=\"-218.234\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">False</text>\r\n",
       "</g>\r\n",
       "<!-- 2 -->\r\n",
       "<g id=\"node3\" class=\"node\"><title>2</title>\r\n",
       "<polygon fill=\"#eeaf81\" stroke=\"black\" points=\"659.167,-486.5 540.167,-486.5 540.167,-418.5 659.167,-418.5 659.167,-486.5\"/>\r\n",
       "<text text-anchor=\"start\" x=\"548.167\" y=\"-471.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">ModelID ≤ 4419.0</text>\r\n",
       "<text text-anchor=\"start\" x=\"564.667\" y=\"-456.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">mse = 0.371</text>\r\n",
       "<text text-anchor=\"start\" x=\"555.667\" y=\"-441.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">samples = 8738</text>\r\n",
       "<text text-anchor=\"start\" x=\"561.667\" y=\"-426.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">value = 9.982</text>\r\n",
       "</g>\r\n",
       "<!-- 1&#45;&gt;2 -->\r\n",
       "<g id=\"edge2\" class=\"edge\"><title>1&#45;&gt;2</title>\r\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M404.207,-357.919C442.447,-376.518 491.378,-400.317 530.59,-419.389\"/>\r\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"529.299,-422.653 539.823,-423.879 532.361,-416.358 529.299,-422.653\"/>\r\n",
       "</g>\r\n",
       "<!-- 5 -->\r\n",
       "<g id=\"node6\" class=\"node\"><title>5</title>\r\n",
       "<polygon fill=\"#e78d4c\" stroke=\"black\" points=\"680.167,-361.5 519.167,-361.5 519.167,-293.5 680.167,-293.5 680.167,-361.5\"/>\r\n",
       "<text text-anchor=\"start\" x=\"527.167\" y=\"-346.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">fiProductClassDesc ≤ 6.5</text>\r\n",
       "<text text-anchor=\"start\" x=\"564.667\" y=\"-331.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">mse = 0.347</text>\r\n",
       "<text text-anchor=\"start\" x=\"555.667\" y=\"-316.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">samples = 9475</text>\r\n",
       "<text text-anchor=\"start\" x=\"558.667\" y=\"-301.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">value = 10.413</text>\r\n",
       "</g>\r\n",
       "<!-- 1&#45;&gt;5 -->\r\n",
       "<g id=\"edge5\" class=\"edge\"><title>1&#45;&gt;5</title>\r\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M404.207,-327.5C435.581,-327.5 474.151,-327.5 508.646,-327.5\"/>\r\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"508.732,-331 518.732,-327.5 508.732,-324 508.732,-331\"/>\r\n",
       "</g>\r\n",
       "<!-- 3 -->\r\n",
       "<g id=\"node4\" class=\"node\"><title>3</title>\r\n",
       "<polygon fill=\"#eb9d65\" stroke=\"black\" points=\"894.667,-550 790.667,-550 790.667,-497 894.667,-497 894.667,-550\"/>\r\n",
       "<text text-anchor=\"start\" x=\"807.667\" y=\"-534.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">mse = 0.381</text>\r\n",
       "<text text-anchor=\"start\" x=\"798.667\" y=\"-519.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">samples = 4544</text>\r\n",
       "<text text-anchor=\"start\" x=\"801.667\" y=\"-504.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">value = 10.206</text>\r\n",
       "</g>\r\n",
       "<!-- 2&#45;&gt;3 -->\r\n",
       "<g id=\"edge3\" class=\"edge\"><title>2&#45;&gt;3</title>\r\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M659.314,-469.778C696.149,-480.63 743.564,-494.598 780.811,-505.572\"/>\r\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"779.957,-508.969 790.538,-508.437 781.935,-502.254 779.957,-508.969\"/>\r\n",
       "</g>\r\n",
       "<!-- 4 -->\r\n",
       "<g id=\"node5\" class=\"node\"><title>4</title>\r\n",
       "<polygon fill=\"#f2c19e\" stroke=\"black\" points=\"894.667,-479 790.667,-479 790.667,-426 894.667,-426 894.667,-479\"/>\r\n",
       "<text text-anchor=\"start\" x=\"807.667\" y=\"-463.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">mse = 0.246</text>\r\n",
       "<text text-anchor=\"start\" x=\"798.667\" y=\"-448.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">samples = 4194</text>\r\n",
       "<text text-anchor=\"start\" x=\"804.667\" y=\"-433.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">value = 9.739</text>\r\n",
       "</g>\r\n",
       "<!-- 2&#45;&gt;4 -->\r\n",
       "<g id=\"edge4\" class=\"edge\"><title>2&#45;&gt;4</title>\r\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M659.314,-452.5C695.992,-452.5 743.16,-452.5 780.335,-452.5\"/>\r\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"780.538,-456 790.538,-452.5 780.538,-449 780.538,-456\"/>\r\n",
       "</g>\r\n",
       "<!-- 6 -->\r\n",
       "<g id=\"node7\" class=\"node\"><title>6</title>\r\n",
       "<polygon fill=\"#eeac7d\" stroke=\"black\" points=\"894.667,-408 790.667,-408 790.667,-355 894.667,-355 894.667,-408\"/>\r\n",
       "<text text-anchor=\"start\" x=\"807.667\" y=\"-392.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">mse = 0.093</text>\r\n",
       "<text text-anchor=\"start\" x=\"798.667\" y=\"-377.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">samples = 2640</text>\r\n",
       "<text text-anchor=\"start\" x=\"801.667\" y=\"-362.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">value = 10.012</text>\r\n",
       "</g>\r\n",
       "<!-- 5&#45;&gt;6 -->\r\n",
       "<g id=\"edge6\" class=\"edge\"><title>5&#45;&gt;6</title>\r\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M680.247,-345.331C712.89,-352.645 750.079,-360.978 780.527,-367.801\"/>\r\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"780.055,-371.282 790.579,-370.053 781.586,-364.451 780.055,-371.282\"/>\r\n",
       "</g>\r\n",
       "<!-- 7 -->\r\n",
       "<g id=\"node8\" class=\"node\"><title>7</title>\r\n",
       "<polygon fill=\"#e58139\" stroke=\"black\" points=\"894.667,-337 790.667,-337 790.667,-284 894.667,-284 894.667,-337\"/>\r\n",
       "<text text-anchor=\"start\" x=\"807.667\" y=\"-321.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">mse = 0.359</text>\r\n",
       "<text text-anchor=\"start\" x=\"798.667\" y=\"-306.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">samples = 6835</text>\r\n",
       "<text text-anchor=\"start\" x=\"801.667\" y=\"-291.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">value = 10.568</text>\r\n",
       "</g>\r\n",
       "<!-- 5&#45;&gt;7 -->\r\n",
       "<g id=\"edge7\" class=\"edge\"><title>5&#45;&gt;7</title>\r\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M680.247,-321.886C712.89,-319.584 750.079,-316.961 780.527,-314.813\"/>\r\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"780.85,-318.299 790.579,-314.104 780.357,-311.316 780.85,-318.299\"/>\r\n",
       "</g>\r\n",
       "<!-- 9 -->\r\n",
       "<g id=\"node10\" class=\"node\"><title>9</title>\r\n",
       "<polygon fill=\"#fef9f6\" stroke=\"black\" points=\"683.667,-255.5 515.667,-255.5 515.667,-187.5 683.667,-187.5 683.667,-255.5\"/>\r\n",
       "<text text-anchor=\"start\" x=\"523.667\" y=\"-240.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">saleElapsed ≤ 975715200.0</text>\r\n",
       "<text text-anchor=\"start\" x=\"564.667\" y=\"-225.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">mse = 0.084</text>\r\n",
       "<text text-anchor=\"start\" x=\"558.667\" y=\"-210.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">samples = 848</text>\r\n",
       "<text text-anchor=\"start\" x=\"561.667\" y=\"-195.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">value = 9.022</text>\r\n",
       "</g>\r\n",
       "<!-- 8&#45;&gt;9 -->\r\n",
       "<g id=\"edge9\" class=\"edge\"><title>8&#45;&gt;9</title>\r\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M404.207,-221.5C434.504,-221.5 471.511,-221.5 505.079,-221.5\"/>\r\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"505.375,-225 515.375,-221.5 505.375,-218 505.375,-225\"/>\r\n",
       "</g>\r\n",
       "<!-- 12 -->\r\n",
       "<g id=\"node13\" class=\"node\"><title>12</title>\r\n",
       "<polygon fill=\"#f9e1d1\" stroke=\"black\" points=\"683.167,-131.5 516.167,-131.5 516.167,-63.5 683.167,-63.5 683.167,-131.5\"/>\r\n",
       "<text text-anchor=\"start\" x=\"524.167\" y=\"-116.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">fiProductClassDesc ≤ 40.5</text>\r\n",
       "<text text-anchor=\"start\" x=\"564.667\" y=\"-101.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">mse = 0.085</text>\r\n",
       "<text text-anchor=\"start\" x=\"558.667\" y=\"-86.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">samples = 939</text>\r\n",
       "<text text-anchor=\"start\" x=\"561.667\" y=\"-71.3\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">value = 9.329</text>\r\n",
       "</g>\r\n",
       "<!-- 8&#45;&gt;12 -->\r\n",
       "<g id=\"edge12\" class=\"edge\"><title>8&#45;&gt;12</title>\r\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M404.207,-191.325C438.724,-174.671 481.952,-153.813 518.881,-135.996\"/>\r\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"520.639,-139.034 528.124,-131.536 517.597,-132.729 520.639,-139.034\"/>\r\n",
       "</g>\r\n",
       "<!-- 10 -->\r\n",
       "<g id=\"node11\" class=\"node\"><title>10</title>\r\n",
       "<polygon fill=\"#fdf3ec\" stroke=\"black\" points=\"891.667,-266 793.667,-266 793.667,-213 891.667,-213 891.667,-266\"/>\r\n",
       "<text text-anchor=\"start\" x=\"807.667\" y=\"-250.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">mse = 0.082</text>\r\n",
       "<text text-anchor=\"start\" x=\"801.667\" y=\"-235.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">samples = 398</text>\r\n",
       "<text text-anchor=\"start\" x=\"804.667\" y=\"-220.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">value = 9.104</text>\r\n",
       "</g>\r\n",
       "<!-- 9&#45;&gt;10 -->\r\n",
       "<g id=\"edge10\" class=\"edge\"><title>9&#45;&gt;10</title>\r\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M683.943,-227.72C716.681,-230.165 753.485,-232.914 783.302,-235.141\"/>\r\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"783.306,-238.651 793.539,-235.905 783.827,-231.67 783.306,-238.651\"/>\r\n",
       "</g>\r\n",
       "<!-- 11 -->\r\n",
       "<g id=\"node12\" class=\"node\"><title>11</title>\r\n",
       "<polygon fill=\"#ffffff\" stroke=\"black\" points=\"891.667,-195 793.667,-195 793.667,-142 891.667,-142 891.667,-195\"/>\r\n",
       "<text text-anchor=\"start\" x=\"807.667\" y=\"-179.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">mse = 0.075</text>\r\n",
       "<text text-anchor=\"start\" x=\"801.667\" y=\"-164.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">samples = 450</text>\r\n",
       "<text text-anchor=\"start\" x=\"804.667\" y=\"-149.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">value = 8.949</text>\r\n",
       "</g>\r\n",
       "<!-- 9&#45;&gt;11 -->\r\n",
       "<g id=\"edge11\" class=\"edge\"><title>9&#45;&gt;11</title>\r\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M683.943,-203.186C716.824,-195.955 753.807,-187.822 783.692,-181.25\"/>\r\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"784.524,-184.65 793.539,-179.084 783.02,-177.814 784.524,-184.65\"/>\r\n",
       "</g>\r\n",
       "<!-- 13 -->\r\n",
       "<g id=\"node14\" class=\"node\"><title>13</title>\r\n",
       "<polygon fill=\"#fae9dc\" stroke=\"black\" points=\"891.667,-124 793.667,-124 793.667,-71 891.667,-71 891.667,-124\"/>\r\n",
       "<text text-anchor=\"start\" x=\"811.167\" y=\"-108.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">mse = 0.08</text>\r\n",
       "<text text-anchor=\"start\" x=\"801.667\" y=\"-93.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">samples = 541</text>\r\n",
       "<text text-anchor=\"start\" x=\"804.667\" y=\"-78.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">value = 9.237</text>\r\n",
       "</g>\r\n",
       "<!-- 12&#45;&gt;13 -->\r\n",
       "<g id=\"edge13\" class=\"edge\"><title>12&#45;&gt;13</title>\r\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M683.268,-97.5C716.178,-97.5 753.286,-97.5 783.309,-97.5\"/>\r\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"783.615,-101 793.615,-97.5 783.615,-94.0001 783.615,-101\"/>\r\n",
       "</g>\r\n",
       "<!-- 14 -->\r\n",
       "<g id=\"node15\" class=\"node\"><title>14</title>\r\n",
       "<polygon fill=\"#f7d8c1\" stroke=\"black\" points=\"891.667,-53 793.667,-53 793.667,-0 891.667,-0 891.667,-53\"/>\r\n",
       "<text text-anchor=\"start\" x=\"807.667\" y=\"-37.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">mse = 0.064</text>\r\n",
       "<text text-anchor=\"start\" x=\"801.667\" y=\"-22.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">samples = 398</text>\r\n",
       "<text text-anchor=\"start\" x=\"804.667\" y=\"-7.8\" font-family=\"Times New Roman,serif\" font-size=\"14.00\">value = 9.454</text>\r\n",
       "</g>\r\n",
       "<!-- 12&#45;&gt;14 -->\r\n",
       "<g id=\"edge14\" class=\"edge\"><title>12&#45;&gt;14</title>\r\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M683.268,-73.1652C716.321,-63.4274 753.611,-52.4418 783.702,-43.5767\"/>\r\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"785.011,-46.8398 793.615,-40.6564 783.033,-40.1251 785.011,-46.8398\"/>\r\n",
       "</g>\r\n",
       "</g>\r\n",
       "</svg>\r\n"
      ],
      "text/plain": [
       "<graphviz.files.Source at 0x1deb8c3d3c8>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "draw_tree(m.estimators_[0], df_trn, precision=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Let's see what happens if we create a bigger tree."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[2.175583928816829e-17, 0.4791917172489859, 1.0, 0.5899218973405747]\n"
     ]
    }
   ],
   "source": [
    "m = RandomForestRegressor(n_estimators=1, bootstrap=False, n_jobs=-1)\n",
    "m.fit(X_train, y_train)\n",
    "print_score(m)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "The training set result looks great! But the validation set is worse than our original model. This is why we need to use *bagging* of multiple trees to get more generalizable results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## Bagging"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Intro to bagging"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "To learn about bagging in random forests, let's start with our basic model again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "e:\\Anaconda\\envs\\fastai-cpu\\lib\\site-packages\\sklearn\\ensemble\\forest.py:245: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.11183168570725253, 0.34419786485966564, 0.9730231802074433, 0.7884250154050879]\n"
     ]
    }
   ],
   "source": [
    "m = RandomForestRegressor(n_jobs=-1)\n",
    "m.fit(X_train, y_train)\n",
    "print_score(m)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "We'll grab the predictions for each individual tree, and look at one example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(array([9.39266, 9.15905, 8.9872 , 9.15905, 8.9872 , 9.15905, 8.9872 , 8.9872 , 9.51044, 8.9359 ]),\n",
       " 9.126493893488487,\n",
       " 9.104979856318357)"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preds = np.stack([t.predict(X_valid) for t in m.estimators_]) # this takes all the predicted values from the first row \n",
    "# and returns the mean of all those values; this mean, predicted price, is very close to the validation dataset first \n",
    "# row \n",
    "preds[:,0], np.mean(preds[:,0]), y_valid[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10, 12000)"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preds.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX0AAAD8CAYAAACb4nSYAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAHvtJREFUeJzt3Xt0nPV95/H3V6O7bVmyLRtjW5bvxiGAjewQk3ApAZwmhWyy2zUpKdAk3mxDktJsNrDdhh6T7mG7myZpQ7MhFMimCW6W5iQOoSFQYkG4BMsGAhbRWL7LBjy6+arbjL77x4zssZCskTXSM5fP65w5mud5fo/mO3Oszzz+zW9+P3N3REQkPxQEXYCIiEwchb6ISB5R6IuI5BGFvohIHlHoi4jkEYW+iEgeUeiLiOQRhb6ISB5R6IuI5JHCVBqZ2Trgm0AIeMDd7x10vAb4HlCZaHOnuz+eOHYX8EkgBnze3Z8422PNmDHDa2trR/k0RETy27Zt21rdvXqkdiOGvpmFgPuAa4EWYKuZbXb3xqRm/x34kbt/28xWAI8DtYn764F3AecDT5nZUnePDfd4tbW1NDQ0jFSWiIgkMbN9qbRLpXtnDdDs7rvdvRfYBNw4qI0DFYn7U4FDifs3Apvcvcfd9wDNid8nIiIBSCX05wAHkrZbEvuS/RVws5m1EL/K/9wozhURkQmSSujbEPsGT815E/Cwu88Ffh/4vpkVpHguZrbBzBrMrCESiaRQkoiInItUQr8FmJe0PZfT3TcDPgn8CMDdXwBKgRkpnou73+/ude5eV1094ucQIiJyjlIJ/a3AEjNbYGbFxD+Y3TyozX7gGgAzu4B46EcS7dabWYmZLQCWAC+lq3gRERmdEUfvuHvUzG4HniA+HPNBd99hZhuBBnffDHwR+K6Z3UG8++ZWj6/OssPMfgQ0AlHgs2cbuSMiIuPLMm3lrLq6OteQTRGR0TGzbe5eN1K7lL6cJSKS6fr7nZ5oPz3RGN19/XT3xehOvt8Xoycav9/T1584Fj/uDhVlhVSUFlFRVkRFaWH8Z+L+5JJCzIYal5J9FPoiknZ9sf54APfF6E78PBW40TOPxUP4dDgPtDszsAfCPOl4Uph3R/vpjfaP2/MpMJhSWnT6jSH5ftlQ25n7pqHQF8kD7k53Xz9Huvo40tXH0e4+unpjZ1wZ90TjV8BnhnPSsaTwHrhS7kk6lhzosf5z7zYOFRilhQWUFoUoGfhZFKK0qIDSwhCTJhVSWpjYLgoljhdQUni6TWnR6eMDv6P0VJsz95UWhQA43h3laHcfR7sGfvYNsR09tX9v68lT+0/0nv2jylTfNOZWlXPtilnn/NqlQqEvkiUGB/dQt6NdfXSe7B20Px5UvbHRXQkXFxZQWlhASSIkS5KCuKQwxPRJxZQUxgO3tHAgeM9sMxC0A8dKhjiW/LO0KERRKJh5IKsmFVM1qficzo3G+jmWhjeNVTWVCn2RXNPdF6Pz5NmDO3k7HuIjB7cZTCkpZGp5EVPL4rfZU8uoKDu9PXCrKCukvDgUD+RBQV1SVEBxqICCgszojsgGhaGCtLxp9I3yjflcKPRFxlFXb4xXDnSybV87W/d28PL+Do52R4dtP1RwLztvSiKoT++rLCt+R5BPKS1UUGepgTeNCXmsCXkUkTzReryHhr0dNOxtp2FfB68fPEI00b+9dNZkPnTRbOZNK39HYJ8O7iJCCm4ZRwp9kXPk7uxuPREP+L0dNOzrYE/rCSDeH37x3Kl8+oqF1M2v4tL5VVSWT8yVnMjZKPRFUtQb7ee1g0dOddVs29dB+4leAKrKi7h0/jT+4+p5rK6t4sI5UykpDAVcscg7KfRFhnHkZB/b93ewNdFV8+qBTnoSY8Frp5dz9bKZrK6toq52GouqJ2XMOGyRs1HoixDvqmnp6KJhX6KrZm8H4cPHcI+PG7/w/Apuvmx+vKumtoqZU0qDLlnknCj0JS9FY/387q1jNOxtZ+u+Drbt7eCto90ATC4pZNX8Kj500Wzqaqu4ZF4l5cX6U5HcoH/JkhdO9ER55UAnW/e2s21fB9v3dZz6QszsqaWsXjCN1bXxD1yXn1ehETSSsxT6knPcnX1tJ9m+vyN+29fJ7946Sr/Hx8EvmzWFj66aS12iP35OZVnQJYtMGIW+ZL2TvVF+23IkEfAdvLy/k7bEqJpJxSEuqanks1cvZtX8KlbVVDG1rCjgikWCo9CXrOLuHGjvOn0Vv7+DN948dmqCr4UzJnHVspmsml/Jqpoqls6aoq4akSQKfclo3X2xM67it+/vpPV4DwDlxSEunlvJZ65cyKqaKlbWVDFtgr7KLpKtFPqSMdydg51dbN/fmQj4DhoPHT01jUHt9HKuWDKDlfOrWFVTybJZUygMaEZGkWyl0JfAdPfFeP3gkVMftm7f38HhY/Gr+LKiEBfNncqGKwau4iuZPrkk4IpFsp9CXybMoc6uMwJ+x6Ej9MXiV/E108pZu2j6qQ9bl5+nq3iR8aDQl3Hj7jyx4202v3qQ7fs6T335qbSogIvmVPLJ9y1kVU0lK2uqqJ6iq3iRiaDQl3Gx49AR7nmskRd3tzN7ailrFkxjVU0lq+ZXccHsisBWRxLJdwp9SavW4z187ZdNbNp6gMqyIu75yIXctHqeumpEMoRCX9KiN9rPw8/v4e//rZmuvhi3rV3AF65ZwtRyfRFKJJMo9GVM3J2n3jjMX/+8kb1tJ7l6WTV/8aEVLJ45OejSRGQICn05Z01vHeOexxr5dXMri6on8fBtq7lq2cygyxKRs1Doy6h1nOjl60+F+cFv9jOpOMTdf7CCmy+brw9nRbJASqFvZuuAbwIh4AF3v3fQ8a8DVyc2y4GZ7l6ZOBYDXksc2+/uN6SjcJl4fbF+vv/CPr7xVJgTvTH+6D013PGBpVRp6gORrDFi6JtZCLgPuBZoAbaa2WZ3bxxo4+53JLX/HLAy6Vd0ufsl6StZgvCrpsN89bFGdkVO8P4lM/jLD69g6awpQZclIqOUypX+GqDZ3XcDmNkm4EagcZj2NwF3p6c8CVrz4eN89eeNbGmKUDu9nAf+uI5rLpip9WBFslQqoT8HOJC03QK8Z6iGZjYfWAA8nbS71MwagChwr7v/5BxrlQl05GQf3/i3MN9/YR9lRSH+4vcv4Ja1tRQXqt9eJJulEvpDXdL5MG3XA4+6eyxpX427HzKzhcDTZvaau+864wHMNgAbAGpqalIoScZLNNbPIy/t52+fDNPZ1cf61TV88bqlzNBkZyI5IZXQbwHmJW3PBQ4N03Y98NnkHe5+KPFzt5ltId7fv2tQm/uB+wHq6uqGe0ORcfbrna3c81gjTW8f47KF0/jKh9/FivMrgi5LRNIoldDfCiwxswXAQeLB/vHBjcxsGVAFvJC0rwo46e49ZjYDuBz4m3QULumzt/UEX/35Gzz1xtvMm1bG/7l5Fde/6zz124vkoBFD392jZnY78ATxIZsPuvsOM9sINLj75kTTm4BN7p58pX4B8B0z6wcKiPfpD/cBsEywo919fOvpZh56bg/FoQK+dP0yPvm+BZQWhYIuTUTGiZ2Z0cGrq6vzhoaGoMvIabF+50cNB/jaL5toO9HLv181ly9dv4yZFaVBlyYi58jMtrl73Ujt9I3cPPPi7jY2/qyRxjePUje/ioduXcO7504NuiwRmSAK/TxxoP0k/+PxN/jX19/i/Kml/P1NK/nwRbPVby+SZxT6Oe54T5R/+FUzD/x6DyEz7vjAUjZcsZCyYvXbi+QjhX6O6u93/mV7C3/zRBORYz185JLz+fIHlzN7alnQpYlIgBT6Ocjd+YufvMYjLx3g4nmVfOcTl7KqpiroskQkAyj0c9C3nm7mkZcO8J+uXMiXr19OQYH67UUkTqGfY/5fwwG+9mSYj66cw53rluuDWhE5g2bPyiHPhCPc9ePXuHzxdO792EUKfBF5B4V+jthx6Ah/+oPtLJ45mW/ffKlmwxSRISkZcsDBzi5ue2grU0oLefi2NVSUFgVdkohkKPXpZ7kjJ/u49cGX6OqL8ehn1nLeVE2lICLD05V+FuuJxvj09xvY13aS+z9Rx7LztHyhiJydrvSzVH+/88UfvcpLe9r55vpLeO+i6UGXJCJZQFf6WereX/yOx377Jnd9cDk3XjIn6HJEJEso9LPQQ8/t4f5ndvPH753PhisWBl2OiGQRhX6W+cXrb7LxsUauXTGLu//gXRqLLyKjotDPItv2tfOFTa9wybxK/m79SkKaXkFERkmhnyV2RY7zye81cH5lGf94y2pNjSwi50ShnwUix3q49aGXCJnx8G2rmTapOOiSRCRLachmhjvRE+VPHt5K67FeNm24jPnTJwVdkohkMV3pZ7BorJ/bf7idHYeO8K2Pr+TieZVBlyQiWU5X+hnK3fnLn77Or5oi/PW/u5BrLpgVdEkikgN0pZ+hBhZCuf3qxfzRe+YHXY6I5AiFfgY6tRDKqjl88bqlQZcjIjlEoZ9hBhZCed/iGdz7US2EIiLppdDPIDsOHeE//9M2Fs+czD/cvEoLoYhI2ilVMsTAQigVZUVaCEVExk1KoW9m68ysycyazezOIY5/3cxeSdzCZtaZdOwWM9uZuN2SzuJzRfJCKA/ftkYLoYjIuBlxyKaZhYD7gGuBFmCrmW1298aBNu5+R1L7zwErE/enAXcDdYAD2xLndqT1WWSx5IVQvvcna7QQioiMq1Su9NcAze6+2917gU3AjWdpfxPwSOL+9cCT7t6eCPongXVjKTiXJC+E8r/+w0VaCEVExl0qoT8HOJC03ZLY9w5mNh9YADw9mnPNbIOZNZhZQyQSSaXunKCFUERkoqUS+kONGfRh2q4HHnX32GjOdff73b3O3euqq6tTKCn7DSyEcosWQhGRCZRK6LcA85K25wKHhmm7ntNdO6M9N28MLIRy3YpZfEULoYjIBEol9LcCS8xsgZkVEw/2zYMbmdkyoAp4IWn3E8B1ZlZlZlXAdYl9eathb3whlJXzKvm7m7QQiohMrBFH77h71MxuJx7WIeBBd99hZhuBBncfeAO4Cdjk7p50bruZ3UP8jQNgo7u3p/cpZI9dkeN86v/GF0J54JbVlBZpIRQRmViWlNEZoa6uzhsaGoIuI+0ix3r46Lef42RPjB//6VrNiy8iaWVm29y9bqR2mlp5AmghFBHJFJqGYZxpIRQRySS60h9HWghFRDKNrvTHkRZCEZFMo9AfJ1oIRUQykUJ/HGghFBHJVAr9NDvQfvLUQijf1kIoIpJhlEhp9q+vv8mJ3hjf+cSlTNFCKCKSYRT6aVYfjrD8vCkaiy8iGUmhn0YneqJs3dPBlUvzY6ZQEck+Cv00emFXG72xfoW+iGQshX4a1YcjlBeHuLS2KuhSRESGpNBPE3dnS/gwaxdNp6RQs2eKSGZS6KfJ3raTHGjvUteOiGQ0hX6a1DcdBuDKpTMDrkREZHgK/TSpD0dYOGMSNdPLgy5FRGRYCv006O6L8cLuNq5Q146IZDiFfhq8tKed7r5+rlym0BeRzKbQT4P6cITiwgIuWzA96FJERM5KoZ8G9eEI71kwjbJiDdUUkcym0B+jlo6TNB8+zlXLNGpHRDKfQn+Mngm3Amh8vohkBYX+GG1pOsycyjIWVWtWTRHJfAr9MeiN9vP8rjauXFat1bFEJCso9Mdg+/4OjvdE1bUjIllDoT8G9eEIhQXG2kUaqiki2SGl0DezdWbWZGbNZnbnMG3+0MwazWyHmf0waX/MzF5J3Danq/BMUN8U4dL5VVoWUUSyRuFIDcwsBNwHXAu0AFvNbLO7Nya1WQLcBVzu7h1mljx+scvdL0lz3YE7fLSbxjeP8uV1y4MuRUQkZalc6a8Bmt19t7v3ApuAGwe1+TRwn7t3ALj74fSWmXme2amhmiKSfVIJ/TnAgaTtlsS+ZEuBpWb2nJm9aGbrko6VmllDYv9HxlhvxtjSdJjqKSVcMHtK0KWIiKRsxO4dYKixiD7E71kCXAXMBZ41swvdvROocfdDZrYQeNrMXnP3XWc8gNkGYANATU3NKJ/CxIv1O8/ubOXaFbM0VFNEskoqV/otwLyk7bnAoSHa/NTd+9x9D9BE/E0Adz+U+Lkb2AKsHPwA7n6/u9e5e111deZ3l7za0smRrj517YhI1kkl9LcCS8xsgZkVA+uBwaNwfgJcDWBmM4h39+w2syozK0nafznQSJarb4pQYPC+xTOCLkVEZFRG7N5x96iZ3Q48AYSAB919h5ltBBrcfXPi2HVm1gjEgC+5e5uZrQW+Y2b9xN9g7k0e9ZOt6sMRLp5XSdWk4qBLEREZlVT69HH3x4HHB+37StJ9B/48cUtu8zzw7rGXmTk6TvTyaksnf3bN0qBLEREZNX0jd5Se2RnBHa2SJSJZSaE/SvXhCFXlRbx7ztSgSxERGTWF/ij09zvPhFt5/5JqQgUaqiki2UehPwqNbx6l9XiPhmqKSNZS6I9CfTgCwBUKfRHJUgr9UagPR7hwTgXVU0qCLkVE5Jwo9FN0tLuPbfs61LUjIllNoZ+i55tbifU7Vy6dOXJjEZEMpdBPUX04wpSSQlbWVAZdiojIOVPop8DdqW+KcPniGRSF9JKJSPZSgqWg+fBxDh3p1rdwRSTrKfRTMDBUUx/iiki2U+inYEtThKWzJnN+ZVnQpYiIjIlCfwQne6O8tKddV/kikhMU+iN4cXcbvbF+DdUUkZyg0B9BfVOEsqIQdbVVQZciIjJmCv0R1IcjvHfRdEqLQkGXIiIyZgr9s9jbeoK9bSfVny8iOUOhfxbP7IwP1bxK4/NFJEco9M9iS1OE2unlzJ8+KehSRETSQqE/jO6+GC/salPXjojkFIX+MBr2dtDVF9PUCyKSUxT6w6gPH6Y4VMBlC6cHXYqISNoo9IdRH46wZsE0yosLgy5FRCRtFPpDONTZRfjt4+rPF5Gco9AfwsCsmhqqKSK5RqE/hPqmCOdPLWXxzMlBlyIiklYphb6ZrTOzJjNrNrM7h2nzh2bWaGY7zOyHSftvMbOdidst6Sp8vPTF+nmuuZUrl1VjZkGXIyKSViN+SmlmIeA+4FqgBdhqZpvdvTGpzRLgLuByd+8ws5mJ/dOAu4E6wIFtiXM70v9U0uPl/Z0c64mqP19EclIqV/prgGZ33+3uvcAm4MZBbT4N3DcQ5u5+OLH/euBJd29PHHsSWJee0sdHffgwoQJj7eIZQZciIpJ2qYT+HOBA0nZLYl+ypcBSM3vOzF40s3WjOBcz22BmDWbWEIlEUq9+HNSHI1xaU0VFaVGgdYiIjIdUQn+ojm0ftF0ILAGuAm4CHjCzyhTPxd3vd/c6d6+rrg6uW+XwsW5eP3hU38IVkZyVSui3APOStucCh4Zo81N373P3PUAT8TeBVM7NGM+GWwEtgC4iuSuV0N8KLDGzBWZWDKwHNg9q8xPgagAzm0G8u2c38ARwnZlVmVkVcF1iX0aqD0eYMbmEFbMrgi5FRGRcjDh6x92jZnY78bAOAQ+6+w4z2wg0uPtmTod7IxADvuTubQBmdg/xNw6Aje7ePh5PZKxi/c6zOyNcvXwmBQUaqikiuSmliWXc/XHg8UH7vpJ034E/T9wGn/sg8ODYyhx/rx08QsfJPnXtiEhO0zdyE+qbIpjB+5co9EUkdyn0E7aED3PR3EqmTSoOuhQRkXGj0Ac6TvTy6oFOrlLXjojkOIU+8OvmVvodjc8XkZyn0Cc+VHNqWREXz60MuhQRkXGV96Hv7tSHI7x/yQxCGqopIjku70P/jTePETnWo6GaIpIX8j70t4TjE4Iq9EUkH+R96Nc3RVgxu4KZFaVBlyIiMu7yOvSPdfexbV+HRu2ISN7I69B/flcb0X5X146I5I28Dv36cITJJYWsqqkKuhQRkQmRt6Hv7tQ3RVi7aDrFhXn7MohInsnbtNsVOcHBzi7154tIXsnb0N/SpKGaIpJ/8jb068MRFs+czNyq8qBLERGZMHkZ+l29MX6zp11X+SKSd/Iy9F/c00ZvtF+hLyJ5Jy9Dv74pQmlRAWsWTAu6FBGRCZWXof9MOMJlC6dTWhQKuhQRkQmVd6G/v+0ku1tPqGtHRPJS3oV+fWJWzauWzQy4EhGRiZeHoR+hZlo5tdM1VFNE8k9ehX5PNMbzu9q4cmk1ZlolS0TyT16F/ra9HZzsjak/X0TyVl6Ffn04QlHIeO+i6UGXIiISiJRC38zWmVmTmTWb2Z1DHL/VzCJm9kri9qmkY7Gk/ZvTWfxobWmKsLp2GpNKCoMsQ0QkMCOmn5mFgPuAa4EWYKuZbXb3xkFN/9ndbx/iV3S5+yVjL3Vs3jzSRdPbx7hr1fKgSxERCUwqV/prgGZ33+3uvcAm4MbxLSv9nglHAA3VFJH8lkrozwEOJG23JPYN9jEz+62ZPWpm85L2l5pZg5m9aGYfGUuxY1EfjnBeRSlLZ00OqgQRkcClEvpDjW30Qds/A2rd/SLgKeB7Scdq3L0O+DjwDTNb9I4HMNuQeGNoiEQiKZaeumisn2d3tmqopojkvVRCvwVIvnKfCxxKbuDube7ek9j8LnBp0rFDiZ+7gS3AysEP4O73u3udu9dVV6d/OOUrBzo51h3VKlkikvdSCf2twBIzW2BmxcB64IxROGY2O2nzBuCNxP4qMytJ3J8BXA4M/gB43G1pihAqMC5fPGOiH1pEJKOMOHrH3aNmdjvwBBACHnT3HWa2EWhw983A583sBiAKtAO3Jk6/APiOmfUTf4O5d4hRP+OuPhxh5bxKppYVTfRDi4hklJQGrLv748Djg/Z9Jen+XcBdQ5z3PPDuMdY4Jq3He3jt4BH+y3VLgyxDRCQj5Pw3cp/dGf9g+MqlGqopIpLzoV/fFGH6pGLedX5F0KWIiAQup0O/v995ZmcrVyytpqBAQzVFRHI69F8/dIT2E72aVVNEJCGnQ39LUwQzeP8SDdUUEYEcD/36cIR3z5nK9MklQZciIpIRcjb0j5zs4+X9HVylrh0RkVNyNvR/3dxKv6OpF0REkuRs6NeHD1NRWsjFcyuDLkVEJGPkZOi7O/XhCO9fUk1hKCefoojIOcnJRPzdW8d4+2iPhmqKiAySk6Ffn1gl6wqFvojIGXIz9JsiLD9vCudNLQ26FBGRjJJzoX+8J0rDvnaN2hERGULOhf4Lu9roi7n680VEhpBzoV8fPkx5cYi6+dOCLkVEJOPkVOi7O1uaIqxdNIPiwpx6aiIiaZFTybi79QQtHV3qzxcRGUZOhX59U3yopubbEREZWm6FfjjCwupJzJtWHnQpIiIZKWdCv7svxou72zRqR0TkLHIm9I929bHuwvO4dsWsoEsREclYhUEXkC4zK0r55vqVQZchIpLRcuZKX0RERqbQFxHJIwp9EZE8otAXEckjKYW+ma0zsyYzazazO4c4fquZRczslcTtU0nHbjGznYnbLeksXkRERmfE0TtmFgLuA64FWoCtZrbZ3RsHNf1nd7990LnTgLuBOsCBbYlzO9JSvYiIjEoqV/prgGZ33+3uvcAm4MYUf//1wJPu3p4I+ieBdedWqoiIjFUqoT8HOJC03ZLYN9jHzOy3Zvaomc0bzblmtsHMGsysIRKJpFi6iIiMVipfzrIh9vmg7Z8Bj7h7j5l9Bvge8Hspnou73w/cD5D4bGBfCnUNZwbQOobzc4leizPp9TiTXo/TcuG1mJ9Ko1RCvwWYl7Q9FziU3MDd25I2vwv8z6Rzrxp07pazPZi7j2nyHDNrcPe6sfyOXKHX4kx6Pc6k1+O0fHotUune2QosMbMFZlYMrAc2Jzcws9lJmzcAbyTuPwFcZ2ZVZlYFXJfYJyIiARjxSt/do2Z2O/GwDgEPuvsOM9sINLj7ZuDzZnYDEAXagVsT57ab2T3E3zgANrp7+zg8DxERSYG5v6OLPauZ2YbEZwR5T6/FmfR6nEmvx2n59FrkXOiLiMjwNA2DiEgeyZnQH2mqiHxiZvPM7Fdm9oaZ7TCzLwRdU9DMLGRmL5vZY0HXEjQzq0x8n+Z3iX8j7w26piCZ2R2Jv5PXzewRMysNuqbxlBOhnzRVxAeBFcBNZrYi2KoCFQW+6O4XAJcBn83z1wPgC5weVZbvvgn8wt2XAxeTx6+Lmc0BPg/UufuFxAerrA+2qvGVE6HP2KaKyDnu/qa7b0/cP0b8j3qob1HnBTObC3wIeCDoWoJmZhXAFcA/Arh7r7t3BltV4AqBMjMrBMoZ9D2kXJMroZ/qVBF5x8xqgZXAb4KtJFDfAP4r0B90IRlgIRABHkp0dz1gZpOCLioo7n4Q+N/AfuBN4Ii7/zLYqsZXroR+StM95Bszmwz8C/Bn7n406HqCYGYfBg67+7aga8kQhcAq4NvuvhI4AeTtZ2CJL43eCCwAzgcmmdnNwVY1vnIl9EecKiLfmFkR8cD/gbv/OOh6AnQ5cIOZ7SXe7fd7ZvZPwZYUqBagxd0H/uf3KPE3gXz1AWCPu0fcvQ/4MbA24JrGVa6E/ohTReQTMzPifbZvuPvfBl1PkNz9Lnef6+61xP9dPO3uOX0ldzbu/hZwwMyWJXZdAwxeGyOf7AcuM7PyxN/NNeT4B9upTLiW8YabKiLgsoJ0OfAJ4DUzeyWx77+5++MB1iSZ43PADxIXSLuB2wKuJzDu/hszexTYTnzU28skZvzNVfpGrohIHsmV7h0REUmBQl9EJI8o9EVE8ohCX0Qkjyj0RUTyiEJfRCSPKPRFRPKIQl9EJI/8f8UoceVyzk37AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.plot([metrics.r2_score(y_valid, np.mean(preds[:i+1], axis=0)) for i in range(10)]);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "The shape of this curve suggests that adding more trees isn't going to help us much. Let's check. (Compare this to our original model on a sample)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.10243205181129802, 0.3412592591624538, 0.9773674883587342, 0.792022256819186]\n"
     ]
    }
   ],
   "source": [
    "m = RandomForestRegressor(n_estimators=20, n_jobs=-1)\n",
    "m.fit(X_train, y_train)\n",
    "print_score(m)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.0964044200924326, 0.33756249420472983, 0.9799527456179769, 0.7965037791411347]\n"
     ]
    }
   ],
   "source": [
    "m = RandomForestRegressor(n_estimators=40, n_jobs=-1)\n",
    "m.fit(X_train, y_train)\n",
    "print_score(m)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.09290943049551428, 0.33575888096052303, 0.9813799602336366, 0.7986725506109265]\n"
     ]
    }
   ],
   "source": [
    "m = RandomForestRegressor(n_estimators=80, n_jobs=-1)\n",
    "m.fit(X_train, y_train)\n",
    "print_score(m)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Out-of-bag (OOB) score"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Is our validation set worse than our training set because we're over-fitting, or because the validation set is for a different time period, or a bit of both? With the existing information we've shown, we can't tell. However, random forests have a very clever trick called *out-of-bag (OOB) error* which can handle this (and more!)\n",
    "\n",
    "The idea is to calculate error on the training set, but only include the trees in the calculation of a row's error where that row was *not* included in training that tree. This allows us to see whether the model is over-fitting, without needing a separate validation set.\n",
    "\n",
    "This also has the benefit of allowing us to see whether our model generalizes, even if we only have a small amount of data so want to avoid separating some out to create a validation set.\n",
    "\n",
    "This is as simple as adding one more parameter to our model constructor. We print the OOB error last in our `print_score` function below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.0960574997183861, 0.3379738084696881, 0.9800967698697566, 0.7960075635128223, 0.8578733687389983]\n"
     ]
    }
   ],
   "source": [
    "m = RandomForestRegressor(n_estimators=40, n_jobs=-1, oob_score=True)\n",
    "m.fit(X_train, y_train)\n",
    "print_score(m)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "This shows that our validation set time difference is making an impact, as is model over-fitting."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "## Reducing over-fitting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Subsampling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "It turns out that one of the easiest ways to avoid over-fitting is also one of the best ways to speed up analysis: *subsampling*. Let's return to using our full dataset, so that we can demonstrate the impact of this technique."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "df_trn, y_trn, nas = proc_df(df_raw, 'SalePrice')\n",
    "X_train, X_valid = split_vals(df_trn, n_trn)\n",
    "y_train, y_valid = split_vals(y_trn, n_trn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "The basic idea is this: rather than limit the total amount of data that our model can access, let's instead limit it to a *different* random subset per tree. That way, given enough trees, the model can still see *all* the data, but for each individual tree it'll be just as fast as if we had cut down our dataset as before."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "set_rf_samples(20000) # this makes each random forest run with 20k rows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "e:\\Anaconda\\envs\\fastai-cpu\\lib\\site-packages\\sklearn\\ensemble\\forest.py:245: FutureWarning: The default value of n_estimators will change from 10 in version 0.20 to 100 in 0.22.\n",
      "  \"10 in version 0.20 to 100 in 0.22.\", FutureWarning)\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Wall time: 11.1 s\n",
      "[0.24160235023702945, 0.2795113338607126, 0.8780063752833811, 0.8604766743646701, 0.8651726762286867]\n"
     ]
    }
   ],
   "source": [
    "m = RandomForestRegressor(n_jobs=-1, oob_score=True)\n",
    "%time m.fit(X_train, y_train)\n",
    "print_score(m)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Since each additional tree allows the model to see more data, this approach can make additional trees more useful."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.22738161126011566, 0.26194922306941437, 0.891944841341451, 0.8774587789901436, 0.8803754784426868]\n"
     ]
    }
   ],
   "source": [
    "m = RandomForestRegressor(n_estimators=40, n_jobs=-1, oob_score=True)\n",
    "m.fit(X_train, y_train)\n",
    "print_score(m)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "heading_collapsed": true,
    "hidden": true
   },
   "source": [
    "### Tree building parameters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "We revert to using a full bootstrap sample in order to show the impact of other over-fitting avoidance methods."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "reset_rf_samples() # to turn off sampling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Let's get a baseline for this full set to compare to."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "def dectree_max_depth(tree):\n",
    "    children_left = tree.children_left\n",
    "    children_right = tree.children_right\n",
    "\n",
    "    def walk(node_id):\n",
    "        if (children_left[node_id] != children_right[node_id]):\n",
    "            left_max = 1 + walk(children_left[node_id])\n",
    "            right_max = 1 + walk(children_right[node_id])\n",
    "            return max(left_max, right_max)\n",
    "        else: # leaf\n",
    "            return 1\n",
    "\n",
    "    root_node_id = 0\n",
    "    return walk(root_node_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.07824141637625177, 0.23619186853017332, 0.9872059444054736, 0.9003728153551043, 0.9086642938958147]\n"
     ]
    }
   ],
   "source": [
    "m = RandomForestRegressor(n_estimators=40, n_jobs=-1, oob_score=True)\n",
    "m.fit(X_train, y_train)\n",
    "print_score(m)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "t=m.estimators_[0].tree_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "44"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dectree_max_depth(t)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.14086668007831762, 0.2351542534801781, 0.958528343302569, 0.9012462374510828, 0.9068399428254218]\n"
     ]
    }
   ],
   "source": [
    "m = RandomForestRegressor(n_estimators=40, min_samples_leaf=5, n_jobs=-1, oob_score=True)\n",
    "m.fit(X_train, y_train)\n",
    "print_score(m)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "hidden": true
   },
   "outputs": [],
   "source": [
    "t=m.estimators_[0].tree_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "36"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dectree_max_depth(t)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "Another way to reduce over-fitting is to grow our trees less deeply. We do this by specifying (with `min_samples_leaf`) that we require some minimum number of rows in every leaf node. This has two benefits:\n",
    "\n",
    "- There are less decision rules for each leaf node; simpler models should generalize better\n",
    "- The predictions are made by averaging more rows in the leaf node, resulting in less volatility\n",
    "\n",
    "With `min_samples_leaf=1` the number of leaf nodes we will have is equal to the number of samples that we are taking and the **depth of the tree** is equal to $log_2(sample size)$. However, for `min_samples_leaf=2`the number of leaf nodes will be $\\frac{samplesize}{2}$ and the **depth of the tree** is equal to $log_2(sample size)-1$. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.11499634352543645, 0.23293967255980852, 0.9723622369454257, 0.903097519260144, 0.9084045414163522]\n"
     ]
    }
   ],
   "source": [
    "m = RandomForestRegressor(n_estimators=40, min_samples_leaf=3, n_jobs=-1, oob_score=True)\n",
    "m.fit(X_train, y_train)\n",
    "print_score(m)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "We can also increase the amount of variation amongst the trees by not only use a sample of rows for each tree, but to also using a sample of *columns* for each *split*. We do this by specifying `max_features`, which is the proportion of features to randomly select from at each split."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "- None\n",
    "- 0.5\n",
    "- 'sqrt'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "- 1, 3, 5, 10, 25, 100"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "hidden": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.11925548849856353, 0.2275062579497403, 0.9702770728170188, 0.9075653782630644, 0.9118017118238015]\n"
     ]
    }
   ],
   "source": [
    "m = RandomForestRegressor(n_estimators=40, min_samples_leaf=3, max_features=0.5, n_jobs=-1, oob_score=True)\n",
    "m.fit(X_train, y_train)\n",
    "print_score(m)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "hidden": true
   },
   "source": [
    "We can't compare our results directly with the Kaggle competition, since it used a different validation set (and we can no longer to submit to this competition) - but we can at least see that we're getting similar results to the winners based on the dataset we have.\n",
    "\n",
    "The sklearn docs [show an example](http://scikit-learn.org/stable/auto_examples/ensemble/plot_ensemble_oob.html) of different `max_features` methods with increasing numbers of trees - as you see, using a subset of features on each split requires using more trees, but results in better models:\n",
    "![sklearn max_features chart](http://scikit-learn.org/stable/_images/sphx_glr_plot_ensemble_oob_001.png)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}