notebooks/Python_04_Analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Python Immersion Course\n",
    "\n",
    "### Data Analysis with Python - Part 4\n",
    "\n",
    "### Joe Blankenship - Just some dude"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We'll be extending our work in part 3 with ETL processes to the realm of analysis. Currently, there is significant overlap between bringing in data and its analysis (often in as near real time as possible). Below we'll be leaning heavily on the work of Wes McKinney. If you have a chance, reach out to Wes and thank him and if you're feeling generous, buy his book (see syllabus; I highly recommend it). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n",
    "\n",
    "## Statistical Concepts\n",
    "\n",
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An understanding of statistical modeling is essential before using Python to process your data. Here are a few key concepts to review:\n",
    "\n",
    "* Hypothesis - unproven or unsubstantiated statement on a problem being investigated that is testable\n",
    "* Model - a device to simplify the problem's reality so that relationships between variables can be studied\n",
    "* Population - collection of all data we are interested in studing\n",
    "* Sample - a subset of a population\n",
    "* Descriptive Stats - representation of patterns existing in the data\n",
    "* Inferential Stats - estimations or predictions about the population based on sample analysis\n",
    "* Accuracy - a degree of correctness for a data set/point based on a standard\n",
    "* Precision - a measure of repeatability\n",
    "* Validity - a measurement of quality of a variable\n",
    "* Reliability - a measurement of errors in data over time\n",
    "* Mean - the average number\n",
    "* Median - the middle number\n",
    "* Mode - the most frequent number\n",
    "* Central Tendency - clustering of values about certain numerical values (such as mean, median or mode)\n",
    "* Variability - the spread and dispersion of data values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n",
    "\n",
    "## Numpy\n",
    "\n",
    "<hr>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import numpy\n",
    "\n",
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### NumPy ndarray"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Here is a sum of a simple list\n",
    "\n",
    "a = list(range(1000000))\n",
    "\n",
    "%timeit sum(a)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# now here is the advantage of using a NumPy array\n",
    "\n",
    "b = np.array(a)\n",
    "\n",
    "%timeit np.sum(b)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#create a numpy array with .array() function\n",
    "# Broadcast a math operation\n",
    "\n",
    "a = np.array([1, 2, 3, 4])\n",
    "\n",
    "a * 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ndarrays is a homogeneous data container\n",
    "\n",
    "a.dtype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# determine the dimension of your array\n",
    "# in this case we get a tuple with our array dimensions\n",
    "# in this case a 2D array (columns, rows)\n",
    "\n",
    "a.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# you can create empty or zero filled arrays with a tuple\n",
    "\n",
    "array_zeros = np.zeros((4, 4))\n",
    "array_zeros"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# an array with 2 columns or 2 nested lists in 2 rows\n",
    "\n",
    "array_empty = np.empty((2, 2, 2))\n",
    "array_empty"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# numpy can handle many data types\n",
    "\n",
    "array_dtype = np.array([1, 2, 3, 4], dtype=np.int64)\n",
    "array_dtype.dtype"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# vectorization allows numpy to express operations without for loops\n",
    "\n",
    "array_demo = np.array([[1, 2, 3], [4, 5, 6]])\n",
    "array_demo * array_demo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Indexing and Slicing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# numpy has its own range function\n",
    "\n",
    "array_range = np.arange(10)\n",
    "array_range"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we can slice and index just like we learned in Part 2\n",
    "\n",
    "array_range[::2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we can apply changes to slices\n",
    "\n",
    "array_range_slice = array_range[5:8]\n",
    "array_range_slice[:] = 42\n",
    "array_range"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we can apply this to higher dimensional arrays\n",
    "\n",
    "array_demo_2D = np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]])\n",
    "array_demo_2D[0][2]  # access item in 3rd column, 1st row"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "array_demo_3D = np.array([[[1, 2, 3], [4, 5, 6]], [[7, 8, 9], [10, 11,12]]])\n",
    "array_demo_3D[0, 1, 0]  # access 1st element of the second list in the first row"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# you can use Boolean operators\n",
    "\n",
    "array_demo_3D > 6"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we can use reshape to provide dimension for an array\n",
    "\n",
    "array_reshape = np.arange(24).reshape((8, 3))\n",
    "array_reshape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we can then transpose our array\n",
    "# this is very useful for matrix math\n",
    "\n",
    "array_reshape.T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we can use .dot() for inner matrix products\n",
    "# ORDER MATTERS! try swapping the order of the transpose\n",
    "\n",
    "np.dot(array_reshape.T, array_reshape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# numpy has many ufuncs or universal functions for fast operations\n",
    "# They are similar to standard library operations (e.g., unary ufuncs)\n",
    "\n",
    "array_funcs = np.arange(10)\n",
    "np.sqrt(array_funcs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# binary ufuncs take two arrays and return one\n",
    "\n",
    "array_funcs_x = np.random.randn(10)\n",
    "array_funcs_y = np.random.randn(10)\n",
    "# return largest number per element between each array index\n",
    "np.maximum(array_funcs_x, array_funcs_y)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Data Processing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# let's continue with examples of vertorization and broadcasting\n",
    "# these are two reasons why numerical computation with numpy is great\n",
    "\n",
    "# create an array of 1000 equally spaced points\n",
    "points = np.arange(-5, 5, 0.01)\n",
    "\n",
    "# create a meshgrid using out 1D points array twice\n",
    "xs, ys = np.meshgrid(points, points)\n",
    "\n",
    "# now we can evaluate the function sqrt(x^2 + y^2)\n",
    "z = np.sqrt(xs ** 2 + ys ** 2)\n",
    "%timeit z"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# numpy.where can replace `x if condition else y` statements for any scale array\n",
    "\n",
    "cond = np.array([True, True, False, False, False, True, False, False, True, False])\n",
    "# create a resultant array\n",
    "# where a value is taken from x where condition is True\n",
    "np.where(cond, array_funcs_x, array_funcs_y)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we can also call math function for arrays such as mean and standard deviation\n",
    "\n",
    "rand_2d_array = np.random.randn(5, 4)\n",
    "print(\"mean: \" + str(rand_2d_array.mean()) + \" - std: \" + str(rand_2d_array.std()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# you can apply these across any number of axes\n",
    "\n",
    "rand_2d_array.mean(axis=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# for Boolean arrays, you can use .any() and .all() to check for True\n",
    "\n",
    "cond.any()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we can sort our values using .sort()\n",
    "\n",
    "array_funcs_x.sort(0)  # sort our 1D array in ascending order\n",
    "array_funcs_x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# We can use .unique() to create an array of unique values from an array\n",
    "\n",
    "array_funcs_unique = np.array([1, 2, 3, 3, 2, 3, 1, 2, 2, 1])\n",
    "np.unique(array_funcs_unique)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# you can use inid to test membership of values in one array\n",
    "\n",
    "np.in1d(array_funcs_unique, [1, 3, 4, 5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### File I/O"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use np.save() and np.load() to write and read data\n",
    "# data is saved as raw binary data files with the .npy extension\n",
    "\n",
    "np.save('data/meshgrid_proof_out', z)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "np.load('data/meshgrid_proof_out.npy')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# You can also use np.loadtxt and np.genfromtxt...\n",
    "# but this is much easier to do in Pandas :)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n",
    "\n",
    "## Pandas\n",
    "\n",
    "<hr>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import pandas\n",
    "\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Series"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# A Series is like a 1D array with an index (or data labels)\n",
    "\n",
    "series_example = pd.Series([10, 12, 13, 24])\n",
    "series_example  # index is on the left, data values on the right"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# if you want to see the array representation, use .values\n",
    "series_example.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use .index to see the index\n",
    "series_example.index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# you can assign an index\n",
    "series_example_2 = pd.Series([10, 11, 12, 13], index=['a', 'b', 'c', 'd'])\n",
    "series_example_2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use math operators\n",
    "series_example_2 * 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use Boolean operators to examine Series\n",
    "series_example_2 > 11"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a Series from a dictionary\n",
    "series_example_3 = pd.Series({'Pat': 100, 'Tad': 100, 'Anna': 101})\n",
    "series_example_3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# check for null values in your data with .isnull() and .notnull()\n",
    "pd.notnull(series_example_3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# assign a name to the Series\n",
    "series_example_3.name = 'names'\n",
    "series_example_3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# assign a new index to Series\n",
    "series_example_3.index = ['Matt', 'Lynn', 'Lori']\n",
    "series_example_3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# a dataframe can be thought of as a dictionary of series\n",
    "# each series is homogenous\n",
    "# a common method of building dataframes is from dictionary of equal length lists\n",
    "\n",
    "data = {\n",
    "    'places': ['Lexington', 'Louisville', 'Bowling Green'],\n",
    "    'established': [1782, 1778, 1798],\n",
    "    'population': [321959, 771158, 58067]\n",
    "}\n",
    "# create dataframe from data. there is the option to identify columns and an index\n",
    "ky_cities = pd.DataFrame(data, columns=['places', 'established', 'population'], index=[1, 2, 3])\n",
    "ky_cities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# You can examing the columns of a dataframe\n",
    "ky_cities.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# you can examine columns as a series using ['column_name'] or .column_name\n",
    "ky_cities['places']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# you can pull rows using .loc(label) and .iloc(position)\n",
    "ky_cities.iloc[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# add a column\n",
    "ky_cities = pd.DataFrame(ky_cities, columns=['places', 'established', 'population', 'rand'])\n",
    "ky_cities['rand'] = np.random.randn()\n",
    "ky_cities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# delete a column\n",
    "del ky_cities['rand']\n",
    "ky_cities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we can transpose the dataframe\n",
    "ky_cities.T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# you can also pull the dataframe's values\n",
    "ky_cities.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Indexing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# you can create an index object for a dataframe\n",
    "test_index = pd.Index(np.arange(10))\n",
    "test_index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# once created indexes are immutable\n",
    "ky_cities.index[0] = 'one'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Functionality"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# you can reindex a series\n",
    "places_ky = ky_cities['places']\n",
    "places_ky = places_ky.reindex([2, 3, 1])\n",
    "places_ky"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# you can drop entries with .drop\n",
    "places_ky_2 = places_ky.drop(3)\n",
    "places_ky_2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# you can index similar to numpy except you can use integers and labels\n",
    "# if you slice with labels, the endpoint is inclusive\n",
    "places_ky[1:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# indexing into a dataframe is much the same way\n",
    "ky_cities['population'][1]  # population for lexington"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# you can use math operators on the dataframe columns\n",
    "ky_cities.population * 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# you can also use lambdas and apply() to apply changes across a series\n",
    "# you can also use map() and applymap()\n",
    "lambda_func = lambda x: x + 200\n",
    "ky_cities.established.apply(lambda_func)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we can sort the values\n",
    "ky_cities.places.sort_index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we can order values using sort_values(by='column_name')\n",
    "ky_cities.sort_values(by='population')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we can establish rank in data using .rank()\n",
    "ky_cities.rank()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# you can check is all indexes are unique\n",
    "ky_cities.places.is_unique"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Descriptive Stats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we can use .describe to get stats for a dataframe or series\n",
    "ky_cities.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are several summary stats for dataframes and series. Refer to the Pandas documentation for further details.\n",
    "Here are a few:\n",
    "\n",
    "* sum (sum of values)\n",
    "* cumsum (cummulative sum of values)\n",
    "* mean (average)\n",
    "* median (median)\n",
    "* var (variance)\n",
    "* std (standard deviation)\n",
    "* skew (skewness)\n",
    "* kurt (kurtosis)\n",
    "* pct_change (percent changes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we can calculate correlation (strength of relationship between variable between -1 and 1)\n",
    "ky_census = pd.read_csv('data/census_2010_ky.csv')\n",
    "ky_census.corr()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we can also calculate covariance (how two variable vary together)\n",
    "ky_census.cov()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# There is also a .unique method for series\n",
    "ky_census.geoid.unique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# value_counts\n",
    "ky_census.label.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we can also test for membership with isin\n",
    "ky_census.geoid.isin(['21005', '21023'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Missing Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we can check for missing data with isnull() or notnull()\n",
    "missing_data = pd.Series([345, 567, np.nan, 890])\n",
    "missing_data.isnull()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# to drop null values, use dropna()\n",
    "missing_data.dropna()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# we can also fill null values with fillna()\n",
    "missing_data.fillna(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n",
    "\n",
    "## StatsModels\n",
    "\n",
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There is much to cover with statistical modeling in Python with StatsModels which is based on SciPy and NumPy functionality. Here we will perform a simple regression on our census data to see how well the condition of single fathers in Kentucky counties is dependent upon single mothers in the same geographic space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import Pandas and statsmodels\n",
    "import pandas as pd\n",
    "import statsmodels.formula.api as smf"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# read in our census data set\n",
    "census_2010_ky = pd.read_csv('data/census_2010_ky.csv')\n",
    "census_2010_ky.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# examine our descriptive stats\n",
    "census_2010_ky.describe()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# use our order of lease squares model to fit the relationship between single moms and single dads\n",
    "model = smf.ols('sindads~sinmoms + totpop + medage', census_2010_ky).fit()\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<hr>\n",
    "\n",
    "## Additional Materials (for future versions)\n",
    "\n",
    "<hr>\n",
    "\n",
    "* [Pandas Crosstab](http://pbpython.com/pandas-crosstab.html)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<hr>\n",
    "\n",
    "## Resources\n",
    "\n",
    "<hr>\n",
    "\n",
    "**Note:** A lot of the open-source materials are provided by people who develop those materials for a living. So please consider sending them a thank you and if you can, a few buck to support their efforts. Thanks! :)    \n",
    "\n",
    "* [Numpy Docs](https://docs.scipy.org/doc/numpy/)\n",
    "* [Pandas Docs](https://pandas.pydata.org/pandas-docs/stable/)\n",
    "* [Pandas Tricks and Features](https://realpython.com/python-pandas-tricks/)\n",
    "* [Speed Up Pandas](https://realpython.com/fast-flexible-pandas/)\n",
    "* [statsmodels](http://www.statsmodels.org/stable/index.html)\n",
    "* [SciPy Lectures](http://www.scipy-lectures.org/)\n",
    "* [SciKit and Pandas](https://medium.com/dunder-data/from-pandas-to-scikit-learn-a-new-exciting-workflow-e88e2271ef62)\n",
    "* [Python for Data Analysis by Wes McKinney](http://wesmckinney.com/pages/book.html)\n",
    "* [Introduction to Statistical Learning with Python by Thomas Haslwanter](https://github.com/thomas-haslwanter/statsintro_python)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}