DataSciUF Python I.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Python Basics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sign in here: https://goo.gl/forms/SFsCpO4Vy7PDa2Rg2\n",
    "\n",
    "## Syntax\n",
    "Python is an object oriented scripting language and does not require a specific first or last line (such as <code>public static void main</code> in Java or <code>return</code> in C).\n",
    "\n",
    "There are no curly braces {} to define code blocks or semi-colons ; to end a line.  Instead of braces, indentation is rigidly enforced to create a block of code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Arbitrary indentation can be used within a code block, as long as the indentation is consistent."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Variables and Types"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Variables can be given alphanumeric names beginning with an underscore or letter.  Variable types do not have to be declared and are inferred at run time."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Int"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Float"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Strings can be declared with either single or double quotes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Strings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The scope of variables is local to the function, class, and file in that increasing order of scope.  Global variables can also be declared."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# First function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Modules and Import\n",
    "Files with a .py extension are known as Modules in Python.  Modules are used to store functions, variables, and class definitions.  \n",
    "\n",
    "Modules that are not part of the standard Python library are included in your program using the <code>import</code> statement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# To use Math, we must import it"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Whoops.  Importing the <code>math</code> module allows us access to all of its functions, but we must call them in this way"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Whole.part"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, you can use the <code>from</code> keyword"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# From with pi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using the <code>from</code> statement we can import everything from the math module.  \n",
    "\n",
    "Disclaimer: many Pythonistas discourage doing this for performance reasons.  Just import what you need"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# From ... *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Strings\n",
    "As you may expect, Python has a powerful, full featured string module.  \n",
    "\n",
    "### Substrings\n",
    "Python strings can be substringed using bracket syntax"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Print 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Python is a 0-index based language.  Generally whenever forming a range of values in Python, the first argument is inclusive whereas the second is not, i.e. <code>mystring[11:25]</code> returns characters 11 through 24.\n",
    "\n",
    "You can omit the first or second argument"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Print first 4"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Characters before 9th"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Characters after 27th"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Omitting start and end"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using negative values, you can count positions backwards"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Print almost last 4 characters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### String Functions\n",
    "Here are some more useful string functions\n",
    "#### find"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Find \"Gators\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks like nothing was found.  -1 is returned by default."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### lower and upper"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### replace"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that replace returned a new string.  Nothing was modified in place"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### split"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### join\n",
    "\n",
    "The <code>join</code> is useful for building strings from lists or other iterables.  Call <code>join</code> on the desired separator"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Join with spaces"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For more information on string functions:\n",
    "\n",
    "https://docs.python.org/2/library/stdtypes.html#string-methods"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Structures\n",
    "### Lists\n",
    "The Python standard library does not have traditional C-style fixed-memory fixed-type arrays.  Instead, lists are used and can contain a mix of any type.\n",
    "\n",
    "Lists are created with square brackets []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# mylist list of 5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# append 6"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# extend the list with the contents of another list\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# insert the number 7 at index 6\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# removes the first matching occurence \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# by default, the last item in the list is removed and returned\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# pops at at index\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# len()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# the range function returns a list from -3 inclusive to 0 non inclusive\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# default list sorting. When more complex objects are in the list, arguments can be used to customize how to sort\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# reverse the list\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For more information on Lists:\n",
    "\n",
    "https://docs.python.org/2/tutorial/datastructures.html#more-on-lists"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tuples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Python supports n-tuple sequences.  These are non-mutable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# mytuple"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# access an item"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# results in error"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###Sets\n",
    "Python includes the set data structure which is an unordered collection with no duplicates"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# in"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# more in"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# set arithmetic"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# set and"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# OR"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# XOR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dictionaries\n",
    "Python supports dictionaries which can be thought of as an unordered list of key, value pairs.  Keys can be any immutable type and are typically integers or strings.  Values can be any object, even dictionaries.\n",
    "\n",
    "Dictionaries are created with curly braces {}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# mydict"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Florida"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# delete"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# assignment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# appending"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# keys"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conditionals\n",
    "Python supports the standard if-else-if conditional expression"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Loops\n",
    "Python supports for, foreach, and while loops\n",
    "### For (counting)\n",
    "Traditional counting loops are accomplished in Python with a combination of the <code>for</code> key word and the <code>range</code> function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#with one argument, range produces integers from 0 to 9\n",
    "    print x"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# with two argumentts, range produces integers from 5 to 11\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# with three arguments, range starts at 1 and goes in steps of 3 until greater than 12"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# can use a negative step size as well\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# with a positive step, all values are less than 1. No integers are produced\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# same goes for a negative step as all values are less than 2\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Foreach\n",
    "As it turns out, counting loops are just foreach loops in Python.  The <code>range</code> function returns a list of integers over which <code>for in</code> iterates.  This can be extended to any other iterable type"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# iterate over a list of strings"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### While\n",
    "Python supports standard <code>while</code> loops"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Python does not have a construct for a do-while loop, though it can be accomplished using the <code>break</code> statement"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Functions\n",
    "Functions in Python do not have a distinction between those that do and do not return a value.  If a value is returned, the type is not declared.\n",
    "\n",
    "Functions can be declared in any module without any distinction between static and non-static.  Functions can even be declared within other functions\n",
    "\n",
    "The syntax is the following"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# define function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# define player, name, number"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Functions can have optional arguments if a default value is provided in the function signature"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "\n",
    "    \n",
    "# no team argument supplied"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# supplying all three arguments"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Python functions can be called using named arguments, instead of positional"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### \\*args and \\**kwargs\n",
    "In Python, there is a special deferencing scheme that allows for defining and calling functions with argument lists or dictionaries.\n",
    "\n",
    "#### \\*args"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# calling with *args"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Argument lists can also be used in defining a function as such"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# define foo, *args, print *args"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### **kwargs\n",
    "Similarly, we can define a dictionary of named parameters"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# calling with kwargs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just as before, we can define a function taking an arbitrary dictionary"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### return\n",
    "In Python functions, an arbitrary number of values can be returned"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# def sum, return a + b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Science Tutorial\n",
    "\n",
    "Sign in here: https://goo.gl/forms/SFsCpO4Vy7PDa2Rg2\n",
    "\n",
    "Now that we've covered some Python basics, we will begin a tutorial going through many tasks a data scientist may perform.  We will obtain real world data and go through the process of auditing, analyzing, visualing, and building classifiers from the data.\n",
    "\n",
    "We will use a database of selected disease statistics of various contries from the Global Health Observatory. The data is organized by country and year, with the number of specific incidents of each disease listed. The attributes and domain of each entry are described by the table below:\n",
    "\n",
    "| Attribute                     | Domain                          |\n",
    "|-------------------------------|---------------------------------|\n",
    "| 1. Country                    | String                          |\n",
    "| 2. Year                       | Year (2009-2014)                |\n",
    "| 3. T.b. gambiense             | Integer                         |\n",
    "| 4. T.b. rhodesiense           | Integer                         |\n",
    "| 5. Cholera                    | Integer                         |\n",
    "| 6. Meningitis (suspected)     | Integer                         |\n",
    "| 7. Congenital Rubella         | Integer                         |\n",
    "| 8. Diphtheria                 | Integer                         |\n",
    "| 9. Japanese encephalitis      | Integer                         |\n",
    "| 10. Leprosy                   | Integer                         |\n",
    "| 11. Malaria                   | Integer                         |\n",
    "| 12. Measles                   | Integer                         |\n",
    "| 13. Mumps                     | Integer                         |\n",
    "| 14. Neonatal Tetanus          | Integer                         |\n",
    "| 15. Pertussis                 | Integer                         |\n",
    "| 16. Plague                    | Integer                         |\n",
    "| 17. Poliomyelitis             | Integer                         |\n",
    "| 18. Rubella                   | Integer                         |\n",
    "| 19. Total Tetanus             | Integer                         |\n",
    "| 20. Tuberculosis              | Integer                         |\n",
    "| 21. Yellow Fever              | Integer                         |\n",
    "| 22. Cutaneous Leishmaniasis   | Integer                         |\n",
    "| 23. Visceral Leishmaniasis    | Integer                         |\n",
    "\n",
    "For more information on this data set:\n",
    "http://apps.who.int/gho/data/node.home\n",
    "\n",
    "## Obtaining the Data\n",
    "Lets begin by programmatically obtaining the data.  Here I'll define a function we can use to make HTTP requests and download the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#define download_file function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we'll specify the url of the file and the file name we will save to"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "#specify url and filename"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And make a call to <code>download_file</code>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#call our download_file function"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note:**  If you see an InsecurePlatformWarning message, ignore it. More info can be found here: https://urllib3.readthedocs.org/en/latest/security.html#insecureplatformwarning\n",
    "\n",
    "Now this might seem like overkill for downloading a single, small csv file, but we can use this same function to access countless APIs available on the World Wide Web by building an API request in the url.\n",
    "\n",
    "## Wrangling the Data\n",
    "Now that we have some data, lets get it into a useful form.  For this task we will use a package called pandas. pandas is an open source, BSD-licensed library providing high-performance, easy-to-use data structures and data analysis tools for Python.  The most fundamental data structure in pandas is the dataframe, which is similar to the data.frame data structure found in the R statistical programming language.\n",
    "\n",
    "For more information: http://pandas.pydata.org\n",
    "\n",
    "pandas dataframes are a 2-dimensional labeled data structures with columns of potentially different types.  Dataframes can be thought of as similar to a spreadsheet or SQL table.\n",
    "\n",
    "There are numerous ways to build a dataframe with pandas.  Since we have already attained a csv file, we can use a parser built into pandas called <code>read_csv</code> which will read the contents of a csv file directly into a data frame.\n",
    "\n",
    "For more information: http://pandas.pydata.org/pandas-docs/dev/generated/pandas.io.parsers.read_csv.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#import pandas and load dataset into a frame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lets take a look at some simple statistics for the **Cholera** column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#describe cholera column"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Referring to the documentation, the data contains 1164 entries. However, if we take a look at the \"count\" section, it shows only 245 entries. This is because the original data is filled with empty strings, which pandas automatically converts to Numpy's <code>nan</code> datatype, or \"Not a Number\". \n",
    "\n",
    "Lets take a look at another column, this time **Pertussis**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#describe pertussis column"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well at least the name is correct.  We were expecting a mean and standard deviation, and now the data type is an object.  \n",
    "\n",
    "Whats up with our data?\n",
    "\n",
    "We have arrived at arguably the most important part of performing data science: dealing with messy data.  One of most important tools in a data scientist's toolbox is the ability to audit, clean, and reshape data.  The real world is full of messy data and your sources may not always have data in the exact format you desire.\n",
    "\n",
    "In this case we are working with csv data, which is a relatively straightforward format, but this will not always be the case when performing real world data science.  Data comes in all varieties from csv all the way to something as unstructured as a collection of emails or documents.  A data scientist must be versed in a wide variety of technologies and methodologies in order to be successful.\n",
    "\n",
    "Now, lets do a little bit of digging into why were are not getting a numeric pandas column"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#find unique values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using <code>unique</code> we can see that '0 0', '5 5', and '2 2' all appear as distinct values in this series. Because of the space between the numbers, Python has classified these as *strings* rather than *integers*. Indeed, it's not immediately obvious that these were meant to be legitimate entries in the first place.\n",
    "\n",
    "Lets see what we can do with these unrecognized values. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#convert column to numeric values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we have attempted to convert the **Pertussis** series to a numeric type.  Lets see what the unique values are now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#find new unique values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The decimal point after each number means that it is an integer value being represented by a floating point number.  Now instead of our pesky *strings* we have <code>nan</code> (not a number).  <code>nan</code> is a construct used by pandas to represent the absence of value.  It is a data type that comes from the package numpy, used internally by pandas, and is not part of the standard Python library.\n",
    "\n",
    "Now that we have <code>nan</code> values in place of strings, we can use some nice features in pandas to deal with these missing values.\n",
    "\n",
    "What we are about to do is what is called \"imputing\" or providing a replacement for missing values so the data set becomes easier to work with.  There are a number of strategies for imputing missing values, all with their own pitfalls.  In general, imputation introduces some degree of bias to the data, so the imputation strategy taken should be in an attempt to minimize that bias.\n",
    "\n",
    "Here, we will simply ignore all of the <code>nan</code> values, however other strategies such as replacing the <code>nan</code>'s with the mean of the data are also commonly used."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#convert whole data frame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<code>health_data.mean().round()</code> will take the mean of each column (this computation ignores the currently present nan values), then round, and return a dataframe indexed by the columns of the original dataframe.\n",
    "\n",
    "This function can be used to replace all missing values with the mean of each column. In this tutorial however, we will not use this method, because the large number of missing values would greatly skew our standard deviations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#find mean values for imputing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have figured out how to impute these missing values, lets start over and quickly apply this technique to the entire dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#quickly load and convert data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#check unique values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Structurally, Pandas dataframes are a collection of Series objects sharing a common index.  In general, the Series object and Dataframe object share a large number of functions with some behavioral differences.  In other words, whatever computation you can do on a single column can generally be applied to the entire dataframe.\n",
    "\n",
    "Now we can use the dataframe version of <code>describe</code> to get an overview of all of our data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "collapsed": false,
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#overview description of data frame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "## Visualizing the Data\n",
    "Another important tool in the data scientist's toolbox is the ability to create visualizations from data.  Visualizing data is often the most logical place to start getting a deeper intuition of the data.  This intuition will shape and drive your analysis.\n",
    "\n",
    "Even more important than visualizing data for your own personal benefit, it is often the job of the data scientist to use the data to tell a story.  Creating illustrative visuals that succinctly convey an idea are the best way to tell that story, especially to stakeholders with less technical skillsets.\n",
    "\n",
    "Here we will be using a Python package called ggplot (https://ggplot.yhathq.com).  The ggplot package is an attempt to bring visuals following the guidelines outlayed in the grammar of graphics (http://vita.had.co.nz/papers/layered-grammar.html) to Python.  It is based off of and intended to mimic the features of the ggplot2 library found in R.  Additionally, ggplot is designed to work with Pandas dataframes, making things nice and simple. \n",
    "\n",
    "We'll start by doing a bit of setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "ename": "ImportError",
     "evalue": "No module named ggplot",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mImportError\u001b[0m                               Traceback (most recent call last)",
      "\u001b[1;32m<ipython-input-72-67eff948785b>\u001b[0m in \u001b[0;36m<module>\u001b[1;34m()\u001b[0m\n\u001b[0;32m      2\u001b[0m \u001b[0mget_ipython\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mmagic\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34mu'matplotlib inline'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      3\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[1;32m----> 4\u001b[1;33m \u001b[1;32mfrom\u001b[0m \u001b[0mggplot\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[1;33m*\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      5\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      6\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mwarnings\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mImportError\u001b[0m: No module named ggplot"
     ]
    }
   ],
   "source": [
    "# The following line is NOT Python code, but a special syntax for enabling inline plotting in IPython\n",
    "%matplotlib inline \n",
    "\n",
    "from ggplot import *\n",
    "\n",
    "import warnings\n",
    "\n",
    "# ggplot usage of pandas throws a future warning\n",
    "warnings.filterwarnings('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So we enabled plotting in IPython and imported everything from the ggplot package.  Now we'll create a plot and then break down the components"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#create our first plot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A plot begins with the <code>ggplot</code> function.  Here, we pass in the cancer_data pandas dataframe and a special function called <code>aes</code> (short for aesthetic).  The values provided to <code>aes</code> change depending on which type of plot is being used.  Here we are going to make a histogram from the **human African trypanosomiasis (T.b. rhodesiense)** column in health_data, so that column name needs to be passed as the x parameter to <code>aes</code>.\n",
    "\n",
    "The grammar of graphics is based off of a concept of \"geoms\" (short for geometric objects).  These geoms provide granular control of the plot and are progressively added to the base call to <code>ggplot</code> with + syntax.\n",
    "\n",
    "\n",
    "Lets say we wanted to show the mean number of cases on this plot.  We could do something like the following"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#plot with vline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, each geom has its own set of parameters specific to the appearance of that geom (also called aesthetics).\n",
    "\n",
    "Lets try a scatter plot to get some multi-variable action"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#scatter plot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With a simple aesthetic addition, we can see how these two variables have changed over the past six years."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "#color scatter plot"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By adding <code>color = 'Year'</code> as a parameter to the aes function, we now give a color to each unique value found in that column and automatically get a legend.\n",
    "\n",
    "We can also do things such as add a title or change the axis labeling with geoms"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false,
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "#plot with custom title and axes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I highly encourage you to check out https://ggplot.yhathq.com/docs/index.html to see all of the available geoms.  The best way to learn is to play with and visualize the data with many different plots and aesthetics."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There doesn't seem to be much correlation between these two variables as a function of the year, however it is difficult to say with such simple statistics. If we wanted to analyze relationships between all of the variables in our data set, we would need to analyze a very high-dimensional space. Using some more complex data analysis and machine learning techniques, we may be able to extract higher order correlations in this high-dimensional data space. This will be the topic of our next workshop, Python II."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summary and Python II\n",
    "\n",
    "In the first half of our two-part Python series, we've learned about variables, data structures, functions, and graphing. While we have introduced these topics in the context of data science with Python, they are central to programming in any language and in any context. We have also laid the foundation for programmatically obtaining, cleaning, and visualizing data sets.\n",
    "\n",
    "Now that we have an understanding of how to obtain and visualize some simple statistical information contained in a dataset, we've set the stage for machine learning and data analysis. These topics will be covered in depth in our next workshop, Python II.\n",
    "\n",
    "Post workshop survey: https://goo.gl/forms/gxGfc9LzDIev11R53"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}