data_analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data analysis and visualization in Jupyter Notebooks\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this lesson you will learn \n",
    "- to do some data analysis using pandas\n",
    " - The important concept of *tidy* data\n",
    "- to produce nice plots with seaborn\n",
    "- how other types of media can be embedded in a notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploratory data analysis in Jupyter"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will use four important Python packages\n",
    "1. `numpy` is the fundamental package for scientific computing with Python\n",
    " - for the purposes of this notebook `numpy` is like the engine of a car, it's under the hood running the show\n",
    "2. `pandas` is a more recently developed package for data manipulation and analysis \n",
    " - powerful high-level tool for data exploration\n",
    " - two fundamental data structures which can be applied to many types of data: `Series` and `DataFrames`  \n",
    "\n",
    "3. `matplotlib` is the standard Python package for plotting, \"grandfather of all Python visualization packages\"\n",
    "4. `seaborn` is a higher-level visualization package based on `matplotlib`\n",
    " - while `matplotlib` is extremely powerful, it can also be complex. \n",
    " - sometimes much effort needed to get good-looking graphs\n",
    " - default visualization much more appealing compared to `matplotlib`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will download and process a dataset on Nobel prizes. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "pandas defines a `read_csv` function that can read any CSV file. By giving the URL to the file, pandas will automatically download and parse the file, and return a `DataFrame` object. We need to specify a few options to make sure the dates are parsed correctly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# dataset from http://oppnadata.se/en/dataset/nobel-prizes/resource/f3da8ba9-a17f-4911-9003-4bcef93619cc\n",
    "nobel = pd.read_csv(\"data/nobels.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `nobel` variable now contains a `DataFrame` object, a Pandas data structure that contains 2D tabular data. The `head(n)` method displays the first `n` rows of this table."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nobel.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each column (and row) of the `DataFrame` is a `Series`. Series can be accessed by their names as follows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nobel[\"year\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type(nobel[\"year\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A Series object can produce statistical information about the datums in it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nobel[\"share\"].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's also somewhat smart about the contents of the data it sees so it can summarize non-numerical data as well."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nobel[\"bornCountryCode\"].describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you call a method on the dataframe like count, it will call the same method on each of the series."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nobel.count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The dataset is clearly not quite complete, especially in the death statistics. Possibly because the laureates are still alive?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also use the function `describe()` to request statistics for the entire dataframe but then it will only give statistics for the numerical variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nobel.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To calculate some more elaborate statistics, we first add a column (one Nobel prize per laureate). This will add the column \"number\" to the dataframe with the value 1 for each row."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "nobel[\"number\"] = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Age statistics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's first look at statistics based on the age of prize recipients.  \n",
    "We need to convert the \"born\" column to datetime format. Datetimes are hardly ever recognized correctly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type(nobel[\"born\"][0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "nobel[\"born\"] = pd.to_datetime(nobel[\"born\"], errors ='coerce')\n",
    "# coercion is necessary because the data is a bit messy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type(nobel[\"born\"][0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nobel[\"born\"].dt.year"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now add a column to the DataFrame with age when prize was received."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nobel[\"age\"] = nobel[\"year\"] - nobel[\"born\"].dt.year\n",
    "nobel[[\"surname\",\"age\"]].head(10)\n",
    "#print(nobel[\"age\"].to_string())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now plot a histogram of the age at which laureates receive their prize, using the inbuilt matplotlib support of pandas "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "nobel.plot?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nobel[\"age\"].plot.hist(bins=[20,30,40,50,60,70,80,90,100],alpha=0.6);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To extract the numbers, use the value_counts method"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nobel[\"age\"].value_counts(bins=[20,30,40,50,60,70,80,90,100])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "An alternative plot that is is better for comparing distributions is the box plot.\n",
    "\n",
    "The \"by\" keyword tells by which value the the observations should be **grouped by**, which is the next topic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nobel.boxplot(column=\"age\", by=\"category\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Country statistics - groupby"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We use the powerful `groupby` method to split data into groups, select the column \"number\", and sum up to get the total sum of Nobel prizes by country "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "nobels_by_country = nobel.groupby('bornCountry',sort=True)[\"number\"].sum()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nobels_by_country.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The pandas Series only shows a limited number of rows. Let's print them all"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(nobels_by_country.to_string())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How many prizes have people born in Sweden received?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nobels_by_country[\"Sweden\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Who were they?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nobel.loc[nobel['bornCountry'] == \"Sweden\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We move on. Let's extract four countries and generate some plots"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "countries = np.array([\"France\", \"USA\", \"United Kingdom\", \"Sweden\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "nobel2 = nobel.loc[nobel['bornCountry'].isin(countries)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now group by both `bornCountry` and `category`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nobels_by_country2 = nobel2.groupby(['bornCountry',\"category\"],sort=True).sum()\n",
    "nobels_by_country2[\"number\"].head(50)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can reshape the `DataFrame` a bit using the pivot_table method to create a spreadsheet-like representation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "table = nobel2.pivot_table(values=\"number\",index=\"bornCountry\", columns=\"category\",aggfunc=np.sum)\n",
    "table"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This representation can be used to make a heatmap visualization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <font color=\"red\"> *Exercise 2.1* </font>\n",
    "\n",
    "Gripped by literally morbid curiosity you want to know where Nobel laureates died and presumably are buried.\n",
    "\n",
    "Group the laureates by country of death, use `sort_values` function of pandas.DataFrame to sort them in ascending order and select the 5 countries with most deaths.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <font color=\"green\"> *Solution* </font>\n",
    "\n",
    "> The solution can be found in the solutions.ipynb notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Single country statistics - filter\n",
    "\n",
    "Often the characteristics of a single subset are interesting.\n",
    "\n",
    "To do this we usually use some Python syntax that may look a bit strange at first. \n",
    "\n",
    "See what the following command produces. The `.head(5)` is just there to limit the results to the first 5."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nobel.head(5)[\"bornCountryCode\"] == \"NL\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It uses some Python syntactic sugar to create a Series of boolean variables, True for the rows for which `[\"bornCountryCode\"] == \"NL\"` is true and False for the rest.\n",
    "\n",
    "If we pass this Series back to the DataFrame using the []-brackets we get all those rows from the DataFrame that were True in the Series"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "dutch_nobelists = nobel[nobel[\"bornCountryCode\"] == \"NL\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It is also possible to combine the clauses for more complex filters/queries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "born_and_died_in_sweden = nobel[(nobel[\"bornCountryCode\"] == \"SE\") \n",
    "                                & (nobel[\"diedCountryCode\"] == \"SE\")]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <font color=\"red\"> *Exercise 2.2* </font>\n",
    "\n",
    "Find out which Swedish Nobel prize winner did not die in Sweden.\n",
    "\n",
    "Hints: \n",
    "- the change is a very minor one, don't think too complicated\n",
    "- a NaN means that either the person is still alive or their place of death isn't known)\n",
    " - the `dropna` function can help you filter those but that is not strictly necessary\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <font color=\"green\"> *Solution* </font>\n",
    "\n",
    "> The solution can be found in the solutions.ipynb notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualizing data with seaborn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.heatmap(table,linewidths=.5, annot=True);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Violin plots can reveal trends (but are not very quantitative)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "sns.violinplot(y=\"year\", x=\"bornCountry\",inner=\"stick\", data=nobel2);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also use multiple conditions. Let's extract only physics prizes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nobel3 = nobel.loc[nobel['bornCountry'].isin(countries) & nobel['category'].isin(['physics'])]\n",
    "sns.violinplot(y=\"year\", x=\"bornCountry\",inner=\"stick\", data=nobel3);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Swarmplots display categorical scatterplots with non-overlapping points"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.swarmplot(y=\"year\", x=\"bornCountry\", data=nobel2, alpha=.5);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.factorplot(x=\"bornCountry\", y=\"year\", col=\"category\", data=nobel2, kind=\"swarm\");"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sns.factorplot(x=\"bornCountry\", col=\"category\", data=nobel2, kind=\"count\");"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Seaborn uses a variant of the The Grammar of Graphics, on which most famously simple graphics libraries are based. This is beyond the scope of this notebook though."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Other visualization packages\n",
    "* [Plotly](https://plot.ly/) - commercial online service for creating and sharing visualizations in notebooks\n",
    "* [Bokeh](http://bokeh.pydata.org/en/latest/) - web-based, general-purpose and fast visualization toolkit\n",
    "* [mpld3](http://mpld3.github.io/examples/index.html) - must be seen..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tidy Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You may have observed that the data analysis seemed ridiculously easy in the example. This is in fact quite true, because the data was already **in the right format**.\n",
    "\n",
    "It is said that 80% of a data analysts time is spent on the gritty details of understanding data and getting it  to the right format and the other 20% on the actual analysis.\n",
    "\n",
    "Both Pandas in Python and the Tidyverse packages in R ascribe to the concept of **tidy** data as presented by Hadley Wickham. The original article on it can be found [here](https://www.jstatsoft.org/article/view/v059i10) and is worth a read.\n",
    "\n",
    "To summarize data is tidy when\n",
    "\n",
    "1. each variable forms a column\n",
    "2. each observation forms a row\n",
    "3. each type of observational unit forms a table\n",
    "\n",
    "What an observation is and what a variable is depend on the semantics of the analytics question you are facing.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Melting\n",
    "\n",
    "Let's assume we have some run time statistics from a 1500 m running event.\n",
    "\n",
    "| Runner   | 400m | 800m | 1200m | 1500m |\n",
    "|----------|------|------|-------|-------|\n",
    "| Runner 1 | 64   | 128  | 192   | 240   |\n",
    "| Runner 2 | 80   | 160  | 240   | 300   |\n",
    "| Runner 3 | 96   | 192  | 288   | 360   |\n",
    "\n",
    "This is a classical table generated for displaying information. The issue here is that the column names 400m, 800m, 1200m and 1500m are, in fact variables.\n",
    "\n",
    "To tidy the data we'd like it to be in the following format.\n",
    "\n",
    "| Runner   | distance | time(s) |\n",
    "|----------|----------|---------|\n",
    "| Runner 1 | 400m     | 64      |\n",
    "| Runner 1 | 800m     | 128     |\n",
    "| ....     | ...      | ...     |\n",
    "\n",
    "That way we can perform  aggregate operations on it, particularly we can **filter**  and **group** the data set. The data is also in a format where it is possible to model relationships between variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.DataFrame([\n",
    "        {'Runner': 'Runner 1', 400: 64, 800: 128, 1200: 192, 1500: 240},\n",
    "        {'Runner': 'Runner 2', 400: 80, 800: 160, 1200: 240, 1500: 300},\n",
    "        {'Runner': 'Runner 3', 400: 96, 800: 192, 1200: 288, 1500: 360},\n",
    "         ])\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.melt(df, id_vars=\"Runner\", \n",
    "             value_vars=[400, 800, 1200, 1500], \n",
    "             var_name=\"distance\", \n",
    "             value_name=\"time\"\n",
    "            )\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, to e.g. compute the time spent on each interval, we c"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.sort_values(\"distance\").groupby(\"Runner\").time.diff()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Merging\n",
    "\n",
    "The database world has the concept of joins and tidy data sometimes needs to be joined as well to create a larger DataFrame with redundant data.\n",
    "\n",
    "In Pandas the function to do this is called `merge`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "orders = pd.DataFrame([\n",
    "        {\"Person\": \"Dick\", \"Pizza\": \"Pepperoni\"},\n",
    "        {\"Person\": \"Tom\", \"Pizza\": \"Hawaii\"},\n",
    "        {\"Person\": \"Harry\", \"Pizza\": \"Capricciosa\"}])\n",
    "\n",
    "toppings = pd.DataFrame([\n",
    "        {\"Pizza\": \"Pepperoni\", \"Ingredient 1\": \"Pepperoni\", \"Ingredient 2\": \"Cheese\"},\n",
    "        {\"Pizza\": \"Margherita\", \"Ingredient 1\": \"Cheese\", \"Ingredient 2\": \"Tomato\"},\n",
    "        {\"Pizza\": \"Hawaii\", \"Ingredient 1\": \"Ham\", \"Ingredient 2\": \"Pineapple\"},\n",
    "        {\"Pizza\": \"Capricciosa\", \"Ingredient 1\": \"Mushrooms\", \"Ingredient 2\": \"Ham\"},\n",
    "])\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "person_toppings = orders.merge(toppings, on=\"Pizza\")\n",
    "person_toppings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The default join type is an **inner** join. We could also do a **left**, **outer** or **right** join.\n",
    "\n",
    "If the concepts are not familiar don't worry, you'll run across them sooner or later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "orders.merge(toppings, on=\"Pizza\", how=\"outer\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <font color=\"red\"> *Exercise 2.3* </font>\n",
    "The abovementioned pizza data is not yet **tidy** because the column labels \"Ingredient 1\" and \"Ingredient 2\" are in fact priority values.\n",
    "\n",
    "Make a tidy version of person_toppings. Who have ordered pizzas that have ham in it?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### <font color=\"green\"> *Solution* </font>\n",
    "\n",
    "> The solution can be found in the solutions.ipynb notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Other concepts\n",
    "\n",
    "This tutorial does not aim to be complete. We only have enough time to show you the way.\n",
    "\n",
    "Other important concepts for tidying data are\n",
    "* **splitting** data when a single column holds multiple variables\n",
    " * e.g. \"male10-18\" contains two variables, gender and age group\n",
    "* parsing dates, timestamps and other nontrivial datums\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}