pandas.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5fCEDCU_qrC0"
   },
   "source": [
    "# Pandas\n",
    "\n",
    "pandas is a software package written for the Python programming language for data manipulation and analysis. In particular, it offers data structures and operations for manipulating tabular data and time series. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "rMExucF44s_c"
   },
   "source": [
    "## Learning objective\n",
    "\n",
    "Learn how to use pandas to work through the steps social scientists often take to process and analyze raw data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "GJBs_flRovLc"
   },
   "source": [
    "## 1. Import pandas\n",
    "\n",
    "To begin, let's look at a few ways to import pandas"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "gJr_9dXGpJ05"
   },
   "outputs": [],
   "source": [
    "# import specific classes and functions: useful if you know you'll only be needing the most common parts of pandas\n",
    "from pandas import DataFrame, read_csv\n",
    "\n",
    "# General syntax to import a package but no functions: \n",
    "#   import (package) as (give the package a nickname/alias)\n",
    "# Most pandas users will import the entire package and give it the short nickname \"pd\" as follows:\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we've seen how to import pandas, let's go over how to use it to work with data using a real example. We'll be using the same survey data we saw from Part 1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "9c2d00fn3zDE"
   },
   "source": [
    "## 3. Import csv data\n",
    "\n",
    "As mentioned in the introduction, pandas is designed to work with tabular data. Tabluar data is most commonly provided to researchers in a simple spreadsheet format known as csv, which stands for *comma separated values*. Because of this, pandas has a function to load tabular data from a csv file:  `read_csv`. The most basic way to use the `read_csv` function is to simply give it the path to the csv file you want to load. Here, we will load the example survey data, which is contained in the file \"survey_responses.csv\" in our GitHub repo:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "7X7PRKhX4kJV"
   },
   "outputs": [],
   "source": [
    "# The following path is valid for our Binder setup. If you are running this notebook in your local machine instead and have the file downloaded to a different location, you may need to change this path.\n",
    "Location = './data/survey_responses.csv'\n",
    "df = pd.read_csv(Location)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "oRsvGySgjl6H"
   },
   "source": [
    "`read_csv` loads the data into a structure known as a **DataFrame**, as we can see if we check the type of the variable `df` that we created:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "8ZmhPaCsjl6I"
   },
   "outputs": [],
   "source": [
    "type(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "cE0uTGw6jl6J"
   },
   "source": [
    "Think of a **DataFrame** as a table or spreadsheet that you can work with inside your Python code! Like any table, it contains data organized into rows and columns.\n",
    "\n",
    "Now you might be wondering what the data actually looks like. The best way to find out is to use the `display` function, which will show a sample of data from the DataFrame. This is always the first thing you should do after loading a new dataset in pandas!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "NGGMN_Ti28h3"
   },
   "outputs": [],
   "source": [
    "display(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "iwv6zshx5ylG"
   },
   "source": [
    "Notice that the top row and the leftmost column are special. The top row is bolded and contains text that looks different from the rest of the table; this is because it is the *header* of the DataFrame. The header does not contain real data; instead, it contains the names of each column. Meanwhile, the leftmost column does not have a name and simply contains consecutive numbers [0,1,2,3,4,...]. This is the *index* of the DataFrame; by default this works likes row numbers in an Excel file, but more generally you can think of the index as containing the name of each row. This index is also similar to the primary key of a sql table with the exception that an index is allowed to have duplicates.\n",
    "\n",
    "You might be wondering where the header and index came from. In pandas, the default behavior is to do the following:\n",
    "- Treat the first row of the CSV file as the header\n",
    "- Create an index from scratch containing consecutive numbers\n",
    "\n",
    "But this behavior is customizable via parameters in `read_csv`, and you may need to customize it depending on what your CSV file looks like. In this case, you might have noticed that the file already contains a column that looks like an index (\"Response_ID\"), which means that pandas' automatically-generated index is redundant. Therefore, we might want to tell pandas to use the \"Response_ID\" column as the index. We can do this using the `index_col` parameter of `read_csv`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(Location, index_col=\"Response_ID\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "display(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that pandas is no longer generating a redundant index column; instead, \"Response_ID\" is being treated as the index, as evidenced by the fact that it has been moved to the left and bolded.\n",
    "\n",
    "This was just one example of how you might need to adjust your usage of `read_csv` depending on what's inside your file. There are other common scenarios, for example some CSV files don't have a header. We won't go into that in detail here, but again, remember you can use the `help` function to see what parameters are available to you."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "0Qfk96MX10T0"
   },
   "outputs": [],
   "source": [
    "help(read_csv)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "One last thing we want to point out before we move on: notice that in some cells in the table contain the strange-looking data \"NaN\". NaN is short for \"Not A Number\" and is pandas' way of representing *missing data*. Think of NaNs as equivalent to an empty cell in an Excel file. The meaning of missing data depends on the context of the dataset. In this case, since this data is from a survey and each column represents a survey question, we can infer that in this case missing data means that the survey taker did not give an answer for that question. In research, we often want to do something special to handle missing data; we will return to this later in the workshop."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OwuxHmxllTwN"
   },
   "source": [
    "## 4. Checking the structure of the data\n",
    "Usually we first do a couple of sanity checks to make sure the data we imported is in good shape and understand its high level structure. \n",
    "\n",
    "Usually we make sure the data type of each variable is sensible, check the total number of observations, and the number of variables in the data set.\n",
    "\n",
    "Let's take a look at the example data set. First, we'll check data types using the `dtypes` variable.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "yCC64Uqy6nAY"
   },
   "outputs": [],
   "source": [
    "# Check data type of the columns\n",
    "df.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NmbZVWO2BSLU"
   },
   "source": [
    "Next we'll check the number of observations (rows) in the dataset. This can be done by examining the length of the DataFrame's *index*:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "QIoJ_maABVOg"
   },
   "outputs": [],
   "source": [
    "len(df.index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IWQpJOKXBkCm"
   },
   "source": [
    "Similarly, we can check the number of variables (columns) in the dataset by examining the length of the DataFrame's *columns* list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "iwu71C2hBmVG"
   },
   "outputs": [],
   "source": [
    "len(df.columns)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "g0z8mSwz6ZvH"
   },
   "source": [
    "## 5. Slicing and Indexing\n",
    "Usually, we will not be working with an entire data set all at the same time. Instead, we usually want to pick out specific rows or columns to analyze and work with. In pandas, this can be done using *slicing and indexing*."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SwTB_xAcjl6N"
   },
   "source": [
    "The most basic indexing operation is selecting a specific column. This can be done using standard Python indexing syntax (square brackets) and giving it the name of the column you want. For example, the following code will select the \"Age\" column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Qr7i1ojWjl6N"
   },
   "outputs": [],
   "source": [
    "age_column = df['Age']\n",
    "display(age_column)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bNOAR5qfjl6N"
   },
   "source": [
    "The syntax for selecting a row is similar, except that you need to tell pandas that you are trying to access a row, since indexing defaults to columns. To do this, use the `loc` variable:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "w0IFERlnjl6O"
   },
   "outputs": [],
   "source": [
    "fifth_row = df.loc[5]\n",
    "display(fifth_row)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "hFNKSwHOjl6O"
   },
   "source": [
    "Both row indexing and column indexing allow you to select multiple rows or columns at once. To do this, you can provide a **list** of row or column names, instead of just one:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "WvDmj7y6HwJi"
   },
   "outputs": [],
   "source": [
    "multi_columns = df[['Age', 'Living_Location']]\n",
    "display(multi_columns)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "m_aQBFI3jl6O"
   },
   "outputs": [],
   "source": [
    "multi_rows = df.loc[[1,2,3,4]]\n",
    "display(multi_rows)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "yAraQcG2jl6O"
   },
   "source": [
    "If a DataFrame's index is numerical (as it is both in our case and in the default settings), Python's **slicing** syntax can also be used to select a range of rows. This can simplify your code a lot because you can specify a range rather than writing out full list of indices! For example, the following code is equivalent to the previous code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "y92xDXcYjl6P"
   },
   "outputs": [],
   "source": [
    "multi_rows = df.loc[1:4] # important difference from regular python: pandas slices include *both* indices!\n",
    "display(multi_rows)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2NYv4_YSKO4f"
   },
   "source": [
    "Additionally, we may sometimes want to pick out a **single** value from the DataFrame. For example, suppose we want to find the Age in the fifth row. We can do this by giving both a row and column to the `loc` indexer. Note that the row and column **must** be given in that order (you cannot give the column first) and should be separated by a comma, as shown below:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "iSrJcPshKg1N"
   },
   "outputs": [],
   "source": [
    "fifth_age = df.loc[5,\"Age\"]\n",
    "print(fifth_age)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xiAc_0QGjl6P"
   },
   "source": [
    "Finally, the same syntax can be used to select multiple rows **and** columns at the same time; you simply need to provide a list or range of rows, followed by a list of column names, separated by a comma.\n",
    "\n",
    "Let's get the first 4 rows, for columns 'Age' and 'Moving_Times':"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "4Y3Ywvp8jl6Q"
   },
   "outputs": [],
   "source": [
    "multi_row_and_col = df.loc[1:4,[\"Age\", \"Moving_Times\"]]\n",
    "display(multi_row_and_col)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "CqhRO0k3BvM2"
   },
   "source": [
    "## Exercise 1\n",
    "\n",
    "Write code to fetch the following data:\n",
    "1. The Hometown in the 10th row\n",
    "2. The Parental_Income and Race_Ethnicity for rows 100 through 110\n",
    "3. The School_Year in rows 8, 42, 47, and 99\n",
    "\n",
    "(Note: each problem should be solved with a *single* indexing operation; for example, you should not need 4 separate indexing operations to answer question 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "qAnp6yhjCL68"
   },
   "outputs": [],
   "source": [
    "# Write your answers here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6lBVwk2sjl6Q"
   },
   "source": [
    "## 6. Summarizing column data\n",
    "Previously, we sanity checked our data by checking its structure (data types and number of rows and columns). A second common step in preliminary exploration of the data is to examine individual columns (with the help of indexing, as covered previously). pandas can help with this process by offering several methods to *summarize* the data in a column."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MdRkt4Prjl6Q"
   },
   "source": [
    "### Summarizing numerical data\n",
    "Some columns contain numerical data. For example, ages in the \"Age\" column are represented as numbers. For numerical data, pandas offers a number of methods that implement common mathematical summary functions, such as finding the mininum, maximum, and arithmetic mean. We can use these, for instance, to find out who the youngest and oldest participants in the survey were, and what the average age was:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "M__x7ovSjl6R"
   },
   "outputs": [],
   "source": [
    "print(df[\"Age\"].min()) # what's the age of the youngest participant?\n",
    "print(df[\"Age\"].max()) # what's the age of the oldest participant?\n",
    "print(df[\"Age\"].mean()) # what's the average age of participants?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_JT9EoIUjl6R"
   },
   "source": [
    "### Summarizing categorical data\n",
    "Mathematical operations like min and max are useful for quickly summarizing numerical data. But not all columns are numerical. For example, the \"College\" column contains text. But notice that the text in the \"College\" column is not just any arbitrary text! It only has specific, fixed values, because in the original survey this was a multiple-choice question. Therefore, \"College\" is what data scientists refer to as a *categorical variable*: one that can only take on values from a fixed, finite set of possibilities. To summarize categorical data, researchers generally want to know what is the set of values the variable can take. In pandas, this can be done using the `unique` method:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "Nib9a0S_jl6R"
   },
   "outputs": [],
   "source": [
    "print(df[\"College\"].unique())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "jRw50ceO_b_k"
   },
   "source": [
    "## Exercise 2\n",
    "\n",
    "- What are the possible values for \"Self_Confidence\" and \"Parental_Income\"?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "IsL_ZU9BAiSY"
   },
   "outputs": [],
   "source": [
    "print(____________) # fill in your code for Self_Confidence here\n",
    "print(____________) # fill in your code for Parental_Income here"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 3\n",
    "While the `unique` method is most commonly used to look at categorical text data, it can actually be applied to any data, even numerical data! One possible reason you might want to use the `unique` method on numerical data is if you have integer data representing a count or other finite data (for example, Moving_Times and Age in our data) and want to see what unique values there are. Let's try running `unique` on the Moving_Times column. How many unique values are there? Do you notice anything here that looks like a problem?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# fill in your code below\n",
    "print(____________)\n",
    "# Looking at the printed output, do you notice any potential problems in the data?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "vIQI7zsSC2FF"
   },
   "source": [
    "## 7. Querying using conditions based on column values\n",
    "\n",
    "Now we can start looking at more advanced data analysis. Previously, we selected subsets of rows by giving the `loc` indexer a list or range of row indices. Most often, however, we don't know ahead of time which rows we want to analyze; instead, we more often want to select rows based on some conditions. For example, if we were interested in studying how undergraduate freshmen specifically answered our survey, we would want to select only the rows where \"School_Year\" is \"1st year undergraduate\". To achieve this, the `loc` indexer can actually be used with conditions, not just with indices! \n",
    "\n",
    "This is best explained through examples. Let's start with the example just discussed, selecting only rows representing undergraduate freshmen. The syntax for the comparison would be `df['School_Year'] == \"1st year undergraduate`, and we can pass this comparison directly to `loc`, as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "IHWjCNENBH43"
   },
   "outputs": [],
   "source": [
    "freshmen = df.loc[df['School_Year'] == \"1st year undergraduate\"]\n",
    "display(freshmen)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pW5ot28Fjl6S"
   },
   "source": [
    "Notice that this syntax is similar to the filtering syntax used by numpy (as seen in the previous Python workshop) and by R. Alternatively, if you are familiar with SQL, you can think of the above code as being equivalent to the SQL query `select * from df where School_Year=\"1st year undergraduate\"`.\n",
    "\n",
    "Like in numpy, you can include multiple comparisons in a single query, combining them using operators `&` for \"and\" and `|` for \"or\". So, for example, if we wanted only responses from freshmen *or* sophomores:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "B4xk3-AJDAPA"
   },
   "outputs": [],
   "source": [
    "fresh_soph = df.loc[(df['School_Year']==\"1st year undergraduate\")|(df['School_Year']==\"2nd year undergraduate\")]\n",
    "display(fresh_soph)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And if we wanted responses only from freshmen in the College of Human Ecology:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "eco_fresh = df.loc[(df['School_Year']==\"1st year undergraduate\")&(df['College']==\"College of Human Ecology\")]\n",
    "display(eco_fresh)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6QlU0jFUjl6S"
   },
   "source": [
    "Notice that when we run a query, the matching rows get returned in a new DataFrame, which is a subset of the original, containing only rows that matched the query. We can confirm that the result is still a DataFrame by checking the type of the variable:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "hG979xWhjl6S"
   },
   "outputs": [],
   "source": [
    "type(eco_fresh)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OppCv7_Ojl6T"
   },
   "source": [
    "This means that we can run all of the previously described DataFrame methods on the query results! For example, this can be useful for finding out how many rows matched the query:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "RuZIkruEjl6T"
   },
   "outputs": [],
   "source": [
    "print(len(eco_fresh.index)) # we can use the same code we saw earlier for counting rows, because eco_fresh is still a DataFrame"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "90HyVcgYjl6T"
   },
   "source": [
    "## Exercise 4\n",
    "\n",
    "How many freshmen in the data have experienced hunger (i.e., answered \"Yes\" for Hunger_Experience)?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "X-DlbQVWjl6T"
   },
   "outputs": [],
   "source": [
    "fresh_hunger = df.loc[______________] # first, fill in your query here\n",
    "print(_____________) # then, fill in the code to count the number of rows"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 5\n",
    "Do more seniors live on-campus or off-campus?\n",
    "\n",
    "(Hint: you will need to run two queries)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write your solution below\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "yZ-cXW6P6Opk"
   },
   "source": [
    "## 8. Modifying and adding rows\n",
    "Queries aren't just useful for accessing data; we may sometimes also want to *modify* data that matches a query! One common reason to do this is for data preprocessing. Let's consider an example that's quite common in survey work. The \"Self_Confidence\" column contains answers to a multiple-choice question meant to represent a scale of *intensity*, with seven options representing increasing levels of confidence: \"Disagree strongly\", \"Disagree somewhat\", \"Disagree slightly\", \"Don't know / not applicable\", \"Agree slightly\", \"Agree somewhat\", and \"Agree strongly\". We might wish to turn this into a numerical Likert Scale, with values from 0 to 6 where 0 is \"Disagree strongly\" and 6 is \"Agree strongly\". This is commonly done in social science research to enable more quantitative comparisons of responses on scale-based questions like this one.\n",
    "\n",
    "Let's consider one step in this Likert scale conversion, changing \"Disagree strongly\" to 0. We can achieve this with the following steps:\n",
    "\n",
    "1. Write a query to find all instances of the thing you want to preprocess (in this case, rows where the Self_Confidence is \"Disagree strongly\") (See Section 7)\n",
    "2. Use row-and-column indexing to combine this query (which selects certain rows) with column selection (See Section 5)\n",
    "3. Use Python assigment syntax (the \"=\" operator) to set the new value (in this case, 0)\n",
    "\n",
    "Now let's look at the code for these steps:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "EVP5mQ1R6Tl6"
   },
   "outputs": [],
   "source": [
    "# Step 1: the query syntax for selecting rows where Self_Confidence is \"Disagree strongly\"\n",
    "query = (df[\"Self_Confidence\"] == \"Disagree strongly\")\n",
    "# Step 2 and 3: row-and-column indexing followed by new value assignment\n",
    "df.loc[query,\"Self_Confidence\"] = 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# To confirm that it worked, show that there now exist some rows where \"Self_Confidence\" is 0\n",
    "print(len(df.loc[df[\"Self_Confidence\"] == 0].index))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that although for clearer illustration we split the code into multiple lines to correspond to the multiple steps, in practice most pandas users will do the entire operation in a single line of code (this concision is part of what makes pandas so powerful!). Thus, the following single line of code would have been equivalent to what we did above:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.loc[(df[\"Self_Confidence\"] == \"Disagree strongly\"),\"Self_Confidence\"] = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 6\n",
    "Repeat this process to convert the remaining 6 response options. The conversions should be as follows:\n",
    "- Disagree somewhat --> 1\n",
    "- Disagree slightly --> 2\n",
    "- Don't know / not applicable --> 3\n",
    "- Agree slightly --> 4\n",
    "- Agree somewhat --> 5\n",
    "- Agree strongly --> 6"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Finish the exercise by filling in the blanks below:\n",
    "df.loc[___________________,\"Self_Confidence\"] = 1\n",
    "df.loc[___________________,\"Self_Confidence\"] = 2\n",
    "df.loc[___________________,\"Self_Confidence\"] = 3\n",
    "df.loc[___________________,\"Self_Confidence\"] = 4\n",
    "df.loc[___________________,\"Self_Confidence\"] = 5\n",
    "df.loc[___________________,\"Self_Confidence\"] = 6\n",
    "# Once you've finished your solution, the following code should be \n",
    "# able to run successfully now that Self_Confidence contains numerical data\n",
    "print(df[\"Self_Confidence\"].mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 7\n",
    "Another common reason for modifying rows is data cleaning. Let's work through an example using the Moving_Times column. Earlier in Exercise 3, we found that while most participants gave a numerical answer for the moving times question (which is what we wanted), some participants instead filled in free text. This prevents us from running numerical summarization methods like min, max, and mean. We can fix this by modifying the Moving_Times data, similar to what we did previously for Self_Confidence. For each non-numerical answer, we will replace it with the best possible numerical interpretation we can come up with; for example, we can replace \"twice\" with 2.\n",
    "\n",
    "As a reminder, in Exercise 3 you should have discovered that the following non-numerical answers exist in the data:\n",
    "- 'twice' \n",
    "- 'None' \n",
    "- 'III'\n",
    "- '1 (but when my family was renovating my house we briefly moved into another house in the same town then moved back, so if that counts as moving then 3 times)'\n",
    "- '2-3 times'\n",
    "- '2 times'\n",
    "\n",
    "We will allow you some freedom to make your own judgments in cases that are ambigious - for example, it is up to you how you want to handle the answer \"2-3 times\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Fill in your answer below\n",
    "\n",
    "\n",
    "# Once you are done, the following lines of code should be able to run successfuly,\n",
    "# printing out the average number of moving times in the data\n",
    "df[\"Moving_Times\"] = pd.to_numeric(df[\"Moving_Times\"])\n",
    "print(df[\"Moving_Times\"].mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "G5lS8Sf8jl6U"
   },
   "source": [
    "In the previous examples, we *replaced* values in an existing column. But there are other cases where instead, you might want to preserve the original data, and instead put the cleaned data in a new column. pandas lets us do this too! As an example, let's consider the \"Moving_Times\" column. The data in this column counts how many times the survey participant has moved prior to taking the survey. You may imagine that for some research questions we don't necessarily care about the exact number of moving times, but instead only need to know whether the survey participant has moved at all (a boolean value). But this does not mean we want to completely replace the \"Moving_Times\" column, because for *other* research questions we might actually care about the exact number of moving times! So instead, we might want to create a *new* column that contains the simplified information of whether or not a survey participant has moved.\n",
    "\n",
    "We will start by creating a new column, let's call it \"Has_Moved\". Initially, we'll create the column as a copy of \"Moving_Times\"; we can then modify it (using code similar to what we did previously for \"Self_Confidence\") without affecting the original column. To create a new column, we can simply use Python assigment syntax:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "hNWb6wQAjl6U"
   },
   "outputs": [],
   "source": [
    "df[\"Has_Moved\"] = df[\"Moving_Times\"]\n",
    "display(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "oIMcZ7Cbjl6U"
   },
   "source": [
    "We see that there is now a new column \"Has_Moved\" that is an exact copy of \"Moving_Times\". Next we will modify it as follows: rows containing nonzero values will be set to 1, representing that the participant has moved (rows containing zero will not need to be changed, as they will remain as 0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "oj2Fb7hhjl6V"
   },
   "outputs": [],
   "source": [
    "df.loc[(df['Has_Moved'] > 0), 'Has_Moved'] = 1\n",
    "display(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TwLU5fCWjl6V"
   },
   "source": [
    "## 9. Fine-grained analysis using groupby\n",
    "In section 6, we showed how to summarize data from an *entire* column, such as taking the min and max of numerical data. This is useful for initial exploration, but when tackling actual research questions, we often want to do more complicated operations involving interactions between multiple variables. For example, instead of just finding average moving times in the data, we might be interested in finding the average moving times among students in each college. Using what we have learned so far, one way to do this would be to use queries: for each college, you could write a query to select only the rows matching that college, and call the `mean` method each time. But this is extremely tedious, given that there are so many colleges in the data! Thankfully, pandas offers a faster alternative: `groupby`.\n",
    "\n",
    "The `groupby` method can be thought of a splitting a DataFrame based on some categorical variable. For example, if we use `groupby` on the \"College\" column, we'll have one group containing rows where the College is \"College of Human Ecology\", another containing rows where the College is \"College of Arts and Sciences\", and so on, for every unique college. Now, if we just run `groupby` on its own, we won't immediately see anything useful:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "D0LNym1Pjl6V"
   },
   "outputs": [],
   "source": [
    "df_grouped = df.groupby(\"College\")\n",
    "display(df_grouped) # prints out some strange code that isn't super useful..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "BG_OSJQ3jl6V"
   },
   "source": [
    "But the key difference happens when we call summarization methods on the grouped variable. Instead of running the summarization on the entire data, as happens normally, the summarization will be run separately on each group! Let's try using the `min` summarization method on \"year\" in the grouped data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "4qk2lYsKjl6V"
   },
   "outputs": [],
   "source": [
    "display(df_grouped['Moving_Times'].mean()) # we can use the same indexing and summarization syntax as we did for regular DataFrames, but the operation now happens separately for each group!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "6zx0vHUmjl6W"
   },
   "source": [
    "As we can see, instead of a single mean, we get multiple: one for each group! Specifically, we are seeing the mean moving times for participants in each college. We can immediately see how this might be useful for running comparative analyses; for example, it appears that on average students in the College of Engineering have moved fewer times than students in the College of Agriculture and Life Sciences."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-QL_oq4Cjl6W"
   },
   "source": [
    "There are other kinds of summarization methods that can be useful when combined with `groupby`. Here are just a few examples:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "D6A5wTMQjl6W"
   },
   "outputs": [],
   "source": [
    "# The count() method counts the total number of items in a column.\n",
    "# When combined with groupby, the counting will happen separately per group.\n",
    "# Let's try using it to find out how many participants we got from each college\n",
    "df_grouped = df.groupby(\"College\")\n",
    "display(df_grouped[\"College\"].count())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "omhvRRokjl6W"
   },
   "outputs": [],
   "source": [
    "# The sum() method computes the sum of all items in a column\n",
    "# When combined with groupby, sums are computed separately per group\n",
    "# Let's try using it to find out how many participants in each college have ever moved.\n",
    "df_grouped = df.groupby(\"College\")\n",
    "display(df_grouped[\"Has_Moved\"].sum())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "lKEE8WO0jl6W"
   },
   "source": [
    "Finally, note that it is possible to group on multiple columns at once! This will create a group for each unique pair of possible values in the two columns. For example, we might be interested in splitting our analysis not just by college, but by different school years in each college:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "gjbzKU5Ejl6W"
   },
   "outputs": [],
   "source": [
    "df_grouped = df.groupby([\"College\", \"School_Year\"])\n",
    "display(df_grouped[\"Moving_Times\"].mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Final Exercise\n",
    "In this wrap-up, you will combine various pandas skills that we have learned in order to explore an example research question on this data.\n",
    "\n",
    "Here is the research question we will be exploring: Is there a relationship between parental income and hunger experience? More concretely, are participants who come from a lower parental income more likely to report having experienced hunger?\n",
    "\n",
    "In order to answer this question, you will need to take the following steps. Each step can be achieved using one of the techniques we have discussed today.\n",
    "1. The Hunger_Experience column is currently coded as a string, with possible answers \"Yes\" and \"No\". In order to perform numeric analyses on it, you will need to convert it to numerical data, like we discussed in Section 8.\n",
    "2. You will then need to use a groupby operation to split the data into the groups we are interested in comparing\n",
    "3. Finally, you will need to pick the right summarization method to use on the grouped DataFrame.\n",
    "\n",
    "HINT: for binary-coded data (that is, data where 1 means yes and 0 means no), the percentage of answers that are \"yes\" can be computed using the mean."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Write your answer below\n"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "collapsed_sections": [],
   "name": "pandas_final.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}