July 2019 Democratic Debate NLP Part 1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# July 2019 Democratic Debate: NLP Part 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A skill I've been interested in learning has been Natural Language Processing (NLP). It has so many great applications in healthcare from being able to summarize and extract themes from medical notes to being able to translate medical terminology into lay language for patients. Given its widespread use in healthcare, I thought it would be a great idea to get experience using these tools. \n",
    "\n",
    "As someone who loves politics, I thought it would be a great starter project to analyze and search for new insights on the democratic debate that occured on July 30-31, 2019. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## a. Creating My Text Corpus\n",
    "\n",
    "The first thing I needed to do was to find a transcript of the two nights of debate. Unfortunately, I could not find a one source place for the transcripts, so I extracted the transcripts from NBC News and Washington Post. The first task was to make sure that they were in a consistent format and only included the information from the actual debate.\n",
    "\n",
    "I downloaded the transcripts into a .txt file. Then, I skimmed over the text to get a sense of the format of the two transcripts. Next, I deleted any text that did not pertain to the actual debate. For example, random text from the website or commentary made by the network pundits after the debate. I wanted to keep the two .txt files separate because it didn't make sense to me to lump all their text together given the stage dynamics and the nature of the questions asked on both nights. \n",
    "\n",
    "The next step was to get them into the same format. My ideal format was: \"[Name]: [Text]\". I copied the text into Microsoft Word and quickly realized how messy the formatting was. I used Microsoft Word's built-in find-and-replace function with wildcards to delete extra spaces, time marks and links, ensure the name format was \"First Last\", and ensuring that their text was all in one line (ie. lines not separated by paragraph breaks). I won't go into all the specifics but Word's find-and-replace function is amazing for widespread text manipulation and saved me a lot of time.\n",
    "\n",
    "Now that my two .txt files are in the format I wanted them in. It's time to create a corpus dataset that separated out the names and the speech into two separated columns. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\Data Science Projects\\\\NLPDemD'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import os\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import nltk\n",
    "import re\n",
    "import pprint\n",
    "import string\n",
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "import pickle"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "DemDebate1raw = open('.\\\\raw text\\\\Democratic Debate July 30 Night 1.txt', encoding=\"utf8\").read()\n",
    "DemDebate1raw = DemDebate1raw.splitlines()\n",
    "str_DemDebate1raw = {'Name':DemDebate1raw}\n",
    "DemDebate1raw_df = pd.DataFrame(str_DemDebate1raw, columns = ['Name'])\n",
    "df1 = DemDebate1raw_df['Name'].str.split(\":\", n=1, expand=True)\n",
    "DemDebate1raw_df['Name'] = df1[0]\n",
    "DemDebate1raw_df['Text'] = df1[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "DemDebate2raw = open('.\\\\raw text\\\\Democratic Debate July 31 Night 2.txt', encoding=\"utf8\").read()\n",
    "DemDebate2raw = DemDebate2raw.splitlines()\n",
    "str_DemDebate2raw = {'Name':DemDebate2raw}\n",
    "DemDebate2raw_df = pd.DataFrame(str_DemDebate2raw, columns = ['Name'])\n",
    "df2 = DemDebate2raw_df['Name'].str.split(\":\", n=1, expand=True)\n",
    "DemDebate2raw_df['Name'] = df2[0]\n",
    "DemDebate2raw_df['Text'] = df2[1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Name</th>\n",
       "      <th>Text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Part 1</td>\n",
       "      <td>Opening Statements, Taxes, Health Insurance, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Jake Tapper</td>\n",
       "      <td>Welcome back to the CNN Democratic Presidenti...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Jake Tapper</td>\n",
       "      <td>You will each receive one minute to answer qu...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Jake Tapper</td>\n",
       "      <td>A candidate infringing on another candidate’s...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Dana Bash</td>\n",
       "      <td>Time now for opening statements, you’ll each ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          Name                                               Text\n",
       "0       Part 1   Opening Statements, Taxes, Health Insurance, ...\n",
       "1  Jake Tapper   Welcome back to the CNN Democratic Presidenti...\n",
       "2  Jake Tapper   You will each receive one minute to answer qu...\n",
       "3  Jake Tapper   A candidate infringing on another candidate’s...\n",
       "4    Dana Bash   Time now for opening statements, you’ll each ..."
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "DemDebate1raw_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## b. Cleaning up the Corpus\n",
    "For easier manipulation and analysis, I'm now going to append all the strings that came from the same person, into one cell. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "DemDebate1_df = DemDebate1raw_df.groupby(['Name'])['Text'].apply(', '.join).reset_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "DemDebate2_df = DemDebate2raw_df.groupby(['Name'])['Text'].apply(', '.join).reset_index()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Deleting the non-named rows\n",
    "DemDebate1_df = DemDebate1_df.drop([6,10,12,13,14,15]).reset_index()\n",
    "DemDebate1_df.set_index('Name', inplace=True)\n",
    "DemDebate1_df = DemDebate1_df.drop('index', axis=1)\n",
    "#DemDebate1_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Text</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Name</th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ANDREW YANG</th>\n",
       "      <td>If you've heard anything about me and my camp...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>BILL DE BLASIO</th>\n",
       "      <td>To the working people of America, tonight I b...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>CORY BOOKER</th>\n",
       "      <td>Thank you, Dana. Last week the president of t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>DANA BASH</th>\n",
       "      <td>Let's start with opening statements. You will...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>DON LEMON</th>\n",
       "      <td>Stand by, Senator. ,  Please stand by. ,  Ple...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>JAKE TAPPER</th>\n",
       "      <td>Welcome back to the CNN Democratic presidenti...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>JAY INSLEE</th>\n",
       "      <td>Good evening. I'm Jay Inslee. I am running fo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>JOE BIDEN</th>\n",
       "      <td>Tonight, I think Democrats are expecting some...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>JULIAN CASTRO</th>\n",
       "      <td>Thank you, Dana, and good evening. You know, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>KAMALA HARRIS</th>\n",
       "      <td>This is an inflection moment in the history o...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>KIRSTEN GILLIBRAND</th>\n",
       "      <td>My grandmother taught me that nothing's impos...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>MICHAEL BENNET</th>\n",
       "      <td>Thank you. Last week, I saw one of those Trum...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>TULSI GABBARD</th>\n",
       "      <td>I love our country. It's why I enlisted after...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                 Text\n",
       "Name                                                                 \n",
       "ANDREW YANG          If you've heard anything about me and my camp...\n",
       "BILL DE BLASIO       To the working people of America, tonight I b...\n",
       "CORY BOOKER          Thank you, Dana. Last week the president of t...\n",
       "DANA BASH            Let's start with opening statements. You will...\n",
       "DON LEMON            Stand by, Senator. ,  Please stand by. ,  Ple...\n",
       "JAKE TAPPER          Welcome back to the CNN Democratic presidenti...\n",
       "JAY INSLEE           Good evening. I'm Jay Inslee. I am running fo...\n",
       "JOE BIDEN            Tonight, I think Democrats are expecting some...\n",
       "JULIAN CASTRO        Thank you, Dana, and good evening. You know, ...\n",
       "KAMALA HARRIS        This is an inflection moment in the history o...\n",
       "KIRSTEN GILLIBRAND   My grandmother taught me that nothing's impos...\n",
       "MICHAEL BENNET       Thank you. Last week, I saw one of those Trum...\n",
       "TULSI GABBARD        I love our country. It's why I enlisted after..."
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "## Deleting the non-named rows\n",
    "DemDebate2_df = DemDebate2_df.drop([0,13]).reset_index()\n",
    "DemDebate2_df.set_index('Name', inplace=True)\n",
    "DemDebate2_df = DemDebate2_df.drop('index', axis=1)\n",
    "DemDebate2_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## c. More Cleaning\n",
    "\n",
    "The next step is to standardize and further clean the text. This involves removing unneeded characters and numbers, and then exploring the text as we continue to tag and break down text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "def clean_text1(text):\n",
    "    '''Make lower case, remove punctuation, remove numbers and numbers with letters next to it'''\n",
    "    text = text.lower()\n",
    "    text = re.sub('[%s]' % re.escape(string.punctuation), '', text)\n",
    "    text = re.sub('\\w*\\d\\w*', '', text)\n",
    "    return text\n",
    "\n",
    "round1 = lambda x: clean_text1(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "DemDebate1_clean = pd.DataFrame(DemDebate1_df.Text.apply(round1))\n",
    "DemDebate2_clean = pd.DataFrame(DemDebate2_df.Text.apply(round1))\n",
    "#DemDebate2_clean.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## d. Tokenization and Document-Term Matrix\n",
    "\n",
    "In this step, I am removing words that are considered \"stop words\" like \"a\", \"the\", etc. Then I am making the remaining words its own column (re: tokenization). And then restructuring the dataframe so that each word column represents a count for every candidate. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>abide</th>\n",
       "      <th>ability</th>\n",
       "      <th>able</th>\n",
       "      <th>abolition</th>\n",
       "      <th>abroad</th>\n",
       "      <th>absolutely</th>\n",
       "      <th>abusers</th>\n",
       "      <th>abusing</th>\n",
       "      <th>aca</th>\n",
       "      <th>accelerated</th>\n",
       "      <th>...</th>\n",
       "      <th>yes</th>\n",
       "      <th>yesterday</th>\n",
       "      <th>york</th>\n",
       "      <th>young</th>\n",
       "      <th>younger</th>\n",
       "      <th>youngest</th>\n",
       "      <th>youngstown</th>\n",
       "      <th>zealand</th>\n",
       "      <th>zero</th>\n",
       "      <th>zone</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Name</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Amy Klobuchar</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>7</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Bernie Sanders</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Beto O’Rourke</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Dana Bash</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Don Lemon</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Elizabeth Warren</th>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Jake Tapper</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>John Delaney</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>John Hickenlooper</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Marianne Williamson</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Pete Buttigieg</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Steve Bullock</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Tim Ryan</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>13 rows × 2477 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                     abide  ability  able  abolition  abroad  absolutely  \\\n",
       "Name                                                                       \n",
       "Amy Klobuchar            0        0     0          0       0           0   \n",
       "Bernie Sanders           0        0     0          0       0           1   \n",
       "Beto O’Rourke            0        0     2          0       1           0   \n",
       "Dana Bash                0        0     0          0       0           0   \n",
       "Don Lemon                1        0     0          0       0           0   \n",
       "Elizabeth Warren         0        2     1          0       0           0   \n",
       "Jake Tapper              0        0     2          0       0           0   \n",
       "John Delaney             0        0     2          0       0           1   \n",
       "John Hickenlooper        0        0     2          0       0           0   \n",
       "Marianne Williamson      0        0     1          1       0           0   \n",
       "Pete Buttigieg           0        0     1          0       0           0   \n",
       "Steve Bullock            0        1     1          0       0           0   \n",
       "Tim Ryan                 0        0     1          0       0           0   \n",
       "\n",
       "                     abusers  abusing  aca  accelerated  ...  yes  yesterday  \\\n",
       "Name                                                     ...                   \n",
       "Amy Klobuchar              0        0    0            0  ...    7          0   \n",
       "Bernie Sanders             0        0    0            0  ...    1          0   \n",
       "Beto O’Rourke              0        0    1            0  ...    0          0   \n",
       "Dana Bash                  0        0    0            0  ...    0          0   \n",
       "Don Lemon                  0        0    0            0  ...    1          0   \n",
       "Elizabeth Warren           0        0    0            0  ...    1          0   \n",
       "Jake Tapper                0        0    0            0  ...    2          1   \n",
       "John Delaney               0        0    0            0  ...    0          0   \n",
       "John Hickenlooper          0        0    0            0  ...    0          0   \n",
       "Marianne Williamson        0        0    0            0  ...    2          0   \n",
       "Pete Buttigieg             1        0    1            1  ...    1          0   \n",
       "Steve Bullock              0        0    0            0  ...    1          0   \n",
       "Tim Ryan                   0        1    0            0  ...    1          0   \n",
       "\n",
       "                     york  young  younger  youngest  youngstown  zealand  \\\n",
       "Name                                                                       \n",
       "Amy Klobuchar           0      0        0         0           0        0   \n",
       "Bernie Sanders          1      2        1         0           0        0   \n",
       "Beto O’Rourke           0      0        0         0           0        0   \n",
       "Dana Bash               0      0        0         0           0        0   \n",
       "Don Lemon               0      0        0         1           0        0   \n",
       "Elizabeth Warren        0      0        0         0           0        0   \n",
       "Jake Tapper             0      0        0         0           0        0   \n",
       "John Delaney            1      0        0         1           0        0   \n",
       "John Hickenlooper       0      0        0         0           0        0   \n",
       "Marianne Williamson     0      1        0         0           0        0   \n",
       "Pete Buttigieg          0      0        1         0           0        1   \n",
       "Steve Bullock           0      0        0         0           0        0   \n",
       "Tim Ryan                0      0        0         0           1        0   \n",
       "\n",
       "                     zero  zone  \n",
       "Name                             \n",
       "Amy Klobuchar           0     0  \n",
       "Bernie Sanders          0     0  \n",
       "Beto O’Rourke           0     0  \n",
       "Dana Bash               0     0  \n",
       "Don Lemon               0     0  \n",
       "Elizabeth Warren        0     0  \n",
       "Jake Tapper             0     0  \n",
       "John Delaney            3     0  \n",
       "John Hickenlooper       0     0  \n",
       "Marianne Williamson     0     0  \n",
       "Pete Buttigieg          0     0  \n",
       "Steve Bullock           0     0  \n",
       "Tim Ryan                0     1  \n",
       "\n",
       "[13 rows x 2477 columns]"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cv = CountVectorizer(stop_words='english')\n",
    "DD1_cv = cv.fit_transform(DemDebate1_clean.Text)\n",
    "DD1_dtm = pd.DataFrame(DD1_cv.toarray(), columns=cv.get_feature_names())\n",
    "DD1_dtm.index = DemDebate1_clean.index\n",
    "DD1_dtm.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>aberration</th>\n",
       "      <th>abhorrent</th>\n",
       "      <th>ability</th>\n",
       "      <th>able</th>\n",
       "      <th>abroad</th>\n",
       "      <th>absolutely</th>\n",
       "      <th>abused</th>\n",
       "      <th>abusing</th>\n",
       "      <th>abusive</th>\n",
       "      <th>accept</th>\n",
       "      <th>...</th>\n",
       "      <th>yesterday</th>\n",
       "      <th>york</th>\n",
       "      <th>youll</th>\n",
       "      <th>young</th>\n",
       "      <th>youngstown</th>\n",
       "      <th>youre</th>\n",
       "      <th>youve</th>\n",
       "      <th>zero</th>\n",
       "      <th>zip</th>\n",
       "      <th>zone</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Name</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>ANDREW YANG</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>BILL DE BLASIO</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>CORY BOOKER</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>DANA BASH</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>DON LEMON</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 2688 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                aberration  abhorrent  ability  able  abroad  absolutely  \\\n",
       "Name                                                                       \n",
       "ANDREW YANG              0          0        0     1       1           0   \n",
       "BILL DE BLASIO           0          0        0     1       0           1   \n",
       "CORY BOOKER              0          0        1     0       0           0   \n",
       "DANA BASH                0          0        0     1       0           0   \n",
       "DON LEMON                0          0        0     0       0           0   \n",
       "\n",
       "                abused  abusing  abusive  accept  ...  yesterday  york  youll  \\\n",
       "Name                                              ...                           \n",
       "ANDREW YANG          0        0        1       0  ...          0     0      0   \n",
       "BILL DE BLASIO       0        0        0       2  ...          0     3      0   \n",
       "CORY BOOKER          0        0        0       0  ...          2     0      0   \n",
       "DANA BASH            0        0        0       0  ...          0     1      0   \n",
       "DON LEMON            0        0        0       0  ...          0     0      0   \n",
       "\n",
       "                young  youngstown  youre  youve  zero  zip  zone  \n",
       "Name                                                              \n",
       "ANDREW YANG         0           0      1      3     3    1     0  \n",
       "BILL DE BLASIO      0           0      4      0     1    0     0  \n",
       "CORY BOOKER         0           0      1      1     0    0     0  \n",
       "DANA BASH           0           0      1      0     0    0     0  \n",
       "DON LEMON           0           0      0      0     0    0     0  \n",
       "\n",
       "[5 rows x 2688 columns]"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "DD2_cv = cv.fit_transform(DemDebate2_clean.Text)\n",
    "DD2_dtm = pd.DataFrame(DD2_cv.toarray(), columns=cv.get_feature_names())\n",
    "DD2_dtm.index = DemDebate2_clean.index\n",
    "DD2_dtm.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## e. Next Steps\n",
    "\n",
    "In this part, I have compiled my text from their sources, cleaned up what I could using Microsoft Word before bringing them in, restructured the information into dataframes, did another around of cleaning, and created a Document-Text Matrix. \n",
    "\n",
    "Now that we've done an initial clean and have two corpus that are mostly ready to use, I'm going to \"pickle\" both the clean and the DTM files so that I can use them in my next notebook. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "DemDebate1_clean.to_pickle(\"DemDebate1_clean.pkl\")\n",
    "DemDebate2_clean.to_pickle(\"DemDebate2_clean.pkl\")\n",
    "DD1_dtm.to_pickle(\"DD1_dtm.pkl\")\n",
    "DD2_dtm.to_pickle(\"DD2_dtm.pkl\")\n",
    "pickle.dump(cv, open(\"cv.pkl\", \"wb\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## f. Acknowledgements\n",
    "Thanks to NBC News and Washington Post for having a publicly available transcript of the debates. And a special thanks to [Alice Zhao](https://www.youtube.com/channel/UCyv-PL-QgkAXEfDRcKrYMeA) for creating very thorough and clear video tutorials for doing NLP. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}