code/1-Processing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![UKDS Logo](images/UKDS_Logos_Col_Grey_300dpi.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "# Text-mining: Basics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Welcome to this <a href=\"https://ukdataservice.ac.uk/\" target=_blank>UK Data Service</a> *Computational Social Science* training series! \n",
    "\n",
    "The various *Computational Social Science* training series, all of which guide you through some of the popular and useful computational techniques, tools, methods and concepts that social science research might want to use. For example, this series covers collecting data from websites and social media platorms, working with text data, conducting simulations (agent based modelling), and more. The series includes recorded video webinars, interactive notebooks containing live programming code, reading lists and more.\n",
    "\n",
    "* To access training materials on our GitHub site: <a href=\"https://github.com/UKDataServiceOpen/computational-social-science\" target=_blank>[Training Materials]</a>\n",
    "\n",
    "* To keep up to date with upcoming and past training events: <a href=\"https://ukdataservice.ac.uk/news-and-events/events\" target=_blank>[Events]</a>\n",
    "\n",
    "* To get in contact with feedback, ideas or to seek assistance: <a href=\"https://ukdataservice.ac.uk/help.aspx\" target=_blank>[Help]</a>\n",
    "\n",
    "<a href=\"https://www.research.manchester.ac.uk/portal/julia.kasmire.html\" target=_blank>Dr J. Kasmire</a>  <br />\n",
    "UK Data Service  <br />\n",
    "University of Manchester <br />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "toc": true
   },
   "source": [
    "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
    "<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#Introduction\" data-toc-modified-id=\"Introduction-1\"><span class=\"toc-item-num\">1&nbsp;&nbsp;</span>Introduction</a></span><ul class=\"toc-item\"><li><span><a href=\"#Guide-to-using-this-resource\" data-toc-modified-id=\"Guide-to-using-this-resource-1.1\"><span class=\"toc-item-num\">1.1&nbsp;&nbsp;</span>Guide to using this resource</a></span></li><li><span><a href=\"#Interaction\" data-toc-modified-id=\"Interaction-1.2\"><span class=\"toc-item-num\">1.2&nbsp;&nbsp;</span>Interaction</a></span></li><li><span><a href=\"#Learn-more\" data-toc-modified-id=\"Learn-more-1.3\"><span class=\"toc-item-num\">1.3&nbsp;&nbsp;</span>Learn more</a></span></li></ul></li><li><span><a href=\"#Retrieval\" data-toc-modified-id=\"Retrieval-2\"><span class=\"toc-item-num\">2&nbsp;&nbsp;</span>Retrieval</a></span></li><li><span><a href=\"#Processing\" data-toc-modified-id=\"Processing-3\"><span class=\"toc-item-num\">3&nbsp;&nbsp;</span>Processing</a></span><ul class=\"toc-item\"><li><span><a href=\"#Tokenisation\" data-toc-modified-id=\"Tokenisation-3.1\"><span class=\"toc-item-num\">3.1&nbsp;&nbsp;</span>Tokenisation</a></span></li><li><span><a href=\"#Standardising\" data-toc-modified-id=\"Standardising-3.2\"><span class=\"toc-item-num\">3.2&nbsp;&nbsp;</span>Standardising</a></span></li><li><span><a href=\"#Removing-irrelevancies\" data-toc-modified-id=\"Removing-irrelevancies-3.3\"><span class=\"toc-item-num\">3.3&nbsp;&nbsp;</span>Removing irrelevancies</a></span></li><li><span><a href=\"#Consolidation\" data-toc-modified-id=\"Consolidation-3.4\"><span class=\"toc-item-num\">3.4&nbsp;&nbsp;</span>Consolidation</a></span></li></ul></li><li><span><a href=\"#Conclusions\" data-toc-modified-id=\"Conclusions-4\"><span class=\"toc-item-num\">4&nbsp;&nbsp;</span>Conclusions</a></span></li><li><span><a href=\"#Further-reading-and-resources\" data-toc-modified-id=\"Further-reading-and-resources-5\"><span class=\"toc-item-num\">5&nbsp;&nbsp;</span>Further reading and resources</a></span></li></ul></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "There is a table of contents provided here at the top of the notebook, but you can also access this menu at any point by clicking the Table of Contents button on the top toolbar (an icon with four horizontal bars, if unsure hover your mouse over the buttons). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-------------------------------------\n",
    "\n",
    "<div style=\"text-align: center\"><i><b>This is notebook 1 of 2 in this lesson</i></b></div>\n",
    "\n",
    "-------------------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the first in a series of jupyter notebooks on text-mining that cover basic preparation processes, common natural language processing tasks, and some more advanced natural language tasks. These interactive code-along notebooks use python as a programming language, but introduce various packages related to text-mining and text processing. Most of those tasks could be done in other packages, so please be aware that the options demonstrated here are not the only way, or even the best way, to accomplish a text-mining task.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Guide to using this resource\n",
    "\n",
    "This learning resource was built using <a href=\"https://jupyter.org/\" target=_blank>Jupyter Notebook</a>, an open-source software application that allows you to mix code, results and narrative in a single document. As <a href=\"https://jupyter4edu.github.io/jupyter-edu-book/\" target=_blank>Barba et al. (2019)</a> espouse:\n",
    "> In a world where every subject matter can have a data-supported treatment, where computational devices are omnipresent and pervasive, the union of natural language and computation creates compelling communication and learning opportunities.\n",
    "\n",
    "If you are familiar with Jupyter notebooks then skip ahead to the main content (*Retrieval*). Otherwise, the following is a quick guide to navigating and interacting with the notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Interaction\n",
    "\n",
    "**You only need to execute the code that is contained in sections which are marked by `In []`.**\n",
    "\n",
    "To execute a cell, click or double-click the cell and press the `Run` button on the top toolbar (you can also use the keyboard shortcut Shift + Enter).\n",
    "\n",
    "Try it for yourself:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Enter your name and press enter:\")\n",
    "name = input()\n",
    "print(\"\\r\")\n",
    "print(\"Hello {}, enjoy learning more about Python and computational social science!\".format(name))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Learn more\n",
    "\n",
    "Jupyter notebooks provide rich, flexible features for conducting and documenting your data analysis workflow. To learn more about additional notebook features, we recommend working through some of the <a href=\"https://github.com/darribas/gds19/blob/master/content/labs/lab_00.ipynb\" target=_blank>materials</a> provided by Dani Arribas-Bel at the University of Liverpool."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Retrieval\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first step in text-mining, or any form of data-mining, is retrieving a data set to work with. Within text-mining, or any language analysis context, one data set is usually referred to as 'a corpus' while multiple data sets are referred to as 'corpora'. 'Corpus' is a latin-root word and therefore has a funny plural. \n",
    "\n",
    "For text-mining, a corpus can be:\n",
    "- a set of tweets, \n",
    "- the full text of an 18th centrury novel,\n",
    "- the contents of a page in the dictionary, \n",
    "- minutes of local council meetings, \n",
    "- random gibberish letters and numbers, or\n",
    "- just about anything else in text format. \n",
    "\n",
    "\n",
    "Retrieval is a very important step, but it is not the focus of this particular training series. If you are interested in creating a corpus from internet data, then you may want to check out our <a href=\"https://github.com/UKDataServiceOpen/web-scraping\" target=_blank>previous training series</a> that covers Web-scraping (available as recordings of webinars or as a code-along jupyter notebook like this one) and API's (also as recording or jupyter notebook). Both of these demonstrate and discuss ways to get data from the internet that you could use to build a corpus. \n",
    "\n",
    "Instead, for the purposes of this session, we will assume that you already have a corpus to analyse. This is easy for us to assume, because we have provided a sample text file that we can use as a corpus for these exercises. \n",
    "\n",
    "First, let's check that it is there. To do that, click in the code cell below and hit the 'Run' button at the top of this page or by holding down the 'Shift' key and hitting the 'Enter' key. \n",
    "\n",
    "For the rest of this notebook, I will use 'Run/Shift+Enter' as short hand for:\n",
    "* click in the code cell below and hit the 'Run' button at the top of this page, or\n",
    "* click in the code cell below and hold down the 'Shift' key while hitting the 'Enter' key'. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# It is good practice to always start by importing the modules and packages you will need. \n",
    "\n",
    "import os                         # os is a module for navigating your machine (e.g., file directories).\n",
    "import nltk                       # nltk stands for natural language tool kit and is useful for text-mining. \n",
    "import re                         # re is for regular expressions, which we use later \n",
    "\n",
    "print(\"1. Succesfully imported necessary modules\")    # The print statement is just a bit of encouragement!\n",
    "\n",
    "print(\"\")\n",
    "\n",
    "# List all of the files in the \"data\" folder that is provided to you\n",
    "for file in os.listdir(\"./data\"):\n",
    "   print(\"2. One of the files in ./data is...\", file)\n",
    "print(\"\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_______________________________________________________________________________________________________________________________\n",
    "Great! We have imported a useful module and used it to check that we have access to the sample_text file. \n",
    "\n",
    "Now we need to load that sample_text file into a variable that we can work with in python. Time to Run/Shift+Enter again!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Open the \"sample_text\" file and read (import) its contents to a variable called \"corpus\"\n",
    "with open(\"./data/sample_text.txt\", \"r\", encoding = \"ISO-8859-1\") as f:\n",
    "    corpus = f.read()\n",
    "    \n",
    "    print(corpus)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_______________________________________________________________________________________________________________________________\n",
    "Hmm. Not excellent literature, but it will do for our purposes. \n",
    "\n",
    "A quick look tells us that there are capital letters, contractions, punctuation, numbers as digits, numbers written out, abbreviations, and other things that, as humans, we know are equivalent but that computers do not know about. \n",
    "\n",
    "Before we go further, it helps to know what kind of variable corpus is. Run/Shift+Enter the next code block to find out!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "type(corpus)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_______________________________________________________________________________________________________________________________\n",
    "This tells us that 'corpus' is one very long string of text characters.  \n",
    "\n",
    "Congratulations! We are done with the retreival portion of this process. The rest won't be quite so straightforward because next up... Processing. \n",
    "\n",
    "Processing is about cleaning, correcting, standardizing and formatting the raw data returned from the retrieval process. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Processing\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_______________________________________________________________________________________________________________________________\n",
    "The string we have as our corpus is a good starting point, but it is not perfect. It has a bunch of errors and punctuation which need to be corrected. But even worse, it is 'one long thing' when statistical analysis typically requires 'lots of short things'. \n",
    "\n",
    "So, clearly, we have a few steps to go through with our raw text. \n",
    "- Tokenisation, (or splitting text into various kinds of 'short things' that can be statistically analysed).\n",
    "- Standardising the next (including converting uppercase to lower, correcting spelling, find-and-replace operations to remove abbreviations, etc.). \n",
    "- Removing irrelevancies (anything from punctuation to stopwords like 'the' or 'to' that are unhelpful for many kinds of analysis).\n",
    "- Consolidating (including stemming and lemmatisation that strip words back to their 'root'). \n",
    "- Basic NLP (that put some of the small things back together into logically useful medium things, like multi-word noun or verb phrases and proper names).\n",
    "\n",
    "In practice, most text-mining work will require that any given corpus undergo multiple steps, but the exact steps and the exact order of steps depends on the desired analysis to be done. Thus, some of the examples that follow will use the raw text corpus as an input to the process while others use a processed corpus as an input. \n",
    "\n",
    "As a side note, it is good practice to create new variables whenever you manipulate an existing variable rather than write over the original. This means that you keep the original and can go back to it anytime you need to if you want to try a different manipulation or correct an error. You will see how this works as we progress through the processing steps. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Tokenisation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our first step is to cut our 'one big thing' into tokens, or 'lots of little things'. As an example, one project I worked involved downloading a file with hundreds of recorded chess games, which I then divided into individual text files with one game each. The games had a very standard format, with every game ending with either '1-0', '0-1' or '1/2-1/2'. Thus, I was able to use regular expressions (covered in more detail later) to iterate over the file, selecting everyithing until it found an instance of '1-0', '0-1' or '1/2-1/2', at which point it would cut what it had selected, write it to a blank file, save it, and start iterating over the original file again. \n",
    "\n",
    "Other options that might make more sense with other kinds of files would be to to cut and write from the large file to new files after a specified number of lines or characters. \n",
    "\n",
    "Whether you have one big file or many smaller ones, most text-mining work will also want to divide the corpus into what are known as 'tokens'. These 'tokens' are the unit of analysis, which might be chapters, sections, paragraphs, sentences, words, or something else. \n",
    "\n",
    "Since we have one file already loaded as a corpus, we can skip the right to tokenising that text into sentences and words. Both options are functions available through the ntlk package that we imported earlier. These are both useful tokens in their own way, so we will see how to produce both kinds. \n",
    " \n",
    "We start by dividing our corpus into words, splitting the string into substrings whenever 'word_tokenize' detects a word. \n",
    "\n",
    "Let's try that. But this time, let's just have a look at the first 100 things it finds instead of the entire text.\n",
    "Run/Shift+Enter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nltk.download('punkt')\n",
    "from nltk import word_tokenize                                           # importing the word_tokenize function from nltk\n",
    "\n",
    "corpus_words = word_tokenize(corpus)                                     # Pass the corpus through word tokenize \n",
    "print(corpus_words[:10])                                                 # the [:100] within the print statement says \n",
    "                                                                         # to print only the first 100 items in the list  \n",
    "print(\"...\")                                                             # the print(\"...\") just improves output readability\n",
    "type(corpus_words)                                                       # Always good to know your variable type!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's have a look. \n",
    "\n",
    "We can see that corpus_words is a list of strings. We know it is a list because it starts and ends with square brackets and we know the things in that list are strings because they are surrounded by single quotes. \n",
    "\n",
    "We can also see that puctuation marks are counted as tokens in that list. For example, the full stop at the end of the first sentence appears as its own token because word_tokenize knows that it does not count as part of the previous word. Interestingly, 'U.K.' is all one token, despite having full stops in. Clever stuff, this tokenisation function!\n",
    "\n",
    "Word_tokenize is a useful function if you want to take a 'bag of words' approach to text-mining. This reduces a lot of the contextual information within the original corpus because it ignores how the words were used or in what order they originally appeared, making it easy to count how often each word occurrs. There is a surprising amount of insight to be gained here, but it does mean that 'building' in the next two sentences will be counted as the \"same\" word. \n",
    "- \"He is building a diorama for a school project.\" where 'building' is a verb\n",
    "- \"The building is a clear example of brutalist architecture.\" where 'building' is a noun\n",
    "\n",
    "There are other kinds of analyses that you could do if you want verb-building and noun-building to be counted as different words. That usually starts with tokenising differently, for example into sentences rather than words. \n",
    "Let's see what that looks like by running the same basic analysis again, but this time with sentence-token things instead of word-token things. \n",
    "\n",
    "Do that funky Run/Shift+Enter thing! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# importing sent_tokenize from nltk\n",
    "from nltk import sent_tokenize\n",
    "\n",
    "# Same again, but this time broken into sentences\n",
    "corpus_sentences = sent_tokenize(corpus)\n",
    "print(corpus_sentences[:10])                                                  # Since these are sentences instead of words, \n",
    "                                                                              # we only want the first 10 items instead of 100.\n",
    "print(\"...\")                                                                  \n",
    "type(corpus_sentences)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_______________________________________________________________________________________________________________________________\n",
    "\n",
    "Corpus_sentences is also a list of strings (starts and ends with square brackets, each item is surrounded by single quotes). \n",
    "\n",
    "This time, the full stops at the end of each sentence are included within the sentence token, which makes sense. \n",
    "\n",
    "Moving forward, some of the next steps make more sense to do on the word-tokens while others on sentence-tokens."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Standardising\n",
    "#### Remove uppercase letters"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we want to focus on the 'bag of words' approach, we don't really care about uppercase or lowercase distinctions. For example, we want 'Privacy' to count as the same word as 'privacy', rather than as two different words. \n",
    "\n",
    "We can remove all uppercase letters with a built in python command on corpus_words. Do this in the next code cell, again returning just the first 100 items instead of the whole thing. \n",
    "\n",
    "Do the Run/Shift+Enter thing. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# You can see that I created a new variable called corpus_lower rather than edit corpus_words directly.\n",
    "# This means I can easily compare two different processes or correct something without going back and re-running earlier steps. \n",
    "\n",
    "corpus_lower = [word.lower() for word in corpus_words]\n",
    "print(corpus_lower[:100])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_______________________________________________________________________________________________________________________________\n",
    "Great! This is another step in the right direction. \n",
    "\n",
    "If you want a bit more practice, you can copy/paste/edit the command above to create a second version that applies to corpus_sentences instead of corpus_words. You will have to think for yourself whether this makes sense to do or not. Uppercase letters are potentially useful in an analysis that looks at sentences, but since the tokens already capture sentences, maybe that value is no longer useful. \n",
    "\n",
    "Anyway, have a go. Knock yourself out! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Spelling correction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_______________________________________________________________________________________________________________________________\n",
    "Everybody loves spelling... RIGHT?!?\n",
    "\n",
    "Fortunately, there are several decent spellchecking packages written for python. They are not automatically installed and ready to import in the same way that the 'os' or 'nltk' packages were, but we just need to install the packages and import the functions we need through an installer called 'pip'. You will see 'pip' in the next code block, but since this is in jupyter notebook rather than directly in a python shell, we need to put a '!' in front of the 'pip' function. Don't worry too much about that now, I just mention  it here in case you find it interesting to know. \n",
    "\n",
    "The next code cell:\n",
    "- installs the 'autocorrect' package,\n",
    "- imports the Speller function, and\n",
    "- creates a one-word command that specifies that the Speller function should use English language. \n",
    "\n",
    "Run/Shift+Enter, as per usual. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install autocorrect\n",
    "from autocorrect import Speller\n",
    "check = Speller(lang='en')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_______________________________________________________________________________________________________________________________\n",
    "Super. Creating that one-word command saves us some time, which is maybe less important here but is a good skill to be aware of if you are working on text-mining every day for weeks on end. Always be on the look out for good ways to save time. \n",
    "\n",
    "Moving on, we need to iterate over our corpus, checking and correcting each token. This is easy to do if you start with a new, empty list (I called mine 'corpus_correct_spell'). As I work through corpus_words, one token at a time, we append (which is just fancy for 'add to the end') the corrected word to our new blank list. \n",
    "\n",
    "Then, as usual, we have a quick look at the first 100 entries in the new 'corpus_correct_spell'. \n",
    "\n",
    "Run/Shift+Enter. You know how to do it. Don't worry if it takes a while... Checking the spelling on each word is not a cakewalk. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus_correct_spell = []\n",
    "\n",
    "for word in corpus_words:\n",
    "    corpus_correct_spell.append(check(word))    \n",
    "\n",
    "print(corpus_correct_spell[:100])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_______________________________________________________________________________________________________________________________\n",
    "How did it do? Well, this spell-checker replaced 'haz' with 'had' rather than 'has'. That is ok, I guess. No automatic spelling correction programme will get it 100% right 100% of the time. Maybe your project has specific research questions that won't work with this decision. \n",
    "\n",
    "In that case, you would have to check out some other spell-checkers like textblob or pyspellchecker. You might even want to custom build or adapt your own spell-checker, especially if you were working with very non-standard text, like comment boards that use a bunch of slang, common typos, or specific terms. \n",
    "\n",
    "But take a moment here and consider the following questions... \n",
    "- Can you apply this spell-checker to corpus_sentences rather than corpus_words? If you are not sure what happens, try it out by copying, editing and re-running the above code block. \n",
    "- Should you have appled this spell-checker to corpus_lower rather than corpus_words? What difference would it make? Again, try it out if you are not sure. \n",
    "\n",
    "Next up, specific replacements with RegEx! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### RegEx replacements"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "RegEx stands for REGular EXpressions, which is probably familiar to you as the basis for how find-and-replace works in text documents. I mentioned this above when I talked about cutting up a large file into smaller files whenever the computer iterating over the large file found one of three specific combinations of numbers and symbols. \n",
    "\n",
    "But RegEx is actually stronger than that because you can use it to identify combinations of letters, numbers, symbols, spaces and more, some of which can be repeated more than once or can be optional. I won't go into RegEx too much more here, because that is a whole set of lessons on its own. But here are a couple of examples that you might find useful in a text like ours where we know that there are mixtures of numbers written as numbers, numbers spelled out, geographic abbreviations and more.\n",
    "\n",
    "As you might expect, do the Run/Shift+Enter thing. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus_numbers = [re.sub(r\"ninety-six\", \"96\", word) for word in corpus_words]   # Defines a new variable create by substituting\n",
    "                                                                                # '96' for 'ninety-six' in corpus_words\n",
    "\n",
    "print(corpus_numbers[:100])                                            # Prints the first 100 items in the newly created corpus\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Super! Now, this only works on 'ninety-six', but there might be other numbers spelled out in the text. We would have to look at it all to be sure, either manually or by using word frequency tables (we'll get to that). If we were to find some, we would have to revise our RegEx to capture more things and substitute them properly. \n",
    "\n",
    "One way to do that might be to define multiple terms to replace and what to replace them with. To do that, I searched on stack overflow and found a function written to multiple items by RegEx in a string. \n",
    "\n",
    "Run/Shift+Enter below!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's try editing this. \n",
    "What happens when we use lowercase letters instead of uppercase letters in \"United Kingdom\"?\n",
    "What happens if you change the order of the entries in 'dict'. What happens if you reverse the order of \n",
    "- \"United Kingdom of Great Britain and Northern Ireland\" : \"U.K.\", and \n",
    "- \"United Kingdom of Great Britain\" : \"U.K.\", ?\n",
    "\n",
    "You should also feel free to add your own lines to 'dict' to exact some substitutions of your own. \n",
    "\n",
    "Note: this function works on strings, so I applied it to 'corpus' our original raw text. \n",
    "We can either put a step like this as the first step in a pipeline, or we can adapt the code to iterate over a list of strings. Both have pros and cons. What do you think those pros and cons might be?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def multiple_replace(dict, text):\n",
    "  # Create a regular expression  from the dictionary keys\n",
    "  regex = re.compile(\"(%s)\" % \"|\".join(map(re.escape, dict.keys())))\n",
    "\n",
    "  # For each match, look-up corresponding value in dictionary\n",
    "  return regex.sub(lambda mo: dict[mo.string[mo.start():mo.end()]], text) \n",
    "\n",
    "if __name__ == \"__main__\": \n",
    "\n",
    "  dict = {\n",
    "    \"CA\" : \"California\",\n",
    "    \"United Kingdom\" : \"U.K.\",\n",
    "    \"United Kingdom of Great Britain and Northern Ireland\" : \"U.K.\",\n",
    "    \"United Kingdom of Great Britain\" : \"U.K.\",\n",
    "    \"UK\" : \"U.K.\",\n",
    "    \"Privacy Policy\" : \"noodle soup\",\n",
    "  } \n",
    "\n",
    "corpus_replace = multiple_replace(dict, corpus)\n",
    "print(corpus_replace)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Removing irrelevancies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Remove punctuation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Punctuation is not always very useful for understanding text, especially if you look at words as tokens because lots of the punctuation ends up being tokenised on its own. \n",
    "\n",
    "We could use RegEx to replace all punctuation with nothing, and that is a valid approach. But, just for variety sake, I demonstrate another way here."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_______________________________________________________________________________________________________________________________\n",
    "Forging ahead, let's filter out punctuation. We can define a string that includes all the standard English language punctuation, and then use that to iterate over corpus_words, removing anything that matches.\n",
    "\n",
    "But wait... Do we really want to remove the:\n",
    "- hyphen in 'ninety-six' or words like 'lactose-free'? \n",
    "- full stops in 'u.k.'? \n",
    "- the apostrophe in contractions or possessives?\n",
    "\n",
    "There are no right or wrong answers here. Every project will have to decide, based on the research questions, what is the right choice for the specific context. In this case, we want to remove the full stops, even from 'u.k.' so that it becomes identical to 'uk'. \n",
    "\n",
    "But, at the same time, we don't necessarily want to remove dashes or apostrophes. Those are punctuation marks that occur in the middle of words and do add meaning to the word, so I want to keep them. \n",
    "\n",
    "Run/Shift+Enter, as is tradition. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "English_punctuation = \"!\\\"#$%&()*+,./:;<=>?@[\\]^_`{|}~“”\"      # Define a variable with all the punctuation to remove.\n",
    "print(English_punctuation)                                     # Print that defined variable, just to check it is correct.\n",
    "print(\"...\")                                                   # Print an ellipsis, just to make the output more readable.\n",
    "\n",
    "table_punctuation = str.maketrans('','', English_punctuation)  # The python function 'maketrans' creates a table that maps\n",
    "print(table_punctuation)                                       # the punctation marks to 'None'. Print the table to check. \n",
    "print(\"...\")                                                   # Just to be clear, '!' is 33 in Unicode, and '\\' is 34, etc.\n",
    "                                                               # 'None' is python for nothing, not a string of the word \"none\".\n",
    "    \n",
    "corpus_no_punct = [w.translate(table_punctuation) for w in corpus_words]  \n",
    "                                                               # Iterate over corpus_words, turning punctuation to nothing.\n",
    "print(corpus_no_punct[:100])                                   # Print the 1st 100 items in corpus_no_punct to check."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_______________________________________________________________________________________________________________________________\n",
    "Super! \n",
    "\n",
    "Do you want to try something else? How about you create a version that *does* filter out dashes and apostrophes. \n",
    "\n",
    "C'mon. You know you can do it. \n",
    "\n",
    "Take each of the steps above and copy/paste/edit them as needed. \n",
    "- Create a copy of the line that defines the English_punctuation variable and edit it to define an All_English_Punctuation variable that includes more punctuation.\n",
    "- Then create a copy of the line that defines the table_punctuation variable and have it create a table_all_punctuation variable.\n",
    "- Then create a copy of the line that creates the corpus_no_punct variable and have it create an absolutely_no_punct variable.\n",
    "- Then ask for the first 100 items of absolutely_no_punct. \n",
    "\n",
    "Feel free to change the variable names as you like. I am going for clarity, but you might prefer brevity. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Did you notice that removing the punctuation has left list items that are empty strings. Between 'corpus' and 'it', for example, is an item shown as ''. This is an empty string item that was a full stop before we removed the punctuation. \n",
    "\n",
    "Why do you think these empty string items are included in the output list? \n",
    "Can you think of how we might remove this?\n",
    "Since those empty strings are python-recognised instances of 'None',  python can find and filter them out. \n",
    "\n",
    "Let's give it a try. Run/Shift+Enter. Do it!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus_no_space = list(filter(None, corpus_no_punct))     # This filters out the empty string from the no_punct list.\n",
    "\n",
    "print(corpus_no_space[:100])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we are cooking with gas (unless that saying is no longer environmentally sustainable? Hmmm. ). \n",
    "\n",
    "But we are not done yet! Next up... Stopwords!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Stopwords"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Stopwords are typically conjunctions ('and', 'or'), prepositions ('to', 'around'), determiners ('the', 'an'), possessives ('s) and the like. The are **REALLY** common in all languages, and tend to occur at about the same ratio in all kinds of writing, regardless of who did the writing or what it is about. These words are definitely important for structure as they make all the difference between \"Freeze *or* I'll shoot!\" and \"Freeze *and* I'll shoot!\". \n",
    "\n",
    "Buuuut... Many for many text-mining analyses, especially those that take the bag of words approach, these words don't have a whole lot of meaning in and of themselves. Thus, we want to remove them. \n",
    "\n",
    "Let's start by downloading the basic stopwords function built into nltk and storing the English language ones in a list called, appropriately enough, 'stop_words'. \n",
    "\n",
    "Then let's have a look at what is in that list with a print command by doing the whole Run/Shift+Enter thing in the next two (two?!?) code cells. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nltk.download('stopwords')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.corpus import stopwords\n",
    "stop_words = set(stopwords.words('english'))\n",
    "print(sorted(stop_words))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_____________________________________________________________________________________________________________________________\n",
    "Great. Now let's remove those stop_words by creating another list called corpus_no_stop_words. Then, we iterate over corpus_correct_spell, looking at them one by one and appending them to corpus_no_stop_words if and only if they do not match any of the items in the stop_words list. \n",
    "\n",
    "As you might expect, you should do the whole Run/Shift+Enter thing. Again. (I know, I know...)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "corpus_no_stop_words = []\n",
    "\n",
    "for word in corpus_lower:\n",
    "    if word not in stop_words:\n",
    "        corpus_no_stop_words.append(word)\n",
    "        \n",
    "        \n",
    "print(corpus_no_stop_words[:100])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_______________________________________________________________________________________________________________________________\n",
    "Hey now! That looks pretty good. Not perfect, but good.\n",
    "\n",
    "Want to try more? Run the same code above, but on 'corpus_words' rather than 'corpus_lower'. What happens? Why do you think that is?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Consolidation\n",
    "#### Stemming words"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can probably imagine what comes next by now. We import a specific tool from nltk (it is not called the natural language tool kit for nuthin'), define a function, create a fresh new corpus by applying the function to an existing corpus, and print the first hundred items to have a nosey. \n",
    "\n",
    "Go ahead. Run/Shift+Enter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from nltk.stem.porter import PorterStemmer\n",
    "\n",
    "porter = PorterStemmer()\n",
    "corpus_stemmed = [porter.stem(word) for word in corpus_no_space]\n",
    "print(corpus_stemmed[:100])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that 'sample' has become 'sampl', which collapses 'sampled' together with 'samples' and 'sampling' and 'sample'. This puts plurals and verb tenses all in the same form so they can be counted as instances of the \"same\" word.\n",
    "\n",
    "If we are happy with this stemming process, we might decide that we are done with the cleaning and can dive into the text-mining. \n",
    "\n",
    "Alternatively, we might decide to do a bit more cleaning, perhaps by downloading packages that replace contractions, so that 'haven't' would become 'have' and 'not'. There are many potentially useful changes like these that you may want to make. \n",
    "\n",
    "Buuuuuuuuuuuuuut... maybe we want to keep the count the verbs together and the nouns separetely? For that, we need the slightly more sophisticated approach of 'lemmatisation'. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Lemmatisation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Lemmatisation is similar to stemming, in that it aims to turn various forms of the same word into a single form. However, lemmatisation is a bit more sophisticated because: \n",
    "- It recognises irregular plurals and returns the correct singular form. Example = 'rocks' --> 'rock' but 'corpora' --> 'corpus' \n",
    "- If part of speech tags are supplied, it treats verbs, adjectives and nouns differenly, even if they have the same surface form. Example - 'caring' would not be changed if used as an adjective (as in 'his caring manner') but would go to 'care' if it was a verb (as in 'he is caring for baby squirrels'. In contrast, stemming would remove the 'ing' and turn 'caring' into 'car'. \n",
    "- If no part of speech tags are supplied, lemmatisation tools tend to assume words as nouns, so the process becomes a sophisticated de-pluraliser. \n",
    "\n",
    "Again, you import a specific tool from nltk, define a short form for its use, apply it to the relevant input variable, saving the output as a new variable with a suitable name. \n",
    "\n",
    "Once more, unto the Run/Shift+Enter!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nltk.download('wordnet')\n",
    "from nltk.corpus import wordnet\n",
    "from nltk.stem import WordNetLemmatizer\n",
    "lemmatizer = WordNetLemmatizer() \n",
    " \n",
    "print('rocks :', lemmatizer.lemmatize('rocks'))              #a few examples of lemmatising as a de-pluraliser\n",
    "print('corpora :', lemmatizer.lemmatize('corpora'))\n",
    "print('cares :', lemmatizer.lemmatize('cares'))              #no part of speech tag supplied, so 'cares' is treated as noun\n",
    "print('caring :', lemmatizer.lemmatize('caring', pos = \"v\")) #when part of speech tag added, 'caring' is treated as verb             \n",
    "print('cared :', lemmatizer.lemmatize('cared', pos = \"v\"))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The results show that our examples produce good output -  'rocks', 'corpora' and 'cares' are all de-pluralised correctly. The examples with part of speech tags also show that 'caring' and 'cared' are both correctly converted to 'care' as the base verb. \n",
    "\n",
    "Let's try it on our corpus, this time applying it to the 'corpus_no_space' variable, which has not had the stemming process applied to it. \n",
    "\n",
    "Run/Shift+Enter. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus_lemmed = [lemmatizer.lemmatize(word) for word in corpus_no_space]\n",
    "\n",
    "print(corpus_lemmed[:100])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well, the results are a bit mixed. There were no part of speech tags in our corpus, so everything was treated as nouns. The corpus has been effectively de-pluralised, but all of the different verb tenses remain. So, I guess we need to mark the corpus for part of speech tags, usually abbreviated to POS. \n",
    "\n",
    "But that is a topic for the next section!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have achieved a whole lot already! This is great work! \n",
    "\n",
    "Now, you will have to think carefully about:\n",
    "- what processes you will need for the analysis you want to run, \n",
    "- what is the right order of processes for your corpus/corpora and your research questions, and \n",
    "- how will you keep track of which processes you run and in which order. Replicability demands clear step-by-steps!\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Further reading and resources"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Books, tutorials, package recommendations, etc. for Python\n",
    "- Programming with Python for Social Scientists. Brooker, 2020. https://study.sagepub.com/brooker\n",
    "- Automate the Boring Stuff with Python: Practical Programming for Total Beginners, Sweigart, 2019. ISBN-13: 9781593279929\n",
    "- SentDex, python programming tutorials on YouTube https://www.youtube.com/user/sentdex\n",
    "- nltk (Natural Language Toolkit) https://www.nltk.org/book/ch01.html\n",
    "- nltk.corpus http://www.nltk.org/howto/corpus.html\n",
    "- spaCy https://nlpforhackers.io/complete-guide-to-spacy/\n",
    "\n",
    "Books and package recommendations for R\n",
    "- Quanteda, an R package for text analysis https://quanteda.io/​\n",
    "- Text Mining with R, a free online book https://www.tidytextmining.com/​"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right\"><a href=\"./4-Social-networks.ipynb\" target=_blank><i>Next section: Social networks</i></a></div>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}