code/2-Extraction.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![UKDS Logo](images/UKDS_Logos_Col_Grey_300dpi.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "# Text-mining: Basics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Welcome to this <a href=\"https://ukdataservice.ac.uk/\" target=_blank>UK Data Service</a> *Computational Social Science* training series! \n",
    "\n",
    "The various *Computational Social Science* training series, all of which guide you through some of the popular and useful computational techniques, tools, methods and concepts that social science research might want to use. For example, this series covers collecting data from websites and social media platorms, working with text data, conducting simulations (agent based modelling), and more. The series includes recorded video webinars, interactive notebooks containing live programming code, reading lists and more.\n",
    "\n",
    "* To access training materials on our GitHub site: <a href=\"https://github.com/UKDataServiceOpen/computational-social-science\" target=_blank>[Training Materials]</a>\n",
    "\n",
    "* To keep up to date with upcoming and past training events: <a href=\"https://ukdataservice.ac.uk/news-and-events/events\" target=_blank>[Events]</a>\n",
    "\n",
    "* To get in contact with feedback, ideas or to seek assistance: <a href=\"https://ukdataservice.ac.uk/help.aspx\" target=_blank>[Help]</a>\n",
    "\n",
    "<a href=\"https://www.research.manchester.ac.uk/portal/julia.kasmire.html\" target=_blank>Dr J. Kasmire</a>  <br />\n",
    "UK Data Service  <br />\n",
    "University of Manchester <br />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "toc": true
   },
   "source": [
    "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
    "<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#Introduction\" data-toc-modified-id=\"Introduction-1\"><span class=\"toc-item-num\">1&nbsp;&nbsp;</span>Introduction</a></span><ul class=\"toc-item\"><li><span><a href=\"#Guide-to-using-this-resource\" data-toc-modified-id=\"Guide-to-using-this-resource-1.1\"><span class=\"toc-item-num\">1.1&nbsp;&nbsp;</span>Guide to using this resource</a></span></li><li><span><a href=\"#Interaction\" data-toc-modified-id=\"Interaction-1.2\"><span class=\"toc-item-num\">1.2&nbsp;&nbsp;</span>Interaction</a></span></li><li><span><a href=\"#Learn-more\" data-toc-modified-id=\"Learn-more-1.3\"><span class=\"toc-item-num\">1.3&nbsp;&nbsp;</span>Learn more</a></span></li></ul></li><li><span><a href=\"#Preliminary-NLP-(or-finishing-up-the-processing)\" data-toc-modified-id=\"Preliminary-NLP-(or-finishing-up-the-processing)-2\"><span class=\"toc-item-num\">2&nbsp;&nbsp;</span>Preliminary NLP (or finishing up the processing)</a></span><ul class=\"toc-item\"><li><span><a href=\"#POS---part-of-speech-tagging\" data-toc-modified-id=\"POS---part-of-speech-tagging-2.1\"><span class=\"toc-item-num\">2.1&nbsp;&nbsp;</span>POS - part of speech tagging</a></span></li><li><span><a href=\"#Named-Entity-Recogntion-and-chunking\" data-toc-modified-id=\"Named-Entity-Recogntion-and-chunking-2.2\"><span class=\"toc-item-num\">2.2&nbsp;&nbsp;</span>Named Entity Recogntion and chunking</a></span></li></ul></li><li><span><a href=\"#Counts-and-(relative)-frequency\" data-toc-modified-id=\"Counts-and-(relative)-frequency-3\"><span class=\"toc-item-num\">3&nbsp;&nbsp;</span>Counts and (relative) frequency</a></span></li><li><span><a href=\"#Similarity\" data-toc-modified-id=\"Similarity-4\"><span class=\"toc-item-num\">4&nbsp;&nbsp;</span>Similarity</a></span></li><li><span><a href=\"#Discovery\" data-toc-modified-id=\"Discovery-5\"><span class=\"toc-item-num\">5&nbsp;&nbsp;</span>Discovery</a></span></li><li><span><a href=\"#Conclusions\" data-toc-modified-id=\"Conclusions-6\"><span class=\"toc-item-num\">6&nbsp;&nbsp;</span>Conclusions</a></span></li><li><span><a href=\"#Further-reading-and-resources\" data-toc-modified-id=\"Further-reading-and-resources-7\"><span class=\"toc-item-num\">7&nbsp;&nbsp;</span>Further reading and resources</a></span></li></ul></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "There is a table of contents provided here at the top of the notebook, but you can also access this menu at any point by clicking the Table of Contents button on the top toolbar (an icon with four horizontal bars, if unsure hover your mouse over the buttons). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-------------------------------------\n",
    "\n",
    "<div style=\"text-align: center\"><i><b>This is notebook 2 of 2 in this lesson</i></b></div>\n",
    "\n",
    "-------------------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At the end of the last section, we got held up by the need to do part-of-speech (POS) tagging in order to get a really effective lemmatisation process. POS tagging is actually basic NLP, as opposed to the kind of cleaning and regularising that we were doing in the last section. So why should a NLP process be needed before the preparatory processing is done? \n",
    "\n",
    "Well, it comes done to choices. Not every analysis will need a sophisticated lemmatiser, so those projects may have a nice finish to the processing step and the start of the extraction or NLP step. Others willneed the lemmatiser or name entity recognisers or other advanced preparatory steps. Those projects will have a less clear distinction between preparation for NLP and NLP. \n",
    "\n",
    "But even if the project ends up having a clear distinction between the processes, researchers may find that after they start doing some NLP processes, they need to go back and run different preparatory processes instead of or in addition to the ones they chose earlier. \n",
    "\n",
    "The main takeaway point here is that researchers need to know that developing a text-mining project can be messy, iterative, and complicated. I recommend that you think about each step as elements in a pipeline (or in multiple pipelines). I recommend that you build your own code functions that concatenate the steps, running each one from the output of the previous one. \n",
    "\n",
    "In this way, you get a fresh clean output at the end of the pipeline each time whenever you need one. It also means that everything you apply the pipeline to gets treated in the same way, with each process done in the same order. This helps replicability!\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Guide to using this resource\n",
    "\n",
    "This learning resource was built using <a href=\"https://jupyter.org/\" target=_blank>Jupyter Notebook</a>, an open-source software application that allows you to mix code, results and narrative in a single document. As <a href=\"https://jupyter4edu.github.io/jupyter-edu-book/\" target=_blank>Barba et al. (2019)</a> espouse:\n",
    "> In a world where every subject matter can have a data-supported treatment, where computational devices are omnipresent and pervasive, the union of natural language and computation creates compelling communication and learning opportunities.\n",
    "\n",
    "If you are familiar with Jupyter notebooks then skip ahead to the main content (*Preliminary NLP*). Otherwise, the following is a quick guide to navigating and interacting with the notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Interaction\n",
    "\n",
    "**You only need to execute the code that is contained in sections which are marked by `In []`.**\n",
    "\n",
    "To execute a cell, click or double-click the cell and press the `Run` button on the top toolbar (you can also use the keyboard shortcut Shift + Enter).\n",
    "\n",
    "Try it for yourself:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Enter your name and press enter:\")\n",
    "name = input()\n",
    "print(\"\\r\")\n",
    "print(\"Hello {}, enjoy learning more about Python and computational social science!\".format(name)) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Learn more\n",
    "\n",
    "Jupyter notebooks provide rich, flexible features for conducting and documenting your data analysis workflow. To learn more about additional notebook features, we recommend working through some of the <a href=\"https://github.com/darribas/gds19/blob/master/content/labs/lab_00.ipynb\" target=_blank>materials</a> provided by Dani Arribas-Bel at the University of Liverpool."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Preliminary NLP (or finishing up the processing)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start off by importing and downloading all the things we will need. \n",
    "\n",
    "Run/Shift+Enter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk                       # get nltk \n",
    "from nltk import word_tokenize    # and some of its key functions\n",
    "from nltk import sent_tokenize    \n",
    "\n",
    "!pip install autocorrect          \n",
    "from autocorrect import Speller   # things we need for spell checking\n",
    "check = Speller(lang='en')\n",
    "\n",
    "import re                         # things we need for RegEx corrections\n",
    "def multiple_replace(dict, text):\n",
    "  regex = re.compile(\"(%s)\" % \"|\".join(map(re.escape, dict.keys())))\n",
    "  return regex.sub(lambda mo: dict[mo.string[mo.start():mo.end()]], text) \n",
    "\n",
    "if __name__ == \"__main__\": \n",
    "  dict = {\n",
    "    \"CA\" : \"California\",\n",
    "    \"United Kingdom\" : \"U.K.\",\n",
    "    \"United Kingdom of Great Britain and Northern Ireland\" : \"U.K.\",\n",
    "    \"United Kingdom of Great Britain\" : \"U.K.\",\n",
    "    \"UK\" : \"U.K.\",\n",
    "    \"Privacy Policy\" : \"noodle soup\",}\n",
    "\n",
    "English_punctuation = \"-!\\\"#$%&()'*+,./:;<=>?@[\\]^_`{|}~''“”\"      # Things for removing punctuation, stopwords and empty strings\n",
    "table_punctuation = str.maketrans('','', English_punctuation)  \n",
    "\n",
    "nltk.download('stopwords')\n",
    "nltk.download('punkt')\n",
    "nltk.download('wordnet')\n",
    "nltk.download('webtext')\n",
    "\n",
    "from nltk.corpus import stopwords\n",
    "stop_words = set(stopwords.words('english'))\n",
    "\n",
    "from nltk.corpus import wordnet                    # Finally, things we need for lemmatising!\n",
    "from nltk.stem import WordNetLemmatizer\n",
    "lemmatizer = WordNetLemmatizer() \n",
    "nltk.download('averaged_perceptron_tagger')        # Like a POS-tagger...\n",
    "\n",
    "print(\"Succesfully imported necessary modules\")    # The print statement is just a bit of encouragement!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### POS - part of speech tagging"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's get back to where we were when we left off last time - with a tokenised corpus on which we need to run a POS tagger. \n",
    "\n",
    "Run/Shift+Enter, as above!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(\"./data/sample_text.txt\", \"r\", encoding = \"ISO-8859-1\") as f:\n",
    "    corpus = f.read()\n",
    "    \n",
    "corpus_words = word_tokenize(corpus)\n",
    "\n",
    "corpus_lower = [word.lower() for word in corpus_words]\n",
    "\n",
    "corpus_correct_spell = []\n",
    "for word in corpus_lower:\n",
    "    corpus_correct_spell.append(check(word))    \n",
    "\n",
    "corpus_no_stopwords = []\n",
    "for word in corpus_correct_spell:\n",
    "    if word not in stop_words:\n",
    "        corpus_no_stopwords.append(word)\n",
    "        \n",
    "corpus_no_punct = [w.translate(table_punctuation) for w in corpus_no_stopwords]  \n",
    "corpus_no_space = list(filter(None, corpus_no_punct))      \n",
    "        \n",
    "print(corpus_no_space[:100])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Excellent. Now it is time to tag that corpus with POS-tags. This is pretty easy, as nltk comes with a POS-tagger. \n",
    "\n",
    "Run/Shift+Enter, as you would expect. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus_pos_tagged = nltk.pos_tag(corpus_no_space)        \n",
    "print(corpus_pos_tagged[:100])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Excellent. That has successfully added POS tags to all off the words in our corpus. Now, let's try lemmatising again with the POS tags. \n",
    "\n",
    "Despite what seems obvious, the nltk POS tagger does not use the same POS tags that the nltk lemmatize function needs. Why? I have no idea. \n",
    "\n",
    "But to move forward, I need a to define a quick little function called get_wordnet_pos to convert the tag format to the right one. I tell a lie. I did not write this function but copied it off of Stack Overflow. This is not cheating so much as being economical. A HUGE number of the things you want to do or the problems you want to solve will be discussed on Stack Overflow. Just use a popular search engine to find them, read through all the answers, try them out. \n",
    "\n",
    "Having defined the get_wordnet_pos function, the code belowe then creates a new, blank list called corpus_lemmed. \n",
    "After that, the code iterates over corpus_pos_tagged, looking at each word and POS-tag pair, uses the get_wordnet_pos function to convert the POS-tag to the right format, and using that to lemmatize correctly. \n",
    "\n",
    "At the end, the lemmatised word is appended to the new list we created. \n",
    "\n",
    "Go ahead. \n",
    "Run/Shift+Enter. \n",
    "You know you want to!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_wordnet_pos(word):\n",
    "    \"\"\"Map POS tag to first character lemmatize() accepts\"\"\"\n",
    "    tag = nltk.pos_tag([word])[0][1][0].upper()\n",
    "    tag_dict = {\"J\": wordnet.ADJ,\n",
    "                \"N\": wordnet.NOUN,\n",
    "                \"V\": wordnet.VERB,\n",
    "                \"R\": wordnet.ADV}\n",
    "    return tag_dict.get(tag, wordnet.NOUN)\n",
    "\n",
    "corpus_lemmed = []\n",
    "for pair in corpus_pos_tagged:\n",
    "    corpus_lemmed.append(lemmatizer.lemmatize(pair[0], get_wordnet_pos(pair[0])))   \n",
    "print(corpus_lemmed[:100])\n",
    "\n",
    "#corpus_lemmed_tagged = []    \n",
    "#for pair in corpus_pos_tagged:\n",
    "#    corpus_lemmed_tagged.append([(lemmatizer.lemmatize(pair[0], get_wordnet_pos(pair[0])), pair[1])])   \n",
    "#print(corpus_lemmed_tagged[:100])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output of the above code returns a list with only words but without any POS-tags. \n",
    "\n",
    "If you want to keep the corpus in pairs of word and POS-tag, you will need to activate the second, commented out lines. This means you will need to  remove the '#' in front of each line of code starting with 'corpus_lemmed_tagged' and re-run the code.\n",
    "\n",
    "Give it a try!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Named Entity Recogntion and chunking"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are really getting somewhere now! Let's try another basic NLP process - CHUNKING!\n",
    "\n",
    "Named Entity Recognition is a specific kind of 'chunk' operation. Chunking operations iterate over a corpus that has been word tokenised and POS-tagged and put it all back together into sentences. Named Entity Recognition does this too, with special attention to building up the noun phrases that capture well-known entities or onganisations. \n",
    "\n",
    "The chunks are returned within sets of nested brackets (both square and round to capture different levels of nesting. \n",
    "\n",
    "So, 'The Cat in the Hat' would come out as (S The/DT (ORGANIZATION Cat/NNP) in/IN the/DT Hat/NNP). \n",
    "The 'S' at the beginning stands for 'sentence' which is the highest level grouping that the chunker can find. \n",
    "the 'Cat' is recognised as the key entity, so is tagged with ORGANIZATION. \n",
    "The 'in the hat' part is captured as belonging to a noun phrase, same as the cat, but it recongsises that this a sentence about a cat, not a sentence about a hat. \n",
    "\n",
    "Clever, eh?\n",
    "Let's try it!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#importing chunk library from nltk\n",
    "nltk.download('words')\n",
    "nltk.download('maxent_ne_chunker')\n",
    "from nltk import ne_chunk                                    # ne_chunk is 'named entity chunk'. Other chunkers are available.\n",
    "\n",
    "# NER and other chunkers only work on word tokenised and POS tagged corpora... \n",
    "corpus_pos_tagged2 = nltk.pos_tag(corpus_words)\n",
    "corpus_chunked = ne_chunk(corpus_pos_tagged2)\n",
    "print(corpus_chunked[57:88])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This time, I specifically only asked for a printout of a key range in the resulting corpus. I wanted to highlight here how the word \"Tree\" preceeds those ORGANIZATION entities that hang together as multi-word entities. See, for example, how 'United Kingdom', 'Great Britain' and 'Northern Ireland' are each within square brackets to identify them as the multi-word entity captured by the ORGANIZATION tag. \n",
    "\n",
    "You might also have noticed that this chunking function is run on corpus_pos_tagged2, which is simply the corpus_words that has been put through the nltk.pos_tag function. This means that corpus_pos_tagged2 still has its stopwords, punctuation, etc. \n",
    "\n",
    "Why do you think this is? What do you think would happen if you ran the chunking procedure on corpus_pos_tagged which DOES have all the stopwords and punctuation removed?\n",
    "\n",
    "Well, guess what? You can find out by doing that Run/Shift+Enter thing!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "corpus_chunked_extra_processes = ne_chunk(corpus_pos_tagged)\n",
    "print(corpus_chunked_extra_processes[:100])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hmmm. No 'Tree' markers, no 'ORGANIZATION' markers, etc. \n",
    "\n",
    "This is because some chunking processes use some of the stopwords (especially determiners like 'an' and 'the') and punctuation, etc. to be useful in determining appropriate chunks. \n",
    "\n",
    "This may create some challenges for your corpus. For example, if you want to: \n",
    "- Count words, then you probably want to remove stopwords, punctuation, etc. \n",
    "- Identify chunks, like named entities, then you probably want to leave some or all of the stopwords, punctuation, etc.\n",
    "- Count chunks (e.g. count named entities), you probably want to combine the processes in the right order. \n",
    "\n",
    "Good to know!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Counts and (relative) frequency"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Excellent! Now, you might be surprised, but a very important function of NLP for analysing text boils down to counting things, often words. This is why so much attention in the last section was focussed on making sure all the words that we want to be counted as 'the same word' appeared in the same form while all the words that we want to count as 'different words' appear in different forms. \n",
    "\n",
    "Thus, we want to apply the count functions to a corpus that has had some of that standardisation, consolidation, lemmatised (or at least stemming) processes applied already. \n",
    "\n",
    "First, we import some counting functions, then we apply them to corpus_lemmed. \n",
    "Run/Shift+Enter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import Counter\n",
    "corpus_counts = Counter(corpus_lemmed)\n",
    "print(corpus_counts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great! You may have noticed that we applied this count function to a list of words rather than word and POS-tag pairs. This is on purpose, but the code could be written so that it only looks at the first item in each word and POS-tag pairs. \n",
    "\n",
    "If you want to try that, go ahead. You may want to refer back to the code block where we defined get_wordnet_pos because the code to create corpus_lemmed_tagged uses indices (in [square brackets]) to refer to only one element within a pair. \n",
    "\n",
    "But, for now, let's have a closer look at the 100 most common words in our corpus by using the 'most_common' function from Counter. \n",
    "Run/Shift+Enter!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "print(corpus_counts.most_common(100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just for comparison, let's find the 100 most common words in 'Emma' by Jane Austen. \n",
    "We do need to import the text as a corpus and process it in the same way as we did your corpus so that they can be seen as comparable. \n",
    "\n",
    "Run/Shift+Enter  - but be patient. This is a lot of processes to run. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nltk.download('gutenberg')\n",
    "import nltk.corpus\n",
    "emma = nltk.corpus.gutenberg.raw('austen-emma.txt')\n",
    "\n",
    "emma_words = word_tokenize(emma)\n",
    "\n",
    "emma_lower = [word.lower() for word in emma_words]\n",
    "\n",
    "emma_correct_spell = []\n",
    "for word in emma_lower:\n",
    "    emma_correct_spell.append(check(word))    \n",
    "\n",
    "    emma_no_stopwords = []\n",
    "for word in emma_lower:\n",
    "    if word not in stop_words:\n",
    "        emma_no_stopwords.append(word)\n",
    "        \n",
    "emma_no_punct = [w.translate(table_punctuation) for w in emma_no_stopwords]  \n",
    "emma_no_space = list(filter(None, emma_no_punct))              \n",
    "emma_pos_tagged = nltk.pos_tag(emma_no_space)   \n",
    "\n",
    "emma_lemmed = []\n",
    "for pair in emma_pos_tagged:\n",
    "    emma_lemmed.append(lemmatizer.lemmatize(pair[0], get_wordnet_pos(pair[0])))   \n",
    "    \n",
    "emma_counts = Counter(emma_lemmed)\n",
    "print(emma_counts)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Excellent! Clearly, the words in Emma are very different than those of our sample corpus, and those words that appear in both occur in very different relative frequencies (not least because one is a page of babble and the other is a full novel).\n",
    "\n",
    "To get a better idea, how about we compare the 20 most common words from both corpora. \n",
    "Run/Shift+Enter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(corpus_counts.most_common(20))\n",
    "print(emma_counts.most_common(20))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok. Clearly very different. But let's try one more thing for now... Let's count how many times each of these texts use the word 'personal'. We could use any word as the target word, but I happen to know that there is a non-zero result for these two texts for this word. \n",
    "\n",
    "Run/Shift+Enter!!!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(corpus_counts['personal'])\n",
    "print(emma_counts['personal'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, despite being much shorter, the sample text corpus uses the word 'personal' over 8 times more often. \n",
    "\n",
    "That sounds very personal. \n",
    "\n",
    "Feel free to choose other words and re-run the code. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Similarity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, comparing the most common words in two documents is one way to compare how similar they are, but there are more sophisticated ways. \n",
    "\n",
    "spaCy is a relatively new option for text-mining in python, but it is very powerful. First off, we need to download and import a few things. \n",
    "\n",
    "Run/Shift+Enter (you are already so good at this!)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install spacy -q\n",
    "import spacy\n",
    "!python -m spacy download en_core_web_lg -q\n",
    "from nltk.corpus import webtext"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Super. Now, let's load the model via spacy-load, and then test it on a trivial corpus that has only three words. \n",
    "\n",
    "Run/Shift+Enter already!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nlp = spacy.load('en_core_web_lg')\n",
    "\n",
    "word_similarity = nlp(\"troll elf rabbit\")\n",
    "\n",
    "\n",
    "for word1 in word_similarity:\n",
    "    for word2 in word_similarity:\n",
    "        print(word1.text, word2.text, word1.similarity(word2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This code does a few things. First, it loads a model of common words in English (this is the 'en_core_web_lg') that has 300 dimension vectors for each word. If that sounds like nonsense, don't worry too much. \n",
    "\n",
    "What it means is that the model has a list of lots of common words in English, each of which comes with a 'scorecard' of how they rank on 300 different features which is a sort of abstract way of capturing the meaning of the word. This is not derived from logical scoring by people, but through an AI sort of analysis of how the words are used in LOTS of text, which finds patterns like:\n",
    "- Is the target word used more often like a noun or a verb? \n",
    "- Is it usually plural (if a noun) or in gerrund ('ing'-form, if a verb)? \n",
    "- Is it frequently preceded by adjectives like 'little' or 'unprecedented' or adverbs like 'always' or 'never'? \n",
    "\n",
    "What comes out of this code is a pair-wise comparison of all the vectors, or scorecards, for the words in our little three-word corpus. This comes out as a number between 0 and 1, with 0 being totally different (or not found in the model) and 1 being a perfect match. \n",
    "\n",
    "Looking at the results, we see that comparing a word to itself (e.g. the first line which has 'dog dog 1.0') scores a 1, or 100 percent match. Not surprising. \n",
    "\n",
    "We also see that 'dog' and 'cat' are a pretty good match at 0.8. Both words are likely to be used in similar ways. For example, both would fit equally well into a sentence like \"I really want to get a pet (dog/cat), but I just don't spend enough time at home to take care of it properly.\"\n",
    "\n",
    "We also see that 'banana' is closer to 'cat' than to 'dog', but not by much. Presumably, bananas are more likely to sit around like cats than to run around like dogs? No idea. \n",
    "\n",
    "Feel free to edit the little three-word corpus and re-run the similarity test. Try 'puppy' to see if it is closer to 'dog' than 'dog' is to 'cat'? Try adding 'apple'? Or 'unprecedented'? Or anything else?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Of course, comparing individual words is all well and good, but what you probably want to compare is one text to another. To do that, first we need to prepare a few texts to do some comparing. You have already seen our sample corpus and 'Emma' by Jane Austen, but let's also add 'Persuasion' by Jane Austen and a selection of text from another nltk.corpus of texts from the web. The specific text is called 'firefox'. \n",
    "\n",
    "All of these texts need to be put through the nlp function we created from spaCy so that it creates a document vector. \n",
    "\n",
    "Document vectors are much like word vectors in that they score the document on a large number of dimensions. However, instead of coming packaged with spaCy, they are created from the text that you pass to spaCy. Which is what we are doing now.\n",
    "\n",
    "Run/Shift+Enter below!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "SimEmma = nlp(nltk.corpus.gutenberg.raw('austen-emma.txt'))\n",
    "SimPers = nlp(nltk.corpus.gutenberg.raw('austen-persuasion.txt'))\n",
    "SimFire = nlp(nltk.corpus.webtext.raw('firefox.txt'))\n",
    "SimCorp = nlp(corpus)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can, of course, have a peek at the contents of 'Persuasion' or 'firefox' if you like. You probably know how to do that with a print command, but maybe you want to run some of the other operations on the text too. \n",
    "\n",
    "Or you can plow on ahead and Run/Shift+Enter to run the document vector similarity comparisons below. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(SimEmma.similarity(SimPers))\n",
    "print(SimEmma.similarity(SimFire))\n",
    "print(SimEmma.similarity(SimCorp))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Each of these compares 'Emma' to one of the other texts (the ones in parentheses at the end). \n",
    "\n",
    "Are you surprised by the results? Feel free to try comparing the other texts to each other, rather than just to 'Emma'. \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Discovery\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, for the final bit of NLP that we cover here, let's talk about discovery. This is about identifying patterns that reveal relationships and applying it more widely to discover additional relationships. Let's start by importing a few things that we need. \n",
    "\n",
    "Run/Shift+Enter. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re \n",
    "import string \n",
    "import nltk \n",
    "import spacy \n",
    "import pandas as pd \n",
    "import numpy as np \n",
    "import math \n",
    "from tqdm import tqdm \n",
    "\n",
    "from spacy.matcher import Matcher \n",
    "from spacy.tokens import Span \n",
    "from spacy import displacy \n",
    "\n",
    "pd.set_option('display.max_colwidth', 200)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's take a look at 'Emma'. We start by tokenising the raw text into sentences, then we create a list of all sentences that contain the sub-string \"like a\", then we create run our list through our nlp function from spaCy. \n",
    "\n",
    "Run/Shift+Enter. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# sample text \n",
    "emma_sentences = nltk.sent_tokenize(nltk.corpus.gutenberg.raw('austen-emma.txt'))\n",
    "emma_such_as =\"\"\n",
    "for sentence in emma_sentences:\n",
    "    if \"like a \" in sentence:\n",
    "        emma_such_as += sentence\n",
    "                \n",
    "# create a spaCy object \n",
    "doc = nlp(emma_such_as)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, let's take a closer look at context around those instances of \"like a\". \n",
    "\n",
    "To do that, we use some spaCy functions that print the word, print its role in the sentence, and print its POS-tag. \n",
    "\n",
    "This lets us see if we can find any patterns in the word roles or POS-tags that might help us understand the patterns relating to \"like a\". \n",
    "\n",
    "Run/Shift+Enter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# print token, dependency, POS tag \n",
    "for tok in doc: \n",
    "  print(tok.text, \"-->\",tok.dep_,\"-->\", tok.pos_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It seems like a good start would be to define a pattern with \"like a\" followed by a noun. So first, we define that pattern. \n",
    "\n",
    "Run/Shift+Enter. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#define the pattern \n",
    "pattern = [{'LOWER': 'like'}, \n",
    "           {'LOWER': 'a'}, \n",
    "           {'POS': 'NOUN'}]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, we run a function called Matcher over the text that returns all of the substrings that match the pattern. \n",
    "\n",
    "Run/Shift+Enter. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Matcher class object \n",
    "matcher = Matcher(nlp.vocab) \n",
    "matcher.add(\"matching_1\", [pattern]) \n",
    "\n",
    "matches = matcher(doc) \n",
    "for match_id, start, end in matches:\n",
    "    string_id = nlp.vocab.strings[match_id]  # Get string representation\n",
    "    span = doc[start:end]  # The matched span\n",
    "    print(span.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you would expect, we get a list of substrings that match the pattern we have defined. Let's create a little more ambitious pattern. \n",
    "\n",
    "This time, we want to capture verbs followed by \"like a\" followed by up to three optional modifiers (adverbs and adjectives) and finally followed by a noun. \n",
    "\n",
    "Run/Shift+Enter. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Matcher class object\n",
    "matcher = Matcher(nlp.vocab)\n",
    "\n",
    "#redefine the pattern\n",
    "pattern2 = [{'POS':'VERB'},\n",
    "           {'LOWER': 'like'},\n",
    "           {'LOWER': 'a'},\n",
    "           {'DEP':'amod', 'OP':\"?\"},\n",
    "           {'DEP':'amod', 'OP':\"?\"},\n",
    "           {'DEP':'amod', 'OP':\"?\"},\n",
    "           {'POS': 'NOUN'}]\n",
    "\n",
    "matcher.add(\"matching_1\", [pattern2])\n",
    "matches = matcher(doc)\n",
    "\n",
    "for match_id, start, end in matches:\n",
    "    string_id = nlp.vocab.strings[match_id]  # Get string representation\n",
    "    span = doc[start:end]  # The matched span\n",
    "    print(span.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well, this is more interesting. We see that some one can 'look like a sensible young man', can 'argue like a young man' and can 'write like a sensible man'. This suggests that the author, or possible society at the time of publication, associates these verbs and these adjectives with men. Potentially, the analysis is even more complicated in that young men might argue while old men do not, or that sensible men write much more often than other men. \n",
    "\n",
    "If we continued to analyse the text in this way, we might also find a similar combinations for verbs and adjectives associated with women. Will there be any evidence that (young) women are sensible? Or that women write or argue in ways that are comparable to how men write and argue? \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have only started to dip our toes into what NLP can do, but hopefully this will whet your appetite to know more. \n",
    "\n",
    "As before, these exercises and this sample code should highlight to you that you need to think about:\n",
    "- your research questions and what you want to show, explore or understand, \n",
    "- your data, texts, corpus, or other research materials to analyse etc. \n",
    "- how your processes are related to your reserch questions, and \n",
    "- how your processes and data can be made available and reproducible. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Further reading and resources"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Books, tutorials, package recommendations, etc. for Python\n",
    "- Natural Language Processing with Python by Steven Bird, Ewan Klein and Edward Loper, http://www.nltk.org/book/\n",
    "- Foundations of Statistical Natural Language Processing by Christopher Manning and Hinrich Schütze, https://nlp.stanford.edu/fsnlp/promo/\n",
    "- Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics and Speech Recognition by Dan Jurafsky and James H. Martin, https://web.stanford.edu/~jurafsky/slp3/ed3book.pdf\n",
    "- Deep Learning in Natural Language Processing by Li Deng, Yang Liu, https://lidengsite.wordpress.com/book-chapters/\n",
    "- nltk.corpus http://www.nltk.org/howto/corpus.html\n",
    "- spaCy https://nlpforhackers.io/complete-guide-to-spacy/\n",
    "\n",
    "Books and package recommendations for R\n",
    "- Quanteda, an R package for text analysis https://quanteda.io/​\n",
    "- Text Mining with R, a free online book https://www.tidytextmining.com/​"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right\"><a href=\"./1-Processing.ipynb\" target=_blank><i>Previous section: Processing text</i></a></div>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}