code/3-Classifiers.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![UKDS Logo](images/UKDS_Logos_Col_Grey_300dpi.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "# Text-mining: Classifiers and sentiment analysis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Welcome to this <a href=\"https://ukdataservice.ac.uk/\" target=_blank>UK Data Service</a> *Computational Social Science* training series! \n",
    "\n",
    "The various *Computational Social Science* training series, all of which guide you through some of the popular and useful computational techniques, tools, methods and concepts that social science research might want to use. For example, this series covers collecting data from websites and social media platorms, working with text data, conducting simulations (agent based modelling), and more. The series includes recorded video webinars, interactive notebooks containing live programming code, reading lists and more.\n",
    "\n",
    "* To access training materials on our GitHub site: <a href=\"https://github.com/UKDataServiceOpen/computational-social-science\" target=_blank>[Training Materials]</a>\n",
    "\n",
    "* To keep up to date with upcoming and past training events: <a href=\"https://ukdataservice.ac.uk/news-and-events/events\" target=_blank>[Events]</a>\n",
    "\n",
    "* To get in contact with feedback, ideas or to seek assistance: <a href=\"https://ukdataservice.ac.uk/help.aspx\" target=_blank>[Help]</a>\n",
    "\n",
    "<a href=\"https://www.research.manchester.ac.uk/portal/julia.kasmire.html\" target=_blank>Dr J. Kasmire</a>  <br />\n",
    "UK Data Service  <br />\n",
    "University of Manchester <br />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "toc": true
   },
   "source": [
    "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
    "<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#Introduction\" data-toc-modified-id=\"Introduction-1\"><span class=\"toc-item-num\">1&nbsp;&nbsp;</span>Introduction</a></span><ul class=\"toc-item\"><li><span><a href=\"#Guide-to-using-this-resource\" data-toc-modified-id=\"Guide-to-using-this-resource-1.1\"><span class=\"toc-item-num\">1.1&nbsp;&nbsp;</span>Guide to using this resource</a></span></li><li><span><a href=\"#Interaction\" data-toc-modified-id=\"Interaction-1.2\"><span class=\"toc-item-num\">1.2&nbsp;&nbsp;</span>Interaction</a></span></li><li><span><a href=\"#Learn-more\" data-toc-modified-id=\"Learn-more-1.3\"><span class=\"toc-item-num\">1.3&nbsp;&nbsp;</span>Learn more</a></span></li></ul></li><li><span><a href=\"#Sentiment-Analysis-as-an-example-of-machine-learning/deep-learning-classification\" data-toc-modified-id=\"Sentiment-Analysis-as-an-example-of-machine-learning/deep-learning-classification-2\"><span class=\"toc-item-num\">2&nbsp;&nbsp;</span>Sentiment Analysis as an example of machine learning/deep learning classification</a></span></li><li><span><a href=\"#Analyse-trivial-documents-with-built-in-sentiment-analysis-tool\" data-toc-modified-id=\"Analyse-trivial-documents-with-built-in-sentiment-analysis-tool-3\"><span class=\"toc-item-num\">3&nbsp;&nbsp;</span>Analyse trivial documents with built-in sentiment analysis tool</a></span></li><li><span><a href=\"#Acquire-and-analyse-trivial-documents\" data-toc-modified-id=\"Acquire-and-analyse-trivial-documents-4\"><span class=\"toc-item-num\">4&nbsp;&nbsp;</span>Acquire and analyse trivial documents</a></span></li><li><span><a href=\"#Train-and-test-a-sentiment-analysis-tool-with-trivial-data\" data-toc-modified-id=\"Train-and-test-a-sentiment-analysis-tool-with-trivial-data-5\"><span class=\"toc-item-num\">5&nbsp;&nbsp;</span>Train and test a sentiment analysis tool with trivial data</a></span></li><li><span><a href=\"#You-can-train-and-test-a-sentiment-analysis-tool-with-more-interesting-data-too...\" data-toc-modified-id=\"You-can-train-and-test-a-sentiment-analysis-tool-with-more-interesting-data-too...-6\"><span class=\"toc-item-num\">6&nbsp;&nbsp;</span>You can train and test a sentiment analysis tool with more interesting data too...</a></span></li><li><span><a href=\"#Conclusions\" data-toc-modified-id=\"Conclusions-7\"><span class=\"toc-item-num\">7&nbsp;&nbsp;</span>Conclusions</a></span></li><li><span><a href=\"#Further-reading-and-resources\" data-toc-modified-id=\"Further-reading-and-resources-8\"><span class=\"toc-item-num\">8&nbsp;&nbsp;</span>Further reading and resources</a></span></li></ul></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "There is a table of contents provided here at the top of the notebook, but you can also access this menu at any point by clicking the Table of Contents button on the top toolbar (an icon with four horizontal bars, if unsure hover your mouse over the buttons). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sentiment analysis is a commonly used example of automatic classification. To be clear, automatic classification means that a model or learning algorithm has been trained on correctly classified documents and it uses this training to return a probability assessment of what class a new document should belong to. \n",
    "\n",
    "Sentiment analysis works the same way, but usually only has two classes - positive and negative. A trained model looks at new data and says whether that new data is likely to be positive or negative. Let's take a look!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Guide to using this resource\n",
    "\n",
    "This learning resource was built using <a href=\"https://jupyter.org/\" target=_blank>Jupyter Notebook</a>, an open-source software application that allows you to mix code, results and narrative in a single document. As <a href=\"https://jupyter4edu.github.io/jupyter-edu-book/\" target=_blank>Barba et al. (2019)</a> espouse:\n",
    "> In a world where every subject matter can have a data-supported treatment, where computational devices are omnipresent and pervasive, the union of natural language and computation creates compelling communication and learning opportunities.\n",
    "\n",
    "If you are familiar with Jupyter notebooks then skip ahead to the main content (*Sentiment Analysis as an example of machine learning/deep learning classification*). Otherwise, the following is a quick guide to navigating and interacting with the notebook."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Interaction\n",
    "\n",
    "**You only need to execute the code that is contained in sections which are marked by `In []`.**\n",
    "\n",
    "To execute a cell, click or double-click the cell and press the `Run` button on the top toolbar (you can also use the keyboard shortcut Shift + Enter).\n",
    "\n",
    "Try it for yourself:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Enter your name and press enter:\n",
      "Louise\n",
      "\n",
      "Hello Louise, enjoy learning more about Python and computational social science!\n"
     ]
    }
   ],
   "source": [
    "print(\"Enter your name and press enter:\")\n",
    "name = input()\n",
    "print(\"\\r\")\n",
    "print(\"Hello {}, enjoy learning more about Python and computational social science!\".format(name)) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Learn more\n",
    "\n",
    "Jupyter notebooks provide rich, flexible features for conducting and documenting your data analysis workflow. To learn more about additional notebook features, we recommend working through some of the <a href=\"https://github.com/darribas/gds19/blob/master/content/labs/lab_00.ipynb\" target=_blank>materials</a> provided by Dani Arribas-Bel at the University of Liverpool."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sentiment Analysis as an example of machine learning/deep learning classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start off by importing and downloading some useful packages, including `textblob`: it is based on `nltk` and has built in sentiment analysis tools. \n",
    "\n",
    "To import the packages, click in the code cell below and hit the 'Run' button at the top of this page or by holding down the 'Shift' key and hitting the 'Enter' key. \n",
    "\n",
    "For the rest of this notebook, I will use 'Run/Shift+Enter' as short hand for 'click in the code cell below and hit the 'Run' button at the top of this page or by hold down the 'Shift' key while hitting the 'Enter' key'. \n",
    "\n",
    "Run/Shift+Enter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "A file we can use is...  testing_set.csv\n",
      "A file we can use is...  training_set.csv\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import os                         # os is a module for navigating your machine (e.g., file directories).\n",
    "import nltk                       # nltk stands for natural language tool kit and is useful for text-mining. \n",
    "import csv                        # csv is for importing and working with csv files\n",
    "import statistics\n",
    "\n",
    "# List all of the files in the \"data\" folder that is provided to you\n",
    "\n",
    "for file in os.listdir(\"./data/sentiment-analysis\"):\n",
    "   print(\"A file we can use is... \", file)\n",
    "print(\"\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package brown to /Users/loucap/nltk_data...\n",
      "[nltk_data]   Package brown is already up-to-date!\n",
      "[nltk_data] Downloading package punkt to /Users/loucap/nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n",
      "[nltk_data] Downloading package wordnet to /Users/loucap/nltk_data...\n",
      "[nltk_data]   Package wordnet is already up-to-date!\n",
      "[nltk_data] Downloading package averaged_perceptron_tagger to\n",
      "[nltk_data]     /Users/loucap/nltk_data...\n",
      "[nltk_data]   Package averaged_perceptron_tagger is already up-to-\n",
      "[nltk_data]       date!\n",
      "[nltk_data] Downloading package conll2000 to\n",
      "[nltk_data]     /Users/loucap/nltk_data...\n",
      "[nltk_data]   Package conll2000 is already up-to-date!\n",
      "[nltk_data] Downloading package movie_reviews to\n",
      "[nltk_data]     /Users/loucap/nltk_data...\n",
      "[nltk_data]   Unzipping corpora/movie_reviews.zip.\n",
      "Finished.\n"
     ]
    }
   ],
   "source": [
    "!pip install -U textblob -q\n",
    "!python -m textblob.download_corpora -q\n",
    "from textblob import TextBlob"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Analyse trivial documents with built-in sentiment analysis tool"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, lets get some data.\n",
    "\n",
    "Run/Shift+Enter, as above!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "...\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "textblob.blob.TextBlob"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "Doc1 = TextBlob(\"Textblob is just super. I love it!\")             # Convert a few basic strings into Textblobs \n",
    "Doc2 = TextBlob(\"Cabbages are the worst. Say no to cabbages!\")    # Textblobs, like other text-mining objects, are often called\n",
    "Doc3 = TextBlob(\"Paris is the capital of France. \")               # 'documents'\n",
    "print(\"...\")\n",
    "type(Doc1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Docs 1 through 3 are Textblobs, which we can see by the output of type(Doc1). \n",
    "\n",
    "We get a Textblob by passing a string to the function that we imported above. Specifically, this is done by using this format --> Textblob('string goes here'). Textblobs are ready for analysis through the textblob tools, such as the built-in sentiment analysis tool that we see in the code below. \n",
    "\n",
    "Run/Shift+Enter on those Textblobs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sentiment(polarity=0.47916666666666663, subjectivity=0.6333333333333333)\n",
      "Sentiment(polarity=-1.0, subjectivity=1.0)\n",
      "Sentiment(polarity=0.0, subjectivity=0.0)\n"
     ]
    }
   ],
   "source": [
    "print(Doc1.sentiment)\n",
    "print(Doc2.sentiment)\n",
    "print(Doc3.sentiment)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The output of the previous code returns two values for each Textblob object. Polarity refers to a positive-negative spectrum while subjectivity refers to an opinion-fact spectrum. \n",
    "\n",
    "We can see, for example, that Doc1 is fairly positive but also quite subjective while Doc2 is very negative and very subjective. Doc3, in contrast, is both neutral and factual. \n",
    "\n",
    "Maybe you don't need both polarity and subjectivity. For example, if you are trying to categorise opinions, you don't need the subjectivity score and would only want the polarity. \n",
    "\n",
    "To get only one of the two values, you can call the appropriate sub-function as shown below. \n",
    "\n",
    "Run/Shift+Enter for sub-functional fun. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.47916666666666663\n",
      "0.6333333333333333\n"
     ]
    }
   ],
   "source": [
    "print(Doc1.sentiment.polarity)\n",
    "print(Doc1.sentiment.subjectivity)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Acquire and analyse trivial documents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Super. We have imported some documents (in our case, just sentences in string format) to textblob and analysed it using the built-in sentiment analyser. But we don't want to import documents one string at a time...that would take forever!\n",
    "\n",
    "Let's import data in .csv format instead! The data here comes from a set of customer reviews of Amazon products. Naturally, not all of the comments in the product reviews are really on topic, but it does not actually matter for our purposes. But, I think it is only fair to warn you...there is some foul language and potentially objectionable personal opinions in the texts if you go through it all. \n",
    "\n",
    "Run/Shift+Enter (if you dare!)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[\"What does pin.mismatch mean? It's an error preventing me from logging into Beejive! \", '0'], [\"I don't feel special at all... That's all I have to say  i thought I was the only person you said sweet things to.\", '0'], [\"@sky327 Mornin' sunshine!  I'm so jealous I won't be in Chicago tomorrow!    Give my boys an extra long squeeze for me!\", '0'], ['@loris_sl We ignore them at our peril. But never fear, our friends are keeping a close eye on them and bark loudly to scare them off ', '4'], ['@jennylk everyone loves him in that ', '4'], ['@nathanmcdonald hopefully better battery life, would be happy if it equaled a BlackBerry on this point ', '4'], [\"@KimPriestap You don't? Well you should! \", '4'], ['@inedia_bella You are very sweet and awesome... ', '4'], ['@ArejayHale http://twitpic.com/48jlr - I saw this guy at the show on 4-29-09.  ', '4'], [\"LOL!!  I'm glad I've seen the original trailer or that wouldn't have made sense     Someone certainly went to a lot... http://bit.ly/A0pyr\", '4']]\n"
     ]
    }
   ],
   "source": [
    "with open('./data/sentiment-analysis/training_set.csv', newline='', encoding = 'ISO-8859-1') as f:  # Import a csv of scored \"product reviews\"\n",
    "    reader = csv.reader(f)\n",
    "    Doc_set = list(reader)\n",
    "\n",
    "print(Doc_set[45:55])  # Look at a subset of the imported data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A very good start (although you will see what I mean about the off-topic comments and foul language). \n",
    "\n",
    "Now, the .csv file has multiple strings per row, the first of which we want to pass to `texblob` to create a Textblob object. The second is a number representing the class that the statement belongs to. '4' represents 'positive', '2' represents neutral and '0' represents negative. Don't worry about this for now as we will come to that in a moment. \n",
    "\n",
    "The code below creates a new list that has the text string and the sentiment score for each item in the imported Doc_set, and also shows you the first 20 results of that new list to look at. \n",
    "\n",
    "Run/Shift+Enter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[\"What does pin.mismatch mean? It's an error preventing me from logging into Beejive! \", '0', Sentiment(polarity=-0.390625, subjectivity=0.6875)], [\"I don't feel special at all... That's all I have to say  i thought I was the only person you said sweet things to.\", '0', Sentiment(polarity=0.23571428571428568, subjectivity=0.7404761904761905)], [\"@sky327 Mornin' sunshine!  I'm so jealous I won't be in Chicago tomorrow!    Give my boys an extra long squeeze for me!\", '0', Sentiment(polarity=-0.03125, subjectivity=0.25)], ['@loris_sl We ignore them at our peril. But never fear, our friends are keeping a close eye on them and bark loudly to scare them off ', '4', Sentiment(polarity=0.1, subjectivity=0.8)], ['@jennylk everyone loves him in that ', '4', Sentiment(polarity=0.0, subjectivity=0.0)], ['@nathanmcdonald hopefully better battery life, would be happy if it equaled a BlackBerry on this point ', '4', Sentiment(polarity=0.65, subjectivity=0.75)], [\"@KimPriestap You don't? Well you should! \", '4', Sentiment(polarity=0.0, subjectivity=0.0)], ['@inedia_bella You are very sweet and awesome... ', '4', Sentiment(polarity=0.7275, subjectivity=0.9225000000000001)], ['@ArejayHale http://twitpic.com/48jlr - I saw this guy at the show on 4-29-09.  ', '4', Sentiment(polarity=0.0, subjectivity=0.0)], [\"LOL!!  I'm glad I've seen the original trailer or that wouldn't have made sense     Someone certainly went to a lot... http://bit.ly/A0pyr\", '4', Sentiment(polarity=0.5223214285714286, subjectivity=0.7553571428571428)]]\n"
     ]
    }
   ],
   "source": [
    "Doc_set_analysed = []\n",
    "\n",
    "for item in Doc_set:\n",
    "    Doc_set_analysed.append([item[0], item[1], TextBlob(item[0]).sentiment])\n",
    "\n",
    "print(Doc_set_analysed[45:55])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, edit the code above so that Doc_set_analysed only has the text string, the number string and the Textblob polarity. \n",
    "\n",
    "We will want to use that to get a sense of whether the polarity judgements are accurate or not. Thus, we want to know whether the judgement assigned to each statement (the '4', '2' or '0') matches with the polarity assigned by the `textblob` sentiment analyser. \n",
    "\n",
    "To do this, we need to convert the second item (the '4', '2' or '0') to a 1, 0 or -1 to match what we get back from the sentiment analyser, compare them to find the difference and then find the average difference. \n",
    "\n",
    "Run\\Shift+Enter. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9319827085773832\n"
     ]
    }
   ],
   "source": [
    "Doc_set_polarity_accuracy = []\n",
    "\n",
    "for item in Doc_set_analysed:\n",
    "    if (item[1] >= '4'):                            # this code checks the string with the provided judgement\n",
    "        x = 1                                       # and replaces it with a number matching textblob's polarity\n",
    "    elif (item[1] == '2'):\n",
    "        x = 0\n",
    "    else:\n",
    "        x = -1\n",
    "    y = item[2].polarity\n",
    "    Doc_set_polarity_accuracy.append(abs(x-y))     # unless my math is entirely wrong, this returns 'accuracy' or\n",
    "                                                    # the difference between the provided and calculated polarity\n",
    "                                                    # Exact matches (-1 and -1 or 1 and 1) return 0, complete opposites\n",
    "                                                    # (1 and -1 or -1 and 1) returning 2, all else proportionally in between. \n",
    "    \n",
    "\n",
    "print(statistics.mean(Doc_set_polarity_accuracy))   # Finding the average of all accuracy shows ... it is not great.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hmmm. If the sentiment analyser were:\n",
    "- entirely accurate, we would have an average difference of 0\n",
    "- entirely inaccurate, we would have an average difference of 2\n",
    "- entirely random, we would expect an average difference of 1\n",
    "\n",
    "As it stands, we have an average difference that suggests we are a bit more accurate than chance... but not my much. \n",
    "\n",
    "However, it is important to remember that we are testing an assigned class against a probable class... The assigned class (the '4', '2' or '0' in the original data set) is an absolute judgement and so is always *exactly* 4, 2, or 0 but never 2.8 or 0.05. In contrast, the polarity judgement returned by the sentiment analyser is a probability: it is 1 if the sentiment analyser is absolutely confident that the statement is positive but only .5 if the sentiment analyser is fairly confident that the statement is positive. \n",
    "\n",
    "In light of this, the fact that we got a better than chance score on our average accuracy test may mean we are doing quite well. We could test this, of course, and convert the polarity scores from the sentiment analyser into 1, 0 or -1 or even into 4, 2 and 0 and then compare those. \n",
    "\n",
    "Heck. Why not? Let's have a go. \n",
    "Run\\Shift+Enter. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.37835\n"
     ]
    }
   ],
   "source": [
    "Doc_set_polarity_accuracy_2 = []\n",
    "\n",
    "for item in Doc_set_analysed:\n",
    "    x = item[1]                                     # This code sets the original judgement assigned to each statement as x\n",
    "    if (item[2].polarity > 0):                               # then converts polarity scores of more than 0 to '4'\n",
    "        y = '4'                                    \n",
    "    elif (item[2].polarity == 0 ):                           # converts polarity scores of exactly 0 to '2'\n",
    "        y = '2'\n",
    "    else:                                           # and converts negative polarity scores to '0'\n",
    "        y = '0'\n",
    "    if x == y:                                      # then compares the assigned judgement to the converted polarity score\n",
    "        Doc_set_polarity_accuracy_2.append(1)       # and adds a 1 if they match exactly\n",
    "    else:\n",
    "        Doc_set_polarity_accuracy_2.append(0)       # or adds a 0 if they do not match exactly. \n",
    "\n",
    "print(statistics.mean(Doc_set_polarity_accuracy_2)) # Finds the average of the match rate. Still not great.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Well, an average close to 1 would be entirely accurate while close to 0 would be entirely wrong (and to be fair, *entirely* wrong would also be accurate too...in a sense). \n",
    "\n",
    "Our average though suggests that our accuracy is still not great. Ah well. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Train and test a sentiment analysis tool with trivial data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we know how to use the built-in analyser, let's have a look back at the sentiment analysis scores for Doc1 and Doc2. \n",
    "- Doc1 = 'Textblob is just super. I love it!' which scored scored .48 on polarity... halfway between neutral and positive. \n",
    "- Doc2 = 'Cabbages are the worst. Say no to cabbages!' which scored -1 on polarity... the most negative it could score. \n",
    "\n",
    "Do we really think Doc2 is so much more negative than Doc1 is positive? Hmmmm. The built-in sentiment analyser is clearly not as accurate as we would want. Let's try to train our own, starting with a small set of trivial training and testing data sets. \n",
    "\n",
    "The following code does a few different things:\n",
    "- It defines 'train' as a data set with 10 sentences, each of which is marked as 'pos' or 'neg'.\n",
    "- It defines 'test' as a data set with 6 completely different sentences, also marked as 'pos' or 'neg'. \n",
    "- It imports NaiveBayesClassifier from the textblob.classifiers.\n",
    "- It defines 'cl' as a brand new NaiveBayesClassifier that is trained on the 'train' data set. \n",
    "\n",
    "Run/Shift+Enter to make it so. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "train = [\n",
    "    ('I love this sandwich.', 'pos'),\n",
    "    ('this is an amazing place!', 'pos'),\n",
    "    ('I feel very good about these beers.', 'pos'),\n",
    "    ('this is my best work.', 'pos'),\n",
    "    (\"what an awesome view\", 'pos'),\n",
    "    ('I do not like this restaurant', 'neg'),\n",
    "    ('I am tired of this stuff.', 'neg'),\n",
    "    (\"I can't deal with this\", 'neg'),\n",
    "    ('he is my sworn enemy!', 'neg'),\n",
    "    ('my boss is horrible.', 'neg')]\n",
    "test = [\n",
    "     ('the beer was good.', 'pos'),\n",
    "     ('I do not enjoy my job', 'neg'),\n",
    "     (\"I ain't feeling dandy today.\", 'neg'),\n",
    "     (\"I feel amazing!\", 'pos'),\n",
    "     ('Gary is a friend of mine.', 'pos'),\n",
    "     (\"I can't believe I'm doing this.\", 'neg')]\n",
    "\n",
    "\n",
    "from textblob.classifiers import NaiveBayesClassifier\n",
    "cl = NaiveBayesClassifier(train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hmm. The code ran but there is nothing to see. This is because we have no output! Let's get some output and see what it did. \n",
    "\n",
    "The next code block plays around with 'cl', the classifier we trained on our 'train' data set.\n",
    "\n",
    "The first line asks 'cl' to return a judgment of one sentence about a library. \n",
    "\n",
    "Then, we ask it to return a judgement of another sentence about something being a doozy. Although both times we get a judgement on whether the sentence is 'pos' or 'neg', the second one has more detailed sub-judgements we can analyse that show us how the positive and negative the sentence is so we can see whether the overall judgement is close or not. \n",
    "\n",
    "Do the Run/Shift+Enter thing that you are so good at doing!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Our 'cl' classifier says 'This is an amazing library!' is  pos\n",
      "...\n",
      "Our 'cl' classifier says 'This one is a doozy.' is probably pos because its positive score is  0.63  and its negative score is  0.37 .\n"
     ]
    }
   ],
   "source": [
    "print(\"Our 'cl' classifier says 'This is an amazing library!' is \", cl.classify(\"This is an amazing library!\"))\n",
    "print('...')\n",
    "\n",
    "prob_dist = cl.prob_classify(\"This one is a doozy.\")\n",
    "print(\"Our 'cl' classifier says 'This one is a doozy.' is probably\",\n",
    "      prob_dist.max(), \"because its positive score is \",\n",
    "      round(prob_dist.prob(\"pos\"), 2),\n",
    "      \" and its negative score is \",\n",
    "      round(prob_dist.prob(\"neg\"), 2),\n",
    "      \".\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Super. Now... What if we want to apply our 'cl' classifier to a document with multiple sentences... What kind of judgements can we get with that? \n",
    "\n",
    "Well, `textblob` is sophisticated enough to give an overall 'pos' or 'neg' judgement, as well as a sentence-by-sentence judgement. \n",
    "\n",
    "Run/Shift+Enter, buddy. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Overall, 'blob' is  pos  because it's sentences are ...\n",
      "The beer is good.\n",
      "pos\n",
      "But the hangover is horrible.\n",
      "neg\n"
     ]
    }
   ],
   "source": [
    "blob = TextBlob(\"The beer is good. But the hangover is horrible.\", classifier=cl)\n",
    "\n",
    "print(\"Overall, 'blob' is \", blob.classify(), \" because it's sentences are ...\")\n",
    "for s in blob.sentences:\n",
    "     print(s)\n",
    "     print(s.classify())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What if we try to classify a document that we converted to Textblob format with the built-in sentiment analyser?\n",
    "\n",
    "Well, we still have Doc1 to try it on.\n",
    "\n",
    "Run/Shift+Enter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Textblob is just super. I love it!\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'pos'"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "print(Doc1)\n",
    "Doc1.classify()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Uh huh. We get an error. \n",
    "\n",
    "The error message says the blob known as Doc1 has no classifier. It suggests we train one first, but we can just apply 'cl'. \n",
    "\n",
    "Run/Shift+Enter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'pos'"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cl_Doc1 = TextBlob('Textblob is just super. I love it!', classifier=cl)\n",
    "cl_Doc1.classify()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Unsurprisingly, when we classify the string that originally went into Doc1 using our 'cl' classifier, we still get a positive judgement. \n",
    "\n",
    "Now, what about accuracy? We have been using 'cl' even though it is trained on a REALLY tiny training data set. What does that do to our accuracy? For that, we need to run an accuracy challenge using our test data set. This time, we are using a built-in accuracy protocol which deals with negative values and everything for us. This means we want our result to be as close to 1 as possible. \n",
    "\n",
    "Run/Shift+Enter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8333333333333334"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cl.accuracy(test)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hmmm. Not perfect.\n",
    "\n",
    "Fortunately, we can add more training data and try again. The code below defines a new training data set and then runs a re-training functiong called 'update' on our 'cl' classifier. \n",
    "\n",
    "Run/Shift+Enter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "True"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "new_data = [('She is my best friend.', 'pos'),\n",
    "            (\"I'm happy to have a new friend.\", 'pos'),\n",
    "            (\"Stay thirsty, my friend.\", 'pos'),\n",
    "            (\"He ain't from around here.\", 'neg')]\n",
    "\n",
    "cl.update(new_data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, copy the code we ran before to get the accuracy check. Paste it into the next code block and Run\\Shift+Enter it.  \n",
    "\n",
    "Not only will this tell us if updating 'cl' with 'new_data' has improved the accuracy, it is also a chance for you to create a code block of your own. Well, done you (I assume). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Copy and paste the accuracy challenge from above into this cell and re-run it to get an updated accuracy score. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## You can train and test a sentiment analysis tool with more interesting data too..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is all well and good, but 'cl' is trained on some seriously trivial data. What if we want to use some more interesting data, like the Doc_set that we imported from .csv earlier?\n",
    "\n",
    "Well, we are in luck! Sort of...\n",
    "\n",
    "We can definitely train a classifier on Doc_set, but let's just have a closer look at Doc_set before we jump right in and try that. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[\"What does pin.mismatch mean? It's an error preventing me from logging into Beejive! \", '0'], [\"I don't feel special at all... That's all I have to say  i thought I was the only person you said sweet things to.\", '0'], [\"@sky327 Mornin' sunshine!  I'm so jealous I won't be in Chicago tomorrow!    Give my boys an extra long squeeze for me!\", '0'], ['@loris_sl We ignore them at our peril. But never fear, our friends are keeping a close eye on them and bark loudly to scare them off ', '4'], ['@jennylk everyone loves him in that ', '4'], ['@nathanmcdonald hopefully better battery life, would be happy if it equaled a BlackBerry on this point ', '4'], [\"@KimPriestap You don't? Well you should! \", '4'], ['@inedia_bella You are very sweet and awesome... ', '4'], ['@ArejayHale http://twitpic.com/48jlr - I saw this guy at the show on 4-29-09.  ', '4'], [\"LOL!!  I'm glad I've seen the original trailer or that wouldn't have made sense     Someone certainly went to a lot... http://bit.ly/A0pyr\", '4']]\n",
      "...\n",
      "20000\n"
     ]
    }
   ],
   "source": [
    "print(Doc_set[45:55])\n",
    "print('...')\n",
    "print(len(Doc_set))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Doc_set is a set of comments that come from 'product reviews'. As we saw earlier, each item has two strings, the first of which is the comment and the second of which is a number 4, 2 or 0 which is written as a string. The second item, the number-written-as-a-string, is the class judgement. These scores may have been manually created, or may be the result of a semi-manual or supervised automation process. Excellent for our purposes, but not ideal because:\n",
    "- These scores are strings rather than integers. You can tell because they are enclosed in quotes.\n",
    "- These scores range from 0 (negative) to 4 (positive) and also contains 2 (neutral), while the textblob sentiment analysis and classifier functions we have been using return scores from -1 (negative) through 0 (neutral) to 1 (positive). \n",
    "\n",
    "Well, we could change 4 to 1, 2 to 0 and 0 to -1 with the use of regular expressions (RegEx) if we wanted. But as you will see, this is not strictly necessary. \n",
    "\n",
    "However, there is another issue. Doc_set has 20,000 items. This is big, but this is actually MUCH smaller than it could be. This is a subset of a 1,000,000+ item data set that you can download for free (see extra resources and reading at the end). The original data set was way too big for Jupyter notebook and was even too big for me to analyse on my laptop. I know because I tried. When you find yourself in a situation like this, you can try: \n",
    "- Accessing proper research computing facilities (good for real research, too much for a code demo). \n",
    "- Dividing a too big data set into chunks, and train/update a chunk at a time. \n",
    "- Processing a too big data set to remove punctuation, stop words, urls, twitter handles, etc. (saving computer power for what matters).\n",
    "- Or a combination of these options. \n",
    "\n",
    "But, you can try training a classifier on the much smaller 'testing_set' if you like. That set has under 5000 entries and so does not max out the computer's memory. \n",
    "\n",
    "I have provided the code below to load 'testing_set' into a new variable called Doc_set_2. Feel free to run the code below, then add more code blocks with processes copied from above. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[['is in San Francisco at Bay to Breakers.', '2'], ['just landed at San Francisco', '2'], ['San Francisco today.  Any suggestions?', '2'], ['?Obama Administration Must Stop Bonuses to AIG Ponzi Schemers ... http://bit.ly/2CUIg', '0'], ['started to think that Citi is in really deep s&amp;^t. Are they gonna survive the turmoil or are they gonna be the next AIG?', '0'], [\"ShaunWoo hate'n on AiG\", '0'], ['@YarnThing you will not regret going to see Star Trek. It was AWESOME!', '4'], ['On my way to see Star Trek @ The Esquire.', '2'], ['Going to see star trek soon with my dad.', '2'], ['annoying new trend on the internets:  people picking apart michael lewis and malcolm gladwell.  nobody wants to read that.', '0']]\n"
     ]
    }
   ],
   "source": [
    "with open('./data/sentiment-analysis/testing_set.csv', newline='') as f:              # Import a csv of scored \"product reviews\"\n",
    "    reader = csv.reader(f)\n",
    "    Doc_set_2 = list(reader)\n",
    "\n",
    "print(Doc_set_2[45:55])                                                             # Look at a subset of the imported data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can train a classifier on whatever data you want and with whatever categories you want. \n",
    "\n",
    "Want to train a classifier to recognise sarcasm? Go for it. \n",
    "How about recognising lies in political speeches? Good idea. \n",
    "How about tweets from bots or from real people? Definitely useful. \n",
    "\n",
    "The hard part is actually getting the data ready to feed to train your classifier. Depending on what you want to train your classifier to do, you may have to manually tag a whole lotta data. But it is always a good idea to start small. 10 items? 100? What can you do quickly that will give you enough of an idea to see if it is worth investing more time. \n",
    "\n",
    "Good luck!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Further reading and resources"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Books, tutorials, package recommendations, etc. for Python\n",
    "\n",
    "- Natural Language Processing with Python by Steven Bird, Ewan Klein and Edward Loper, http://www.nltk.org/book/\n",
    "- Foundations of Statistical Natural Language Processing by Christopher Manning and Hinrich Schütze, https://nlp.stanford.edu/fsnlp/promo/\n",
    "- Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics and Speech Recognition by Dan Jurafsky and James H. Martin, https://web.stanford.edu/~jurafsky/slp3/ed3book.pdf\n",
    "- Deep Learning in Natural Language Processing by Li Deng, Yang Liu, https://lidengsite.wordpress.com/book-chapters/\n",
    "- Sentiment Analysis data sets https://blog.cambridgespark.com/50-free-machine-learning-datasets-sentiment-analysis-b9388f79c124\n",
    "\n",
    "NLTK options\n",
    "- nltk.corpus http://www.nltk.org/howto/corpus.html\n",
    "- Data Camp tutorial on sentiment analysis with nltk https://www.datacamp.com/community/tutorials/simplifying-sentiment-analysis-python\n",
    "- Vader sentiment analysis script available on github (nltk) https://www.nltk.org/_modules/nltk/sentiment/vader.html\n",
    "- TextBlob https://textblob.readthedocs.io/en/dev/\n",
    "- Flair, a NLP script available on github https://github.com/flairNLP/flair\n",
    "\n",
    "spaCy options\n",
    "- spaCy https://nlpforhackers.io/complete-guide-to-spacy/\n",
    "- Data Quest tutorial on sentiment analysis with spaCy https://www.dataquest.io/blog/tutorial-text-classification-in-python-using-spacy/\n",
    "\n",
    "\n",
    "Books and package recommendations for R\n",
    "- Quanteda, an R package for text analysis https://quanteda.io/​\n",
    "- Text Mining with R, a free online book https://www.tidytextmining.com/​"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div style=\"text-align: right\"><a href=\"./2-Extraction.ipynb\" target=_blank><i>Next section: Extracting text</i></a></div>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.13"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}