SocialMedia - Part 3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " <table><tr><td><img src=\"images/dbmi_logo.png\" width=\"75\" height=\"73\" alt=\"Pitt Biomedical Informatics logo\"></td><td><img src=\"images/pitt_logo.png\" width=\"75\" height=\"75\" alt=\"University of Pittsburgh logo\"></td></tr></table>\n",
    " \n",
    "\n",
    "# Social Media and Data Science - Part 3\n",
    "\n",
    "Data science modules developed by the University of Pittsburgh Biomedical Informatics Training Program with the support of the National Library of Medicine data science supplement to the University of Pittsburgh (Grant # T15LM007059-30S1). \n",
    "\n",
    "Developed by Harry Hochheiser, harryh@pitt.edu. All errors are my responsibility.\n",
    "\n",
    "<a rel=\"license\" href=\"http://creativecommons.org/licenses/by-nc/4.0/\"><img alt=\"Creative Commons License\" style=\"border-width:0\" src=\"https://i.creativecommons.org/l/by-nc/4.0/88x31.png\" /></a><br />This work is licensed under a <a rel=\"license\" href=\"http://creativecommons.org/licenses/by-nc/4.0/\">Creative Commons Attribution-NonCommercial 4.0 International License</a>.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "###  *Goal*: Use social media posts to explore the appplication of text and natural language processing to see what might be learned from online interactions.\n",
    "\n",
    "Specifically, we will retrieve, annotate, process, and interpret Twitter data on health-related issues such as depression."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "--- \n",
    "References:\n",
    "* [Mining Twitter Data with Python (Part 1: Collecting data)](https://marcobonzanini.com/2015/03/02/mining-twitter-data-with-python-part-1/)\n",
    "* The [Tweepy Python API for Twitter](http://www.tweepy.org/)\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import operator\n",
    "import numpy as np\n",
    "import matplotlib\n",
    "import matplotlib.pyplot as plt\n",
    "import jsonpickle\n",
    "import json\n",
    "import random\n",
    "import tweepy\n",
    "import spacy\n",
    "import time\n",
    "from datetime import datetime"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3.0 Introduction\n",
    "\n",
    "This module continues the Social Media Data Science module started in [Part 1](SocialMedia%20-%20Part%201.ipynb) and [Part 2](SocialMedia%20-%20Part%202.ipynb), covering the natural language processing analysis of our tweet corpus, providing an introduction to basic concepts of Natural Language Processing.\n",
    "  \n",
    "Our case study will apply these topics to Twitter discussions of smoking and vaping. Although details of the tools used to access data and the format and content of the data may differ for various services, the strategies and procedures used to analyze the data will generalize to other tools."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.0.1 Setup\n",
    "\n",
    "Before we dig in, we must grab a bit of code from [Part 1](SocialMedia%20-%20Part%201.ipynb)and [Part 2](SocialMedia%20-%20Part%202.ipynb):\n",
    "\n",
    "1. Our Tweets class\n",
    "3. Our twitter API Keys - be sure to copy the keys that you generated when you completed [Part 1](SocialMedia%20-%20Part%201.ipynb).\n",
    "4. Configuration of our Twitter connection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Tweets:\n",
    "    \n",
    "    \n",
    "    def __init__(self,term=\"\",corpus_size=100):\n",
    "        self.tweets={}\n",
    "        if term !=\"\":\n",
    "            self.searchTwitter(term,corpus_size)\n",
    "                \n",
    "    def searchTwitter(self,term,corpus_size):\n",
    "        searchTime=datetime.now()\n",
    "        while (self.countTweets() < corpus_size):\n",
    "            new_tweets = api.search(term,lang=\"en\",tweet_mode='extended',count=corpus_size)\n",
    "            for nt_json in new_tweets:\n",
    "                nt = nt_json._json\n",
    "                if self.getTweet(nt['id_str']) is None and self.countTweets() < corpus_size:\n",
    "                    self.addTweet(nt,searchTime,term)\n",
    "            time.sleep(30)\n",
    "                \n",
    "    def addTweet(self,tweet,searchTime,term=\"\",count=0):\n",
    "        id = tweet['id_str']\n",
    "        if id not in self.tweets.keys():\n",
    "            self.tweets[id]={}\n",
    "            self.tweets[id]['tweet']=tweet\n",
    "            self.tweets[id]['count']=0\n",
    "            self.tweets[id]['searchTime']=searchTime\n",
    "            self.tweets[id]['searchTerm']=term\n",
    "        self.tweets[id]['count'] = self.tweets[id]['count'] +1\n",
    "        \n",
    "    def combineTweets(self,other):\n",
    "        for otherid in other.getIds():\n",
    "            tweet = other.getTweet(otherid)\n",
    "            searchTerm = other.getSearchTerm(otherid)\n",
    "            searchTime = other.getSearchTime(otherid)\n",
    "            self.addTweet(tweet,searchTime,searchTerm)\n",
    "        \n",
    "    def getTweet(self,id):\n",
    "        if id in self.tweets:\n",
    "            return self.tweets[id]['tweet']\n",
    "        else:\n",
    "            return None\n",
    "    \n",
    "    def getTweetCount(self,id):\n",
    "        return self.tweets[id]['count']\n",
    "    \n",
    "    def countTweets(self):\n",
    "        return len(self.tweets)\n",
    "    \n",
    "    # return a sorted list of tupes of the form (id,count), with the occurrence counts sorted in decreasing order\n",
    "    def mostFrequent(self):\n",
    "        ps = []\n",
    "        for t,entry in self.tweets.items():\n",
    "            count = entry['count']\n",
    "            ps.append((t,count))  \n",
    "        ps.sort(key=lambda x: x[1],reverse=True)\n",
    "        return ps\n",
    "    \n",
    "    # reeturns tweet IDs as a set\n",
    "    def getIds(self):\n",
    "        return set(self.tweets.keys())\n",
    "    \n",
    "    # save the tweets to a file\n",
    "    def saveTweets(self,filename):\n",
    "        json_data =jsonpickle.encode(self.tweets)\n",
    "        with open(filename,'w') as f:\n",
    "            json.dump(json_data,f)\n",
    "    \n",
    "    # read the tweets from a file \n",
    "    def readTweets(self,filename):\n",
    "        with open(filename,'r') as f:\n",
    "            json_data = json.load(f)\n",
    "            incontents = jsonpickle.decode(json_data)   \n",
    "            self.tweets=incontents\n",
    "        \n",
    "    def getSearchTerm(self,id):\n",
    "        return self.tweets[id]['searchTerm']\n",
    "    \n",
    "    def getSearchTime(self,id):\n",
    "        return self.tweets[id]['searchTime']\n",
    "    \n",
    "    def getText(self,id):\n",
    "        tweet = self.getTweet(id)\n",
    "        text=tweet['full_text']\n",
    "        if 'retweeted_status'in tweet:\n",
    "            original = tweet['retweeted_status']\n",
    "            text=original['full_text']\n",
    "        return text\n",
    "                \n",
    "    def addCode(self,id,code):\n",
    "        tweet=self.getTweet(id)\n",
    "        if 'codes' not in tweet:\n",
    "            tweet['codes']=set()\n",
    "        tweet['codes'].add(code)\n",
    "        \n",
    "   \n",
    "    def addCodes(self,id,codes):\n",
    "        for code in codes:\n",
    "            self.addCode(id,code)\n",
    "        \n",
    " \n",
    "    def getCodes(self,id):\n",
    "        tweet=self.getTweet(id)\n",
    "        if 'codes' in tweet:\n",
    "            return tweet['codes']\n",
    "        else:\n",
    "            return None\n",
    "        \n",
    "    # NEW -ROUTINE TO GET PROFILE\n",
    "    def getCodeProfile(self):\n",
    "        summary={}\n",
    "        for id in self.tweets.keys():\n",
    "            tweet=self.getTweet(id)\n",
    "            if 'codes' in tweet:\n",
    "                for code in tweet['codes']:\n",
    "                    if code not in summary:\n",
    "                            summary[code] =0\n",
    "                    summary[code]=summary[code]+1\n",
    "        sortedsummary = sorted(summary.items(),key=operator.itemgetter(0),reverse=True)\n",
    "        return sortedsummary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Put the values of your keys into these variables"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "consumer_key = 'YOUR-CONSUMER-KEY'\n",
    "consumer_secret = 'YOUR-CONSUMER-SECRET'\n",
    "access_token = 'YOUR-ACCESS-TOKEN'\n",
    "access_secret = 'YOUR-ACCESS-SECRET'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tweepy import OAuthHandler\n",
    "\n",
    "auth = OAuthHandler(consumer_key, consumer_secret)\n",
    "auth.set_access_token(access_token, access_secret)\n",
    "\n",
    "api = tweepy.API(auth)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3.1 Natural langauge processing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our goal is to build a classifier capable of distinguishing tweets related to tobacco smoing from other, unrelated tweets. To do this, we will use ome basic natural language processing to explore the types of words and language found in the tweets. \n",
    "\n",
    "To do this, we will use the [spaCy](https://spaCy.io/) Python NLP package. spaCy provides significant NLP power out-of-the box, with customization facilities offering greater flexibility at various stages of the pipeline. Details can be found at the  [spaCy web site](https://spaCy.io/), and in this [tutorial](https://nicschrading.com/project/Intro-to-NLP-with-spaCy/). spaCy is built on a neural network model based on recent developments in NLP research. See the [spaCy architecture](https://spaCy.io/api/) description for an overview.\n",
    "\n",
    "Before we get into the deails, a bit of an introduction. \n",
    "\n",
    "Natural Language Processing involves a series of operations on an input text, each building off of the previous step to add additional insight and undertanding.  Thus, many NLP packages run as pipeline processors providing modular components at each stage of the process. Separating key steps into discrete packages provides needed modularity, as developers can modify and customize individual components as needed. spaCy, like other NLP tools including [GATE](https://gate.ac.uk/) and [cTAKES](https://ctaes.apache.org)  operate on such a model. Although the specific components of each pipeline vary from system to system (and from tasks to task, the key tasks are roughly similar:\n",
    "\n",
    "1. *Tokenizing*: splitting the text into words, punctuation, and other markers.\n",
    "2. *Part of speech tagging*: Classifying terms as nouns, verbs, adjective, adverbs, ec.\n",
    "3. *Dependency Parsing* or *Chunking*: Defining relationships between tokens (subject and object of sentence) and grouping into noun and veb phrases.\n",
    "4. *Named Entity Recognition*: Mapping words or phrases to standard vocabularies or other common, known values. This step is often key for linking free text to accepted terms for diseases, symptoms, and/or anatomic locations.\n",
    "\n",
    "Each of these steps might be accomplished through rules, machine learning models, or some combination of approaches. After these initial steps are complete, results might be used to identify relationships between items in the text, build classifiers, or otherwise conduct further analysis. We'll get into these topics later.\n",
    "\n",
    "The [spaCy documentation](https://spaCy.io/usage/spaCy-101) and [cTAKES default pipeline description](https://cwiki.apache.org/confluence/display/CTAKES/Default+Clinical+Pipeline) provide two examples of how these components might be arranged in practice.  For more information on NLP theory and methods, see [Speech and Language Processing (3rd ed. draft)](https://web.stanford.edu/~jurafsky/slp3/), perhaps the leading NLP textbook.\n",
    "\n",
    "Given this introduction, we can read in our tweets and get to work."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3.1.1 Reading in data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's start by reading in the 'smoking' tweets that you created in [Part 1](SocialMedia%20-%20Part%201.ipynb)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "smoking=Tweets()\n",
    "smoking.readTweets(\"tweets.json\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "smoking.countTweets()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To go along with these tweets, let's search for, and save, a set of tweets for the search term 'vaping':"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vaping = Tweets(\"vaping\",100)\n",
    "vaping.saveTweets(\"tweets-vaping.json\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3.1.2 NLP Roadmap\n",
    "\n",
    "\n",
    "spaCy, like many other natural laguage processing tools, operates as a *pipeline* - a sequential series of operations, each of which conducts some analysis and passes results on to the next.  Each of the steps on the pipeline can operate both on the original text and on any of the results of the previous stages. The basic Spacy pipeline starts with the following steps:\n",
    "\n",
    "1. Tokenizing - splitting into individual elements.\n",
    "2. Tagging - assigning part-of-speech tags\n",
    "3. Parsing - identifying relaionships between elements of a sentence.\n",
    "4. Named Entity Recogntion (NER) - identifying domain-specific nounds and concepts. In biomedical literature, this might mean diseases, symptoms, anatomic locations, etc. \n",
    "\n",
    "Tokenizing is the assumed first stage of every pipeline. To see the contents of a pipeline, we can create an NLP object for the English language and iterate over the components of the pipeline. Although we'll usually use all of the components of the pipeline, they can be [customized](https://spacy.io/usage/processing-pipelines)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import spacy\n",
    "nlp = spacy.load('en')\n",
    "for name,proc in nlp.pipeline:\n",
    "    print(name,proc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3.1.3 Tokenizing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tokenizing is the process of splitting a text into individual components - words - for further processing. Although this might sound simple, the pecularities of the English language and how it is used often make tokenizing more complex than we might expect.\n",
    "\n",
    "To see some of the challenges, we will grab a specifc pre-chosen tweet and process it. For demonstration purposes, we will just use the text of the tweet.  \n",
    "\n",
    "This will give us a beginning feel for what [Spacy](https://spacy.io) can do, how we might use it, and how we might want to extend and revise the tokenizing process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample='#Smoking affects multiple parts of our body. Know more: https://t.co/hwTeRdC9Hf \\n#SwasthaBharat #NHPIndia #mCessation #QuitSmoking https://t.co/x7xHO9G2Cr'\n",
    "sample"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tweets have usage patterns that are non-standard English - URLs, hashtags, user references (this particularly tweet was not selected accidentally). These patterns create challenges for extracting content - we might want to know that \"#QuitSmoking\" is, in a tweet, a hashtag that should be considered as a complete unit.  \n",
    "\n",
    "We'll see soon how we might do this, but first, to start the NLP process, we can import the spaCy components and create an NLP object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import spacy\n",
    "nlp = spacy.load('en')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we can then parse out the text from the first tweet."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "parsed = nlp(sample)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The result is a list of tokens. We can print out each token to start:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print([token.text for token in parsed])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see right away that this parsing isn't quite what we would like. Default English parsing treats  `#Smoking`  as two separate tokens - `#` and `Smoking`. Similar problems happen for other hashtags.\n",
    "\n",
    "To treat this as a hashtag, we will indeed need to revise the tokenizer. \n",
    "\n",
    "For another example of potential problems, consider this tweet text:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "smoketweet='E-cigarette use by teens linked to later tobacco smoking, study says https://t.co/AhTpFUw0TW'\n",
    "parsed=nlp(smoketweet)\n",
    "print( [tok.text for tok in parsed])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that \"E-cigarette\" becomes three tokens. This is not what we want - we want it to be held together as one. \n",
    "\n",
    "We will revise the spaCy tokenizer to handle these two difficulties - hashtags and \"E-cigarette\" tokenizing. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.1.3.1 Exception rules\n",
    "\n",
    "\"E-cigarette\" can be handled with some simple exception rules.\n",
    "\n",
    "To do this, we can refer to the spaCy docuentation, which describes the process for adding a [special-case tokenizer rule](https://spacy.io/usage/linguistic-features#section-tokenization). Essentially, these rules allow for the possibility of adding new rules to customize parsing for specific domains:\n",
    "\n",
    "Each new rule will be a dictionary with three fields:\n",
    "    * `ORTH` is the text that will be matched\n",
    "    * `LEMMA` is the lemma form\n",
    "    * `POS` is the part-of-speech\n",
    "    \n",
    "These can then be added to the tokenizer:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from spacy.symbols import ORTH, LEMMA, POS\n",
    "special_case = [{ORTH: u'e-cigarette', LEMMA: u'e-cigarette', POS: u'NOUN'}]\n",
    "nlp.tokenizer.add_special_case(u'e-cigarette', special_case)\n",
    "nlp.tokenizer.add_special_case(u'E-cigarette', special_case)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These commands suggest the text \"e-cigarette\" should be handled by the special case rule saying that it is a single token. Now, let's take a look at the result:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "parsed=nlp(smoketweet)\n",
    "print( [tok.text for tok in parsed])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we capture \"E-cigarette\" as one token. Note the importance of including both capitalizations in revised rules."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3.1.3.2 Tokenizing hashtags"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As indicators of the progress and content of Twitter conversations, hashtags are important in tweets. For example, some analyses might want to use trends in hashtags, and their mentions in tweets and retweets, to understand conversational dynamics and the spread of ideas. However, as we saw, they are not handled properly by the deafult tokenier. As a reminder: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "parsed = nlp(sample)\n",
    "print( [tok.text for tok in parsed])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "we can look specifically at \"#Smoking\", which becomes two tokens:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(parsed[0])\n",
    "print(parsed[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note how \"#Smoking\" is split into \"#\" and \"Smoking\". To avoid this, we will can add a specialized processing component as a member of a [spaCy pipeline](https://spacy.io/usage/processing-pipelines).\n",
    "\n",
    "To process hashtags, we will use code suggested by a [spaCy\n",
    "GitHub issue](https://github.com/explosion/spaCy/issues/503). To see how this should work, let's walkt through some steps:\n",
    "\n",
    "First, let's look at the tokens in the tweet parsed above. We can iterate through with enumerate. We can also look at a few interesting elements:\n",
    "\n",
    "* `nbor` gets the next token after a token.\n",
    "* `idx ` is the position of the token in the list of characters, starting at 0."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(str(parsed[0].idx)+\" \"+parsed[0].text)\n",
    "print(str(parsed[0].nbor().idx)+\" \"+str(parsed[0].nbor().text))\n",
    "print(str(parsed[1].nbor().idx)+\" \"+str(parsed[1].nbor().text))\n",
    "print(str(parsed[2].nbor().idx)+\" \"+str(parsed[2].nbor().text))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thus, '#' starts of the string,  and 'Smoking' occupies characters 7 characters starting with character 1.  The 9th characer (index 8) is a space, so the next token ('affects') starts on the 10th character, which has index 9, etc.\n",
    "\n",
    "We can use this information to find a hash tag. essentially, we can look for a tag that has the text '#'. If we find one, we can look at the next tag and merge all of the characters from the start of the first tag to the end of the second tag. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "start=parsed[0].idx\n",
    "length = len(parsed[1].text)\n",
    "end = start+length+1\n",
    "print(str(start))\n",
    "print(str(end))\n",
    "parsed.merge(start,end)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This combines the character starting with 0 up until the character before the character at index 8 (which is a space) to form a new token."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, if we look at the list of tokens, we see that the first two are merged:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "print( [tok.text for tok in parsed])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To get this to work for all of the tokens in a tweet, we need a routine that will repeatedly iterate over the tokens until we can't find anymore hashtags:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nlp = spacy.load('en')\n",
    "def hashtag_pipe(doc):\n",
    "    merged_hashtag = True\n",
    "    while merged_hashtag == True:\n",
    "        merged_hashtag = False\n",
    "        for token_index,token in enumerate(doc):\n",
    "            if token.text == '#':\n",
    "                try:\n",
    "                    nbor = token.nbor()\n",
    "                    start_index = token.idx\n",
    "                    end_index = start_index + len(token.nbor().text) + 1\n",
    "                    if doc.merge(start_index, end_index) is not None:\n",
    "                        merged_hashtag = True\n",
    "                        break\n",
    "                except:\n",
    "                    pass\n",
    "    return doc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This routine might require a bit of explanation. The main routine in lines 6-16 does the bulk of the work shown above - we find a token that contains only the single character '#', we find the end of the next token, and we merge the two.\n",
    "\n",
    "There is one catch in that inner loop. If the last token in the string is a '#', the attempt to read the next token (on line 9) will cause an exception. If this happens, we're done anyway. So we `try` to get the next token. If it fails, we must be at the end of the document, so the `except`  clause does nothing, as indicated by the `pass`.\n",
    "\n",
    "However, this is not the whole story. The merging of these two tokens removes one from the list of tokens returned by `enumerate(doc)`. If we continue on, the result of the enumeration will evenutally blow  up, as the code will try to access an element in the set of tokens that is no longer there (try it and see). \n",
    "\n",
    "To get around this, we change the inner loop to `break` out as soon as a pair of tokens are merged. This will start the process over with a new enumeration. This process will repeat until we make it all the way through lines 6-16 - in other words, all of the way through the tweet -  without finding a pair of tokens to merge. When this happens, `merged_hashtag` will stay False, and the outer loop will exit."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we have this routine written, we can then add it to the first position in the pipeline, which will put it after the default tokenizer, but before the part of speech tagger and other components."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nlp.add_pipe(hashtag_pipe,first=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And then we can try it out..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc = nlp(\"twitter #hashtag\")\n",
    "print(doc[0].text)\n",
    "print(doc[1].text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Returning to our first example..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample='#Smoking affects multiple parts of our body. Know more: https://t.co/hwTeRdC9Hf \\n#SwasthaBharat #NHPIndia #mCessation #QuitSmoking https://t.co/x7xHO9G2Cr'\n",
    "print(sample+\"\\n\")\n",
    "parsed = nlp(sample)\n",
    "print( [tok.text for tok in parsed])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can try a tweet that ends with a '#':"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "doc = nlp(\"twitter #hashtag #\")\n",
    "print([tok.text for tok in doc])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Great! We can also try a pathological example."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "parsed = nlp(\"weird hashtag ###tag\")\n",
    "print( [tok.text for tok in parsed])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Oops. That doesn't work. It's not even clear that this is a legal hashtag. \n",
    "\n",
    "**BONUS CHALLENGE**: Perhaps you can extend the routine to make it handle hashtags started by multiple '#' symbols?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Summarizing, we can combine the changes to the tokenizer, wrapping them up in a subroutine as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from spacy.symbols import ORTH, LEMMA, POS\n",
    "\n",
    "def getTwitterNLP():\n",
    "    nlp = spacy.load('en')\n",
    "    special_case = [{ORTH: u'e-cigarette', LEMMA: u'e-cigarette', POS: u'NOUN'}]\n",
    "    nlp.tokenizer.add_special_case(u'e-cigarette', special_case)\n",
    "    nlp.tokenizer.add_special_case(u'E-cigarette', special_case)\n",
    "    def hashtag_pipe(doc):\n",
    "        merged_hashtag = True\n",
    "        while merged_hashtag == True:\n",
    "            merged_hashtag = False\n",
    "            for token_index,token in enumerate(doc):\n",
    "                if token.text == '#':\n",
    "                    try:\n",
    "                        nbor = token.nbor()\n",
    "                        start_index = token.idx\n",
    "                        end_index = start_index + len(token.nbor().text) + 1\n",
    "                        if doc.merge(start_index, end_index) is not None:\n",
    "                            merged_hashtag = True\n",
    "                            break\n",
    "                    except:\n",
    "                        pass\n",
    "        return doc\n",
    "    nlp.add_pipe(hashtag_pipe,first=True)\n",
    "    return nlp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "nlp = getTwitterNLP()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "parsed = nlp(\"weird e-cigarette hashtag ###tag\")\n",
    "print( [tok.text for tok in parsed])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that spaCy can also detect sentences. If you have multiple sentences, they will be found in the results of the parser as spans, each with a start and endpoint, given in terms of the positions of the tokens: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "parsed= nlp(\"This is an example of parsing two sentences. Here is the second sentence.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for span in parsed.sents:\n",
    "    print(str(span.start)+\" \"+str(span.end))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thus the first sentence includes token 0-8 and the second includes 9-14:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's also possible to access the text of the sentences directly:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sents = list(parsed.sents)\n",
    "sents[0].text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(parsed[0].text)\n",
    "print(parsed[8].text)\n",
    "print(parsed[9].text)\n",
    "print(parsed[14].text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tokenizers are traditionally built using optimized [regular expressions](https://www.regular-expressions.info/). For more information about tokenizing in paCy, see [spaCy 101](https://spacy.io/usage/spacy-101#section-features) and the [detailed discussion of the spaCy tokenizer](https://spacy.io/usage/linguistic-features#tokenization). For a more general introduction, see [Chapter 2 of Speech and Language Processing (3rd ed. draft)](https://web.stanford.edu/~jurafsky/slp3/)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3.1.3.3 Lemmatization, stop words, and alpha characterization\n",
    "\n",
    "The spaCy tokenizer proivdes a few other useful features along the way:\n",
    "\n",
    "* Lemmatization: For each token, spaCy can find the *lemma*: the \"standard\" or \"base\" form, reducing verb forms to their base verb, plurals to appropriate singular nouns, etc.  \n",
    "* Stop word identification - labelling words as commonly-found words taht add little or no information.\n",
    "* Alphanumeric identification - identifying those tokens that contain only alphanumeric values.\n",
    "\n",
    "To see these in action, let's review a few tokens:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample='#Smoking affects multiple parts of our body. Know more: https://t.co/hwTeRdC9Hf \\n#SwasthaBharat #NHPIndia #mCessation #QuitSmoking https://t.co/x7xHO9G2Cr'\n",
    "parsed=nlp(sample)\n",
    "print(sample)\n",
    "print(parsed[1].text)\n",
    "print(parsed[1].lemma)\n",
    "print(parsed[1].lemma_)\n",
    "print(parsed[1].is_stop)\n",
    "print(parsed[1].is_alpha)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, `affects` has the lemma `affect`.  Note that spaCy stores many fields as both hashes for efficiency and as text  for readability. You'll want to use the text form for interpreting results, but the hash for computing. They differ only in the use of the trailing underscore - thus `lemma` is the hash while `lemma_` is the human readable form.\n",
    "\n",
    "We can also see that `affect` is not a stop word, and it is alphabetic.\n",
    "\n",
    "Some NLP systems will go a bit further than spaCy's lemmatization, using a process called \"stemming\" to reduce words to base forms. With a stemming algorithm, \"scared\" might be reduced to \"scare\" - see this description of [Porter's stemming algorithm](https://tartarus.org/martin/PorterStemmer/) for more detail. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you play around a bit, you might notice that even very common words don't get called stop words:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text=\"This is a test of the stop word tool\"\n",
    "doc=nlp(text)\n",
    "for d in doc:\n",
    "    print(d.text+\", \"+str(d.is_stop))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Clearly, 'This', 'a',' of', and 'the' should be considered stop words. This is a bit of a minor bug. This [Stack Overflow](https://stackoverflow.com/questions/41170726/add-remove-stop-words-with-spacy) post provides a workaround:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for word in nlp.Defaults.stop_words:\n",
    "    lex = nlp.vocab[word]\n",
    "    lex.is_stop = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now try it..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text=\"This is a test of the stop word tool\"\n",
    "doc=nlp(text)\n",
    "for d in doc:\n",
    "    print(d.text+\", \"+str(d.is_stop))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looks much better. Tying this together with the hashtag pipe routine above:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def getTwitterNLP():\n",
    "    nlp = spacy.load('en')\n",
    "    \n",
    "    for word in nlp.Defaults.stop_words:\n",
    "        lex = nlp.vocab[word]\n",
    "        lex.is_stop = True\n",
    "    \n",
    "    special_case = [{ORTH: u'e-cigarette', LEMMA: u'e-cigarette', POS: u'NOUN'}]\n",
    "    nlp.tokenizer.add_special_case(u'e-cigarette', special_case)\n",
    "    nlp.tokenizer.add_special_case(u'E-cigarette', special_case)\n",
    "    def hashtag_pipe(doc):\n",
    "        merged_hashtag = True\n",
    "        while merged_hashtag == True:\n",
    "            merged_hashtag = False\n",
    "            for token_index,token in enumerate(doc):\n",
    "                if token.text == '#':\n",
    "                    try:\n",
    "                        nbor = token.nbor()\n",
    "                        start_index = token.idx\n",
    "                        end_index = start_index + len(token.nbor().text) + 1\n",
    "                        if doc.merge(start_index, end_index) is not None:\n",
    "                            merged_hashtag = True\n",
    "                            break\n",
    "                    except:\n",
    "                        pass\n",
    "        return doc\n",
    "    nlp.add_pipe(hashtag_pipe,first=True)\n",
    "    return nlp"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "text=\"This is a test of the stop word tool\"\n",
    "nlp=getTwitterNLP()\n",
    "doc=nlp(text)\n",
    "for d in doc:\n",
    "    print(d.text+\", \"+str(d.is_stop))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3.1.4  Part-Of-Speech Tagging "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The next step in NLP is *Part of speech tagging* - classifying each token as one of the parts of speech that we all learned in elementrary school. Parts of speech are assigned to attributes of each token:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample='#Smoking affects multiple parts of our body. Know more: https://t.co/hwTeRdC9Hf \\n#SwasthaBharat #NHPIndia #mCessation #QuitSmoking https://t.co/x7xHO9G2Cr'\n",
    "parsed=nlp(sample)\n",
    "print(parsed[1].text)\n",
    "print(parsed[1].pos)\n",
    "print(parsed[1].pos_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As discussed before, we have two attributes here - `pos` is the hash code for the part of speech, used for efficiency, while `pos_` is the human readable form. Other attributes derived by spaCy follow the same pattern.\n",
    "\n",
    "A second attribute - `tag` - provide es additional information.\n",
    "\n",
    "As described in the [spaCy documentation for part-of-speech tags](https://spacy.io/api/annotation#pos-tagging), the tags associated with these two fields come from different sources. 'tag_' uses parts-of-speech from a version of the [Penn Treebank](https://www.seas.upenn.edu/~pdtb/), a well-known corpus of annotated text. 'pos_' uses a simpler set of tags from [A Universal Part-of-Speech Tagset](https://arxiv.org/abs/1104.2086), published by researchers from Google.  \n",
    "\n",
    "The tags for `affects` provide an example of the difference. According to the [spaCy documentation ](https://spacy.io/api/annotation#pos-tagging) `VBZ` from the Penn tag set indicates a 'verb, 3rd person singular present', while 'the 'VERB' result for 'pos_' is a more general tag from the Google set. There are many types of verbs in the Penn Treebank that correspond tot the 'VERB' tag from the Google set. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(parsed[1].text)\n",
    "print(parsed[1].tag_)\n",
    "print(parsed[1].pos_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you want to learn more about a part of spech tag, you can use `spacy.explain`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(spacy.explain(parsed[1].pos_))\n",
    "print(spacy.explain(parsed[1].tag_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at token 0 (\"#Smoking\"), token 3 (\"parts\"), token 11 (\"https://t.co/hwTeRdC9Hf'\"),  and token 13(\"#SwasthaBharat\") to see a few more tokens in action."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "t0 = parsed[0]\n",
    "t3 = parsed[3]\n",
    "t11= parsed[11]\n",
    "t13 = parsed[13]\n",
    "print (t0.text,t0.lemma_,t0.pos_,t0.tag_,t0.is_stop,t0.is_alpha)\n",
    "print (t3.text,t3.lemma_,t3.pos_,t3.tag_,t3.is_stop,t3.is_alpha)\n",
    "print (t11.text,t11.lemma_,t11.pos_,t11.tag_,t11.is_stop,t11.is_alpha)\n",
    "print (t13.text,t13.lemma_,t13.pos_,t13.tag_,t13.is_stop,t13.is_alpha)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that URLS are neither alphabetical  nor stop-words, but they are proper nouns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "Let's turn the code that we used above into a routine, along with a routine to print out token details and try another tweet or two. To make things easy to read, we'll use some spaces to format things in columns. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def printTokDetails(parsed):\n",
    "    print(\"{:25} {:25} {:7}{:7}{:7}{:7}\".format(\"Token text\",\"Lemma\",\"POS\",\"Tag\",\"Stop?\",\"Alpha?\"))\n",
    "    for tok in parsed:\n",
    "        print(\"{:25} {:25} {:7}{:7}{:7}{:7}\".format(str(tok.text),str(tok.lemma_),str(tok.pos_),str(tok.tag_),str(tok.is_stop),str(tok.is_alpha)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tweet_id=random.choice(list(smoking.getIds()))\n",
    "sample2 = smoking.getText(tweet_id)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "parsed2=nlp(sample2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "printTokDetails(parsed2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You might see some interesting pattners arising here.  For example:\n",
    "\n",
    "* We see many different type of speech. Initially, we might want to focus on the nouns alone, as they provide much of the content.  \n",
    "\n",
    "* Look for words like \"is\" or \"was\" - these might all refer to a common lemma term - \"be\", corresponding to the generic form of he verb. Do you see any other incidents of lemma forms that differ from the parsed text?\n",
    "\n",
    "* URLs and icons might be present in tweets. Are they classified as alphanumeric? Should we include them as part of the \"useful\" text from a tweet? \n",
    "\n",
    "How about another?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tweet_id=random.choice(list(smoking.getIds()))\n",
    "sample2 = smoking.getText(tweet_id)\n",
    "sample2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "parsed2=nlp(sample2)\n",
    "printTokDetails(parsed2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Try a few more of these to get a bit more of a feel for he distribution of lemmas and POS tags. The following shortcut routine will make this a bit easier. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def getRandomTweetText(tweets):\n",
    "    tweet_id = random.choice(list(tweets.getIds()))\n",
    "    return tweets.getText(tweet_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## EXERCISE 3.1: Filtering tokens\n",
    "\n",
    "Although NLP parsing is often a good start, further filtering is often necessary to focus on data relevant for specific tasks. In this problem, we will review some additional tweets and develop a post-processing routine capable of filtering tweets as necessary for our needs. \n",
    "\n",
    "3.1.1 Using the `getRandomTweetText`, and `printTokDetails` routines above, aong with the spaCy `parser` command, examine several tweets to decide which tokens should be included or not.  List criteria for keeeping/removing tokens. Remember to use `spacy.explain()` for any unfamiliar POS or tag entries. Note that your  criteria will not be perfect, and will likely need refinining. Examine enough tweets to feel confident in your criteria. Because we are parsing tweets, please don't forget hashtags and user mentions.\n",
    "\n",
    "3.1.2 Write a routine  `includeToken` that will return a token to be inclued if it matches the criteria that you identified in 3.11, and false otherwise.  Assume for now that we are only interested in nouns and verbs, as they might be a good starting point to find information about vaping or smoking. For any tokens that are included,`includeToken` should return the lemmatized-version of the token, converted to all lower-case and stripped of any whitespace, using `strip()`. Zero-length tokens should not be included.\n",
    "\n",
    "3.1.3 Write a routine `filterTweetTokens` that will filter the parsed tokens from a single tweet, returning a list of the tokens to be included, based on your criteria from `includeToken`. To standardize matters, `filterTweetTokens` should also return the lemmatized-version of the token, converted to all lower-case.\n",
    "\n",
    "3.1.4 Run `filterTweetTokens` on a few tweets. Identify any inaccuracies and explain them. When possible, identify an approach for improving performance, and implement it in a revision version of `filterTweetTokens`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "*ANSWER FOLLOWS insert here*\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*END OF ANSWER*\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will come back to these routines in [Part 4](SocialMedia%20-%20Part%203.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3.1.6  Dependency parsing"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "*Dependency parsing* is the process of identifying the syntactic linkages between elements in a sentence. Dependency parsers lin noun phrases and modifiers, subjects to objects, etc. The [spaCy description of dependency parsing](https://spacy.io/usage/linguistic-features#dependency-parse) provides a detailed introduction - here, we provide a brief summary.\n",
    "\n",
    "To see the dependencies in action, we can iterate through the tokens, printing out the dependencies, and the head (ie, the token that a token depens upon, and "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample='#Smoking affects multiple parts of our body. Know more: https://t.co/hwTeRdC9Hf \\n#SwasthaBharat #NHPIndia #mCessation #QuitSmoking https://t.co/x7xHO9G2Cr'\n",
    "print(sample)\n",
    "print(\"----\")\n",
    "parsed=nlp(sample)\n",
    "for chunk in parsed.noun_chunks:\n",
    "    print(chunk.text, chunk.root.text, chunk.root.dep_,chunk.root.head.text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for token in parsed:\n",
    "    print(token.text,token.dep_,token.head.text, token.pos_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see a few things from this example:\n",
    "\n",
    "1. `#Smoking` is a noun subject of the sentence, dependent on the verb `affects`.\n",
    "2. `affects` is a verb at the ROOT level.\n",
    "3. `multiple` is an adjective modifier that modifies `parts`.\n",
    "4. `parts` is a noun that is the direct object of `affects`, etc..\n",
    "\n",
    "We can look in more deail a the text, dependency,  head, and children, of each token.   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def printParseTree(parsed):\n",
    "    print(\"{:10} {:10} {:7} {:7} {:30}\".format(\"Token text\",\"dep\",\"Head text\",\"POS\",\"Children\"))\n",
    "    for tok in parsed:\n",
    "        children=[child.text for child in tok.children]\n",
    "        children=\",\".join(children)\n",
    "        print(\"{:10} {:10} {:7} {:7} {:30}\".format(str(tok.text),str(tok.dep_),str(tok.head.text),str(tok.head.pos_),children))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sents =list(parsed.sents)\n",
    "print(sents[0])\n",
    "printParseTree(sents[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we can see that that 'affects' is the root verb, with `#Smoking` as a noun subjects and `parts` as the object. `Parts`  is modified by `muliple` and `of our body`'.  \n",
    "\n",
    "We can use the [displacy](https://spacy.io/usage/visualizers#section-dep) renderer to show a graphical depiction of the dependencies. Since displacy seems to prefer showing thepare tree fror an entire document, we'll try it on a single sentence.\n",
    "\n",
    "Note - the \"%%capture\" line below tells Jupyter to hide some very ugly errors, whie still displaying the nice graphical result. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture --no-display\n",
    "from spacy import displacy\n",
    "\n",
    "text=\"#Smoking affects multiple parts of our body.\"\n",
    "parsed=nlp(text)\n",
    "displacy.render(docs=[parsed],jupyter=True, options={'distance': 90})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This diagram shows the structure given above in the printed version of the parse tree. \n",
    "\n",
    "These relationships might be useful for some NLP goals, particularly those involving relationships between concpets. \n",
    "\n",
    "A variety of approaches - including greedy algorithms, graph-based methods, and machine learning - can be used to extract dependencies. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3.1.7 Named Entity Recognition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Named entity recognition* is the process of extracting categories to known entities - places, people, things, ec. spaCy provides a statistical model capable of assigning an [entity type](https://spacy.io/api/annotation#named-entities) to many of the terms in a document. For an example, let's look at the entities found in a tweet:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample ='Scott Gottlieb of @FDA says 8 million lives could be saved by cutting nicotine levels #publichealth https://phony.url/123'\n",
    "print(sample)\n",
    "parsed=nlp(sample)\n",
    "print(\"----\")\n",
    "for ent in parsed.ents:\n",
    "    print(ent.text,ent.label_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that entities are not equivalent to tokens: `Scott Gottlieb` and `8 million` are entities, but not tokens. For comparison:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print([tok.text for tok in parsed])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thus, two tokens - `Scott` and `Gottlieb` are combined to form a single entity - `Scott Gottlieb'.' We can modify the above to see where each entity starts and ends:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"{:15} {:5} {:5} {:5}\".format(\"Text\",\"Start\",\"End\",\"Type\"))\n",
    "for ent in parsed.ents:\n",
    "    print(\"{:15} {:5} {:5} {:5}\".format(ent.text,ent.start_char,ent.end_char,ent.label_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Thus, `Scott Gottlieb` starts at character 0 and goes up through (but not including) character 14.\n",
    "\n",
    "We can also use the spaCy visualizer to look at the named entities in a sentence:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture --no-display\n",
    "displacy.render(docs=[parsed],jupyter=True, style='ent')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "\n",
    "Let's try another."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sample='How many people in New York are smokers?'\n",
    "print(sample)\n",
    "parsed=nlp(sample)\n",
    "print(\"----\")\n",
    "print(\"{:15} {:5} {:5} {:5}\".format(\"Text\",\"Start\",\"End\",\"Type\"))\n",
    "for ent in parsed.ents:\n",
    "    print(\"{:15} {:5} {:5} {:5}\".format(ent.text,ent.start_char,ent.end_char,ent.label_))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a named entity type, `GPE` stands for geopolitical entity.\n",
    "\n",
    "Here, we note that hashtags are not necessarily categorized as entities.  This might be a shortcoming if we were going to use named entities as part of our strategy for classiying tweets. The spaCy named entity recognizer is based on statistical models that can be extended given enough training data. See the discussion of [training the named entity recognizer](https://spacy.io/usage/training#section-ner) for details on how this might be done. \n",
    "\n",
    "*Challenge*: Collect some tweets with hashtags and train the spaCy named entity recognizer add a `HASHTAG` as a new entity type."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise 3.2\n",
    "\n",
    "The natural language processing pipeline consists of several processes that add substantial structure to our understanding of these Tweets. Tokenizing, part of speech tagging, lemmatiziation, dependency parsing, and named entity recognition each add different details that might be used to understand and classify documents, while also providing some hints as to interesting questions that we might ask.\n",
    "\n",
    "Review some tweets and discuss any patterns or questions that arise. You might consider some of the following:\n",
    "\n",
    "* Are there terms that show up more frequently in the vaping tweets as opposed to the smoking tweets?\n",
    "* Are the tokens that we filtered (in Exercise 3.1) useful, or do we need the whole set of tokens to inerpret\n",
    "* Are the named entities informative?\n",
    "\n",
    "Describe any other interesting phenomena that you think you might see in the corpus. Note that this question is not asking for fully statistically supported models. Rather, we're just looking for things that might be interesting to pursue further: it may turn out that any \"patterns\" you identify here are just incidental.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "*ANSWER FOLLOWS - insert answer here *\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*END OF ANSWER*\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3.2 Comparing Vocabularies\n",
    "\n",
    "Although the examination of a few selected tweets might help us understand some of the trends in terminology and how they differ between the `smoking` and the `vaping` sets, these spot checks may not give a balanced picture of text usage across both of the corpora.  Here we will try to more systematically address the questions that you considered in Exercise 3.2.\n",
    "\n",
    "\n",
    "A most systematic way to go about this would be to identify frequently-occurring tokens in  both corpora, using methods similar to those used in our examination of frequent authors from  [Part 1](SocialMedia%20-%20Part%201.ipynb) and in `getCodeProfile()` from [Part 2](SocialMedia%20-%20.ipynb). Specifically, we will write a routine that iterates through all of the tweets in a Tweets object and does the following:\n",
    "\n",
    "1. Parse the tweet\n",
    "2. Filter tokens (using the routines developed above).\n",
    "3. Adds each token to a hash assoicating each token with a count of the number of times it has appeared in the corpus\n",
    "\n",
    "The result will be a hash with the number of times each term occurs in the corpus. We can then sort this hash by descending values of the count to find the most frequent terms, and we can comapare results for the two sets. We'll return this information in two forms - a hash (for quick access) and a list (for sorting):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def getFrequentTerms(tweets,filtered=True):\n",
    "    frequents={}\n",
    "    for id in tweets.getIds():\n",
    "        text = tweets.getText(id)\n",
    "        parsed=nlp(text)\n",
    "        if filtered ==True:\n",
    "            toks = filterTweetTokens(parsed)\n",
    "        else:\n",
    "            toks = [tok for tok in parsed]\n",
    "        \n",
    "        for tok in toks:\n",
    "            if tok not in frequents:\n",
    "                frequents[tok]=0\n",
    "            frequents[tok]=frequents[tok]+1\n",
    "    sorts=sorted(frequents.items(),key=operator.itemgetter(1),reverse=True)\n",
    "    return frequents,sorts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "smokFreqs,smokSorted=getFrequentTerms(smoking)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "len(smokFreqs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "smokSorted[:20]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vapFreqs,vapSorted = getFrequentTerms(vaping)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "vapSorted[:20]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can go through these lists to get an idea of some of the commonalities. One way to do this would be to create a new list containing all of the terms found in both lists, along with their counts for each list:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "merged=[]\n",
    "for w,count in smokFreqs.items():\n",
    "    if w in vapFreqs:\n",
    "        vcount=vapFreqs[w]\n",
    "        item= (w,count,vcount)\n",
    "        merged.append(item)\n",
    "merged"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From these lists, a few observations come to mind:\n",
    "\n",
    "1. There are significant repeats, particularly in the top 20 terms.\n",
    "\n",
    "2. Frequent occurrences of terms like 'cigarette' in both list suggest that distinguishing between the two sets of tweets might be difficult.\n",
    "\n",
    "3. The vaping datasset contains many similar frequent terms like 'vap','vaping', 'vapor'. These similarities are also seen in related hashtags - `#vape`, `#vaping`, `#vapelife`, `#vapor`, etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Exercise 3.3\n",
    "\n",
    "Based on these observations of the frequent terms, we will consider some of the questions raised in exercise 3.2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "3.3.1 Exercise 3.1 and the `getFrequentTerms` method developed above take the relevant tokens from each tweet to be only those that meet certain criteria. However, we do not separately include the named entities.  One possible improvement would be to add any named entities to the list of tokens to be included. Revise `filterTweetTokens` to add any named entities to the end of the token list.  \n",
    "\n",
    "*Note* - be sure to add entities to the list only if they have a length of greater than zero! \n",
    "\n",
    "Try the result on a few tokens.  Do you see any potential problems with the simple approach to doing this? Does this strategy seem worth pursuing?\n",
    "\n",
    "3.3.2 We noticed that there are many repeated patterns in the tweets for vaping, including many terms and hashtags prefixed with 'vap'. One possible approach to this would be to further revise the lemmatizer to reduce these entries to common forms - perhaps 'vape' and '#vape'. Revise the `getTwitterNLP` routine above to include a lemmatizer that handles these cases."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "*ANSWER FOLLOWS - insert answer here*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*END OF ANSWER cut above here*\n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 3.3 Final Notes\n",
    "\n",
    "Now that you've seen the basics of using natural language processing to extract understanding from the tweets, you're ready to move on to the next step.  [Part 4](SocialMedia%20-%20Part%204.ipynb) will take the results of the NLP output and create basic classifier machine learning models. "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.6",
   "language": "python",
   "name": "python3.6"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}