code/4-Social-networks.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![UKDS Logo](images/UKDS_Logos_Col_Grey_300dpi.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "# Text-mining: Named Entity Extraction and Social Network Creation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Welcome to this <a href=\"https://ukdataservice.ac.uk/\" target=_blank>UK Data Service</a> *Computational Social Science* training series! \n",
    "\n",
    "The various *Computational Social Science* training series, all of which guide you through some of the popular and useful computational techniques, tools, methods and concepts that social science research might want to use. For example, this series covers collecting data from websites and social media platorms, working with text data, conducting simulations (agent based modelling), and more. The series includes recorded video webinars, interactive notebooks containing live programming code, reading lists and more.\n",
    "\n",
    "* To access training materials on our GitHub site: <a href=\"https://github.com/UKDataServiceOpen/computational-social-science\" target=_blank>[Training Materials]</a>\n",
    "\n",
    "* To keep up to date with upcoming and past training events: <a href=\"https://ukdataservice.ac.uk/news-and-events/events\" target=_blank>[Events]</a>\n",
    "\n",
    "* To get in contact with feedback, ideas or to seek assistance: <a href=\"https://ukdataservice.ac.uk/help.aspx\" target=_blank>[Help]</a>\n",
    "\n",
    "<a href=\"https://www.research.manchester.ac.uk/portal/julia.kasmire.html\" target=_blank>Dr J. Kasmire</a>  <br />\n",
    "UK Data Service  <br />\n",
    "University of Manchester <br />"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "toc": true
   },
   "source": [
    "<h1>Table of Contents<span class=\"tocSkip\"></span></h1>\n",
    "<div class=\"toc\"><ul class=\"toc-item\"><li><span><a href=\"#Introduction\" data-toc-modified-id=\"Introduction-1\"><span class=\"toc-item-num\">1&nbsp;&nbsp;</span>Introduction</a></span></li><li><span><a href=\"#Acquire-and-prepare-a-set-of-documents-with-named-entities.\" data-toc-modified-id=\"Acquire-and-prepare-a-set-of-documents-with-named-entities.-2\"><span class=\"toc-item-num\">2&nbsp;&nbsp;</span>Acquire and prepare a set of documents with named entities.</a></span></li><li><span><a href=\"#Extract-the-desired-chunks\" data-toc-modified-id=\"Extract-the-desired-chunks-3\"><span class=\"toc-item-num\">3&nbsp;&nbsp;</span>Extract the desired chunks</a></span></li><li><span><a href=\"#Identify-which-extracted-proper-nouns-are-named-in-the-same-document\" data-toc-modified-id=\"Identify-which-extracted-proper-nouns-are-named-in-the-same-document-4\"><span class=\"toc-item-num\">4&nbsp;&nbsp;</span>Identify which extracted proper nouns are named in the same document</a></span></li><li><span><a href=\"#Create-a-network-graph-and-add-the-nodes\" data-toc-modified-id=\"Create-a-network-graph-and-add-the-nodes-5\"><span class=\"toc-item-num\">5&nbsp;&nbsp;</span>Create a network graph and add the nodes</a></span></li><li><span><a href=\"#Add-edges-to-the-network-graph\" data-toc-modified-id=\"Add-edges-to-the-network-graph-6\"><span class=\"toc-item-num\">6&nbsp;&nbsp;</span>Add edges to the network graph</a></span></li><li><span><a href=\"#Have-a-look-at-graph-info\" data-toc-modified-id=\"Have-a-look-at-graph-info-7\"><span class=\"toc-item-num\">7&nbsp;&nbsp;</span>Have a look at graph info</a></span></li><li><span><a href=\"#Draw-the-graph\" data-toc-modified-id=\"Draw-the-graph-8\"><span class=\"toc-item-num\">8&nbsp;&nbsp;</span>Draw the graph</a></span></li><li><span><a href=\"#Conclusions\" data-toc-modified-id=\"Conclusions-9\"><span class=\"toc-item-num\">9&nbsp;&nbsp;</span>Conclusions</a></span></li><li><span><a href=\"#Further-reading\" data-toc-modified-id=\"Further-reading-10\"><span class=\"toc-item-num\">10&nbsp;&nbsp;</span>Further reading</a></span></li></ul></div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "There is a table of contents provided here at the top of the notebook, but you can also access this menu at any point by clicking the Table of Contents button on the top toolbar (an icon with four horizontal bars, if unsure hover your mouse over the buttons). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We already saw how to add part-of-speech tags and named entity labels to a corpus, but now we will explore a practical applications for those POS-tags and entity labels - extracted the names of people and creating a social network based on which people are mentioned in the same document. \n",
    "\n",
    "There are, of course, many other very useful and practical applications for POS-tags and/or entity labels. However, as always, readers should interpret this notebook as being a demonstration of a popular option rather than an exhaustive or comprehensive guide to all possibilities."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Acquire and prepare a set of documents with named entities."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As always, let's start by importing and downloading some useful packages. Many of these will be familiar to you if you have worked through the 'Basic Extraction' notebook, but there are always some new tools to explore. \n",
    "\n",
    "Run/Shift+Enter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import nltk                                          # get nltk \n",
    "from nltk import word_tokenize, pos_tag, ne_chunk    # import some of our old favourte functions\n",
    "from nltk import Tree                                # and import some new functions\n",
    "nltk.download('punkt')\n",
    "nltk.download('averaged_perceptron_tagger')\n",
    "nltk.download('maxent_ne_chunker')\n",
    "nltk.download('words')\n",
    "\n",
    "from itertools import chain, tee\n",
    "from operator import itemgetter\n",
    "\n",
    "!pip install networkx\n",
    "import networkx as nx                                # Just notice this line for now... We will refer to it again later.\n",
    "from networkx.algorithms import community\n",
    "\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "social_data = ['Archibald walked through Manchester with Beryl.', \n",
    "               'Tariq saw Beryl when she was playing tennis.', \n",
    "               'Archibald shares a house with Beryl and Cerys.',\n",
    "               'Cerys works with both Tariq and Edith.', \n",
    "               'Edith drives past Archibald and Denise on her morning commute.',\n",
    "               'Fadwa listens to podcasts while running.', \n",
    "               'Guo-feng and Hita often drive to the Welsh coast at weekends.',\n",
    "               'Icarus shops at the same supermarket as Janyu and Edith.', \n",
    "               'Kelsey and Edith both used to live in London.',\n",
    "               'Laia and Icarus are on the same bowling team.', \n",
    "               'Archibald and Kelsey are both keen gardeners.',\n",
    "               'Laia and Montserrat are both Catalonians. '\n",
    "              ]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Named Entity chunkers require data that is:\n",
    "* word tokenised and \n",
    "* POS-tagged. \n",
    "\n",
    "Named Entity chunkers return a list of nested trees. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tagged_chunked_data = []\n",
    "for item in social_data:\n",
    "    tokens = word_tokenize(item)\n",
    "    tags = nltk.pos_tag(tokens)\n",
    "    chunks = ne_chunk(tags)\n",
    "    tagged_chunked_data.append(chunks)\n",
    "    \n",
    "print(tagged_chunked_data)                       # print everything, since this is a small enough list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Looking at the results, we can see that each item in the list is a tree because each sentence starts with a \"Tree('S')\" indicating that the highest level of tree is a Sentence Tree. \n",
    "\n",
    "But Sentence Trees can have sub-trees. Each sub-tree also starts with \"Tree('TREE_TYPE')\" and we can see that there are \"Tree('PERSON')\", \"Tree('ORGANISATION')\" and \"Tree('GPE')\". Unfortunately, the ne_chunker is not perfect. 'Edith' is listed as a 'GPE' in one place but a 'PERSON' in another. Likewise, 'Archibald' is both a 'PERSON' and an 'ORGANISATION', while 'Guo-feng' and 'Montserrat' are not identified as sub-trees at all. You can probably find more mis-classifications. \n",
    "\n",
    "Let's take a closer look at those sentences. Run/Shift+Enter."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(tagged_chunked_data[0])\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Feel free to change the number in the above code block and re-run it to look at other sentences. \n",
    "\n",
    "Ultimately, a NER chunker is a classifier and can be trained on custom data as we have already seen how to do (check the classifier jupyter notebook in the same folder as this one!). \n",
    "\n",
    "Feel free to try training your own NER chunker classifier. You'll need to put social_data through a word_tokenisation process, then use its output to create a training data set. Then train a classifier on your training data set. \n",
    "\n",
    "But for now, we carry on!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Extract the desired chunks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have some reasonable chunks all chunked up nicely, we want to extract the desired chunks so that they can become the nodes in our network. \n",
    "\n",
    "Run/Shift+Enter!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "extracted_persons = []\n",
    "for tagged_tree in tagged_chunked_data:\n",
    "    people = []\n",
    "    for leaf in tagged_tree.leaves():\n",
    "        if 'NNP' in leaf[1]:\n",
    "            people.append(leaf[0])        \n",
    "    extracted_persons.append(sorted(people))\n",
    "    \n",
    "print(extracted_persons)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Have a look at the results of the code above. It is a list of lists. Each of the sub-lists contains all the nouns that occur in a given sentence. For example, the first sentence was 'Archibald walked through Manchester with Beryl.' and the first sub-list in our extracted_persons list contains 'Archibald', 'Beryl', 'Manchester'. Seems about right. \n",
    "\n",
    "Well, maybe not perfect. Perhaps we have been a bit too generous... Extracted_persons has accurately extracted all the peoples' names, but it has also extracted place names and proper noun categories too. 'Manchester', 'Welsh', 'London' and 'Catalonians' are extracted too. This is because the above code looks for chunks that have POS-tags indicating they are proper nouns (NNP) and extracts those, rather than strictly looking for the names of people. \n",
    "\n",
    "We could try to use the Named Entity Recognition labels, extracting only those labelled as 'PERSON'... But we saw that those are not working reliably for our data set. As an alternative to training our own NER chunker, we could just manually review the list and remove the place or category names.\n",
    "\n",
    "But... maybe we are happy to leave the place names and proper noun categories too. After all, we can infer a kind of relationship between  people and places or categories. When Archibald and Beryl go walking through Manchester, they have a relationship with Manchester. If someone else also has a relationship with Manchester, then it is not exactly wrong to suggest that the third person has a somewhat distant relationship with Archibald and Beryl by virtue of their shared link to Manchester. \n",
    "S\n",
    "So, as an exacutive decision, I am going to leave all the proper nouns in. They will all become nodes in our network. But to do that, we need to find all of the unique entries, which I find is helpful to view in alphabetical order. \n",
    "\n",
    "Run/Shift+Enter, duuuuuuuuuuuuude!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "unique_people = sorted(list(set(chain(*extracted_persons))))\n",
    "print(unique_people)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Identify which extracted proper nouns are named in the same document"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, the alphabetised list of all unique proper nouns will become the nodes list in when we go to create a network. But we also need an edge list telling the network with nodes are connected. In practical terms, an edge list is a list of tuples, each containing two nodes. \n",
    "\n",
    "To do this, we can use the itertools.permutations which looks a list and creates a new list of tuples with all possible permutations of a fixed length that can be made from the original list. To be specific, the code below looks at each of the sub-lists of extracted_persons and creates 2 item tuples from all possible combinations of the items in the sub-list. \n",
    "\n",
    "Run/Shift+Enter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import itertools\n",
    "\n",
    "co_occurring_pairs = []\n",
    "for people in extracted_persons:\n",
    "        for each_permutation in itertools.permutations(people, 2):\n",
    "            co_occurring_pairs.append(each_permutation)\n",
    "\n",
    "print(co_occurring_pairs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a closer look. \n",
    "\n",
    "The first sub-list in extracted_persons was:\n",
    "- 'Archibald', 'Beryl', 'Manchester'\n",
    "\n",
    "At the start of our co_occurring_pairs, we have:\n",
    "- ('Archibald', 'Beryl'),\n",
    "- ('Archibald', 'Manchester'),\n",
    "- ('Beryl', 'Archibald'),\n",
    "- ('Beryl', 'Manchester'),\n",
    "- ('Manchester', 'Archibald'),\n",
    "- ('Manchester', 'Beryl')\n",
    "\n",
    "This means that we have an edge between 'Archibald' and 'Beryl', but also another edge between 'Beryl' and 'Archibald'.\n",
    "\n",
    "There are also no edges involving 'Fadwa' as she only appears in one sentence that has no other proper nouns. Thus, we will have at least one node with no edges.  \n",
    "\n",
    "These two points may or may not be a problem for you, depending on how you want your network to function. For example, you may want some or all of your links to be directed, meaning that the link only goes one way. \n",
    "In our network, this might be reasonable for sentences like  'Tariq saw Beryl when she was playing tennis.' since we don't know that Beryl also saw Tariq. Directed links like this would be especially important for networks based on scientific citations or other links that are clearly one-way.\n",
    "\n",
    "Likewise, if you want your network to be weighted, you may want to add a third value to the tuples with how strong you want the link to be. When you go to create the network, you will need to sum up edges so that multiple instances of an edge between the same two nodes has a higher weight. \n",
    "\n",
    "To create a weighted edge list, run the code below. \n",
    "\n",
    "Run/Shift+Enter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "co_occurring_pairs_weighted = []\n",
    "\n",
    "for pair in co_occurring_pairs:\n",
    "    x = pair[0], pair[1], 1\n",
    "    co_occurring_pairs_weighted.append(x)\n",
    "    \n",
    "print(co_occurring_pairs_weighted)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a network graph and add the nodes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, we initialise an empty network graph object. The 'nx' part of 'nx.Graph()' relies on code at the start of the notebook that imported networkx as nx. If you prefer, you can replace 'nx' with 'networkx'. \n",
    "\n",
    "Run/Shift+Enter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "social_network = nx.Graph()                             # Initialize an empty networkx graph object called 'social_network'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we need to start filling up our empty graph object with details like the nodes list. \n",
    "\n",
    "Run/Shift+Enter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "social_network.add_nodes_from(unique_people)            # Add nodes to social_network from our extracted 'unique_people' list"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hmmm. Nothing happened.\n",
    "\n",
    "Well, that is not true. Something did happen, but we have to call extra functions to see what happened. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "social_network.nodes                                    # Use a graph object functions to see the nodes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Add edges to the network graph"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Assuming that we don't want a directed or weighted graph, we can add edges quite simply with code that is almost identical to the code we used to add nodes. \n",
    "\n",
    "This time, there is another line of code that calls on a different, but obviously similar, function to look at the edges. \n",
    "\n",
    "Run/Shift+Enter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "social_network.add_edges_from(co_occurring_pairs) # Add edges to social_network from our co-occurrence tuples\n",
    "social_network.edges                                     # Another quick look, this time at the just-imported edges"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Not only does this list all of the edges in a helpful way (alphabetically by the first node) it is clear that there is only one edge between any two pairs of nodes. \n",
    "\n",
    "There is an Archibald-Beryl edge, but no Beryl-Archibald edge. Good to know, eh?\n",
    "\n",
    "But what if we want a weighted graph?\n",
    "\n",
    "Well, we need to run a more complicated code that checks if an edge exists and then either creates it or adds weight to it, as appropriate. To keep our weighted and unweighted graphs separate, we will also create a new empty graph called social_network_weighted, add nodes, and then add edges by checking to see if one already exists first. \n",
    "\n",
    "Run/Shift+Enter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "social_network_weighted = nx.Graph()\n",
    "social_network_weighted.add_nodes_from(unique_people) \n",
    "\n",
    "for edge_pair in co_occurring_pairs_weighted:\n",
    "    if social_network_weighted.has_edge(edge_pair[0], edge_pair[1]):\n",
    "            # we added this one before, just increase the weight by one\n",
    "            w = int(edge_pair[2])\n",
    "            social_network_weighted[edge_pair[0]][edge_pair[1]]['weight'] += edge_pair[2]\n",
    "    else:\n",
    "            # new edge. add with weight=1\n",
    "            social_network_weighted.add_edge(edge_pair[0], edge_pair[1], weight = edge_pair[2])\n",
    "            \n",
    "social_network_weighted.edges"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Have a look at graph info"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, some basic info about our two graphs using the nx.info function. \n",
    "\n",
    "Run/Shift+Enter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(social_network)                         # the nx.info prints some basics about social_network\n",
    "print('...')                                            # nx.info doesn't have everything you might want...\n",
    "\n",
    "print(social_network_weighted)     "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Of course, you may want some additional info that is not included in the basics. \n",
    "\n",
    "Run/Shift+Enter"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Network density:\", nx.density(social_network))               # but extra info is easy to get.\n",
    "print(\"Network density:\", nx.density(social_network_weighted)) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Draw the graph"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now, we need different visualisations for our different graphs. The two code blocks below show various ways that you can change the graph visualisation (layout, colour, node size, etc. )\n",
    "\n",
    "Run/Shift+Ente in the next 2 blocks. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "social_network_positions = nx.circular_layout(social_network)  # Define positions for nodes according to a circular layout\n",
    "\n",
    "nx.draw_networkx_nodes(social_network, social_network_positions, # draw nodes according to position, size\n",
    "                       node_size=700) \n",
    "nx.draw_networkx_edges(social_network, social_network_positions,  # draw edges according to position, line width                      \n",
    "                       width=2)      \n",
    "nx.draw_networkx_labels(social_network, social_network_positions,  # draw labels according to position, font choices\n",
    "                        font_size=10, font_family='sans-serif')\n",
    "plt.show()                                                        # show the network as drawn"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pylab import rcParams\n",
    "rcParams['figure.figsize'] = 10, 10\n",
    "\n",
    "weighted_pos = nx.kamada_kawai_layout(social_network_weighted)   # Define positions for force directed node layout\n",
    "\n",
    "elarge = [(u, v) for (u, v, d) in social_network_weighted.edges(data=True) if d['weight'] > 2] #define a 'heavy edge' style\n",
    "esmall = [(u, v) for (u, v, d) in social_network_weighted.edges(data=True) if d['weight'] <= 2] # and a 'light edge' style\n",
    "# nodes\n",
    "nx.draw_networkx_nodes(social_network_weighted, weighted_pos, node_size=400)                    # draw the nodes\n",
    "\n",
    "# edges\n",
    "nx.draw_networkx_edges(social_network_weighted, weighted_pos, edgelist=elarge,                 #draw the heavy edges\n",
    "                       width=3)\n",
    "nx.draw_networkx_edges(social_network_weighted, weighted_pos, edgelist=esmall,                 #draw the light edges   \n",
    "                       width=1, alpha=0.5, edge_color='b', style='dashed')\n",
    "\n",
    "# labels\n",
    "nx.draw_networkx_labels(social_network_weighted, weighted_pos, font_size=15, font_family='serif')\n",
    "\n",
    "plt.axis('off')\n",
    "figure = plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hopefully, this will give you some ideas about what to do with the NLP processes that you have put your corpus through. There is clear value there, related to which things occur together. Different kinds of processing might help you get directed graphs (although that would take some clever classification relating to subjects, objects, etc. ). \n",
    "\n",
    "Please do feel free to start back at the beginning, adding more sentences with the same names or even with new names. \n",
    "\n",
    "As before, these exercises and this sample code should highlight to you that you need to think about:\n",
    "- your research questions and what you want to show, explore or understand, \n",
    "- your data, texts, corpus, or other research materials to analyse etc. \n",
    "- how your processes are related to your reserch questions, and \n",
    "- how your processes and data can be made available and reproducible. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Further reading"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Books, tutorials, package recommendations, etc. for Python\n",
    "\n",
    "- Natural Language Processing with Python by Steven Bird, Ewan Klein and Edward Loper, http://www.nltk.org/book/\n",
    "- Foundations of Statistical Natural Language Processing by Christopher Manning and Hinrich Schütze, https://nlp.stanford.edu/fsnlp/promo/\n",
    "- Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics and Speech Recognition by Dan Jurafsky and James H. Martin, https://web.stanford.edu/~jurafsky/slp3/ed3book.pdf\n",
    "- Deep Learning in Natural Language Processing by Li Deng, Yang Liu, https://lidengsite.wordpress.com/book-chapters/\n",
    "- Sentiment Analysis data sets https://blog.cambridgespark.com/50-free-machine-learning-datasets-sentiment-analysis-b9388f79c124\n",
    "\n",
    "NLTK options\n",
    "- nltk.corpus http://www.nltk.org/howto/corpus.html\n",
    "- Data Camp tutorial on sentiment analysis with nltk https://www.datacamp.com/community/tutorials/simplifying-sentiment-analysis-python\n",
    "- Vader sentiment analysis script available on github (nltk) https://www.nltk.org/_modules/nltk/sentiment/vader.html\n",
    "- TextBlob https://textblob.readthedocs.io/en/dev/\n",
    "- Flair, a NLP script available on github https://github.com/flairNLP/flair\n",
    "\n",
    "networkx\n",
    "- package details https://networkx.github.io/documentation/stable/index.html\n",
    "- info about drawing graphs, including links to dedicated graph visualisation software https://networkx.github.io/documentation/stable/reference/drawing.html\n",
    "- drawing examples and specific tutorials https://networkx.github.io/documentation/latest/auto_examples/index.html\n",
    "- All the graph measures you can ask for https://networkx.github.io/documentation/stable/reference/algorithms/index.html\n",
    "\n",
    "\n",
    "Books and package recommendations for R\n",
    "- Quanteda, an R package for text analysis https://quanteda.io/​\n",
    "- Text Mining with R, a free online book https://www.tidytextmining.com/​"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.9"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": true,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": true,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}