nltk-session-4.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br>\n",
    "<img style=\"float:left\" src=\"http://ipython.org/_static/IPy_header.png\" />\n",
    "<br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Session 4: Getting the most out of what we've learned"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, now you know Python and NLTK! The main things we still have to do are:\n",
    "\n",
    "1. Run through a workflow that addresses some of your specific questions\n",
    "2. Brainstorm some use-cases for NLTK\n",
    "3. Manage resources and results\n",
    "4. Integrate IPython into your existing workflow\n",
    "5. Have an open discussion about what we've done\n",
    "6. Summarise and say goodbye!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import nltk\n",
    "from IPython.display import (display, clear_output, Image, display_pretty, \n",
    "                 display_html, display_jpeg, display_png, display_svg, HTML)\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## A toy workflow: Project Gutenberg"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We're going to start the session today by mining [Project Gutenberg](https://www.gutenberg.org/wiki/Technology_%28Bookshelf%29)'s *Food Processing* books."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    ""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    ""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    ""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    ""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    ""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    ""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    ""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    ""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    ""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    ""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    ""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    ""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    ""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    ""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    ""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    ""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    ""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    ""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    ""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    ""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    ""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    ""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    ""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    ""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    ""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    ""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [Tags used in the Brown Corpus](http://en.wikipedia.org/wiki/Brown_Corpus#Part-of-speech_tags_used)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "HTML('<iframe src=http://en.wikipedia.org/wiki/Brown_Corpus#Part-of-speech_tags_used width=700 height=350></iframe>')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Getting your data into Python/NLTK"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Scenario 1: You have some old books."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Are they machine readable?\n",
    "* OCR options&mdash;institutional or DIY?\n",
    "* Structure them in a meaningful way&mdash;by author, by year, by language ... \n",
    "* Start querying!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Scenario 2: You're interested in an online community."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Explore the site. Sign up for it, maybe.\n",
    "* Download it: *Wget*, *curl*, *crawlers, spiders* ...\n",
    "* Extract relevant data and metadata: Python's [*Beautiful Soup*](http://www.crummy.com/software/BeautifulSoup/bs4/doc/) library.\n",
    "* **Structure your data!**\n",
    "* Annotate your data, save these annotations\n",
    "* Start querying!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Scenario 3: Something of interest breaks in the news"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* It will start being discussed all over the web.\n",
    "* You can use the Twitter API to harvest tweets containing a term or hashtag of interest.\n",
    "* You can get a list of RSS feeds and mine news articles\n",
    "* You can use something like *WebBootCat* to harvest search engine results and make a plain text corpus\n",
    "* Process these into a manageable form\n",
    "* Structure them\n",
    "* Start querying!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Wrap up"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Managing resources and results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You generate huge amounts of code, data and findings. Often, it's hard to know what to do with it all. In this section, we'll provide some suggestions designed to keep your work:\n",
    "\n",
    "1. Reproducible\n",
    "2. Reusable\n",
    "3. Comprehensible"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Your code"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Most importantly, **write comments on your code**. You **will** forget what bits of code are supposed to do. Using others' code is much easier if it's commented up. \n",
    "2. A related point is to name your variables meaningfully: *variablexxy* does not tell us much about what it will contain. *For image in images:*  is a very comprehensible line.\n",
    "3. Also, write docstrings for your functions. Help messages come in very handy for not only others, but yourself. Simply stating what\n",
    "2. **Version control**. When editing your code, you may sometimes break it. [Here](https://drclimate.wordpress.com/2012/11/16/version-control/)'s a write-up about version control from Damien Irving.\n",
    "3. **Share your code**. You are often doing novel things when you code, and sharing what you've done can save somebody else a lot of work. *GitHub* is free for open-source projects. GitHub provides version control, which is especially useful when you are working with a team."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Developing as a programmer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've only scratched the surface of Python, to be honest. In fact, we've only been treating Python as a programming language. Many of its users, however, see it as more than just a programming language: it is an ideology and culture, as well. \n",
    "\n",
    "You'll notice on Stack Overflow, people will remark that some solutions are more 'pythonic' than others. By this, they typically mean that the code is easy to read and broken into discrete functions. More broadly, *pythonic* refers to code that adheres to the *Zen of Python*:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import this"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So, as you explore Python more and more, you learn not only new ways to get tasks done, but also what ways are better to others. While at first you'll be content with making code that works, you'll later want to make sure your code is elegant as well. Fixing up your old code becomes a form of procrastination from thesis writing. Luckily, of all the kinds of procrastination, it's one of the better kinds.\n",
    "\n",
    "Another change you might notice is a switch toward *defensive programming*, where you write code to handle potential errors, and to provide useful messages when people do something wrong. This is a really awesome thing to do.\n",
    "\n",
    "Some code authors also try to use *test-driven development*. From [the Wikipedia article](http://en.wikipedia.org/wiki/Test-driven_development):\n",
    "\n",
    "> First the developer writes an (initially failing) automated test case that defines a desired improvement or new function, then produces the minimum amount of code to pass that test, and finally refactors the new code to acceptable standards.\n",
    "\n",
    "This helps stop feature-creep, builds your confidence, and encourages the division of long code into well-defined functions.\n",
    "\n",
    "Oh, and you'll probably start dreaming in code. *Not* a joke."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Your data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It should now be clear to you that you have data!\n",
    "Think about how you structure it. Without necessarily becoming an archivist, do think about your metadata. It will help you to manage your data later.\n",
    "*Cloud computing* offers you access to more storage and compute-power than you might want to own. Plus you're unlikely to spill coffee on it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Your findings"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "[*Figshare*](http://www.figshare.com) is a site for storing tables and figures. It's particularly useful for working with large datasets, as we often generate far more raw tables and statistics than we can possibly publish.\n",
    "\n",
    "It's becoming more and more common to link journal publications to additional online resources such as GitHub code or Figshares. It's also more and more common to cite GitHub and Figshare&mdash;always nice to bump up your citation count!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Integrating IPython into your workflow"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What you've learned here isn't much good unless you can pull things out of it and put them into your own research workflow.\n",
    "\n",
    "It's important to remember that IPython code may be a little different from vanilla Python, as it can contain Magics, shell commands, and the like.\n",
    "\n",
    "Perhaps the coolest thing about programming is you are simultaneously researching and developing. The functions that you write can be uploaded to the web and used by others who encounter the problem that necessitated your writing the function in the first place.\n",
    "\n",
    "In reality, NLTK is nothing more than a lot of Python functions, coupled with some datasets (corpora, stopword lists, etc.). You can even visit NLTK on GitHub, fork their repository, and start playing around with the code! If you find bugs in the code, or if you think documentation is lacking, you can either write directly to the people who maintain the code, or fix the problem yourself and request that they review your fix and integrate it into NLTK."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Using IPython locally"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've done everything on the cloud so far, and it's been pretty good to us. You may also want to use IPython locally. To do this, you need to install it. There are many ways to install it, and these vary depending on your OS and what you already have installed. See the [IPython website](http://ipython.org/ipython-doc/2/install/install.html#installnotebook) for detailed instructions.\n",
    "\n",
    "> *[Anaconda](http://continuum.io/downloads)* is a large package of Python-based tools (including IPython and Matplotlib) that is easy to install. \n",
    "\n",
    "Once you have IPython installed, it's very easy to start using it. All you need to do is open up Terminal, navigate to the notebook directory and type:\n",
    "\n",
    "     ipython notebook filename.ipynb\n",
    "\n",
    "This will open up a blank notebook, exactly the same as the kind of notebook we've been using on the cloud. The only difference will be that if you enter:\n",
    "\n",
    "     os.listdir('.')\n",
    "\n",
    "you'll get a list of files in the directory of your notebook file, rather than a directory of your part of the cloud."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next steps - keep going!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "Image(url='http://starecat.com/content/wp-content/uploads/two-states-of-every-programmer-i-am-god-i-have-no-idea-what-im-doing.jpg')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We hope you've learned enough in these two days to be excited about what NLTK can add to your work and you're feeling confident to start working on your own.\n",
    "Code breaks. Often. Be patient and try not to get discouraged.\n",
    "The good thing about code breaking so often is that you can find help. Try:\n",
    "* Coming back to these notebooks and refreshing your memory\n",
    "* Checking the NLTK book\n",
    "* Googling your error messages. This will often lead you to Stack Overflow, the major online community for sharing coding questions.\n",
    "* NLTK also has a Google group where people share their experiences and ask for help\n",
    "* Keep in touch! Your community is a wonderful resource."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Summaries and goodbye"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we go, we should summarise what we've learned. Add all this to your CV!\n",
    "\n",
    "* Navigating the IPython notebook\n",
    "* Python commands - defining a variable; building a function\n",
    "* Using Python to perform basic quantitative analysis of text\n",
    "* Tagging and parsing to perform more sophisticated analysis of language\n",
    "* A crash course in corpus linguistics!\n",
    "* An appreciation of clean vs messy data and data structure\n",
    "* Data management practices"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Bragging rights "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You have learned a lot of stuff in under a week:\n",
    "\n",
    "* IPython\n",
    "* Python\n",
    "* NLTK\n",
    "* Linguistic theory\n",
    "\n",
    "Put it all on your resume. Be proud. And if you feel like writing up your findings, do it!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Thanks!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's the end of of course. Thank you to everybody for your participation.\n",
    "\n",
    "Please let us know how you found the course.\n",
    "\n",
    "Come along to *Hacky Hour* (Tsubu, Thursdays at 3pm).\n",
    "\n",
    "Also, [submit a pull request](https://github.com/resbaz/nltk) and improve our teaching materials! "
   ]
  }
 ],
 "metadata": {},
 "nbformat": 4,
 "nbformat_minor": 0
}