RecSys_Collaborative_Filtering_movies_v1.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "RecSys-Collaborative-Filtering-movies-v1.ipynb",
      "version": "0.3.2",
      "provenance": [],
      "collapsed_sections": [],
      "include_colab_link": true
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.6.6"
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/Blackman9t/Machine_Learning/blob/master/RecSys_Collaborative_Filtering_movies_v1.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "v49e_dNQGyro",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://www.bigdatauniversity.com\"><img src=\"https://ibm.box.com/shared/static/cw2c7r3o20w9zn8gkecaeyjhgw3xdgbj.png\" width=\"400\" align=\"center\"></a>\n",
        "\n",
        "<h1 align=\"center\"><font size=\"5\">COLLABORATIVE FILTERING</font></h1>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "0tTnVoUfGyrs",
        "colab_type": "text"
      },
      "source": [
        "Recommendation systems are a collection of algorithms used to recommend items to users based on information taken from the user. These systems have become ubiquitous can be commonly seen in online stores, movies databases and job finders. In this notebook, we will explore recommendation systems based on Collaborative Filtering and implement simple version of one using Python and the Pandas library."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "q6ZkWe0BGyrx",
        "colab_type": "text"
      },
      "source": [
        "<h1>Table of contents</h1>\n",
        "\n",
        "<div class=\"alert alert-block alert-info\" style=\"margin-top: 20px\">\n",
        "    <ol>\n",
        "        <li><a href=\"#ref1\">Acquiring the Data</a></li>\n",
        "        <li><a href=\"#ref2\">Preprocessing</a></li>\n",
        "        <li><a href=\"#ref3\">Collaborative Filtering</a></li>\n",
        "    </ol>\n",
        "</div>\n",
        "<br>\n",
        "<hr>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "ZQjCfG2QGyr0",
        "colab_type": "text"
      },
      "source": [
        "\n",
        "\n",
        "<a id=\"ref1\"></a>\n",
        "# Acquiring the Data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "gJiGQX92Gyr1",
        "colab_type": "text"
      },
      "source": [
        "To acquire and extract the data, simply run the following Bash scripts:  \n",
        "Dataset acquired from [GroupLens](http://grouplens.org/datasets/movielens/). Lets download the dataset. To download the data, we will use **`!wget`** to download it from IBM Object Storage.  \n",
        "__Did you know?__ When it comes to Machine Learning, you will likely be working with large datasets. As a business, where can you host your data? IBM is offering a unique opportunity for businesses, with 10 Tb of IBM Cloud Object Storage: [Sign up now for free](http://cocl.us/ML0101EN-IBM-Offer-CC)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "Gng2XQ1jGyr2",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 319
        },
        "outputId": "f949efc7-0ba9-462c-b552-499b24ba1b62"
      },
      "source": [
        "!wget -O moviedataset.zip https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/ML0101ENv3/labs/moviedataset.zip\n",
        "print('unziping ...')\n",
        "!unzip -o -j moviedataset.zip "
      ],
      "execution_count": 1,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "--2019-05-17 18:44:07--  https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/ML0101ENv3/labs/moviedataset.zip\n",
            "Resolving s3-api.us-geo.objectstorage.softlayer.net (s3-api.us-geo.objectstorage.softlayer.net)... 67.228.254.193\n",
            "Connecting to s3-api.us-geo.objectstorage.softlayer.net (s3-api.us-geo.objectstorage.softlayer.net)|67.228.254.193|:443... connected.\n",
            "HTTP request sent, awaiting response... 200 OK\n",
            "Length: 160301210 (153M) [application/zip]\n",
            "Saving to: ‘moviedataset.zip’\n",
            "\n",
            "moviedataset.zip    100%[===================>] 152.88M  25.9MB/s    in 5.8s    \n",
            "\n",
            "2019-05-17 18:44:13 (26.6 MB/s) - ‘moviedataset.zip’ saved [160301210/160301210]\n",
            "\n",
            "unziping ...\n",
            "Archive:  moviedataset.zip\n",
            "  inflating: links.csv               \n",
            "  inflating: movies.csv              \n",
            "  inflating: ratings.csv             \n",
            "  inflating: README.txt              \n",
            "  inflating: tags.csv                \n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "o5X6w3lhGyr-",
        "colab_type": "text"
      },
      "source": [
        "Now you're ready to start working with the data!"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "x-2Vey93Gyr_",
        "colab_type": "text"
      },
      "source": [
        "<hr>\n",
        "\n",
        "<a id=\"ref2\"></a>\n",
        "# Preprocessing"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "biY58EBKGysG",
        "colab_type": "text"
      },
      "source": [
        "First, let's get all of the imports out of the way:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "Ob26I9zcGysI",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Dataframe manipulation library\n",
        "import pandas as pd\n",
        "#Math functions, we'll only need the sqrt function so let's import only that\n",
        "from math import sqrt\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "%matplotlib inline"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "42iWUXK-GysR",
        "colab_type": "text"
      },
      "source": [
        "Now let's read each file into their Dataframes:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "9cj0y1J4GysV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Storing the movie information into a pandas dataframe\n",
        "movies_df = pd.read_csv('movies.csv')\n",
        "#Storing the user information into a pandas dataframe\n",
        "ratings_df = pd.read_csv('ratings.csv')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "KsaSYOFaGysZ",
        "colab_type": "text"
      },
      "source": [
        "Let's also take a peek at how each of them are organized:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "9jjRToDkGysa",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 195
        },
        "outputId": "8e311ada-97da-4426-c0d2-3f440f118d52"
      },
      "source": [
        "#Head is a function that gets the first N rows of a dataframe. N's default is 5.\n",
        "movies_df.head()"
      ],
      "execution_count": 5,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>movieId</th>\n",
              "      <th>title</th>\n",
              "      <th>genres</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>Toy Story (1995)</td>\n",
              "      <td>Adventure|Animation|Children|Comedy|Fantasy</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>Jumanji (1995)</td>\n",
              "      <td>Adventure|Children|Fantasy</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>Grumpier Old Men (1995)</td>\n",
              "      <td>Comedy|Romance</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>4</td>\n",
              "      <td>Waiting to Exhale (1995)</td>\n",
              "      <td>Comedy|Drama|Romance</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>5</td>\n",
              "      <td>Father of the Bride Part II (1995)</td>\n",
              "      <td>Comedy</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   movieId  ...                                       genres\n",
              "0        1  ...  Adventure|Animation|Children|Comedy|Fantasy\n",
              "1        2  ...                   Adventure|Children|Fantasy\n",
              "2        3  ...                               Comedy|Romance\n",
              "3        4  ...                         Comedy|Drama|Romance\n",
              "4        5  ...                                       Comedy\n",
              "\n",
              "[5 rows x 3 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "hDOHAWvDGyse",
        "colab_type": "text"
      },
      "source": [
        "So each movie has a unique ID, a title with its release year along with it (Which may contain unicode characters) and several different genres in the same field. Let's remove the year from the title column and place it into its own one by using the handy [extract](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.str.extract.html#pandas.Series.str.extract) function that Pandas has."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "_G-CrXSwGysf",
        "colab_type": "text"
      },
      "source": [
        "Let's remove the year from the __title__ column by using pandas' replace function and store in a new __year__ column."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "scrolled": false,
        "id": "zke1WY41Gysg",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 136
        },
        "outputId": "788a67c3-bda6-4c06-aa25-f7cd0b6a5264"
      },
      "source": [
        "#Using regular expressions to find a year stored between parentheses\n",
        "#We specify the parantheses so we don't conflict with movies that have years in their titles\n",
        "movies_df['year'] = movies_df.title.str.extract('(\\(\\d\\d\\d\\d\\))',expand=False)\n",
        "movies_df.head(3)"
      ],
      "execution_count": 6,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>movieId</th>\n",
              "      <th>title</th>\n",
              "      <th>genres</th>\n",
              "      <th>year</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>Toy Story (1995)</td>\n",
              "      <td>Adventure|Animation|Children|Comedy|Fantasy</td>\n",
              "      <td>(1995)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>Jumanji (1995)</td>\n",
              "      <td>Adventure|Children|Fantasy</td>\n",
              "      <td>(1995)</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>Grumpier Old Men (1995)</td>\n",
              "      <td>Comedy|Romance</td>\n",
              "      <td>(1995)</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   movieId  ...    year\n",
              "0        1  ...  (1995)\n",
              "1        2  ...  (1995)\n",
              "2        3  ...  (1995)\n",
              "\n",
              "[3 rows x 4 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zUJdryh1HeQN",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 136
        },
        "outputId": "a07ba995-594c-49d9-dbd5-ac0bdbf8b211"
      },
      "source": [
        "#Removing the parentheses\n",
        "movies_df['year'] = movies_df.year.str.extract('(\\d\\d\\d\\d)',expand=False)\n",
        "movies_df.head(3)"
      ],
      "execution_count": 7,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>movieId</th>\n",
              "      <th>title</th>\n",
              "      <th>genres</th>\n",
              "      <th>year</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>Toy Story (1995)</td>\n",
              "      <td>Adventure|Animation|Children|Comedy|Fantasy</td>\n",
              "      <td>1995</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>Jumanji (1995)</td>\n",
              "      <td>Adventure|Children|Fantasy</td>\n",
              "      <td>1995</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>Grumpier Old Men (1995)</td>\n",
              "      <td>Comedy|Romance</td>\n",
              "      <td>1995</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   movieId  ...  year\n",
              "0        1  ...  1995\n",
              "1        2  ...  1995\n",
              "2        3  ...  1995\n",
              "\n",
              "[3 rows x 4 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 7
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MmutFthlX11i",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 136
        },
        "outputId": "1ac00e11-d8c9-4d66-bf7b-61bd0c1d6e3b"
      },
      "source": [
        "#Removing the years from the 'title' column\n",
        "movies_df['title'] = movies_df.title.str.replace('(\\(\\d\\d\\d\\d\\))', '')\n",
        "movies_df.head(3)"
      ],
      "execution_count": 8,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>movieId</th>\n",
              "      <th>title</th>\n",
              "      <th>genres</th>\n",
              "      <th>year</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>Toy Story</td>\n",
              "      <td>Adventure|Animation|Children|Comedy|Fantasy</td>\n",
              "      <td>1995</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>Jumanji</td>\n",
              "      <td>Adventure|Children|Fantasy</td>\n",
              "      <td>1995</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>Grumpier Old Men</td>\n",
              "      <td>Comedy|Romance</td>\n",
              "      <td>1995</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   movieId  ...  year\n",
              "0        1  ...  1995\n",
              "1        2  ...  1995\n",
              "2        3  ...  1995\n",
              "\n",
              "[3 rows x 4 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 8
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0of-OMxPYBtl",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Applying the strip function to get rid of any ending whitespace characters that may have appeared\n",
        "movies_df['title'] = movies_df['title'].apply(lambda x: x.strip())"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "EpBBW4XlGysi",
        "colab_type": "text"
      },
      "source": [
        "Let's look at the result!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "scrolled": false,
        "id": "yMJxIwl9Gysk",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 195
        },
        "outputId": "75290794-f3d8-4cd7-f2f0-f5c195c5f660"
      },
      "source": [
        "movies_df.head()"
      ],
      "execution_count": 10,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>movieId</th>\n",
              "      <th>title</th>\n",
              "      <th>genres</th>\n",
              "      <th>year</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>Toy Story</td>\n",
              "      <td>Adventure|Animation|Children|Comedy|Fantasy</td>\n",
              "      <td>1995</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>Jumanji</td>\n",
              "      <td>Adventure|Children|Fantasy</td>\n",
              "      <td>1995</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>Grumpier Old Men</td>\n",
              "      <td>Comedy|Romance</td>\n",
              "      <td>1995</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>4</td>\n",
              "      <td>Waiting to Exhale</td>\n",
              "      <td>Comedy|Drama|Romance</td>\n",
              "      <td>1995</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>5</td>\n",
              "      <td>Father of the Bride Part II</td>\n",
              "      <td>Comedy</td>\n",
              "      <td>1995</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   movieId  ...  year\n",
              "0        1  ...  1995\n",
              "1        2  ...  1995\n",
              "2        3  ...  1995\n",
              "3        4  ...  1995\n",
              "4        5  ...  1995\n",
              "\n",
              "[5 rows x 4 columns]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 10
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "XBbkMUmNGysn",
        "colab_type": "text"
      },
      "source": [
        "With that, let's also drop the genres column since we won't need it for this particular recommendation system."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "scrolled": false,
        "id": "XkZ51-TNGyso",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Dropping the genres column\n",
        "movies_df = movies_df.drop('genres', 1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "0jVMx-G4Gysr",
        "colab_type": "text"
      },
      "source": [
        "Here's the final movies dataframe:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "8rQEmz7RGysr",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 195
        },
        "outputId": "10cb71be-f97b-45b0-c6ee-11521707a5ac"
      },
      "source": [
        "movies_df.head()"
      ],
      "execution_count": 12,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>movieId</th>\n",
              "      <th>title</th>\n",
              "      <th>year</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>Toy Story</td>\n",
              "      <td>1995</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>Jumanji</td>\n",
              "      <td>1995</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>3</td>\n",
              "      <td>Grumpier Old Men</td>\n",
              "      <td>1995</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>4</td>\n",
              "      <td>Waiting to Exhale</td>\n",
              "      <td>1995</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>5</td>\n",
              "      <td>Father of the Bride Part II</td>\n",
              "      <td>1995</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   movieId                        title  year\n",
              "0        1                    Toy Story  1995\n",
              "1        2                      Jumanji  1995\n",
              "2        3             Grumpier Old Men  1995\n",
              "3        4            Waiting to Exhale  1995\n",
              "4        5  Father of the Bride Part II  1995"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 12
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "hxKoLMKVGysu",
        "colab_type": "text"
      },
      "source": [
        "<br>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "QFf_4rDPGysu",
        "colab_type": "text"
      },
      "source": [
        "Next, let's look at the ratings dataframe."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "e1SWwTvxGysv",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 195
        },
        "outputId": "01d17cad-0a5f-43c8-b53d-bad11331e042"
      },
      "source": [
        "ratings_df.head()"
      ],
      "execution_count": 13,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>userId</th>\n",
              "      <th>movieId</th>\n",
              "      <th>rating</th>\n",
              "      <th>timestamp</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>169</td>\n",
              "      <td>2.5</td>\n",
              "      <td>1204927694</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>2471</td>\n",
              "      <td>3.0</td>\n",
              "      <td>1204927438</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>1</td>\n",
              "      <td>48516</td>\n",
              "      <td>5.0</td>\n",
              "      <td>1204927435</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>2</td>\n",
              "      <td>2571</td>\n",
              "      <td>3.5</td>\n",
              "      <td>1436165433</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>2</td>\n",
              "      <td>109487</td>\n",
              "      <td>4.0</td>\n",
              "      <td>1436165496</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   userId  movieId  rating   timestamp\n",
              "0       1      169     2.5  1204927694\n",
              "1       1     2471     3.0  1204927438\n",
              "2       1    48516     5.0  1204927435\n",
              "3       2     2571     3.5  1436165433\n",
              "4       2   109487     4.0  1436165496"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 13
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "ZEVJupqhGysy",
        "colab_type": "text"
      },
      "source": [
        "Every row in the ratings dataframe has a user id associated with at least one movie, a rating and a timestamp showing when they reviewed it. We won't be needing the timestamp column, so let's drop it to save on memory."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "x1RB6EV6Gysz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Drop removes a specified row or column from a dataframe\n",
        "ratings_df = ratings_df.drop('timestamp', 1)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "wZw4Mkn5Gys2",
        "colab_type": "text"
      },
      "source": [
        "Here's how the final ratings Dataframe looks like:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "scrolled": true,
        "id": "y303w-cSGys3",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 195
        },
        "outputId": "099b4f9f-1b82-4aed-fc44-31756815d6ca"
      },
      "source": [
        "ratings_df.head()"
      ],
      "execution_count": 15,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>userId</th>\n",
              "      <th>movieId</th>\n",
              "      <th>rating</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>169</td>\n",
              "      <td>2.5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>1</td>\n",
              "      <td>2471</td>\n",
              "      <td>3.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>1</td>\n",
              "      <td>48516</td>\n",
              "      <td>5.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>2</td>\n",
              "      <td>2571</td>\n",
              "      <td>3.5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>2</td>\n",
              "      <td>109487</td>\n",
              "      <td>4.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   userId  movieId  rating\n",
              "0       1      169     2.5\n",
              "1       1     2471     3.0\n",
              "2       1    48516     5.0\n",
              "3       2     2571     3.5\n",
              "4       2   109487     4.0"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 15
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "RLVj4b6lGys5",
        "colab_type": "text"
      },
      "source": [
        "<hr>\n",
        "\n",
        "<a id=\"ref3\"></a>\n",
        "# Collaborative Filtering"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "0D0by-mjGys6",
        "colab_type": "text"
      },
      "source": [
        "Now, time to start our work on recommendation systems. \n",
        "\n",
        "The first technique we're going to take a look at is called __Collaborative Filtering__, which is also known as __User-User Filtering__. As hinted by its alternate name, this technique uses other users to recommend items to the input user. It attempts to find users that have similar preferences and opinions as the input and then recommends items that they have liked to the input. There are several methods of finding similar users (Even some making use of Machine Learning), and the one we will be using here is going to be based on the __Pearson Correlation Function__.\n",
        "\n",
        "<img src=\"https://ibm.box.com/shared/static/1ql8cbwhtkmbr6nge5e706ikzm5mua5w.png\" width=800px>\n",
        "\n",
        "\n",
        "The process for creating a User Based recommendation system is as follows:\n",
        "- Select a user with the movies the user has watched\n",
        "- Based on his rating to movies, find the top X neighbours \n",
        "- Get the watched movie record of the user for each neighbour.\n",
        "- Calculate a similarity score using some formula\n",
        "- Recommend the items with the highest score\n",
        "\n",
        "\n",
        "Let's begin by creating an input user to recommend movies to:\n",
        "\n",
        "Notice: To add more movies, simply increase the amount of elements in the userInput. Feel free to add more in! Just be sure to write it in with capital letters and if a movie starts with a \"The\", like \"The Matrix\" then write it in like this: 'Matrix, The' ."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "GBNjBo64Gys7",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 284
        },
        "outputId": "f7057ad5-426a-4096-9a36-7eba080729d1"
      },
      "source": [
        "userInput = [\n",
        "            {'title':'Breakfast Club, The', 'rating':5},\n",
        "            {'title':'Toy Story', 'rating':3.5},\n",
        "            {'title':'Jumanji', 'rating':2},\n",
        "            {'title':\"Pulp Fiction\", 'rating':5},\n",
        "            {'title':'Akira', 'rating':4.5},\n",
        "            {'title':'Juice','rating':3.5},\n",
        "            {'title':'Hidden, The','rating':4},\n",
        "            {'title':'Sound of Music, The','rating':4.5}\n",
        "         ] \n",
        "inputMovies = pd.DataFrame(userInput)\n",
        "inputMovies"
      ],
      "execution_count": 24,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>rating</th>\n",
              "      <th>title</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>5.0</td>\n",
              "      <td>Breakfast Club, The</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>3.5</td>\n",
              "      <td>Toy Story</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2.0</td>\n",
              "      <td>Jumanji</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>5.0</td>\n",
              "      <td>Pulp Fiction</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>4.5</td>\n",
              "      <td>Akira</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>3.5</td>\n",
              "      <td>Juice</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>4.0</td>\n",
              "      <td>Hidden, The</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>4.5</td>\n",
              "      <td>Sound of Music, The</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   rating                title\n",
              "0     5.0  Breakfast Club, The\n",
              "1     3.5            Toy Story\n",
              "2     2.0              Jumanji\n",
              "3     5.0         Pulp Fiction\n",
              "4     4.5                Akira\n",
              "5     3.5                Juice\n",
              "6     4.0          Hidden, The\n",
              "7     4.5  Sound of Music, The"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 24
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "h5nFNoCJGys-",
        "colab_type": "text"
      },
      "source": [
        "#### Add movieId to input user\n",
        "With the input complete, let's extract the input movies's ID's from the movies dataframe and add them into it.\n",
        "\n",
        "We can achieve this by first filtering out the rows that contain the input movies' title and then merging this subset with the input dataframe. We also drop unnecessary columns for the input to save memory space."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "scrolled": true,
        "id": "OcIuFhffGytA",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 284
        },
        "outputId": "6b69b5c9-b908-489c-df19-cd00a07c532a"
      },
      "source": [
        "#Filtering out the movies by title\n",
        "inputId = movies_df[movies_df['title'].isin(inputMovies['title'].tolist())]\n",
        "inputId"
      ],
      "execution_count": 25,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>movieId</th>\n",
              "      <th>title</th>\n",
              "      <th>year</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>Toy Story</td>\n",
              "      <td>1995</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>Jumanji</td>\n",
              "      <td>1995</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>293</th>\n",
              "      <td>296</td>\n",
              "      <td>Pulp Fiction</td>\n",
              "      <td>1994</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1016</th>\n",
              "      <td>1035</td>\n",
              "      <td>Sound of Music, The</td>\n",
              "      <td>1965</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1246</th>\n",
              "      <td>1274</td>\n",
              "      <td>Akira</td>\n",
              "      <td>1988</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1885</th>\n",
              "      <td>1968</td>\n",
              "      <td>Breakfast Club, The</td>\n",
              "      <td>1985</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3486</th>\n",
              "      <td>3576</td>\n",
              "      <td>Hidden, The</td>\n",
              "      <td>1987</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7911</th>\n",
              "      <td>8593</td>\n",
              "      <td>Juice</td>\n",
              "      <td>1992</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "      movieId                title  year\n",
              "0           1            Toy Story  1995\n",
              "1           2              Jumanji  1995\n",
              "293       296         Pulp Fiction  1994\n",
              "1016     1035  Sound of Music, The  1965\n",
              "1246     1274                Akira  1988\n",
              "1885     1968  Breakfast Club, The  1985\n",
              "3486     3576          Hidden, The  1987\n",
              "7911     8593                Juice  1992"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 25
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kV0XrsEwcbpg",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 284
        },
        "outputId": "183a2894-c536-43d2-f094-c4fff0cc86b5"
      },
      "source": [
        "#Then merging it so we can get the movieId. It's implicitly merging it by title.\n",
        "inputMovies = pd.merge(inputId, inputMovies)\n",
        "inputMovies"
      ],
      "execution_count": 26,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>movieId</th>\n",
              "      <th>title</th>\n",
              "      <th>year</th>\n",
              "      <th>rating</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>Toy Story</td>\n",
              "      <td>1995</td>\n",
              "      <td>3.5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>Jumanji</td>\n",
              "      <td>1995</td>\n",
              "      <td>2.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>296</td>\n",
              "      <td>Pulp Fiction</td>\n",
              "      <td>1994</td>\n",
              "      <td>5.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>1035</td>\n",
              "      <td>Sound of Music, The</td>\n",
              "      <td>1965</td>\n",
              "      <td>4.5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>1274</td>\n",
              "      <td>Akira</td>\n",
              "      <td>1988</td>\n",
              "      <td>4.5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>1968</td>\n",
              "      <td>Breakfast Club, The</td>\n",
              "      <td>1985</td>\n",
              "      <td>5.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>3576</td>\n",
              "      <td>Hidden, The</td>\n",
              "      <td>1987</td>\n",
              "      <td>4.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>8593</td>\n",
              "      <td>Juice</td>\n",
              "      <td>1992</td>\n",
              "      <td>3.5</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   movieId                title  year  rating\n",
              "0        1            Toy Story  1995     3.5\n",
              "1        2              Jumanji  1995     2.0\n",
              "2      296         Pulp Fiction  1994     5.0\n",
              "3     1035  Sound of Music, The  1965     4.5\n",
              "4     1274                Akira  1988     4.5\n",
              "5     1968  Breakfast Club, The  1985     5.0\n",
              "6     3576          Hidden, The  1987     4.0\n",
              "7     8593                Juice  1992     3.5"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 26
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "41-qqYQzdr0b",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 284
        },
        "outputId": "84cb9215-8c48-43aa-fa4b-e4872389a174"
      },
      "source": [
        "#Dropping information we won't use from the input dataframe\n",
        "inputMovies = inputMovies.drop('year', 1)\n",
        "#Final input dataframe\n",
        "#If a movie you added in above isn't here, then it might not be in the original \n",
        "#dataframe or it might spelled differently, please check capitalisation.\n",
        "inputMovies"
      ],
      "execution_count": 27,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>movieId</th>\n",
              "      <th>title</th>\n",
              "      <th>rating</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>1</td>\n",
              "      <td>Toy Story</td>\n",
              "      <td>3.5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>2</td>\n",
              "      <td>Jumanji</td>\n",
              "      <td>2.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>296</td>\n",
              "      <td>Pulp Fiction</td>\n",
              "      <td>5.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>1035</td>\n",
              "      <td>Sound of Music, The</td>\n",
              "      <td>4.5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>1274</td>\n",
              "      <td>Akira</td>\n",
              "      <td>4.5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>1968</td>\n",
              "      <td>Breakfast Club, The</td>\n",
              "      <td>5.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>6</th>\n",
              "      <td>3576</td>\n",
              "      <td>Hidden, The</td>\n",
              "      <td>4.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>8593</td>\n",
              "      <td>Juice</td>\n",
              "      <td>3.5</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   movieId                title  rating\n",
              "0        1            Toy Story     3.5\n",
              "1        2              Jumanji     2.0\n",
              "2      296         Pulp Fiction     5.0\n",
              "3     1035  Sound of Music, The     4.5\n",
              "4     1274                Akira     4.5\n",
              "5     1968  Breakfast Club, The     5.0\n",
              "6     3576          Hidden, The     4.0\n",
              "7     8593                Juice     3.5"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 27
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "tI9zxAbZGytG",
        "colab_type": "text"
      },
      "source": [
        "#### The users who has seen the same movies\n",
        "Now with the movie ID's in our input, we can now get the subset of users that have watched and reviewed the movies in our input.\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "v5vHO71DGytH",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 195
        },
        "outputId": "240a7643-a351-4aa9-eba3-ed4933c7c327"
      },
      "source": [
        "#Filtering out users that have watched movies that the input has watched and storing it\n",
        "userSubset = ratings_df[ratings_df['movieId'].isin(inputMovies['movieId'].tolist())]\n",
        "userSubset.head()"
      ],
      "execution_count": 28,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>userId</th>\n",
              "      <th>movieId</th>\n",
              "      <th>rating</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>19</th>\n",
              "      <td>4</td>\n",
              "      <td>296</td>\n",
              "      <td>4.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>441</th>\n",
              "      <td>12</td>\n",
              "      <td>1968</td>\n",
              "      <td>3.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>479</th>\n",
              "      <td>13</td>\n",
              "      <td>2</td>\n",
              "      <td>2.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>531</th>\n",
              "      <td>13</td>\n",
              "      <td>1274</td>\n",
              "      <td>5.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>681</th>\n",
              "      <td>14</td>\n",
              "      <td>296</td>\n",
              "      <td>2.0</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "     userId  movieId  rating\n",
              "19        4      296     4.0\n",
              "441      12     1968     3.0\n",
              "479      13        2     2.0\n",
              "531      13     1274     5.0\n",
              "681      14      296     2.0"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 28
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Na1Wx2fjeVm9",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "63c08b87-48ba-45a2-cc2b-29a8a203e663"
      },
      "source": [
        "userSubset.shape"
      ],
      "execution_count": 29,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(213139, 3)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 29
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gxHY8RGdeb5i",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "8946a508-799c-49d1-8fe9-21d45397b7bd"
      },
      "source": [
        "ratings_df.shape"
      ],
      "execution_count": 30,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(22884377, 3)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 30
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "XmibUWdmfFBy",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "6f86e9a5-2efe-4cef-a451-cc9a9d498500"
      },
      "source": [
        "movies_df.shape"
      ],
      "execution_count": 32,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "(34208, 3)"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 32
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "qRjEfI6jGytL",
        "colab_type": "text"
      },
      "source": [
        "We now group up the rows by user ID."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "n10OwSzmGytL",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        },
        "outputId": "ac1a6543-1138-4b16-8756-dac6e1723cfa"
      },
      "source": [
        "#Groupby creates several sub dataframes where they all have the same value in the column specified as the parameter\n",
        "userSubsetGroup = userSubset.groupby(['userId'])\n",
        "userSubsetGroup"
      ],
      "execution_count": 34,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "<pandas.core.groupby.generic.DataFrameGroupBy object at 0x7f9c8ef8e0f0>"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 34
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "i8AU4Up4GytN",
        "colab_type": "text"
      },
      "source": [
        "lets look at one of the users, e.g. the one with userID=1130"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "FnC2hRDJGytO",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 225
        },
        "outputId": "94f90da3-fc39-430e-fffe-725545c1f7f8"
      },
      "source": [
        "userSubsetGroup.get_group(1130)"
      ],
      "execution_count": 52,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>userId</th>\n",
              "      <th>movieId</th>\n",
              "      <th>rating</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>104167</th>\n",
              "      <td>1130</td>\n",
              "      <td>1</td>\n",
              "      <td>0.5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>104168</th>\n",
              "      <td>1130</td>\n",
              "      <td>2</td>\n",
              "      <td>4.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>104214</th>\n",
              "      <td>1130</td>\n",
              "      <td>296</td>\n",
              "      <td>4.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>104302</th>\n",
              "      <td>1130</td>\n",
              "      <td>1035</td>\n",
              "      <td>4.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>104363</th>\n",
              "      <td>1130</td>\n",
              "      <td>1274</td>\n",
              "      <td>4.5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>104443</th>\n",
              "      <td>1130</td>\n",
              "      <td>1968</td>\n",
              "      <td>4.5</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "        userId  movieId  rating\n",
              "104167    1130        1     0.5\n",
              "104168    1130        2     4.0\n",
              "104214    1130      296     4.0\n",
              "104302    1130     1035     4.0\n",
              "104363    1130     1274     4.5\n",
              "104443    1130     1968     4.5"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 52
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "grbbaS5aGytQ",
        "colab_type": "text"
      },
      "source": [
        "Let's also sort these groups so the users that share the most movies in common with the input have higher priority. This provides a richer recommendation since we won't go through every single user."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "9CBm1vz8GytR",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Sorting it so users with movie most in common with the input will have priority\n",
        "userSubsetGroup = sorted(userSubsetGroup,  key=lambda x: len(x[1]), reverse=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "gekRpLyFGytd",
        "colab_type": "text"
      },
      "source": [
        "Now lets look at the first user"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "wxTVs7VCGyte",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 437
        },
        "outputId": "f4ad514f-fc28-416d-844b-9d40b7411d8c"
      },
      "source": [
        "userSubsetGroup[0:3]"
      ],
      "execution_count": 60,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "[(38875,          userId  movieId  rating\n",
              "  3590618   38875        1     4.5\n",
              "  3590619   38875        2     2.0\n",
              "  3590719   38875      296     5.0\n",
              "  3590850   38875     1035     3.0\n",
              "  3590902   38875     1274     3.5\n",
              "  3591041   38875     1968     4.0\n",
              "  3591291   38875     3576     4.0\n",
              "  3591550   38875     8593     4.0), (74551,          userId  movieId  rating\n",
              "  6953204   74551        1     4.5\n",
              "  6953205   74551        2     2.5\n",
              "  6953376   74551      296     4.5\n",
              "  6953751   74551     1035     3.5\n",
              "  6953895   74551     1274     3.5\n",
              "  6954179   74551     1968     5.0\n",
              "  6955027   74551     3576     3.5\n",
              "  6956586   74551     8593     2.5), (84039,          userId  movieId  rating\n",
              "  7815686   84039        1     4.0\n",
              "  7815687   84039        2     2.0\n",
              "  7815789   84039      296     5.0\n",
              "  7815989   84039     1035     5.0\n",
              "  7816091   84039     1274     2.0\n",
              "  7816302   84039     1968     3.0\n",
              "  7816781   84039     3576     3.5\n",
              "  7817632   84039     8593     4.0)]"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 60
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "S8vDPIk1Gytg",
        "colab_type": "text"
      },
      "source": [
        "#### Similarity of users to input user\n",
        "Next, we are going to compare all users (not really all !!!) to our specified user and find the one that is most similar.  \n",
        "we're going to find out how similar each user is to the input through the __Pearson Correlation Coefficient__. It is used to measure the strength of a linear association between two variables. The formula for finding this coefficient between sets X and Y with N values can be seen in the image below. \n",
        "\n",
        "Why Pearson Correlation?\n",
        "\n",
        "Pearson correlation is invariant to scaling, i.e. multiplying all elements by a nonzero constant or adding any constant to all elements. For example, if you have two vectors X and Y,then, pearson(X, Y) == pearson(X, 2 * Y + 3). This is a pretty important property in recommendation systems because for example two users might rate two series of items totally different in terms of absolute rates, but they would be similar users (i.e. with similar ideas) with similar rates in various scales .\n",
        "\n",
        "![alt text](https://wikimedia.org/api/rest_v1/media/math/render/svg/bd1ccc2979b0fd1c1aec96e386f686ae874f9ec0 \"Pearson Correlation\")\n",
        "\n",
        "The values given by the formula vary from r = -1 to r = 1, where 1 forms a direct correlation between the two entities (it means a perfect positive correlation) and -1 forms a perfect negative correlation. \n",
        "\n",
        "In our case, a 1 means that the two users have similar tastes while a -1 means the opposite."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "vr24hR81Gyth",
        "colab_type": "text"
      },
      "source": [
        "We will select a subset of users to iterate through. This limit is imposed because we don't want to waste too much time going through every single user."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "NJHZuL8zGyti",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "userSubsetGroup = userSubsetGroup[0:100]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "lvfVYou-Gytl",
        "colab_type": "text"
      },
      "source": [
        "Now, we calculate the Pearson Correlation between input user and subset group, and store it in a dictionary, where the key is the user Id and the value is the coefficient\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "scrolled": true,
        "id": "CC1nB89FGytm",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "#Store the Pearson Correlation in a dictionary, where the key is the user Id and the value is the coefficient\n",
        "pearsonCorrelationDict = {}\n",
        "\n",
        "#For every user group in our subset\n",
        "for name, group in userSubsetGroup:\n",
        "    #Let's start by sorting the input and current user group so the values aren't mixed up later on\n",
        "    group = group.sort_values(by='movieId')\n",
        "    inputMovies = inputMovies.sort_values(by='movieId')\n",
        "    #Get the N for the formula\n",
        "    nRatings = len(group)\n",
        "    #Get the review scores for the movies that they both have in common\n",
        "    temp_df = inputMovies[inputMovies['movieId'].isin(group['movieId'].tolist())]\n",
        "    #And then store them in a temporary buffer variable in a list format to facilitate future calculations\n",
        "    tempRatingList = temp_df['rating'].tolist()\n",
        "    #Let's also put the current user group reviews in a list format\n",
        "    tempGroupList = group['rating'].tolist()\n",
        "    #Now let's calculate the pearson correlation between two users, so called, x and y\n",
        "    Sxx = sum([i**2 for i in tempRatingList]) - pow(sum(tempRatingList),2)/float(nRatings)\n",
        "    Syy = sum([i**2 for i in tempGroupList]) - pow(sum(tempGroupList),2)/float(nRatings)\n",
        "    Sxy = sum( i*j for i, j in zip(tempRatingList, tempGroupList)) - sum(tempRatingList)*sum(tempGroupList)/float(nRatings)\n",
        "    \n",
        "    #If the denominator is different than zero, then divide, else, 0 correlation.\n",
        "    if Sxx != 0 and Syy != 0:\n",
        "        pearsonCorrelationDict[name] = Sxy/sqrt(Sxx*Syy)\n",
        "    else:\n",
        "        pearsonCorrelationDict[name] = 0\n"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Gq7CWLwcGytq",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 54
        },
        "outputId": "219453ef-4de2-4ceb-ab21-7dc9678cb46e"
      },
      "source": [
        "pearsonCorrelationDict.items()"
      ],
      "execution_count": 64,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "dict_items([(38875, 0.6172133998483676), (74551, 0.6961804114562816), (84039, 0.4243398504520545), (125410, 0.7482387262365794), (185430, 0.12027695586485274), (216818, 0.8229511997978235), (224750, 0.4040610178208843), (246533, 0.7715167498104595), (815, 0.48936170212766095), (2432, 0.1364442057274539), (3734, -0.15790348519997205), (10315, 0.3638512152732104), (13053, 0.38148850487162356), (13768, 0.638885507778408), (19208, 0.5348131870734851), (20319, -0.21200662374236923), (23297, 0.39828475756996007), (29218, 0.7826237921249259), (30651, 0.8703882797784898), (35213, 0.2962531982051989), (37148, -0.38364304085653106), (38304, 0.5544788620954262), (38450, -0.06316139407998851), (40527, 0.6525426193750007), (43287, 0.3789683644799346), (46750, 0.5560670160293553), (59164, 0.29508355326173347), (60514, 0.7023379029930387), (61364, 0.37714451164442975), (62531, 0.03528787735930545), (63110, 0.3938804945305509), (67756, 0.5870952657480973), (71498, 0.8210527751457085), (72230, -0.24812138923975677), (72256, 0.08421519210665161), (73435, 0.5975576932277314), (77231, 0.2105379802666301), (78629, 0.47293785296374496), (80261, 0.07715167498104596), (80485, 0.7592514510736501), (88318, -0.45770365415250003), (88741, 0.4912803506947182), (90645, -0.03241018617760822), (90863, 0.675364361864465), (94166, -0.027036903521793762), (98720, -0.3449302046176462), (108959, 0.5052911526399118), (110657, 0.07504691899113708), (115074, 0.6033918684573832), (116283, 0.8835604291022476), (119215, 0.224793049033144), (119741, 0.8873409830255652), (123058, 0), (129203, 0.509433992240644), (130490, 0.897285810469096), (131424, 0.18921406922703551), (132400, 0.09102525683205703), (132717, 0.5954913341754146), (135145, 0.57888373874097), (140075, 0.2767593624086669), (144628, 0.733506588173288), (145123, 0.23159177829329408), (149218, 0.7245144565800904), (150733, 0.35229742431261835), (151615, -0.32001773000527667), (154431, 0.17864740025262535), (157553, 0.1364442057274539), (160435, -0.18798593995805155), (167523, 0.8922178162191937), (169107, 0.3690124832115526), (171537, 0.5773502691896267), (174903, 0.5128206131061158), (180987, 0.6733027231936333), (181813, 0.7636364387231598), (187299, 0.5742710457985442), (187969, 0.23632829671833053), (189328, 0.44661850063155856), (192465, 0.5459626263084689), (193834, 0.6943722744631395), (197263, 0.5742710457985443), (202494, 0.5160924896186915), (207819, 0.7092081432669752), (221191, 0.4074108571913332), (221320, 0.509433992240644), (222322, 0.5306618532579177), (227452, 0.02977456670876978), (227614, -0.5996452852187765), (229481, 0.47324852428030245), (229872, -0.6534596056871291), (230236, 0.27675936240866816), (231015, 0.505291152639911), (232104, 0.5458309647504257), (232531, -0.20756952180650035), (239662, 0.5186877230929468), (239802, 0.012804154455501046), (244606, 0.46166499639462794), (245274, -0.3367238909305437), (1040, 0.8604572206191307), (1130, 0.30559571381473644), (1599, 0.387699057267636)])"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 64
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2XBvtZHhGyts",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 195
        },
        "outputId": "466f1947-2dc7-4e2b-bf21-a7421a237a7c"
      },
      "source": [
        "pearsonDF = pd.DataFrame.from_dict(pearsonCorrelationDict, orient='index')\n",
        "pearsonDF.columns = ['similarityIndex']\n",
        "pearsonDF['userId'] = pearsonDF.index\n",
        "pearsonDF.index = range(len(pearsonDF))\n",
        "pearsonDF.head()"
      ],
      "execution_count": 65,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>similarityIndex</th>\n",
              "      <th>userId</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0.617213</td>\n",
              "      <td>38875</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0.696180</td>\n",
              "      <td>74551</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0.424340</td>\n",
              "      <td>84039</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>0.748239</td>\n",
              "      <td>125410</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0.120277</td>\n",
              "      <td>185430</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   similarityIndex  userId\n",
              "0         0.617213   38875\n",
              "1         0.696180   74551\n",
              "2         0.424340   84039\n",
              "3         0.748239  125410\n",
              "4         0.120277  185430"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 65
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "M1ksKRN2Gytu",
        "colab_type": "text"
      },
      "source": [
        "#### The top x similar users to input user\n",
        "Now let's get the top 50 users that are most similar to the input."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "Z1Lc7zRLGytu",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 343
        },
        "outputId": "e2ff70b8-81c2-4364-e5c5-ad1012bf10b0"
      },
      "source": [
        "topUsers=pearsonDF.sort_values(by='similarityIndex', ascending=False)[0:50]\n",
        "topUsers.head(10)"
      ],
      "execution_count": 66,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>similarityIndex</th>\n",
              "      <th>userId</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>54</th>\n",
              "      <td>0.897286</td>\n",
              "      <td>130490</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>68</th>\n",
              "      <td>0.892218</td>\n",
              "      <td>167523</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>51</th>\n",
              "      <td>0.887341</td>\n",
              "      <td>119741</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>49</th>\n",
              "      <td>0.883560</td>\n",
              "      <td>116283</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>18</th>\n",
              "      <td>0.870388</td>\n",
              "      <td>30651</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>97</th>\n",
              "      <td>0.860457</td>\n",
              "      <td>1040</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>0.822951</td>\n",
              "      <td>216818</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>32</th>\n",
              "      <td>0.821053</td>\n",
              "      <td>71498</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>17</th>\n",
              "      <td>0.782624</td>\n",
              "      <td>29218</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>7</th>\n",
              "      <td>0.771517</td>\n",
              "      <td>246533</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "    similarityIndex  userId\n",
              "54         0.897286  130490\n",
              "68         0.892218  167523\n",
              "51         0.887341  119741\n",
              "49         0.883560  116283\n",
              "18         0.870388   30651\n",
              "97         0.860457    1040\n",
              "5          0.822951  216818\n",
              "32         0.821053   71498\n",
              "17         0.782624   29218\n",
              "7          0.771517  246533"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 66
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "zltwPxFfGytw",
        "colab_type": "text"
      },
      "source": [
        "Now, let's start recommending movies to the input user.\n",
        "\n",
        "#### Rating of selected users to all movies\n",
        "We're going to do this by taking the weighted average of the ratings of the movies using the Pearson Correlation as the weight. But to do this, we first need to get the movies watched by the users in our __pearsonDF__ from the ratings dataframe and then store their correlation in a new column called _similarityIndex\". This is achieved below by merging of these two tables."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "scrolled": true,
        "id": "-HAkyMu7Gytx",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 195
        },
        "outputId": "4d294b0a-e273-4c4e-c44a-46f48550053e"
      },
      "source": [
        "topUsersRating=topUsers.merge(ratings_df, left_on='userId', right_on='userId', how='inner')\n",
        "topUsersRating.head()"
      ],
      "execution_count": 67,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>similarityIndex</th>\n",
              "      <th>userId</th>\n",
              "      <th>movieId</th>\n",
              "      <th>rating</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0.897286</td>\n",
              "      <td>130490</td>\n",
              "      <td>1</td>\n",
              "      <td>3.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0.897286</td>\n",
              "      <td>130490</td>\n",
              "      <td>2</td>\n",
              "      <td>2.5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0.897286</td>\n",
              "      <td>130490</td>\n",
              "      <td>3</td>\n",
              "      <td>3.0</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>0.897286</td>\n",
              "      <td>130490</td>\n",
              "      <td>5</td>\n",
              "      <td>2.5</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0.897286</td>\n",
              "      <td>130490</td>\n",
              "      <td>10</td>\n",
              "      <td>2.5</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   similarityIndex  userId  movieId  rating\n",
              "0         0.897286  130490        1     3.0\n",
              "1         0.897286  130490        2     2.5\n",
              "2         0.897286  130490        3     3.0\n",
              "3         0.897286  130490        5     2.5\n",
              "4         0.897286  130490       10     2.5"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 67
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "CFe6QGEAGyt0",
        "colab_type": "text"
      },
      "source": [
        "Now all we need to do is simply multiply the movie rating by its weight (The similarity index), then sum up the new ratings and divide it by the sum of the weights.\n",
        "\n",
        "We can easily do this by simply multiplying two columns, then grouping up the dataframe by movieId and then dividing two columns:\n",
        "\n",
        "It shows the idea of all similar users to candidate movies for the input user:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "0wr_7OaxGyt0",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 195
        },
        "outputId": "63433131-b6ff-44f6-d912-e335db0d1e98"
      },
      "source": [
        "#Multiplies the similarity by the user's ratings\n",
        "topUsersRating['weightedRating'] = topUsersRating['similarityIndex']*topUsersRating['rating']\n",
        "topUsersRating.head()"
      ],
      "execution_count": 68,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>similarityIndex</th>\n",
              "      <th>userId</th>\n",
              "      <th>movieId</th>\n",
              "      <th>rating</th>\n",
              "      <th>weightedRating</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>0.897286</td>\n",
              "      <td>130490</td>\n",
              "      <td>1</td>\n",
              "      <td>3.0</td>\n",
              "      <td>2.691857</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>0.897286</td>\n",
              "      <td>130490</td>\n",
              "      <td>2</td>\n",
              "      <td>2.5</td>\n",
              "      <td>2.243215</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>0.897286</td>\n",
              "      <td>130490</td>\n",
              "      <td>3</td>\n",
              "      <td>3.0</td>\n",
              "      <td>2.691857</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>0.897286</td>\n",
              "      <td>130490</td>\n",
              "      <td>5</td>\n",
              "      <td>2.5</td>\n",
              "      <td>2.243215</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>0.897286</td>\n",
              "      <td>130490</td>\n",
              "      <td>10</td>\n",
              "      <td>2.5</td>\n",
              "      <td>2.243215</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "   similarityIndex  userId  movieId  rating  weightedRating\n",
              "0         0.897286  130490        1     3.0        2.691857\n",
              "1         0.897286  130490        2     2.5        2.243215\n",
              "2         0.897286  130490        3     3.0        2.691857\n",
              "3         0.897286  130490        5     2.5        2.243215\n",
              "4         0.897286  130490       10     2.5        2.243215"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 68
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "cTsurMd9Gyt3",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 225
        },
        "outputId": "f6c6e277-f097-4631-9923-9d74d8c5cbe9"
      },
      "source": [
        "#Applies a sum to the topUsers after grouping it up by userId\n",
        "tempTopUsersRating = topUsersRating.groupby('movieId').sum()[['similarityIndex','weightedRating']]\n",
        "tempTopUsersRating.columns = ['sum_similarityIndex','sum_weightedRating']\n",
        "tempTopUsersRating.head()"
      ],
      "execution_count": 69,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>sum_similarityIndex</th>\n",
              "      <th>sum_weightedRating</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>movieId</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>32.199621</td>\n",
              "      <td>128.527810</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>31.737956</td>\n",
              "      <td>79.089079</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>15.798521</td>\n",
              "      <td>41.417088</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>5.453482</td>\n",
              "      <td>14.903832</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>16.095377</td>\n",
              "      <td>38.191849</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "         sum_similarityIndex  sum_weightedRating\n",
              "movieId                                         \n",
              "1                  32.199621          128.527810\n",
              "2                  31.737956           79.089079\n",
              "3                  15.798521           41.417088\n",
              "4                   5.453482           14.903832\n",
              "5                  16.095377           38.191849"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 69
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "KQNnq58-Gyt6",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 225
        },
        "outputId": "f3af2cc3-98dc-4727-cfd2-daee4af372a6"
      },
      "source": [
        "#Creates an empty dataframe\n",
        "recommendation_df = pd.DataFrame()\n",
        "#Now we take the weighted average\n",
        "recommendation_df['weighted average recommendation score'] = tempTopUsersRating['sum_weightedRating']/tempTopUsersRating['sum_similarityIndex']\n",
        "recommendation_df['movieId'] = tempTopUsersRating.index\n",
        "recommendation_df.head()"
      ],
      "execution_count": 70,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>weighted average recommendation score</th>\n",
              "      <th>movieId</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>movieId</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>3.991594</td>\n",
              "      <td>1</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>2.491940</td>\n",
              "      <td>2</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>2.621580</td>\n",
              "      <td>3</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>2.732902</td>\n",
              "      <td>4</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>2.372846</td>\n",
              "      <td>5</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "         weighted average recommendation score  movieId\n",
              "movieId                                                \n",
              "1                                     3.991594        1\n",
              "2                                     2.491940        2\n",
              "3                                     2.621580        3\n",
              "4                                     2.732902        4\n",
              "5                                     2.372846        5"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 70
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "fMiWnvv9Gyt8",
        "colab_type": "text"
      },
      "source": [
        "Now let's sort it and see the top 20 movies that the algorithm recommended!"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "scrolled": false,
        "id": "Wi4c44GiGyt9",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 373
        },
        "outputId": "c2dcb2be-540b-4056-d6a8-aaec2007a6c2"
      },
      "source": [
        "recommendation_df = recommendation_df.sort_values(by='weighted average recommendation score', ascending=False)\n",
        "recommendation_df.head(10)"
      ],
      "execution_count": 71,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>weighted average recommendation score</th>\n",
              "      <th>movieId</th>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>movieId</th>\n",
              "      <th></th>\n",
              "      <th></th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>2711</th>\n",
              "      <td>5.0</td>\n",
              "      <td>2711</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4806</th>\n",
              "      <td>5.0</td>\n",
              "      <td>4806</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>26208</th>\n",
              "      <td>5.0</td>\n",
              "      <td>26208</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>93988</th>\n",
              "      <td>5.0</td>\n",
              "      <td>93988</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>27156</th>\n",
              "      <td>5.0</td>\n",
              "      <td>27156</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1181</th>\n",
              "      <td>5.0</td>\n",
              "      <td>1181</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8785</th>\n",
              "      <td>5.0</td>\n",
              "      <td>8785</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>85312</th>\n",
              "      <td>5.0</td>\n",
              "      <td>85312</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1156</th>\n",
              "      <td>5.0</td>\n",
              "      <td>1156</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>49513</th>\n",
              "      <td>5.0</td>\n",
              "      <td>49513</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "         weighted average recommendation score  movieId\n",
              "movieId                                                \n",
              "2711                                       5.0     2711\n",
              "4806                                       5.0     4806\n",
              "26208                                      5.0    26208\n",
              "93988                                      5.0    93988\n",
              "27156                                      5.0    27156\n",
              "1181                                       5.0     1181\n",
              "8785                                       5.0     8785\n",
              "85312                                      5.0    85312\n",
              "1156                                       5.0     1156\n",
              "49513                                      5.0    49513"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 71
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "scrolled": true,
        "id": "Czk8OAzxGyt_",
        "colab_type": "code",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 343
        },
        "outputId": "715777b0-de7e-4419-857a-1285576f6d3c"
      },
      "source": [
        "movies_df.loc[movies_df['movieId'].isin(recommendation_df.head(10)['movieId'].tolist())]"
      ],
      "execution_count": 72,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>movieId</th>\n",
              "      <th>title</th>\n",
              "      <th>year</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>1133</th>\n",
              "      <td>1156</td>\n",
              "      <td>Children Are Watching Us, The (Bambini ci guar...</td>\n",
              "      <td>1944</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1158</th>\n",
              "      <td>1181</td>\n",
              "      <td>Shooter, The</td>\n",
              "      <td>1997</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2626</th>\n",
              "      <td>2711</td>\n",
              "      <td>My Life So Far</td>\n",
              "      <td>1999</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4711</th>\n",
              "      <td>4806</td>\n",
              "      <td>Shop on Main Street, The (Obchod na korze)</td>\n",
              "      <td>1965</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8103</th>\n",
              "      <td>8785</td>\n",
              "      <td>Early Summer (Bakushû)</td>\n",
              "      <td>1951</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>8688</th>\n",
              "      <td>26208</td>\n",
              "      <td>My Night At Maud's (Ma Nuit Chez Maud)</td>\n",
              "      <td>1969</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>9219</th>\n",
              "      <td>27156</td>\n",
              "      <td>Neon Genesis Evangelion: The End of Evangelion...</td>\n",
              "      <td>1997</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>11489</th>\n",
              "      <td>49513</td>\n",
              "      <td>Brighter Summer Day, A (Gu ling jie shao nian ...</td>\n",
              "      <td>1991</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>16898</th>\n",
              "      <td>85312</td>\n",
              "      <td>House of Small Cubes, The (Tsumiki no ie)</td>\n",
              "      <td>2008</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>18942</th>\n",
              "      <td>93988</td>\n",
              "      <td>North &amp; South</td>\n",
              "      <td>2004</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "       movieId                                              title  year\n",
              "1133      1156  Children Are Watching Us, The (Bambini ci guar...  1944\n",
              "1158      1181                                       Shooter, The  1997\n",
              "2626      2711                                     My Life So Far  1999\n",
              "4711      4806         Shop on Main Street, The (Obchod na korze)  1965\n",
              "8103      8785                             Early Summer (Bakushû)  1951\n",
              "8688     26208             My Night At Maud's (Ma Nuit Chez Maud)  1969\n",
              "9219     27156  Neon Genesis Evangelion: The End of Evangelion...  1997\n",
              "11489    49513  Brighter Summer Day, A (Gu ling jie shao nian ...  1991\n",
              "16898    85312          House of Small Cubes, The (Tsumiki no ie)  2008\n",
              "18942    93988                                      North & South  2004"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 72
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "Tug6HO5vGyuA",
        "colab_type": "text"
      },
      "source": [
        "### Advantages and Disadvantages of Collaborative Filtering\n",
        "\n",
        "##### Advantages\n",
        "* Takes other user's ratings into consideration\n",
        "* Doesn't need to study or extract information from the recommended item\n",
        "* Adapts to the user's interests which might change over time\n",
        "\n",
        "##### Disadvantages\n",
        "* Approximation function can be slow\n",
        "* There might be a low of amount of users to approximate\n",
        "* Privacy issues when trying to learn the user's preferences"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "button": false,
        "deletable": true,
        "new_sheet": false,
        "run_control": {
          "read_only": false
        },
        "id": "7if2zdMVGyuB",
        "colab_type": "text"
      },
      "source": [
        "<h2>Want to learn more?</h2>\n",
        "\n",
        "IBM SPSS Modeler is a comprehensive analytics platform that has many machine learning algorithms. It has been designed to bring predictive intelligence to decisions made by individuals, by groups, by systems – by your enterprise as a whole. A free trial is available through this course, available here: <a href=\"http://cocl.us/ML0101EN-SPSSModeler\">SPSS Modeler</a>\n",
        "\n",
        "Also, you can use Watson Studio to run these notebooks faster with bigger datasets. Watson Studio is IBM's leading cloud solution for data scientists, built by data scientists. With Jupyter notebooks, RStudio, Apache Spark and popular libraries pre-packaged in the cloud, Watson Studio enables data scientists to collaborate on their projects without having to install anything. Join the fast-growing community of Watson Studio users today with a free account at <a href=\"https://cocl.us/ML0101EN_DSX\">Watson Studio</a>\n",
        "\n",
        "<h3>Thanks for completing this lesson!</h3>\n",
        "\n",
        "<h4>Author:  <a href=\"https://ca.linkedin.com/in/saeedaghabozorgi\">Saeed Aghabozorgi</a></h4>\n",
        "<p><a href=\"https://ca.linkedin.com/in/saeedaghabozorgi\">Saeed Aghabozorgi</a>, PhD is a Data Scientist in IBM with a track record of developing enterprise level applications that substantially increases clients’ ability to turn data into actionable knowledge. He is a researcher in data mining field and expert in developing advanced analytic methods like machine learning and statistical modelling on large datasets.</p>\n",
        "\n",
        "<hr>\n",
        "\n",
        "<p>Copyright &copy; 2018 <a href=\"https://cocl.us/DX0108EN_CC\">Cognitive Class</a>. This notebook and its source code are released under the terms of the <a href=\"https://bigdatauniversity.com/mit-license/\">MIT License</a>.</p>"
      ]
    }
  ]
}