Recommender-System.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Importing the required libraries\n",
    "\n",
    "%matplotlib inline\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "from scipy import stats\n",
    "from ast import literal_eval\n",
    "from sklearn.feature_extraction.text import TfidfVectorizer, CountVectorizer\n",
    "from sklearn.metrics.pairwise import linear_kernel, cosine_similarity\n",
    "from nltk.stem.snowball import SnowballStemmer\n",
    "from nltk.stem.wordnet import WordNetLemmatizer\n",
    "from nltk.corpus import wordnet\n",
    "from surprise import Reader, Dataset, SVD\n",
    "from surprise.model_selection import cross_validate\n",
    "\n",
    "import warnings; warnings.simplefilter('ignore')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Simple Recommender system \n",
    "##### A Recommender system that recommends movies based on movie ratings and other basic criteria."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "meta = pd.read_csv('Data/movies_metadata.csv')\n",
    "meta['genres'] = meta['genres'].fillna('[]').apply(literal_eval).apply(lambda x: [i['name'] for i in x] if isinstance(x, list) else [])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Average rating (out of 10) required by the movie to qualify is: 5.245\n",
      "Minimum number of votes required by the movie to qualify is: 434.0\n"
     ]
    }
   ],
   "source": [
    "vote_counts = meta[meta['vote_count'].notnull()]['vote_count'].astype('int')\n",
    "vote_averages = meta[meta['vote_average'].notnull()]['vote_average'].astype('int')\n",
    "avg_rating = round(vote_averages.mean(),3)\n",
    "min_votes = vote_counts.quantile(0.95)\n",
    "print(\"Average rating (out of 10) required by the movie to qualify is:\",avg_rating)\n",
    "print(\"Minimum number of votes required by the movie to qualify is:\",min_votes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "meta['year'] = pd.to_datetime(meta['release_date'], errors='coerce').apply(lambda x: str(x).split('-')[0] if x != np.nan else np.nan)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(2181, 6)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Creating a new df called qualified_movies with the following criteria:\n",
    "# vote_count >= min_votes and is not null\n",
    "# vote_average >= avg_rating and is not null\n",
    "\n",
    "qualified_movies = meta[(meta['vote_count'] >= min_votes) & (meta['vote_count'].notnull()) & (meta['vote_average'].notnull())& (meta['vote_average']>= avg_rating)][['title', 'year', 'vote_count', 'vote_average', 'popularity', 'genres']]\n",
    "qualified_movies['vote_count'] = qualified_movies['vote_count'].astype('int')\n",
    "qualified_movies['vote_average'] = qualified_movies['vote_average'].astype('int')\n",
    "qualified_movies.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Creating a new attribute called 'weighted_rating' to show top movies. \n",
    "$\\\\$\n",
    "Weighted_Rating = $(\\frac{votes}{votes + min\\_votes} . rating) + (\\frac{min\\_votes}{votes + min\\_votes} . avg\\_rating)$\n",
    "$\\\\$\n",
    "$\\\\$\n",
    "where,\n",
    "* *votes* is the number of votes for the movie\n",
    "* *min_votes* is the minimum votes required to be listed in the chart\n",
    "* *rating* is the average rating of the movie\n",
    "* *avg_rating* is the mean vote across the whole report"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "def weighted_rating(df):\n",
    "    votes = df['vote_count']\n",
    "    rating = df['vote_average']\n",
    "    return (votes/(votes+min_votes)*rating) + (min_votes/(min_votes+votes)*avg_rating)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "qualified_movies['Weighted_Rating'] = qualified_movies.apply(weighted_rating,axis = 1)\n",
    "qualified_movies = qualified_movies.sort_values('Weighted_Rating', ascending= False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>year</th>\n",
       "      <th>vote_count</th>\n",
       "      <th>vote_average</th>\n",
       "      <th>popularity</th>\n",
       "      <th>genres</th>\n",
       "      <th>Weighted_Rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>15480</th>\n",
       "      <td>Inception</td>\n",
       "      <td>2010</td>\n",
       "      <td>14075</td>\n",
       "      <td>8</td>\n",
       "      <td>29.108149</td>\n",
       "      <td>[Action, Thriller, Science Fiction, Mystery, A...</td>\n",
       "      <td>7.917591</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12481</th>\n",
       "      <td>The Dark Knight</td>\n",
       "      <td>2008</td>\n",
       "      <td>12269</td>\n",
       "      <td>8</td>\n",
       "      <td>123.167259</td>\n",
       "      <td>[Drama, Action, Crime, Thriller]</td>\n",
       "      <td>7.905875</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>22879</th>\n",
       "      <td>Interstellar</td>\n",
       "      <td>2014</td>\n",
       "      <td>11187</td>\n",
       "      <td>8</td>\n",
       "      <td>32.213481</td>\n",
       "      <td>[Adventure, Drama, Science Fiction]</td>\n",
       "      <td>7.897111</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2843</th>\n",
       "      <td>Fight Club</td>\n",
       "      <td>1999</td>\n",
       "      <td>9678</td>\n",
       "      <td>8</td>\n",
       "      <td>63.869599</td>\n",
       "      <td>[Drama]</td>\n",
       "      <td>7.881757</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4863</th>\n",
       "      <td>The Lord of the Rings: The Fellowship of the Ring</td>\n",
       "      <td>2001</td>\n",
       "      <td>8892</td>\n",
       "      <td>8</td>\n",
       "      <td>32.070725</td>\n",
       "      <td>[Adventure, Fantasy, Action]</td>\n",
       "      <td>7.871792</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>292</th>\n",
       "      <td>Pulp Fiction</td>\n",
       "      <td>1994</td>\n",
       "      <td>8670</td>\n",
       "      <td>8</td>\n",
       "      <td>140.950236</td>\n",
       "      <td>[Thriller, Crime]</td>\n",
       "      <td>7.868665</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>314</th>\n",
       "      <td>The Shawshank Redemption</td>\n",
       "      <td>1994</td>\n",
       "      <td>8358</td>\n",
       "      <td>8</td>\n",
       "      <td>51.645403</td>\n",
       "      <td>[Drama, Crime]</td>\n",
       "      <td>7.864005</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7000</th>\n",
       "      <td>The Lord of the Rings: The Return of the King</td>\n",
       "      <td>2003</td>\n",
       "      <td>8226</td>\n",
       "      <td>8</td>\n",
       "      <td>29.324358</td>\n",
       "      <td>[Adventure, Fantasy, Action]</td>\n",
       "      <td>7.861932</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>351</th>\n",
       "      <td>Forrest Gump</td>\n",
       "      <td>1994</td>\n",
       "      <td>8147</td>\n",
       "      <td>8</td>\n",
       "      <td>48.307194</td>\n",
       "      <td>[Comedy, Drama, Romance]</td>\n",
       "      <td>7.860661</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5814</th>\n",
       "      <td>The Lord of the Rings: The Two Towers</td>\n",
       "      <td>2002</td>\n",
       "      <td>7641</td>\n",
       "      <td>8</td>\n",
       "      <td>29.423537</td>\n",
       "      <td>[Adventure, Fantasy, Action]</td>\n",
       "      <td>7.851929</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>256</th>\n",
       "      <td>Star Wars</td>\n",
       "      <td>1977</td>\n",
       "      <td>6778</td>\n",
       "      <td>8</td>\n",
       "      <td>42.149697</td>\n",
       "      <td>[Adventure, Action, Science Fiction]</td>\n",
       "      <td>7.834211</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1225</th>\n",
       "      <td>Back to the Future</td>\n",
       "      <td>1985</td>\n",
       "      <td>6239</td>\n",
       "      <td>8</td>\n",
       "      <td>25.778509</td>\n",
       "      <td>[Adventure, Comedy, Science Fiction, Family]</td>\n",
       "      <td>7.820820</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>834</th>\n",
       "      <td>The Godfather</td>\n",
       "      <td>1972</td>\n",
       "      <td>6024</td>\n",
       "      <td>8</td>\n",
       "      <td>41.109264</td>\n",
       "      <td>[Drama, Crime]</td>\n",
       "      <td>7.814854</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1154</th>\n",
       "      <td>The Empire Strikes Back</td>\n",
       "      <td>1980</td>\n",
       "      <td>5998</td>\n",
       "      <td>8</td>\n",
       "      <td>19.470959</td>\n",
       "      <td>[Adventure, Action, Science Fiction]</td>\n",
       "      <td>7.814106</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>Se7en</td>\n",
       "      <td>1995</td>\n",
       "      <td>5915</td>\n",
       "      <td>8</td>\n",
       "      <td>18.45743</td>\n",
       "      <td>[Crime, Mystery, Thriller]</td>\n",
       "      <td>7.811676</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                   title  year  vote_count  \\\n",
       "15480                                          Inception  2010       14075   \n",
       "12481                                    The Dark Knight  2008       12269   \n",
       "22879                                       Interstellar  2014       11187   \n",
       "2843                                          Fight Club  1999        9678   \n",
       "4863   The Lord of the Rings: The Fellowship of the Ring  2001        8892   \n",
       "292                                         Pulp Fiction  1994        8670   \n",
       "314                             The Shawshank Redemption  1994        8358   \n",
       "7000       The Lord of the Rings: The Return of the King  2003        8226   \n",
       "351                                         Forrest Gump  1994        8147   \n",
       "5814               The Lord of the Rings: The Two Towers  2002        7641   \n",
       "256                                            Star Wars  1977        6778   \n",
       "1225                                  Back to the Future  1985        6239   \n",
       "834                                        The Godfather  1972        6024   \n",
       "1154                             The Empire Strikes Back  1980        5998   \n",
       "46                                                 Se7en  1995        5915   \n",
       "\n",
       "       vote_average  popularity  \\\n",
       "15480             8   29.108149   \n",
       "12481             8  123.167259   \n",
       "22879             8   32.213481   \n",
       "2843              8   63.869599   \n",
       "4863              8   32.070725   \n",
       "292               8  140.950236   \n",
       "314               8   51.645403   \n",
       "7000              8   29.324358   \n",
       "351               8   48.307194   \n",
       "5814              8   29.423537   \n",
       "256               8   42.149697   \n",
       "1225              8   25.778509   \n",
       "834               8   41.109264   \n",
       "1154              8   19.470959   \n",
       "46                8    18.45743   \n",
       "\n",
       "                                                  genres  Weighted_Rating  \n",
       "15480  [Action, Thriller, Science Fiction, Mystery, A...         7.917591  \n",
       "12481                   [Drama, Action, Crime, Thriller]         7.905875  \n",
       "22879                [Adventure, Drama, Science Fiction]         7.897111  \n",
       "2843                                             [Drama]         7.881757  \n",
       "4863                        [Adventure, Fantasy, Action]         7.871792  \n",
       "292                                    [Thriller, Crime]         7.868665  \n",
       "314                                       [Drama, Crime]         7.864005  \n",
       "7000                        [Adventure, Fantasy, Action]         7.861932  \n",
       "351                             [Comedy, Drama, Romance]         7.860661  \n",
       "5814                        [Adventure, Fantasy, Action]         7.851929  \n",
       "256                 [Adventure, Action, Science Fiction]         7.834211  \n",
       "1225        [Adventure, Comedy, Science Fiction, Family]         7.820820  \n",
       "834                                       [Drama, Crime]         7.814854  \n",
       "1154                [Adventure, Action, Science Fiction]         7.814106  \n",
       "46                            [Crime, Mystery, Thriller]         7.811676  "
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "qualified_movies.head(15)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "genres = meta.apply(lambda x: pd.Series(x['genres']),axis=1).stack().reset_index(level=1, drop=True)\n",
    "genres.name = 'genre'\n",
    "gen_meta = meta.drop('genres', axis=1).join(genres)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def top_movies(genre,percentile = 0.80):\n",
    "    genres = meta.apply(lambda x: pd.Series(x['genres']),axis=1).stack().reset_index(level=1, drop=True)\n",
    "    genres.name = 'genre'\n",
    "    gen_meta = meta.drop('genres', axis=1).join(genres)\n",
    "\n",
    "    df = gen_meta[gen_meta['genre'] == genre]\n",
    "    vote_counts = df[df['vote_count'].notnull()]['vote_count'].astype('int')\n",
    "    vote_averages = df[df['vote_average'].notnull()]['vote_average'].astype('int')\n",
    "    avg_rating = round(vote_averages.mean(),3)\n",
    "    min_votes = vote_counts.quantile(percentile)\n",
    "\n",
    "    qualified_movies = df[(df['vote_count'] >= min_votes) & (df['vote_count'].notnull()) & (df['vote_average'].notnull())& (df['vote_average']>= avg_rating)][['title', 'year', 'vote_count', 'vote_average', 'popularity', 'genre']]\n",
    "    qualified_movies['vote_count'] = qualified_movies['vote_count'].astype('int')\n",
    "    qualified_movies['vote_average'] = qualified_movies['vote_average'].astype('int')\n",
    "    \n",
    "    qualified_movies['Weighted_Rating'] = qualified_movies.apply(weighted_rating,axis = 1)\n",
    "    qualified_movies = qualified_movies.sort_values('Weighted_Rating', ascending= False)\n",
    "\n",
    "    return qualified_movies\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>year</th>\n",
       "      <th>vote_count</th>\n",
       "      <th>vote_average</th>\n",
       "      <th>popularity</th>\n",
       "      <th>genre</th>\n",
       "      <th>Weighted_Rating</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>15480</th>\n",
       "      <td>Inception</td>\n",
       "      <td>2010</td>\n",
       "      <td>14075</td>\n",
       "      <td>8</td>\n",
       "      <td>29.108149</td>\n",
       "      <td>Thriller</td>\n",
       "      <td>7.917591</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12481</th>\n",
       "      <td>The Dark Knight</td>\n",
       "      <td>2008</td>\n",
       "      <td>12269</td>\n",
       "      <td>8</td>\n",
       "      <td>123.167259</td>\n",
       "      <td>Thriller</td>\n",
       "      <td>7.905875</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>292</th>\n",
       "      <td>Pulp Fiction</td>\n",
       "      <td>1994</td>\n",
       "      <td>8670</td>\n",
       "      <td>8</td>\n",
       "      <td>140.950236</td>\n",
       "      <td>Thriller</td>\n",
       "      <td>7.868665</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>46</th>\n",
       "      <td>Se7en</td>\n",
       "      <td>1995</td>\n",
       "      <td>5915</td>\n",
       "      <td>8</td>\n",
       "      <td>18.45743</td>\n",
       "      <td>Thriller</td>\n",
       "      <td>7.811676</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>24860</th>\n",
       "      <td>The Imitation Game</td>\n",
       "      <td>2014</td>\n",
       "      <td>5895</td>\n",
       "      <td>8</td>\n",
       "      <td>31.59594</td>\n",
       "      <td>Thriller</td>\n",
       "      <td>7.811081</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42309</th>\n",
       "      <td>Ghost in the Shell</td>\n",
       "      <td>2017</td>\n",
       "      <td>2547</td>\n",
       "      <td>5</td>\n",
       "      <td>68.726676</td>\n",
       "      <td>Thriller</td>\n",
       "      <td>5.035669</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11864</th>\n",
       "      <td>Fantastic 4: Rise of the Silver Surfer</td>\n",
       "      <td>2007</td>\n",
       "      <td>2648</td>\n",
       "      <td>5</td>\n",
       "      <td>12.924254</td>\n",
       "      <td>Thriller</td>\n",
       "      <td>5.034500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>42902</th>\n",
       "      <td>Alien: Covenant</td>\n",
       "      <td>2017</td>\n",
       "      <td>2677</td>\n",
       "      <td>5</td>\n",
       "      <td>72.884078</td>\n",
       "      <td>Thriller</td>\n",
       "      <td>5.034179</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20536</th>\n",
       "      <td>G.I. Joe: Retaliation</td>\n",
       "      <td>2013</td>\n",
       "      <td>3045</td>\n",
       "      <td>5</td>\n",
       "      <td>10.560608</td>\n",
       "      <td>Thriller</td>\n",
       "      <td>5.030563</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25865</th>\n",
       "      <td>Terminator Genisys</td>\n",
       "      <td>2015</td>\n",
       "      <td>3677</td>\n",
       "      <td>5</td>\n",
       "      <td>30.188198</td>\n",
       "      <td>Thriller</td>\n",
       "      <td>5.025865</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1383 rows × 7 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                        title  year  vote_count  vote_average  \\\n",
       "15480                               Inception  2010       14075             8   \n",
       "12481                         The Dark Knight  2008       12269             8   \n",
       "292                              Pulp Fiction  1994        8670             8   \n",
       "46                                      Se7en  1995        5915             8   \n",
       "24860                      The Imitation Game  2014        5895             8   \n",
       "...                                       ...   ...         ...           ...   \n",
       "42309                      Ghost in the Shell  2017        2547             5   \n",
       "11864  Fantastic 4: Rise of the Silver Surfer  2007        2648             5   \n",
       "42902                         Alien: Covenant  2017        2677             5   \n",
       "20536                   G.I. Joe: Retaliation  2013        3045             5   \n",
       "25865                      Terminator Genisys  2015        3677             5   \n",
       "\n",
       "       popularity     genre  Weighted_Rating  \n",
       "15480   29.108149  Thriller         7.917591  \n",
       "12481  123.167259  Thriller         7.905875  \n",
       "292    140.950236  Thriller         7.868665  \n",
       "46       18.45743  Thriller         7.811676  \n",
       "24860    31.59594  Thriller         7.811081  \n",
       "...           ...       ...              ...  \n",
       "42309   68.726676  Thriller         5.035669  \n",
       "11864   12.924254  Thriller         5.034500  \n",
       "42902   72.884078  Thriller         5.034179  \n",
       "20536   10.560608  Thriller         5.030563  \n",
       "25865   30.188198  Thriller         5.025865  \n",
       "\n",
       "[1383 rows x 7 columns]"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "top_movies('Thriller')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Content Based Recommender (Content Based Filtering)\n",
    "##### A recommender system that computes the similarity between movies based on certain criteria and metrics and recommends movies that are extremely similar to a movie that a user liked."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Two types of systems can be created:\n",
    "\n",
    "##### 1) Movie description and taglines based recommender\n",
    "##### 2) Movie cast, crew, keywords and genre based recommender"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "links_small = pd.read_csv('Data/links_small.csv')\n",
    "links_small = links_small[links_small['tmdbId'].notnull()]['tmdbId'].astype('int')\n",
    "\n",
    "meta = pd.read_csv('Data/movies_metadata.csv')\n",
    "meta['genres'] = meta['genres'].fillna('[]').apply(literal_eval).apply(lambda x: [i['name'] for i in x] if isinstance(x, list) else [])\n",
    "meta['year'] = pd.to_datetime(meta['release_date'], errors='coerce').apply(lambda x: str(x).split('-')[0] if x != np.nan else np.nan)\n",
    "\n",
    "meta = meta.drop([19730, 29503, 35587]) #Incorrect data so remove it.\n",
    "meta['id'] = meta['id'].astype('int')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>adult</th>\n",
       "      <th>belongs_to_collection</th>\n",
       "      <th>budget</th>\n",
       "      <th>genres</th>\n",
       "      <th>homepage</th>\n",
       "      <th>id</th>\n",
       "      <th>imdb_id</th>\n",
       "      <th>original_language</th>\n",
       "      <th>original_title</th>\n",
       "      <th>overview</th>\n",
       "      <th>...</th>\n",
       "      <th>revenue</th>\n",
       "      <th>runtime</th>\n",
       "      <th>spoken_languages</th>\n",
       "      <th>status</th>\n",
       "      <th>tagline</th>\n",
       "      <th>title</th>\n",
       "      <th>video</th>\n",
       "      <th>vote_average</th>\n",
       "      <th>vote_count</th>\n",
       "      <th>year</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>False</td>\n",
       "      <td>{'id': 10194, 'name': 'Toy Story Collection', ...</td>\n",
       "      <td>30000000</td>\n",
       "      <td>[Animation, Comedy, Family]</td>\n",
       "      <td>http://toystory.disney.com/toy-story</td>\n",
       "      <td>862</td>\n",
       "      <td>tt0114709</td>\n",
       "      <td>en</td>\n",
       "      <td>Toy Story</td>\n",
       "      <td>Led by Woody, Andy's toys live happily in his ...</td>\n",
       "      <td>...</td>\n",
       "      <td>373554033.0</td>\n",
       "      <td>81.0</td>\n",
       "      <td>[{'iso_639_1': 'en', 'name': 'English'}]</td>\n",
       "      <td>Released</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Toy Story</td>\n",
       "      <td>False</td>\n",
       "      <td>7.7</td>\n",
       "      <td>5415.0</td>\n",
       "      <td>1995</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>65000000</td>\n",
       "      <td>[Adventure, Fantasy, Family]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>8844</td>\n",
       "      <td>tt0113497</td>\n",
       "      <td>en</td>\n",
       "      <td>Jumanji</td>\n",
       "      <td>When siblings Judy and Peter discover an encha...</td>\n",
       "      <td>...</td>\n",
       "      <td>262797249.0</td>\n",
       "      <td>104.0</td>\n",
       "      <td>[{'iso_639_1': 'en', 'name': 'English'}, {'iso...</td>\n",
       "      <td>Released</td>\n",
       "      <td>Roll the dice and unleash the excitement!</td>\n",
       "      <td>Jumanji</td>\n",
       "      <td>False</td>\n",
       "      <td>6.9</td>\n",
       "      <td>2413.0</td>\n",
       "      <td>1995</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>False</td>\n",
       "      <td>{'id': 119050, 'name': 'Grumpy Old Men Collect...</td>\n",
       "      <td>0</td>\n",
       "      <td>[Romance, Comedy]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>15602</td>\n",
       "      <td>tt0113228</td>\n",
       "      <td>en</td>\n",
       "      <td>Grumpier Old Men</td>\n",
       "      <td>A family wedding reignites the ancient feud be...</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>101.0</td>\n",
       "      <td>[{'iso_639_1': 'en', 'name': 'English'}]</td>\n",
       "      <td>Released</td>\n",
       "      <td>Still Yelling. Still Fighting. Still Ready for...</td>\n",
       "      <td>Grumpier Old Men</td>\n",
       "      <td>False</td>\n",
       "      <td>6.5</td>\n",
       "      <td>92.0</td>\n",
       "      <td>1995</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>16000000</td>\n",
       "      <td>[Comedy, Drama, Romance]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>31357</td>\n",
       "      <td>tt0114885</td>\n",
       "      <td>en</td>\n",
       "      <td>Waiting to Exhale</td>\n",
       "      <td>Cheated on, mistreated and stepped on, the wom...</td>\n",
       "      <td>...</td>\n",
       "      <td>81452156.0</td>\n",
       "      <td>127.0</td>\n",
       "      <td>[{'iso_639_1': 'en', 'name': 'English'}]</td>\n",
       "      <td>Released</td>\n",
       "      <td>Friends are the people who let you be yourself...</td>\n",
       "      <td>Waiting to Exhale</td>\n",
       "      <td>False</td>\n",
       "      <td>6.1</td>\n",
       "      <td>34.0</td>\n",
       "      <td>1995</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>False</td>\n",
       "      <td>{'id': 96871, 'name': 'Father of the Bride Col...</td>\n",
       "      <td>0</td>\n",
       "      <td>[Comedy]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>11862</td>\n",
       "      <td>tt0113041</td>\n",
       "      <td>en</td>\n",
       "      <td>Father of the Bride Part II</td>\n",
       "      <td>Just when George Banks has recovered from his ...</td>\n",
       "      <td>...</td>\n",
       "      <td>76578911.0</td>\n",
       "      <td>106.0</td>\n",
       "      <td>[{'iso_639_1': 'en', 'name': 'English'}]</td>\n",
       "      <td>Released</td>\n",
       "      <td>Just When His World Is Back To Normal... He's ...</td>\n",
       "      <td>Father of the Bride Part II</td>\n",
       "      <td>False</td>\n",
       "      <td>5.7</td>\n",
       "      <td>173.0</td>\n",
       "      <td>1995</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40224</th>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>15000000</td>\n",
       "      <td>[Action, Adventure, Drama, Horror, Science Fic...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>315011</td>\n",
       "      <td>tt4262980</td>\n",
       "      <td>ja</td>\n",
       "      <td>シン・ゴジラ</td>\n",
       "      <td>From the mind behind Evangelion comes a hit la...</td>\n",
       "      <td>...</td>\n",
       "      <td>77000000.0</td>\n",
       "      <td>120.0</td>\n",
       "      <td>[{'iso_639_1': 'it', 'name': 'Italiano'}, {'is...</td>\n",
       "      <td>Released</td>\n",
       "      <td>A god incarnate. A city doomed.</td>\n",
       "      <td>Shin Godzilla</td>\n",
       "      <td>False</td>\n",
       "      <td>6.6</td>\n",
       "      <td>152.0</td>\n",
       "      <td>2016</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>40503</th>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>[Documentary, Music]</td>\n",
       "      <td>http://www.thebeatlesliveproject.com/</td>\n",
       "      <td>391698</td>\n",
       "      <td>tt2531318</td>\n",
       "      <td>en</td>\n",
       "      <td>The Beatles: Eight Days a Week - The Touring Y...</td>\n",
       "      <td>The band stormed Europe in 1963, and, in 1964,...</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>99.0</td>\n",
       "      <td>[{'iso_639_1': 'en', 'name': 'English'}]</td>\n",
       "      <td>Released</td>\n",
       "      <td>The band you know. The story you don't.</td>\n",
       "      <td>The Beatles: Eight Days a Week - The Touring Y...</td>\n",
       "      <td>False</td>\n",
       "      <td>7.6</td>\n",
       "      <td>92.0</td>\n",
       "      <td>2016</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44821</th>\n",
       "      <td>False</td>\n",
       "      <td>{'id': 34055, 'name': 'Pokémon Collection', 'p...</td>\n",
       "      <td>16000000</td>\n",
       "      <td>[Adventure, Fantasy, Animation, Action, Family]</td>\n",
       "      <td>http://movies.warnerbros.com/pk3/</td>\n",
       "      <td>10991</td>\n",
       "      <td>tt0235679</td>\n",
       "      <td>ja</td>\n",
       "      <td>Pokémon 3: The Movie</td>\n",
       "      <td>When Molly Hale's sadness of her father's disa...</td>\n",
       "      <td>...</td>\n",
       "      <td>68411275.0</td>\n",
       "      <td>93.0</td>\n",
       "      <td>[{'iso_639_1': 'en', 'name': 'English'}]</td>\n",
       "      <td>Released</td>\n",
       "      <td>Pokémon: Spell of the Unknown</td>\n",
       "      <td>Pokémon: Spell of the Unknown</td>\n",
       "      <td>False</td>\n",
       "      <td>6.0</td>\n",
       "      <td>144.0</td>\n",
       "      <td>2000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>44826</th>\n",
       "      <td>False</td>\n",
       "      <td>{'id': 34055, 'name': 'Pokémon Collection', 'p...</td>\n",
       "      <td>0</td>\n",
       "      <td>[Adventure, Fantasy, Animation, Science Fictio...</td>\n",
       "      <td>http://www.pokemon.com/us/movies/movie-pokemon...</td>\n",
       "      <td>12600</td>\n",
       "      <td>tt0287635</td>\n",
       "      <td>ja</td>\n",
       "      <td>劇場版ポケットモンスター セレビィ 時を越えた遭遇（であい）</td>\n",
       "      <td>All your favorite Pokémon characters are back,...</td>\n",
       "      <td>...</td>\n",
       "      <td>28023563.0</td>\n",
       "      <td>75.0</td>\n",
       "      <td>[{'iso_639_1': 'ja', 'name': '日本語'}]</td>\n",
       "      <td>Released</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Pokémon 4Ever: Celebi - Voice of the Forest</td>\n",
       "      <td>False</td>\n",
       "      <td>5.7</td>\n",
       "      <td>82.0</td>\n",
       "      <td>2001</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45265</th>\n",
       "      <td>False</td>\n",
       "      <td>NaN</td>\n",
       "      <td>0</td>\n",
       "      <td>[Comedy, Drama]</td>\n",
       "      <td>NaN</td>\n",
       "      <td>265189</td>\n",
       "      <td>tt2121382</td>\n",
       "      <td>sv</td>\n",
       "      <td>Turist</td>\n",
       "      <td>While holidaying in the French Alps, a Swedish...</td>\n",
       "      <td>...</td>\n",
       "      <td>1359497.0</td>\n",
       "      <td>118.0</td>\n",
       "      <td>[{'iso_639_1': 'fr', 'name': 'Français'}, {'is...</td>\n",
       "      <td>Released</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Force Majeure</td>\n",
       "      <td>False</td>\n",
       "      <td>6.8</td>\n",
       "      <td>255.0</td>\n",
       "      <td>2014</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>9099 rows × 25 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       adult                              belongs_to_collection    budget  \\\n",
       "0      False  {'id': 10194, 'name': 'Toy Story Collection', ...  30000000   \n",
       "1      False                                                NaN  65000000   \n",
       "2      False  {'id': 119050, 'name': 'Grumpy Old Men Collect...         0   \n",
       "3      False                                                NaN  16000000   \n",
       "4      False  {'id': 96871, 'name': 'Father of the Bride Col...         0   \n",
       "...      ...                                                ...       ...   \n",
       "40224  False                                                NaN  15000000   \n",
       "40503  False                                                NaN         0   \n",
       "44821  False  {'id': 34055, 'name': 'Pokémon Collection', 'p...  16000000   \n",
       "44826  False  {'id': 34055, 'name': 'Pokémon Collection', 'p...         0   \n",
       "45265  False                                                NaN         0   \n",
       "\n",
       "                                                  genres  \\\n",
       "0                            [Animation, Comedy, Family]   \n",
       "1                           [Adventure, Fantasy, Family]   \n",
       "2                                      [Romance, Comedy]   \n",
       "3                               [Comedy, Drama, Romance]   \n",
       "4                                               [Comedy]   \n",
       "...                                                  ...   \n",
       "40224  [Action, Adventure, Drama, Horror, Science Fic...   \n",
       "40503                               [Documentary, Music]   \n",
       "44821    [Adventure, Fantasy, Animation, Action, Family]   \n",
       "44826  [Adventure, Fantasy, Animation, Science Fictio...   \n",
       "45265                                    [Comedy, Drama]   \n",
       "\n",
       "                                                homepage      id    imdb_id  \\\n",
       "0                   http://toystory.disney.com/toy-story     862  tt0114709   \n",
       "1                                                    NaN    8844  tt0113497   \n",
       "2                                                    NaN   15602  tt0113228   \n",
       "3                                                    NaN   31357  tt0114885   \n",
       "4                                                    NaN   11862  tt0113041   \n",
       "...                                                  ...     ...        ...   \n",
       "40224                                                NaN  315011  tt4262980   \n",
       "40503              http://www.thebeatlesliveproject.com/  391698  tt2531318   \n",
       "44821                  http://movies.warnerbros.com/pk3/   10991  tt0235679   \n",
       "44826  http://www.pokemon.com/us/movies/movie-pokemon...   12600  tt0287635   \n",
       "45265                                                NaN  265189  tt2121382   \n",
       "\n",
       "      original_language                                     original_title  \\\n",
       "0                    en                                          Toy Story   \n",
       "1                    en                                            Jumanji   \n",
       "2                    en                                   Grumpier Old Men   \n",
       "3                    en                                  Waiting to Exhale   \n",
       "4                    en                        Father of the Bride Part II   \n",
       "...                 ...                                                ...   \n",
       "40224                ja                                             シン・ゴジラ   \n",
       "40503                en  The Beatles: Eight Days a Week - The Touring Y...   \n",
       "44821                ja                               Pokémon 3: The Movie   \n",
       "44826                ja                     劇場版ポケットモンスター セレビィ 時を越えた遭遇（であい）   \n",
       "45265                sv                                             Turist   \n",
       "\n",
       "                                                overview  ...      revenue  \\\n",
       "0      Led by Woody, Andy's toys live happily in his ...  ...  373554033.0   \n",
       "1      When siblings Judy and Peter discover an encha...  ...  262797249.0   \n",
       "2      A family wedding reignites the ancient feud be...  ...          0.0   \n",
       "3      Cheated on, mistreated and stepped on, the wom...  ...   81452156.0   \n",
       "4      Just when George Banks has recovered from his ...  ...   76578911.0   \n",
       "...                                                  ...  ...          ...   \n",
       "40224  From the mind behind Evangelion comes a hit la...  ...   77000000.0   \n",
       "40503  The band stormed Europe in 1963, and, in 1964,...  ...          0.0   \n",
       "44821  When Molly Hale's sadness of her father's disa...  ...   68411275.0   \n",
       "44826  All your favorite Pokémon characters are back,...  ...   28023563.0   \n",
       "45265  While holidaying in the French Alps, a Swedish...  ...    1359497.0   \n",
       "\n",
       "      runtime                                   spoken_languages    status  \\\n",
       "0        81.0           [{'iso_639_1': 'en', 'name': 'English'}]  Released   \n",
       "1       104.0  [{'iso_639_1': 'en', 'name': 'English'}, {'iso...  Released   \n",
       "2       101.0           [{'iso_639_1': 'en', 'name': 'English'}]  Released   \n",
       "3       127.0           [{'iso_639_1': 'en', 'name': 'English'}]  Released   \n",
       "4       106.0           [{'iso_639_1': 'en', 'name': 'English'}]  Released   \n",
       "...       ...                                                ...       ...   \n",
       "40224   120.0  [{'iso_639_1': 'it', 'name': 'Italiano'}, {'is...  Released   \n",
       "40503    99.0           [{'iso_639_1': 'en', 'name': 'English'}]  Released   \n",
       "44821    93.0           [{'iso_639_1': 'en', 'name': 'English'}]  Released   \n",
       "44826    75.0               [{'iso_639_1': 'ja', 'name': '日本語'}]  Released   \n",
       "45265   118.0  [{'iso_639_1': 'fr', 'name': 'Français'}, {'is...  Released   \n",
       "\n",
       "                                                 tagline  \\\n",
       "0                                                    NaN   \n",
       "1              Roll the dice and unleash the excitement!   \n",
       "2      Still Yelling. Still Fighting. Still Ready for...   \n",
       "3      Friends are the people who let you be yourself...   \n",
       "4      Just When His World Is Back To Normal... He's ...   \n",
       "...                                                  ...   \n",
       "40224                    A god incarnate. A city doomed.   \n",
       "40503            The band you know. The story you don't.   \n",
       "44821                      Pokémon: Spell of the Unknown   \n",
       "44826                                                NaN   \n",
       "45265                                                NaN   \n",
       "\n",
       "                                                   title  video vote_average  \\\n",
       "0                                              Toy Story  False          7.7   \n",
       "1                                                Jumanji  False          6.9   \n",
       "2                                       Grumpier Old Men  False          6.5   \n",
       "3                                      Waiting to Exhale  False          6.1   \n",
       "4                            Father of the Bride Part II  False          5.7   \n",
       "...                                                  ...    ...          ...   \n",
       "40224                                      Shin Godzilla  False          6.6   \n",
       "40503  The Beatles: Eight Days a Week - The Touring Y...  False          7.6   \n",
       "44821                      Pokémon: Spell of the Unknown  False          6.0   \n",
       "44826        Pokémon 4Ever: Celebi - Voice of the Forest  False          5.7   \n",
       "45265                                      Force Majeure  False          6.8   \n",
       "\n",
       "      vote_count  year  \n",
       "0         5415.0  1995  \n",
       "1         2413.0  1995  \n",
       "2           92.0  1995  \n",
       "3           34.0  1995  \n",
       "4          173.0  1995  \n",
       "...          ...   ...  \n",
       "40224      152.0  2016  \n",
       "40503       92.0  2016  \n",
       "44821      144.0  2000  \n",
       "44826       82.0  2001  \n",
       "45265      255.0  2014  \n",
       "\n",
       "[9099 rows x 25 columns]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sub_meta = meta[meta['id'].isin(links_small)]\n",
    "sub_meta"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Movie description based recommender"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "sub_meta['tagline'] = sub_meta['tagline'].fillna('')\n",
    "sub_meta['description'] = sub_meta['overview'] + sub_meta['tagline']\n",
    "sub_meta['description'] = sub_meta['description'].fillna('')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "tf = TfidfVectorizer(analyzer='word',ngram_range=(1, 2),min_df=0, stop_words='english')\n",
    "tfidf_matrix = tf.fit_transform(sub_meta['description'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(9099, 268124)"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tfidf_matrix.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Here, I will be using cosine similarity to find the similarity between two movies. Cosine similarity calculates the numeric quantity that denoted the similarity between two movies.\n",
    "#### $cosine(x,y) = \\frac{x. y^\\intercal}{||x||.||y||} $\n",
    "\n",
    "#### Since we have used the TF-IDF Vectorizer, calculating the Dot Product will directly give us the Cosine Similarity Score. Therefore, we will use sklearn's **linear_kernel** instead of cosine_similarities since it is much faster.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "cosine_sim = linear_kernel(tfidf_matrix, tfidf_matrix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(9099, 9099)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cosine_sim.shape\n",
    "\n",
    "# We now have a pairwise cosine similarity matrix for all the movies in our dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "sub_meta = sub_meta.reset_index()\n",
    "titles = sub_meta['title']\n",
    "indices = pd.Series(sub_meta.index,index = sub_meta['title'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Recommending top 30 movies\n",
    "def get_recommendations(movie_title):\n",
    "    idx = indices[movie_title]\n",
    "    sim_scores = list(enumerate(cosine_sim[idx]))\n",
    "    sim_scores = sorted(sim_scores, key=lambda x: x[1], reverse=True)\n",
    "    sim_scores = sim_scores[1:31]\n",
    "    movie_indices = [i[0] for i in sim_scores]\n",
    "    return titles.iloc[movie_indices]    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "7931                      The Dark Knight Rises\n",
       "132                              Batman Forever\n",
       "1113                             Batman Returns\n",
       "8227    Batman: The Dark Knight Returns, Part 2\n",
       "7565                 Batman: Under the Red Hood\n",
       "524                                      Batman\n",
       "7901                           Batman: Year One\n",
       "2579               Batman: Mask of the Phantasm\n",
       "2696                                        JFK\n",
       "8165    Batman: The Dark Knight Returns, Part 1\n",
       "Name: title, dtype: object"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_recommendations('The Dark Knight').head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the above results for **The Dark Knight** we can see that our system is able to identlfy it as a series of Batman movies and recommends other batman movies as its top recommendations. This system is not upto the mark as it is not considering key factors like director, crew, cast and genre, which determines the rating and popularity of a movie.\n",
    "\n",
    "We can improve our system by using metadata than **Overview** and **Tagline**. In the next subsection, we will build a more sophisticated recommender that takes **genre**, **keywords**, **cast** and **crew** into consideration."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Metadata based Recommender\n",
    "\n",
    "To build our standard metadata based content recommender, we will need to merge our current dataset with the crew and the keyword datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "credits = pd.read_csv('Data/credits.csv')\n",
    "keywords = pd.read_csv('Data/keywords.csv')\n",
    "\n",
    "keywords['id'] = keywords['id'].astype('int')\n",
    "credits['id'] = credits['id'].astype('int')\n",
    "meta['id'] = meta['id'].astype('int')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(9219, 28)"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "meta = meta.merge(credits, on='id')\n",
    "meta = meta.merge(keywords, on='id')\n",
    "\n",
    "sub_meta = meta[meta['id'].isin(links_small)]\n",
    "sub_meta.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have our cast, crew, genres and credits, all in one dataframe. Let us wrangle this a little more using the following intuitions:\n",
    "\n",
    "1. **Crew:** From the crew, we will only pick the director as our feature since the others don't contribute that much to the *feel* of the movie.\n",
    "2. **Cast:** Choosing Cast is a little more tricky. Lesser known actors and minor roles do not really affect people's opinion of a movie. Therefore, we must only select the major characters and their respective actors. Arbitrarily we will choose the top 3 actors that appear in the credits list. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "sub_meta['cast'] = sub_meta['cast'].apply(literal_eval)\n",
    "sub_meta['crew'] = sub_meta['crew'].apply(literal_eval)\n",
    "sub_meta['keywords'] = sub_meta['keywords'].apply(literal_eval)\n",
    "sub_meta['cast_size'] = sub_meta['cast'].apply(lambda x: len(x))\n",
    "sub_meta['crew_size'] = sub_meta['crew'].apply(lambda x: len(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_director(x):\n",
    "    for i in x:\n",
    "        if i['job'] == 'Director':\n",
    "            return i['name']\n",
    "    return np.nan"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "sub_meta['director'] = sub_meta['crew'].apply(get_director)\n",
    "\n",
    "sub_meta['cast'] = sub_meta['cast'].apply(lambda x: [i['name'] for i in x] if isinstance(x, list) else [])\n",
    "sub_meta['cast'] = sub_meta['cast'].apply(lambda x: x[:3] if len(x) >=3 else x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "sub_meta['keywords'] = sub_meta['keywords'].apply(lambda x: [i['name'] for i in x] if isinstance(x, list) else [])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "sub_meta['cast'] = sub_meta['cast'].apply(lambda x: [str.lower(i.replace(\" \", \"\")) for i in x])\n",
    "\n",
    "sub_meta['director'] = sub_meta['director'].astype('str').apply(lambda x: str.lower(x.replace(\" \", \"\")))\n",
    "sub_meta['director'] = sub_meta['director'].apply(lambda x: [x,x, x])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "independent film        610\n",
       "woman director          550\n",
       "murder                  399\n",
       "duringcreditsstinger    327\n",
       "based on novel          318\n",
       "                       ... \n",
       "boarder                   2\n",
       "social climbing           2\n",
       "covert operation          2\n",
       "prisoners                 2\n",
       "crystal                   2\n",
       "Name: keyword, Length: 6709, dtype: int64"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Preprocessing of keywords\n",
    "\n",
    "key = sub_meta.apply(lambda x: pd.Series(x['keywords']),axis=1).stack().reset_index(level=1, drop=True)\n",
    "key.name = 'keyword'\n",
    "key = key.value_counts()\n",
    "\n",
    "# Only taking keywords into considerations whose count is more than 1.\n",
    "key = key[key > 1]\n",
    "key"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "def filter_keywords(x):\n",
    "    words = []\n",
    "    for i in x:\n",
    "        if i in key:\n",
    "            words.append(i)\n",
    "    return words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "stemmer = SnowballStemmer('english')\n",
    "\n",
    "sub_meta['keywords'] = sub_meta['keywords'].apply(filter_keywords)\n",
    "sub_meta['keywords'] = sub_meta['keywords'].apply(lambda x: [stemmer.stem(i) for i in x])\n",
    "sub_meta['keywords'] = sub_meta['keywords'].apply(lambda x: [str.lower(i.replace(\" \", \"\")) for i in x])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "sub_meta['soup'] = sub_meta['keywords'] + sub_meta['cast'] + sub_meta['director'] + sub_meta['genres']\n",
    "sub_meta['soup'] = sub_meta['soup'].apply(lambda x: ' '.join(x))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "count = CountVectorizer(analyzer='word',ngram_range=(1, 2),min_df=0, stop_words='english')\n",
    "count_matrix = count.fit_transform(sub_meta['soup'])\n",
    "\n",
    "cosine_sim = cosine_similarity(count_matrix, count_matrix)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "sub_meta = sub_meta.reset_index()\n",
    "titles = sub_meta['title']\n",
    "indices = pd.Series(sub_meta.index, index=sub_meta['title'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "8031         The Dark Knight Rises\n",
       "6218                 Batman Begins\n",
       "6623                  The Prestige\n",
       "2085                     Following\n",
       "7648                     Inception\n",
       "4145                      Insomnia\n",
       "3381                       Memento\n",
       "8613                  Interstellar\n",
       "7659    Batman: Under the Red Hood\n",
       "1134                Batman Returns\n",
       "Name: title, dtype: object"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_recommendations('The Dark Knight').head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The recommendations seem to have recognized other Christopher Nolan movies (due to the high weightage given to director) and put them as top recommendations. I enjoyed watching **The Dark Knight** as well as some of the other ones in the list including **Batman Begins**, **The Prestige** and **The Dark Knight Rises**. \n",
    "\n",
    "We can of course experiment on this engine by trying out different weights for our features (directors, actors, genres), limiting the number of keywords that can be used in the soup, weighing genres based on their frequency, only showing movies with the same languages, etc."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Popularity and Ratings\n",
    "\n",
    "One thing that we notice about our recommendation system is that it recommends movies regardless of ratings and popularity. It is true that **Batman and Robin** has a lot of similar characters as compared to **The Dark Knight** but it was a terrible movie that shouldn't be recommended to anyone.\n",
    "\n",
    "Therefore, we will add a mechanism to remove bad movies and return movies which are popular and have had a good critical response.\n",
    "\n",
    "I will take the top 25 movies based on similarity scores and calculate the vote of the 60th percentile movie. Then, using this as the value of $m$, we will calculate the weighted rating of each movie using IMDB's formula like we did in the Simple Recommender section."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "def weighted_rating(df,min_votes,avg_rating):\n",
    "    votes = df['vote_count']\n",
    "    rating = df['vote_average']\n",
    "    return (votes/(votes+min_votes)*rating) + (min_votes/(min_votes+votes)*avg_rating)\n",
    "\n",
    "def improved_recommendations(title):\n",
    "    idx = indices[title]\n",
    "    sim_scores = list(enumerate(cosine_sim[idx]))\n",
    "    sim_scores = sorted(sim_scores, key=lambda x: x[1], reverse=True)\n",
    "    sim_scores = sim_scores[1:26]\n",
    "    movie_indices = [i[0] for i in sim_scores]\n",
    "    \n",
    "    movies = sub_meta.iloc[movie_indices][['title', 'vote_count', 'vote_average', 'year']]\n",
    "    vote_counts = movies[movies['vote_count'].notnull()]['vote_count'].astype('int')\n",
    "    vote_averages = movies[movies['vote_average'].notnull()]['vote_average'].astype('int')\n",
    "    \n",
    "    avg_rating = vote_averages.mean()\n",
    "    min_votes = vote_counts.quantile(0.60)\n",
    "    \n",
    "    qualified = movies[(movies['vote_count'] >= min_votes) & (movies['vote_count'].notnull()) & (movies['vote_average'].notnull())]\n",
    "    qualified['vote_count'] = qualified['vote_count'].astype('int')\n",
    "    qualified['vote_average'] = qualified['vote_average'].astype('int')\n",
    "    qualified['wr'] = qualified.apply(weighted_rating,args=(min_votes,avg_rating), axis=1)\n",
    "    qualified = qualified.sort_values('wr', ascending=False).head(10)\n",
    "    return qualified"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>vote_count</th>\n",
       "      <th>vote_average</th>\n",
       "      <th>year</th>\n",
       "      <th>wr</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>7648</th>\n",
       "      <td>Inception</td>\n",
       "      <td>14075</td>\n",
       "      <td>8</td>\n",
       "      <td>2010</td>\n",
       "      <td>7.872621</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8613</th>\n",
       "      <td>Interstellar</td>\n",
       "      <td>11187</td>\n",
       "      <td>8</td>\n",
       "      <td>2014</td>\n",
       "      <td>7.843131</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6623</th>\n",
       "      <td>The Prestige</td>\n",
       "      <td>4510</td>\n",
       "      <td>8</td>\n",
       "      <td>2006</td>\n",
       "      <td>7.662461</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3381</th>\n",
       "      <td>Memento</td>\n",
       "      <td>4168</td>\n",
       "      <td>8</td>\n",
       "      <td>2000</td>\n",
       "      <td>7.641301</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8031</th>\n",
       "      <td>The Dark Knight Rises</td>\n",
       "      <td>9263</td>\n",
       "      <td>7</td>\n",
       "      <td>2012</td>\n",
       "      <td>6.936548</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6218</th>\n",
       "      <td>Batman Begins</td>\n",
       "      <td>7511</td>\n",
       "      <td>7</td>\n",
       "      <td>2005</td>\n",
       "      <td>6.923913</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1134</th>\n",
       "      <td>Batman Returns</td>\n",
       "      <td>1706</td>\n",
       "      <td>6</td>\n",
       "      <td>1992</td>\n",
       "      <td>6.206438</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>132</th>\n",
       "      <td>Batman Forever</td>\n",
       "      <td>1529</td>\n",
       "      <td>5</td>\n",
       "      <td>1995</td>\n",
       "      <td>5.676520</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9024</th>\n",
       "      <td>Batman v Superman: Dawn of Justice</td>\n",
       "      <td>7189</td>\n",
       "      <td>5</td>\n",
       "      <td>2016</td>\n",
       "      <td>5.224783</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1260</th>\n",
       "      <td>Batman &amp; Robin</td>\n",
       "      <td>1447</td>\n",
       "      <td>4</td>\n",
       "      <td>1997</td>\n",
       "      <td>5.167624</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                   title  vote_count  vote_average  year  \\\n",
       "7648                           Inception       14075             8  2010   \n",
       "8613                        Interstellar       11187             8  2014   \n",
       "6623                        The Prestige        4510             8  2006   \n",
       "3381                             Memento        4168             8  2000   \n",
       "8031               The Dark Knight Rises        9263             7  2012   \n",
       "6218                       Batman Begins        7511             7  2005   \n",
       "1134                      Batman Returns        1706             6  1992   \n",
       "132                       Batman Forever        1529             5  1995   \n",
       "9024  Batman v Superman: Dawn of Justice        7189             5  2016   \n",
       "1260                      Batman & Robin        1447             4  1997   \n",
       "\n",
       "            wr  \n",
       "7648  7.872621  \n",
       "8613  7.843131  \n",
       "6623  7.662461  \n",
       "3381  7.641301  \n",
       "8031  6.936548  \n",
       "6218  6.923913  \n",
       "1134  6.206438  \n",
       "132   5.676520  \n",
       "9024  5.224783  \n",
       "1260  5.167624  "
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "improved_recommendations('The Dark Knight')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Collaborative Filtering\n",
    "\n",
    "Our content based engine suffers from some severe limitations. It is only capable of suggesting movies which are *close* to a certain movie. That is, it is not capable of capturing tastes and providing recommendations across genres.\n",
    "\n",
    "Also, the engine that we built is not really personal in that it doesn't capture the personal tastes and biases of a user. Anyone querying our engine for recommendations based on a movie will receive the same recommendations for that movie, regardless of who s/he is.\n",
    "\n",
    "Therefore, in this section, we will use a technique called **Collaborative Filtering** to make recommendations to Movie Watchers. Collaborative Filtering is based on the idea that users similar to a me can be used to predict how much I will like a particular product or service those users have used/experienced but I have not.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "reader = Reader()\n",
    "ratings = pd.read_csv('Data/ratings_small.csv')\n",
    "data = Dataset.load_from_df(ratings[['userId', 'movieId', 'rating']], reader)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'test_rmse': array([0.89348242, 0.89081121, 0.89862841, 0.89207053, 0.90270164]),\n",
       " 'test_mae': array([0.68783566, 0.68535607, 0.69040736, 0.68736954, 0.69724592]),\n",
       " 'fit_time': (3.9010720252990723,\n",
       "  3.8928470611572266,\n",
       "  3.906195878982544,\n",
       "  3.9486989974975586,\n",
       "  3.9032769203186035),\n",
       " 'test_time': (0.21734213829040527,\n",
       "  0.10002994537353516,\n",
       "  0.09884381294250488,\n",
       "  0.2146930694580078,\n",
       "  0.10783529281616211)}"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "svd = SVD()\n",
    "cross_validate(svd, data, measures=['RMSE','MAE'], cv=5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<surprise.prediction_algorithms.matrix_factorization.SVD at 0x7fb620a6b0d0>"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "trainset = data.build_full_trainset()\n",
    "svd.fit(trainset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>movieId</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>7987</th>\n",
       "      <td>50</td>\n",
       "      <td>10</td>\n",
       "      <td>4.0</td>\n",
       "      <td>847412607</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7988</th>\n",
       "      <td>50</td>\n",
       "      <td>21</td>\n",
       "      <td>3.0</td>\n",
       "      <td>847412676</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7989</th>\n",
       "      <td>50</td>\n",
       "      <td>39</td>\n",
       "      <td>2.0</td>\n",
       "      <td>847412692</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7990</th>\n",
       "      <td>50</td>\n",
       "      <td>47</td>\n",
       "      <td>3.0</td>\n",
       "      <td>847412643</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7991</th>\n",
       "      <td>50</td>\n",
       "      <td>95</td>\n",
       "      <td>3.0</td>\n",
       "      <td>847412837</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      userId  movieId  rating  timestamp\n",
       "7987      50       10     4.0  847412607\n",
       "7988      50       21     3.0  847412676\n",
       "7989      50       39     2.0  847412692\n",
       "7990      50       47     3.0  847412643\n",
       "7991      50       95     3.0  847412837"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratings[ratings['userId'] == 50].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Prediction(uid=1, iid=302, r_ui=3, est=2.803362475666917, details={'was_impossible': False})"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "svd.predict(1, 302, 3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For movie with ID 302, we get an estimated prediction of **2.686**. One startling feature of this recommender system is that it doesn't care what the movie is (or what it contains). It works purely on the basis of an assigned movie ID and tries to predict ratings based on how the other users have predicted the movie."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "_cell_guid": "248ec145-20e4-429c-8718-5c20834be1be",
    "_uuid": "b6fe882ef8417e28bb024306a3879dc67affb7c6"
   },
   "source": [
    "## Hybrid Recommender\n",
    "\n",
    "In this section, I will try to build a simple hybrid recommender that brings together techniques we have implemented in the content based and collaborative filter based engines. This is how it will work:\n",
    "\n",
    "* **Input:** User ID and the Title of a Movie\n",
    "* **Output:** Similar movies sorted on the basis of expected ratings by that particular user."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "def convert_int(x):\n",
    "    try:\n",
    "        return int(x)\n",
    "    except:\n",
    "        return np.nan"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "id_map = pd.read_csv('Data/links_small.csv')[['movieId', 'tmdbId']]\n",
    "id_map['tmdbId'] = id_map['tmdbId'].apply(convert_int)\n",
    "id_map.columns = ['movieId', 'id']\n",
    "id_map = id_map.merge(sub_meta[['title', 'id']], on='id').set_index('title')\n",
    "\n",
    "indices_map = id_map.set_index('id')\n",
    "\n",
    "def hybrid(userId, title):\n",
    "    idx = indices[title]\n",
    "    tmdbId = id_map.loc[title]['id']\n",
    "    movie_id = id_map.loc[title]['movieId']\n",
    "    \n",
    "    sim_scores = list(enumerate(cosine_sim[int(idx)]))\n",
    "    sim_scores = sorted(sim_scores, key=lambda x: x[1], reverse=True)\n",
    "    sim_scores = sim_scores[1:26]\n",
    "    movie_indices = [i[0] for i in sim_scores]\n",
    "    \n",
    "    movies = sub_meta.iloc[movie_indices][['title', 'vote_count', 'vote_average', 'year', 'id']]\n",
    "    movies['est'] = movies['id'].apply(lambda x: svd.predict(userId, indices_map.loc[x]['movieId']).est)\n",
    "    movies = movies.sort_values('est', ascending=False)\n",
    "    return movies.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>vote_count</th>\n",
       "      <th>vote_average</th>\n",
       "      <th>year</th>\n",
       "      <th>id</th>\n",
       "      <th>est</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>8658</th>\n",
       "      <td>X-Men: Days of Future Past</td>\n",
       "      <td>6155.0</td>\n",
       "      <td>7.5</td>\n",
       "      <td>2014</td>\n",
       "      <td>127585</td>\n",
       "      <td>3.171352</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>522</th>\n",
       "      <td>Terminator 2: Judgment Day</td>\n",
       "      <td>4274.0</td>\n",
       "      <td>7.7</td>\n",
       "      <td>1991</td>\n",
       "      <td>280</td>\n",
       "      <td>3.168131</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>974</th>\n",
       "      <td>Aliens</td>\n",
       "      <td>3282.0</td>\n",
       "      <td>7.7</td>\n",
       "      <td>1986</td>\n",
       "      <td>679</td>\n",
       "      <td>3.122791</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1011</th>\n",
       "      <td>The Terminator</td>\n",
       "      <td>4208.0</td>\n",
       "      <td>7.4</td>\n",
       "      <td>1984</td>\n",
       "      <td>218</td>\n",
       "      <td>2.989583</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8401</th>\n",
       "      <td>Star Trek Into Darkness</td>\n",
       "      <td>4479.0</td>\n",
       "      <td>7.4</td>\n",
       "      <td>2013</td>\n",
       "      <td>54138</td>\n",
       "      <td>2.969383</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1621</th>\n",
       "      <td>Darby O'Gill and the Little People</td>\n",
       "      <td>35.0</td>\n",
       "      <td>6.7</td>\n",
       "      <td>1959</td>\n",
       "      <td>18887</td>\n",
       "      <td>2.897440</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2014</th>\n",
       "      <td>Fantastic Planet</td>\n",
       "      <td>140.0</td>\n",
       "      <td>7.6</td>\n",
       "      <td>1973</td>\n",
       "      <td>16306</td>\n",
       "      <td>2.843647</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>922</th>\n",
       "      <td>The Abyss</td>\n",
       "      <td>822.0</td>\n",
       "      <td>7.1</td>\n",
       "      <td>1989</td>\n",
       "      <td>2756</td>\n",
       "      <td>2.754509</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1668</th>\n",
       "      <td>Return from Witch Mountain</td>\n",
       "      <td>38.0</td>\n",
       "      <td>5.6</td>\n",
       "      <td>1978</td>\n",
       "      <td>14822</td>\n",
       "      <td>2.726167</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>344</th>\n",
       "      <td>True Lies</td>\n",
       "      <td>1138.0</td>\n",
       "      <td>6.8</td>\n",
       "      <td>1994</td>\n",
       "      <td>36955</td>\n",
       "      <td>2.716496</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                   title  vote_count  vote_average  year  \\\n",
       "8658          X-Men: Days of Future Past      6155.0           7.5  2014   \n",
       "522           Terminator 2: Judgment Day      4274.0           7.7  1991   \n",
       "974                               Aliens      3282.0           7.7  1986   \n",
       "1011                      The Terminator      4208.0           7.4  1984   \n",
       "8401             Star Trek Into Darkness      4479.0           7.4  2013   \n",
       "1621  Darby O'Gill and the Little People        35.0           6.7  1959   \n",
       "2014                    Fantastic Planet       140.0           7.6  1973   \n",
       "922                            The Abyss       822.0           7.1  1989   \n",
       "1668          Return from Witch Mountain        38.0           5.6  1978   \n",
       "344                            True Lies      1138.0           6.8  1994   \n",
       "\n",
       "          id       est  \n",
       "8658  127585  3.171352  \n",
       "522      280  3.168131  \n",
       "974      679  3.122791  \n",
       "1011     218  2.989583  \n",
       "8401   54138  2.969383  \n",
       "1621   18887  2.897440  \n",
       "2014   16306  2.843647  \n",
       "922     2756  2.754509  \n",
       "1668   14822  2.726167  \n",
       "344    36955  2.716496  "
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hybrid(1, 'Avatar')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that for our hybrid recommender, we get different recommendations for different users although the movie is the same. Hence, our recommendations are more personalized and tailored towards particular users."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "In this notebook, I have built 4 different recommendation engines based on different ideas and algorithms. They are as follows:\n",
    "\n",
    "1. **Simple Recommender:** This system used overall TMDB Vote Count and Vote Averages to build Top Movies Charts, in general and for a specific genre. The IMDB Weighted Rating System was used to calculate ratings on which the sorting was finally performed.\n",
    "2. **Content Based Recommender:** We built two content based engines; one that took movie overview and taglines as input and the other which took metadata such as cast, crew, genre and keywords to come up with predictions. We also deviced a simple filter to give greater preference to movies with more votes and higher ratings.\n",
    "3. **Collaborative Filtering:** We used the powerful Surprise Library to build a collaborative filter based on single value decomposition. The RMSE obtained was less than 1 and the engine gave estimated ratings for a given user and movie.\n",
    "4. **Hybrid Engine:** We brought together ideas from content and collaborative filterting to build an engine that gave movie suggestions to a particular user based on the estimated ratings that it had internally calculated for that user."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": []
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"
  },
  "kernelspec": {
   "display_name": "Python 3.8.9 64-bit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}