examples/02_model_collaborative_filtering/lightgcn_deep_dive.ipynb

{
    "cells": [
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "<i>Copyright (c) Recommenders contributors.</i>\n",
                "\n",
                "<i>Licensed under the MIT License.</i>"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "# LightGCN - simplified GCN model for recommendation\n",
                "\n",
                "This notebook serves as an introduction to LightGCN [1], which is an simple, linear and neat Graph Convolution Network (GCN) [3] model for recommendation."
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## 0 Global Settings and Imports"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 1,
            "metadata": {},
            "outputs": [
                {
                    "name": "stderr",
                    "output_type": "stream",
                    "text": [
                        "/anaconda/envs/tf2/lib/python3.7/site-packages/papermill/iorw.py:50: FutureWarning: pyarrow.HadoopFileSystem is deprecated as of 2.0.0, please use pyarrow.fs.HadoopFileSystem instead.\n",
                        "  from pyarrow import HadoopFileSystem\n"
                    ]
                },
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "System version: 3.7.11 (default, Jul 27 2021, 14:32:16) \n",
                        "[GCC 7.5.0]\n",
                        "Pandas version: 1.3.4\n",
                        "Tensorflow version: 2.6.1\n"
                    ]
                }
            ],
            "source": [
                "import sys\n",
                "import os\n",
                "import pandas as pd\n",
                "import numpy as np\n",
                "import tensorflow as tf\n",
                "tf.get_logger().setLevel('ERROR') # only show error messages\n",
                "\n",
                "from recommenders.utils.timer import Timer\n",
                "from recommenders.models.deeprec.models.graphrec.lightgcn import LightGCN\n",
                "from recommenders.models.deeprec.DataModel.ImplicitCF import ImplicitCF\n",
                "from recommenders.datasets import movielens\n",
                "from recommenders.datasets.python_splitters import python_stratified_split\n",
                "from recommenders.evaluation.python_evaluation import map, ndcg_at_k, precision_at_k, recall_at_k\n",
                "from recommenders.utils.constants import SEED as DEFAULT_SEED\n",
                "from recommenders.models.deeprec.deeprec_utils import prepare_hparams\n",
                "from recommenders.utils.notebook_utils import store_metadata\n",
                "\n",
                "print(f\"System version: {sys.version}\")\n",
                "print(f\"Pandas version: {pd.__version__}\")\n",
                "print(f\"Tensorflow version: {tf.__version__}\")"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 2,
            "metadata": {
                "tags": [
                    "parameters"
                ]
            },
            "outputs": [],
            "source": [
                "# top k items to recommend\n",
                "TOP_K = 10\n",
                "\n",
                "# Select MovieLens data size: 100k, 1m, 10m, or 20m\n",
                "MOVIELENS_DATA_SIZE = '100k'\n",
                "\n",
                "# Model parameters\n",
                "EPOCHS = 50\n",
                "BATCH_SIZE = 1024\n",
                "\n",
                "SEED = DEFAULT_SEED  # Set None for non-deterministic results\n",
                "\n",
                "yaml_file = \"../../recommenders/models/deeprec/config/lightgcn.yaml\"\n",
                "user_file = \"../../tests/resources/deeprec/lightgcn/user_embeddings.csv\"\n",
                "item_file = \"../../tests/resources/deeprec/lightgcn/item_embeddings.csv\""
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## 1 LightGCN model\n",
                "\n",
                "LightGCN is a simplified version of Neural Graph Collaborative Filtering (NGCF) [4], which adapts GCNs in recommendation systems.\n",
                "\n",
                "### 1.1 Graph Networks in Recommendation Systems\n",
                "\n",
                "GCN are networks that can learn patterns in graph data. They can be applied in many fields, but they are particularly well suited for Recommendation Systems, because of their ability to encode relationships.\n",
                "\n",
                "In traditional models like matrix factorization [5], user and items are represented as embeddings. And the interaction, which is the signal that encodes the behavior, is not part of the embeddings, but it is represented in the loss function, typically as a dot product. \n",
                "\n",
                "Despite their effectiveness, some authors [1,4] argue that these methods are not sufficient to yield satisfactory embeddings for collaborative filtering. The key reason is that the embedding function lacks an explicit encoding of the crucial collaborative signal, which is latent in user-item interactions to reveal the behavioral similarity between users (or items). \n",
                "\n",
                "**GCNs can be used to encode the interaction signal in the embeddings**. Interacted items can be seen as user´s features, because they provide direct evidence on a user’s preference. Similarly, the users that consume an item can be treated as the item’s features and used to measure the collaborative similarity of two items. A natural way to incorporate the interaction signal in the embedding is by exploiting the high-order connectivity from user-item interactions.\n",
                "\n",
                "In the figure below, the user-item interaction is shown (to the left) as well as the concept of higher-order connectivity (to the right).\n",
                "\n",
                "<img src=\"https://recodatasets.z20.web.core.windows.net/images/High_order_connectivity.png\" width=500 style=\"display:block; margin-left:auto; margin-right:auto;\">\n",
                "\n",
                "The high-order connectivity shows the collaborative signal in a graph form. For example, the path $u_1 ← i_2 ← u2$ indicates the behavior\n",
                "similarity between $u_1$ and $u_2$, as both users have interacted with $i_2$; the longer path $u_1 ← i_2 ← u_2 ← i_4$ suggests that $u_1$ is likely to adopt $i_4$, since her similar user $u_2$ has consumed $i_4$ before. Moreover, from the holistic view of $l = 3$, item $i_4$ is more likely to be of interest to $u_1$ than item $i_5$, since there are two paths connecting $(i_4,u_1)$, while only one path connects $(i_5,u_1)$.\n",
                "\n",
                "Based on this high-order connectivity, NGCF [4] defines an embedding propagation layer, which refines a user’s (or an item’s) embedding by aggregating the embeddings of the interacted items (or users). By stacking multiple embedding propagation layers, we can enforce the embeddings\n",
                "to capture the collaborative signal in high-order connectivities.\n",
                "\n",
                "More formally, let $\\mathbf{e}_{u}^{0}$ denote the original embedding of user $u$ and $\\mathbf{e}_{i}^{0}$ denote the original embedding of item $i$. The embedding propagation can be computed recursively as:\n",
                "\n",
                "$$\n",
                "\\begin{array}{l}\n",
                "\\mathbf{e}_{u}^{(k+1)}=\\sigma\\bigl( \\mathbf{W}_{1}\\mathbf{e}_{u}^{(k)} + \\sum_{i \\in \\mathcal{N}_{u}} \\frac{1}{\\sqrt{\\left|\\mathcal{N}_{u}\\right|} \\sqrt{\\left|\\mathcal{N}_{i}\\right|}} (\\mathbf{W}_{1}\\mathbf{e}_{i}^{(k)} + \\mathbf{W}_{2}(\\mathbf{e}_{i}^{(k)}\\cdot\\mathbf{e}_{u}^{(k)}) ) \\bigr)\n",
                "\\\\\n",
                "\\mathbf{e}_{i}^{(k+1)}=\\sigma\\bigl( \\mathbf{W}_{1}\\mathbf{e}_{i}^{(k)} +\\sum_{u \\in \\mathcal{N}_{i}} \\frac{1}{\\sqrt{\\left|\\mathcal{N}_{i}\\right|} \\sqrt{\\left|\\mathcal{N}_{u}\\right|}} (\\mathbf{W}_{1}\\mathbf{e}_{u}^{(k)} + \\mathbf{W}_{2}(\\mathbf{e}_{u}^{(k)}\\cdot\\mathbf{e}_{i}^{(k)}) ) \\bigr)\n",
                "\\end{array}\n",
                "$$\n",
                "\n",
                "where $\\mathbf{W}_{1}$ and $\\mathbf{W}_{2}$ are trainable weight matrices, $\\frac{1}{\\sqrt{\\left|\\mathcal{N}_{i}\\right|} \\sqrt{\\left|\\mathcal{N}_{u}\\right|}}$ is a discount factor expressed as the graph Laplacian norm, $\\mathcal{N}_{u}$ and $\\mathcal{N}_{i}$ denote the first-hop neighbors of user $u$ and item $i$, and $\\sigma$ is a non-linearity that in the paper is set as a LeakyReLU. \n",
                "\n",
                "To obtain the final representation, each propagated embedding is concatenated (i.e., $\\mathbf{e}_{u}^{(*)}=\\mathbf{e}_{u}^{(0)}||...||\\mathbf{e}_{u}^{(l)}$), and then the final user's preference over an item is computed as a dot product: $\\hat y_{u i} = \\mathbf{e}_{u}^{(*)T}\\mathbf{e}_{i}^{(*)}$.\n",
                "\n",
                "### 1.2 LightGCN architecture\n",
                "\n",
                "LightGCN is a simplified version of NGCF [4] to make it more concise and appropriate for recommendations. The model architecture is illustrated below.\n",
                "\n",
                "<img src=\"https://recodatasets.z20.web.core.windows.net/images/lightGCN-model.jpg\" width=600 style=\"display:block; margin-left:auto; margin-right:auto;\">\n",
                "\n",
                "In Light Graph Convolution, only the normalized sum of neighbor embeddings is performed towards next layer; other operations like self-connection, feature transformation via weight matrices, and nonlinear activation are all removed, which largely simplifies NGCF. In the layer combination step, instead of concatenating the embeddings, we sum over the embeddings at each layer to obtain the final representations.\n",
                "\n",
                "### 1.3 Light Graph Convolution (LGC)\n",
                "\n",
                "In LightGCN, we adopt the simple weighted sum aggregator and abandon the use of feature transformation and nonlinear activation. The graph convolution operation in LightGCN is defined as:\n",
                "\n",
                "$$\n",
                "\\begin{array}{l}\n",
                "\\mathbf{e}_{u}^{(k+1)}=\\sum_{i \\in \\mathcal{N}_{u}} \\frac{1}{\\sqrt{\\left|\\mathcal{N}_{u}\\right|} \\sqrt{\\left|\\mathcal{N}_{i}\\right|}} \\mathbf{e}_{i}^{(k)} \\\\\n",
                "\\mathbf{e}_{i}^{(k+1)}=\\sum_{u \\in \\mathcal{N}_{i}} \\frac{1}{\\sqrt{\\left|\\mathcal{N}_{i}\\right|} \\sqrt{\\left|\\mathcal{N}_{u}\\right|}} \\mathbf{e}_{u}^{(k)}\n",
                "\\end{array}\n",
                "$$\n",
                "\n",
                "The symmetric normalization term $\\frac{1}{\\sqrt{\\left|\\mathcal{N}_{u}\\right|} \\sqrt{\\left|\\mathcal{N}_{i}\\right|}}$ follows the design of standard GCN, which can avoid the scale of embeddings increasing with graph convolution operations.\n",
                "\n",
                "\n",
                "### 1.4 Layer Combination and Model Prediction\n",
                "\n",
                "In LightGCN, the only trainable model parameters are the embeddings at the 0-th layer, i.e., $\\mathbf{e}_{u}^{(0)}$ for all users and $\\mathbf{e}_{i}^{(0)}$ for all items. When they are given, the embeddings at higher layers can be computed via LGC. After $K$ layers LGC, we further combine the embeddings obtained at each layer to form the final representation of a user (an item):\n",
                "\n",
                "$$\n",
                "\\mathbf{e}_{u}=\\sum_{k=0}^{K} \\alpha_{k} \\mathbf{e}_{u}^{(k)} ; \\quad \\mathbf{e}_{i}=\\sum_{k=0}^{K} \\alpha_{k} \\mathbf{e}_{i}^{(k)}\n",
                "$$\n",
                "\n",
                "where $\\alpha_{k} \\geq 0$ denotes the importance of the $k$-th layer embedding in constituting the final embedding. In our experiments, we set $\\alpha_{k}$ uniformly as $1 / (K+1)$.\n",
                "\n",
                "The model prediction is defined as the inner product of user and item final representations:\n",
                "\n",
                "$$\n",
                "\\hat{y}_{u i}=\\mathbf{e}_{u}^{T} \\mathbf{e}_{i}\n",
                "$$\n",
                "\n",
                "which is used as the ranking score for recommendation generation.\n",
                "\n",
                "\n",
                "### 1.5 Matrix Form\n",
                "\n",
                "Let the user-item interaction matrix be $\\mathbf{R} \\in \\mathbb{R}^{M \\times N}$ where $M$ and $N$ denote the number of users and items, respectively, and each entry $R_{ui}$ is 1 if $u$ has interacted with item $i$ otherwise 0. We then obtain the adjacency matrix of the user-item graph as\n",
                "\n",
                "$$\n",
                "\\mathbf{A}=\\left(\\begin{array}{cc}\n",
                "\\mathbf{0} & \\mathbf{R} \\\\\n",
                "\\mathbf{R}^{T} & \\mathbf{0}\n",
                "\\end{array}\\right)\n",
                "$$\n",
                "\n",
                "Let the 0-th layer embedding matrix be $\\mathbf{E}^{(0)} \\in \\mathbb{R}^{(M+N) \\times T}$, where $T$ is the embedding size. Then we can obtain the matrix equivalent form of LGC as:\n",
                "\n",
                "$$\n",
                "\\mathbf{E}^{(k+1)}=\\left(\\mathbf{D}^{-\\frac{1}{2}} \\mathbf{A} \\mathbf{D}^{-\\frac{1}{2}}\\right) \\mathbf{E}^{(k)}\n",
                "$$\n",
                "\n",
                "where $\\mathbf{D}$ is a $(M+N) \\times(M+N)$ diagonal matrix, in which each entry $D_{ii}$ denotes the number of nonzero entries in the $i$-th row vector of the adjacency matrix $\\mathbf{A}$ (also named as degree matrix). Lastly, we get the final embedding matrix used for model prediction as:\n",
                "\n",
                "$$\n",
                "\\begin{aligned}\n",
                "\\mathbf{E} &=\\alpha_{0} \\mathbf{E}^{(0)}+\\alpha_{1} \\mathbf{E}^{(1)}+\\alpha_{2} \\mathbf{E}^{(2)}+\\ldots+\\alpha_{K} \\mathbf{E}^{(K)} \\\\\n",
                "&=\\alpha_{0} \\mathbf{E}^{(0)}+\\alpha_{1} \\tilde{\\mathbf{A}} \\mathbf{E}^{(0)}+\\alpha_{2} \\tilde{\\mathbf{A}}^{2} \\mathbf{E}^{(0)}+\\ldots+\\alpha_{K} \\tilde{\\mathbf{A}}^{K} \\mathbf{E}^{(0)}\n",
                "\\end{aligned}\n",
                "$$\n",
                "\n",
                "where $\\tilde{\\mathbf{A}}=\\mathbf{D}^{-\\frac{1}{2}} \\mathbf{A} \\mathbf{D}^{-\\frac{1}{2}}$ is the symmetrically normalized matrix.\n",
                "\n",
                "### 1.6 Model Training\n",
                "\n",
                "We employ the Bayesian Personalized Ranking (BPR) loss which is a pairwise loss that encourages the prediction of an observed entry to be higher than its unobserved counterparts:\n",
                "\n",
                "$$\n",
                "L_{B P R}=-\\sum_{u=1}^{M} \\sum_{i \\in \\mathcal{N}_{u}} \\sum_{j \\notin \\mathcal{N}_{u}} \\ln \\sigma\\left(\\hat{y}_{u i}-\\hat{y}_{u j}\\right)+\\lambda\\left\\|\\mathbf{E}^{(0)}\\right\\|^{2}\n",
                "$$\n",
                "\n",
                "Where $\\lambda$ controls the $L_2$ regularization strength. We employ the Adam optimizer and use it in a mini-batch manner.\n"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## 2 TensorFlow implementation of LightGCN with MovieLens dataset\n",
                "\n",
                "We will use the MovieLens dataset, which is composed of integer ratings from 1 to 5.\n",
                "\n",
                "We convert MovieLens into implicit feedback for model training and evaluation.\n",
                "\n",
                "### 2.1 Load and split data\n",
                "\n",
                "We split the full dataset into a `train` and `test` dataset to evaluate performance of the algorithm against a held-out set not seen during training. Because SAR generates recommendations based on user preferences, all users that are in the test set must also exist in the training set. For this case, we can use the provided `python_stratified_split` function which holds out a percentage (in this case 25%) of items from each user, but ensures all users are in both `train` and `test` datasets. Other options are available in the `dataset.python_splitters` module which provide more control over how the split occurs."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 3,
            "metadata": {},
            "outputs": [
                {
                    "name": "stderr",
                    "output_type": "stream",
                    "text": [
                        "100%|██████████| 4.81k/4.81k [00:01<00:00, 4.50kKB/s]\n"
                    ]
                },
                {
                    "data": {
                        "text/html": [
                            "<div>\n",
                            "<style scoped>\n",
                            "    .dataframe tbody tr th:only-of-type {\n",
                            "        vertical-align: middle;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe tbody tr th {\n",
                            "        vertical-align: top;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe thead th {\n",
                            "        text-align: right;\n",
                            "    }\n",
                            "</style>\n",
                            "<table border=\"1\" class=\"dataframe\">\n",
                            "  <thead>\n",
                            "    <tr style=\"text-align: right;\">\n",
                            "      <th></th>\n",
                            "      <th>userID</th>\n",
                            "      <th>itemID</th>\n",
                            "      <th>rating</th>\n",
                            "      <th>timestamp</th>\n",
                            "    </tr>\n",
                            "  </thead>\n",
                            "  <tbody>\n",
                            "    <tr>\n",
                            "      <th>0</th>\n",
                            "      <td>196</td>\n",
                            "      <td>242</td>\n",
                            "      <td>3.0</td>\n",
                            "      <td>881250949</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>1</th>\n",
                            "      <td>186</td>\n",
                            "      <td>302</td>\n",
                            "      <td>3.0</td>\n",
                            "      <td>891717742</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>2</th>\n",
                            "      <td>22</td>\n",
                            "      <td>377</td>\n",
                            "      <td>1.0</td>\n",
                            "      <td>878887116</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>3</th>\n",
                            "      <td>244</td>\n",
                            "      <td>51</td>\n",
                            "      <td>2.0</td>\n",
                            "      <td>880606923</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>4</th>\n",
                            "      <td>166</td>\n",
                            "      <td>346</td>\n",
                            "      <td>1.0</td>\n",
                            "      <td>886397596</td>\n",
                            "    </tr>\n",
                            "  </tbody>\n",
                            "</table>\n",
                            "</div>"
                        ],
                        "text/plain": [
                            "   userID  itemID  rating  timestamp\n",
                            "0     196     242     3.0  881250949\n",
                            "1     186     302     3.0  891717742\n",
                            "2      22     377     1.0  878887116\n",
                            "3     244      51     2.0  880606923\n",
                            "4     166     346     1.0  886397596"
                        ]
                    },
                    "execution_count": 3,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "df = movielens.load_pandas_df(size=MOVIELENS_DATA_SIZE)\n",
                "\n",
                "df.head()"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 4,
            "metadata": {},
            "outputs": [],
            "source": [
                "train, test = python_stratified_split(df, ratio=0.75)"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "### 2.2 Process data\n",
                "\n",
                "`ImplicitCF` is a class that intializes and loads data for the training process. During the initialization of this class, user IDs and item IDs are reindexed, ratings greater than zero are converted into implicit positive interaction, and adjacency matrix $R$ of user-item graph is created. Some important methods of `ImplicitCF` are:\n",
                "\n",
                "`get_norm_adj_mat`, load normalized adjacency matrix of user-item graph if it already exists in `adj_dir`, otherwise call `create_norm_adj_mat` to create the matrix and save the matrix if `adj_dir` is not `None`. This method will be called during the initialization process of LightGCN model.\n",
                "\n",
                "`create_norm_adj_mat`, create normalized adjacency matrix of user-item graph by calculating $D^{-\\frac{1}{2}} A D^{-\\frac{1}{2}}$, where $\\mathbf{A}=\\left(\\begin{array}{cc}\\mathbf{0} & \\mathbf{R} \\\\ \\mathbf{R}^{T} & \\mathbf{0}\\end{array}\\right)$.\n",
                "\n",
                "`train_loader`, generate a batch of training data — sample a batch of users and then sample one positive item and one negative item for each user. This method will be called before each epoch of the training process.\n"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 5,
            "metadata": {},
            "outputs": [],
            "source": [
                "data = ImplicitCF(train=train, test=test, seed=SEED)"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "### 2.3 Prepare hyper-parameters\n",
                "\n",
                "Important parameters of `LightGCN` model are:\n",
                "\n",
                "`data`, initialized LightGCNDataset object.\n",
                "\n",
                "`epochs`, number of epochs for training.\n",
                "\n",
                "`n_layers`, number of layers of the model.\n",
                "\n",
                "`eval_epoch`, if it is not None, evaluation metrics will be calculated on test set every \"eval_epoch\" epochs. In this way, we can observe the effect of the model during the training process.\n",
                "\n",
                "`top_k`, the number of items to be recommended for each user when calculating ranking metrics.\n",
                "\n",
                "A complete list of parameters can be found in `yaml_file`. We use `prepare_hparams` to read the yaml file and prepare a full set of parameters for the model. Parameters passed as the function's parameters will overwrite yaml settings."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 6,
            "metadata": {},
            "outputs": [],
            "source": [
                "hparams = prepare_hparams(yaml_file,\n",
                "                          n_layers=3,\n",
                "                          batch_size=BATCH_SIZE,\n",
                "                          epochs=EPOCHS,\n",
                "                          learning_rate=0.005,\n",
                "                          eval_epoch=5,\n",
                "                          top_k=TOP_K,\n",
                "                         )"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "### 2.4 Create and train model\n",
                "\n",
                "With data and parameters prepared, we can create the LightGCN model.\n",
                "\n",
                "To train the model, we simply need to call the `fit()` method."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 7,
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "Already create adjacency matrix.\n",
                        "Already normalize adjacency matrix.\n",
                        "Using xavier initialization.\n"
                    ]
                }
            ],
            "source": [
                "model = LightGCN(hparams, data, seed=SEED)"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 8,
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "Epoch 1 (train)1.4s: train loss = 0.47162 = (mf)0.47137 + (embed)0.00025\n",
                        "Epoch 2 (train)0.9s: train loss = 0.26998 = (mf)0.26926 + (embed)0.00072\n",
                        "Epoch 3 (train)0.9s: train loss = 0.24061 = (mf)0.23969 + (embed)0.00093\n",
                        "Epoch 4 (train)0.9s: train loss = 0.22621 = (mf)0.22512 + (embed)0.00109\n",
                        "Epoch 5 (train)0.8s + (eval)0.4s: train loss = 0.22219 = (mf)0.22098 + (embed)0.00121, recall = 0.16303, ndcg = 0.35414, precision = 0.30817, map = 0.09479\n",
                        "Epoch 6 (train)0.7s: train loss = 0.21284 = (mf)0.21152 + (embed)0.00131\n",
                        "Epoch 7 (train)0.8s: train loss = 0.20306 = (mf)0.20162 + (embed)0.00144\n",
                        "Epoch 8 (train)0.7s: train loss = 0.19611 = (mf)0.19454 + (embed)0.00157\n",
                        "Epoch 9 (train)0.8s: train loss = 0.18534 = (mf)0.18364 + (embed)0.00170\n",
                        "Epoch 10 (train)0.8s + (eval)0.2s: train loss = 0.18289 = (mf)0.18104 + (embed)0.00185, recall = 0.17857, ndcg = 0.38568, precision = 0.33669, map = 0.10658\n",
                        "Epoch 11 (train)0.9s: train loss = 0.17330 = (mf)0.17132 + (embed)0.00198\n",
                        "Epoch 12 (train)0.9s: train loss = 0.16902 = (mf)0.16691 + (embed)0.00211\n",
                        "Epoch 13 (train)0.8s: train loss = 0.16659 = (mf)0.16437 + (embed)0.00222\n",
                        "Epoch 14 (train)0.8s: train loss = 0.16300 = (mf)0.16066 + (embed)0.00233\n",
                        "Epoch 15 (train)0.8s + (eval)0.2s: train loss = 0.16449 = (mf)0.16207 + (embed)0.00242, recall = 0.18845, ndcg = 0.39969, precision = 0.35090, map = 0.11312\n",
                        "Epoch 16 (train)0.8s: train loss = 0.15979 = (mf)0.15729 + (embed)0.00250\n",
                        "Epoch 17 (train)0.8s: train loss = 0.15828 = (mf)0.15568 + (embed)0.00259\n",
                        "Epoch 18 (train)0.9s: train loss = 0.15787 = (mf)0.15519 + (embed)0.00268\n",
                        "Epoch 19 (train)0.9s: train loss = 0.15500 = (mf)0.15224 + (embed)0.00276\n",
                        "Epoch 20 (train)1.0s + (eval)0.3s: train loss = 0.15026 = (mf)0.14740 + (embed)0.00286, recall = 0.19753, ndcg = 0.42133, precision = 0.36723, map = 0.12255\n",
                        "Epoch 21 (train)1.0s: train loss = 0.14811 = (mf)0.14515 + (embed)0.00296\n",
                        "Epoch 22 (train)0.9s: train loss = 0.14676 = (mf)0.14370 + (embed)0.00306\n",
                        "Epoch 23 (train)0.9s: train loss = 0.14456 = (mf)0.14139 + (embed)0.00317\n",
                        "Epoch 24 (train)0.8s: train loss = 0.14317 = (mf)0.13991 + (embed)0.00326\n",
                        "Epoch 25 (train)0.9s + (eval)0.3s: train loss = 0.14045 = (mf)0.13708 + (embed)0.00337, recall = 0.20329, ndcg = 0.42989, precision = 0.37731, map = 0.12708\n",
                        "Epoch 26 (train)0.9s: train loss = 0.13827 = (mf)0.13478 + (embed)0.00349\n",
                        "Epoch 27 (train)0.8s: train loss = 0.13663 = (mf)0.13304 + (embed)0.00359\n",
                        "Epoch 28 (train)0.9s: train loss = 0.13429 = (mf)0.13059 + (embed)0.00370\n",
                        "Epoch 29 (train)1.0s: train loss = 0.13277 = (mf)0.12896 + (embed)0.00381\n",
                        "Epoch 30 (train)0.9s + (eval)0.2s: train loss = 0.12975 = (mf)0.12583 + (embed)0.00393, recall = 0.20393, ndcg = 0.43713, precision = 0.38282, map = 0.12933\n",
                        "Epoch 31 (train)1.0s: train loss = 0.12694 = (mf)0.12289 + (embed)0.00405\n",
                        "Epoch 32 (train)1.0s: train loss = 0.12595 = (mf)0.12178 + (embed)0.00417\n",
                        "Epoch 33 (train)1.0s: train loss = 0.12526 = (mf)0.12098 + (embed)0.00428\n",
                        "Epoch 34 (train)0.9s: train loss = 0.12110 = (mf)0.11669 + (embed)0.00441\n",
                        "Epoch 35 (train)0.9s + (eval)0.2s: train loss = 0.12038 = (mf)0.11584 + (embed)0.00454, recall = 0.20941, ndcg = 0.44525, precision = 0.38865, map = 0.13433\n",
                        "Epoch 36 (train)0.9s: train loss = 0.11915 = (mf)0.11450 + (embed)0.00465\n",
                        "Epoch 37 (train)0.8s: train loss = 0.12037 = (mf)0.11561 + (embed)0.00476\n",
                        "Epoch 38 (train)0.9s: train loss = 0.11479 = (mf)0.10990 + (embed)0.00489\n",
                        "Epoch 39 (train)0.9s: train loss = 0.11375 = (mf)0.10873 + (embed)0.00502\n",
                        "Epoch 40 (train)0.9s + (eval)0.3s: train loss = 0.11331 = (mf)0.10817 + (embed)0.00514, recall = 0.21353, ndcg = 0.45133, precision = 0.39597, map = 0.13544\n",
                        "Epoch 41 (train)0.9s: train loss = 0.11028 = (mf)0.10503 + (embed)0.00525\n",
                        "Epoch 42 (train)0.9s: train loss = 0.11127 = (mf)0.10588 + (embed)0.00539\n",
                        "Epoch 43 (train)0.9s: train loss = 0.10781 = (mf)0.10232 + (embed)0.00549\n",
                        "Epoch 44 (train)0.9s: train loss = 0.10583 = (mf)0.10021 + (embed)0.00562\n",
                        "Epoch 45 (train)0.9s + (eval)0.2s: train loss = 0.10706 = (mf)0.10133 + (embed)0.00573, recall = 0.21276, ndcg = 0.45422, precision = 0.39799, map = 0.13683\n",
                        "Epoch 46 (train)0.9s: train loss = 0.10515 = (mf)0.09929 + (embed)0.00586\n",
                        "Epoch 47 (train)0.9s: train loss = 0.10151 = (mf)0.09552 + (embed)0.00599\n",
                        "Epoch 48 (train)0.9s: train loss = 0.10157 = (mf)0.09544 + (embed)0.00613\n",
                        "Epoch 49 (train)0.9s: train loss = 0.10073 = (mf)0.09449 + (embed)0.00624\n",
                        "Epoch 50 (train)1.0s + (eval)0.2s: train loss = 0.10069 = (mf)0.09432 + (embed)0.00636, recall = 0.21320, ndcg = 0.45300, precision = 0.39926, map = 0.13531\n",
                        "Took 47.25114500336349 seconds for training.\n"
                    ]
                }
            ],
            "source": [
                "with Timer() as train_time:\n",
                "    model.fit()\n",
                "\n",
                "print(\"Took {} seconds for training.\".format(train_time.interval))"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "### 2.5 Recommendation and Evaluation"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "Recommendation and evaluation have been performed on the specified test set during training. After training, we can also use the model to perform recommendation and evalution on other data. Here we still use `test` as test data, but `test` can be replaced by other data with similar data structure."
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "#### 2.5.1 Recommendation\n",
                "\n",
                "We can call `recommend_k_items` to recommend k items for each user passed in this function. We set `remove_seen=True` to remove the items already seen by the user. The function returns a dataframe, containing each user and top k items recommended to them and the corresponding ranking scores."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 9,
            "metadata": {},
            "outputs": [
                {
                    "data": {
                        "text/html": [
                            "<div>\n",
                            "<style scoped>\n",
                            "    .dataframe tbody tr th:only-of-type {\n",
                            "        vertical-align: middle;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe tbody tr th {\n",
                            "        vertical-align: top;\n",
                            "    }\n",
                            "\n",
                            "    .dataframe thead th {\n",
                            "        text-align: right;\n",
                            "    }\n",
                            "</style>\n",
                            "<table border=\"1\" class=\"dataframe\">\n",
                            "  <thead>\n",
                            "    <tr style=\"text-align: right;\">\n",
                            "      <th></th>\n",
                            "      <th>userID</th>\n",
                            "      <th>itemID</th>\n",
                            "      <th>prediction</th>\n",
                            "    </tr>\n",
                            "  </thead>\n",
                            "  <tbody>\n",
                            "    <tr>\n",
                            "      <th>0</th>\n",
                            "      <td>1</td>\n",
                            "      <td>7</td>\n",
                            "      <td>5.597007</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>1</th>\n",
                            "      <td>1</td>\n",
                            "      <td>475</td>\n",
                            "      <td>5.193120</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>2</th>\n",
                            "      <td>1</td>\n",
                            "      <td>89</td>\n",
                            "      <td>5.147971</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>3</th>\n",
                            "      <td>1</td>\n",
                            "      <td>210</td>\n",
                            "      <td>5.133258</td>\n",
                            "    </tr>\n",
                            "    <tr>\n",
                            "      <th>4</th>\n",
                            "      <td>1</td>\n",
                            "      <td>127</td>\n",
                            "      <td>5.024765</td>\n",
                            "    </tr>\n",
                            "  </tbody>\n",
                            "</table>\n",
                            "</div>"
                        ],
                        "text/plain": [
                            "   userID  itemID  prediction\n",
                            "0       1       7    5.597007\n",
                            "1       1     475    5.193120\n",
                            "2       1      89    5.147971\n",
                            "3       1     210    5.133258\n",
                            "4       1     127    5.024765"
                        ]
                    },
                    "execution_count": 9,
                    "metadata": {},
                    "output_type": "execute_result"
                }
            ],
            "source": [
                "topk_scores = model.recommend_k_items(test, top_k=TOP_K, remove_seen=True)\n",
                "\n",
                "topk_scores.head()"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "#### 2.5.2 Evaluation\n",
                "\n",
                "With `topk_scores` predicted by the model, we can evaluate how LightGCN performs on this test set."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 10,
            "metadata": {},
            "outputs": [
                {
                    "name": "stdout",
                    "output_type": "stream",
                    "text": [
                        "MAP:\t0.135305\n",
                        "NDCG:\t0.453004\n",
                        "Precision@K:\t0.399258\n",
                        "Recall@K:\t0.213203\n"
                    ]
                }
            ],
            "source": [
                "eval_map = map(test, topk_scores, k=TOP_K)\n",
                "eval_ndcg = ndcg_at_k(test, topk_scores, k=TOP_K)\n",
                "eval_precision = precision_at_k(test, topk_scores, k=TOP_K)\n",
                "eval_recall = recall_at_k(test, topk_scores, k=TOP_K)\n",
                "\n",
                "print(\"MAP:\\t%f\" % eval_map,\n",
                "      \"NDCG:\\t%f\" % eval_ndcg,\n",
                "      \"Precision@K:\\t%f\" % eval_precision,\n",
                "      \"Recall@K:\\t%f\" % eval_recall, sep='\\n')"
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 11,
            "metadata": {
                "scrolled": true
            },
            "outputs": [
                {
                    "data": {
                        "application/scrapbook.scrap.json+json": {
                            "data": 0.13530545945796316,
                            "encoder": "json",
                            "name": "map",
                            "version": 1
                        }
                    },
                    "metadata": {
                        "scrapbook": {
                            "data": true,
                            "display": false,
                            "name": "map"
                        }
                    },
                    "output_type": "display_data"
                },
                {
                    "data": {
                        "application/scrapbook.scrap.json+json": {
                            "data": 0.45300447729353804,
                            "encoder": "json",
                            "name": "ndcg",
                            "version": 1
                        }
                    },
                    "metadata": {
                        "scrapbook": {
                            "data": true,
                            "display": false,
                            "name": "ndcg"
                        }
                    },
                    "output_type": "display_data"
                },
                {
                    "data": {
                        "application/scrapbook.scrap.json+json": {
                            "data": 0.3992576882290562,
                            "encoder": "json",
                            "name": "precision",
                            "version": 1
                        }
                    },
                    "metadata": {
                        "scrapbook": {
                            "data": true,
                            "display": false,
                            "name": "precision"
                        }
                    },
                    "output_type": "display_data"
                },
                {
                    "data": {
                        "application/scrapbook.scrap.json+json": {
                            "data": 0.2132025994579782,
                            "encoder": "json",
                            "name": "recall",
                            "version": 1
                        }
                    },
                    "metadata": {
                        "scrapbook": {
                            "data": true,
                            "display": false,
                            "name": "recall"
                        }
                    },
                    "output_type": "display_data"
                }
            ],
            "source": [
                "# Record results for tests - ignore this cell\n",
                "store_metadata(\"map\", eval_map)\n",
                "store_metadata(\"ndcg\", eval_ndcg)\n",
                "store_metadata(\"precision\", eval_precision)\n",
                "store_metadata(\"recall\", eval_recall)"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "### 2.6 Infer embeddings\n",
                "\n",
                "With `infer_embedding` method of LightGCN model, we can export the embeddings of users and items in the training set to CSV files for future use."
            ]
        },
        {
            "cell_type": "code",
            "execution_count": 12,
            "metadata": {},
            "outputs": [],
            "source": [
                "model.infer_embedding(user_file, item_file)"
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "## 3. Compare LightGCN with SAR and NCF\n",
                "\n",
                "Here there are the performances of LightGCN compared to [SAR](../00_quick_start/sar_movielens.ipynb) and [NCF](../00_quick_start/ncf_movielens.ipynb) on MovieLens dataset of 100k and 1m. The method of data loading and splitting is the same as that described above and the GPU used was a GeForce GTX 1080Ti.\n",
                "\n",
                "Settings common to the three models: `epochs=15, seed=42`.\n",
                "\n",
                "Settings for LightGCN: `embed_size=64, n_layers=3, batch_size=1024, decay=0.0001, learning_rate=0.015 `.\n",
                "\n",
                "Settings for SAR: `similarity_type=\"jaccard\", time_decay_coefficient=30, time_now=None, timedecay_formula=True`.\n",
                "\n",
                "Settings for NCF: `n_factors=4, layer_sizes=[16, 8, 4], batch_size=1024, learning_rate=0.001`.\n",
                "\n",
                "| Data Size | Model    | Training time | Recommending time | MAP@10   | nDCG@10  | Precision@10 | Recall@10 |\n",
                "| --------- | -------- | ------------- | ----------------- | -------- | -------- | ------------ | --------- |\n",
                "| 100k      | LightGCN | 27.8865       | 0.6445            | 0.129236 | 0.436297 | 0.381866     | 0.205816  |\n",
                "| 100k      | SAR      | 0.4895        | 0.1144            | 0.110591 | 0.382461 | 0.330753     | 0.176385  |\n",
                "| 100k      | NCF      | 116.3174      | 7.7660            | 0.105725 | 0.387603 | 0.342100     | 0.174580  |\n",
                "| 1m        | LightGCN | 396.7298      | 1.4343            | 0.075012 | 0.377501 | 0.345679     | 0.128096  |\n",
                "| 1m        | SAR      | 4.5593        | 2.8357            | 0.060579 | 0.299245 | 0.270116     | 0.104350  |\n",
                "| 1m        | NCF      | 1601.5846     | 85.4567           | 0.062821 | 0.348770 | 0.320613     | 0.108121  |\n",
                "\n",
                "From the above results, we can see that LightGCN performs better than the other two models."
            ]
        },
        {
            "cell_type": "markdown",
            "metadata": {},
            "source": [
                "### References: \n",
                "1. Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yongdong Zhang & Meng Wang, LightGCN: Simplifying and Powering Graph Convolution Network for Recommendation, 2020, https://arxiv.org/abs/2002.02126\n",
                "2. LightGCN implementation [TensorFlow]: https://github.com/kuandeng/lightgcn\n",
                "3. Thomas N. Kipf and Max Welling, Semi-Supervised Classification with Graph Convolutional Networks, ICLR, 2017, https://arxiv.org/abs/1609.02907\n",
                "4. Xiang Wang, Xiangnan He, Meng Wang, Fuli Feng, and Tat-Seng Chua, Neural Graph Collaborative Filtering, SIGIR, 2019, https://arxiv.org/abs/1905.08108\n",
                "5. Y. Koren, R. Bell and C. Volinsky, \"Matrix Factorization Techniques for Recommender Systems\", in Computer, vol. 42, no. 8, pp. 30-37, Aug. 2009, doi: 10.1109/MC.2009.263.  url: https://datajobs.com/data-science-repo/Recommender-Systems-%5BNetflix%5D.pdf"
            ]
        }
    ],
    "metadata": {
        "celltoolbar": "Tags",
        "interpreter": {
            "hash": "3a9a0c422ff9f08d62211b9648017c63b0a26d2c935edc37ebb8453675d13bb5"
        },
        "kernelspec": {
            "display_name": "Python 3.7.11 64-bit ('tf2': conda)",
            "language": "python",
            "name": "python3"
        },
        "language_info": {
            "codemirror_mode": {
                "name": "ipython",
                "version": 3
            },
            "file_extension": ".py",
            "mimetype": "text/x-python",
            "name": "python",
            "nbconvert_exporter": "python",
            "pygments_lexer": "ipython3",
            "version": "3.7.11"
        }
    },
    "nbformat": 4,
    "nbformat_minor": 2
}