analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a id='toc1_'></a>[YouTube History Analysis](#toc0_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Table of contents**<a id='toc0_'></a>    \n",
    "- [YouTube History Analysis](#toc1_)    \n",
    "- [Imports and variables](#toc2_)    \n",
    "- [General watching stats](#toc3_)    \n",
    "  - [Amounts](#toc3_1_)    \n",
    "  - [Moments](#toc3_2_)    \n",
    "- [Topic category frequency](#toc4_)    \n",
    "  - [Most frequent in period](#toc4_1_)    \n",
    "  - [Most frequent by period](#toc4_2_)    \n",
    "  - [Frequency over time](#toc4_3_)    \n",
    "- [Tag frequency](#toc5_)    \n",
    "  - [Most frequent in period](#toc5_1_)    \n",
    "  - [Most frequent by period](#toc5_2_)    \n",
    "  - [Frequency over time](#toc5_3_)    \n",
    "- [Clustering utilities](#toc6_)    \n",
    "  - [Visualization](#toc6_1_)    \n",
    "  - [K-Means](#toc6_2_)    \n",
    "  - [Agglomerative clustering](#toc6_3_)    \n",
    "- [Topic category clustering](#toc7_)    \n",
    "  - [Category embedding](#toc7_1_)    \n",
    "  - [K-Means](#toc7_2_)    \n",
    "  - [Agglomerative clustering](#toc7_3_)    \n",
    "- [Tag clustering](#toc8_)    \n",
    "  - [Tag embedding](#toc8_1_)    \n",
    "  - [K-Means](#toc8_2_)    \n",
    "  - [Agglomerative clustering](#toc8_3_)    \n",
    "\n",
    "<!-- vscode-jupyter-toc-config\n",
    "\tnumbering=false\n",
    "\tanchor=true\n",
    "\tflat=false\n",
    "\tminLevel=1\n",
    "\tmaxLevel=2\n",
    "\t/vscode-jupyter-toc-config -->\n",
    "<!-- THIS CELL WILL BE REPLACED ON TOC UPDATE. DO NOT WRITE YOUR TEXT IN THIS CELL -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a id='toc2_'></a>[Imports and variables](#toc0_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 468,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "from collections import Counter\n",
    "from datetime import datetime, timedelta\n",
    "from dateutil.relativedelta import relativedelta\n",
    "from statistics import mean\n",
    "import spacy\n",
    "from sklearn.cluster import KMeans\n",
    "from sklearn.decomposition import PCA\n",
    "from sklearn.metrics import pairwise_distances_argmin_min\n",
    "from scipy.cluster.hierarchy import dendrogram, linkage, fcluster\n",
    "from file_manager import FileManager"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 469,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.style.use('ggplot')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 470,
   "metadata": {},
   "outputs": [],
   "source": [
    "processed_history = FileManager.load_processed_history()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 471,
   "metadata": {},
   "outputs": [],
   "source": [
    "last_time_str = processed_history[0]['time']\n",
    "\n",
    "# The last time you watched a video\n",
    "last_time = datetime.fromisoformat(last_time_str.replace(\"Z\", \"+00:00\"))\n",
    "\n",
    "# The beginning of the day of last_time\n",
    "last_time_zero = last_time.replace(hour=0, minute=0, second=0, microsecond=0)\n",
    "\n",
    "# The end of the day of last_time\n",
    "last_time_midnight = last_time.replace(hour=23, minute=59, second=59)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 472,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Number of categories and tags to consider for frequency analysis\n",
    "max_most_common = 25"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a id='toc3_'></a>[General watching stats](#toc0_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc3_1_'></a>[Amounts](#toc0_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Understand how many videos you have watched recently. All videos that appear in your history count as watched, even if you immediately closed them.\n",
    "Shorts are included."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Utilities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 473,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the number of watched videos between two dates\n",
    "\n",
    "def get_count_in_period(start = None, end = None):\n",
    "    count = 0\n",
    "    for video in processed_history:\n",
    "        time = datetime.fromisoformat(video['time'].replace(\"Z\", \"+00:00\"))\n",
    "        if (start is None or time >= start) and (end is None or time <= end):\n",
    "            count += 1\n",
    "    return count"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 483,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the number of watched videos for intervals between two dates\n",
    "\n",
    "def get_count_by_interval(start, end, min_start, interval, date_format='%d/%m'):\n",
    "    intervals, counts = [], []\n",
    "\n",
    "    while start >= min_start:\n",
    "        _counts = get_count_in_period(start, end)\n",
    "        counts.append(_counts)\n",
    "        intervals.append(start.strftime(date_format))\n",
    "        start = start - interval\n",
    "        end = end - interval\n",
    "    \n",
    "    return intervals[::-1], counts[::-1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 484,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot counts for intervals\n",
    "\n",
    "def plot_amount_by_interval(intervals, counts, title):\n",
    "    plt.figure(figsize=(10, 5))\n",
    "    plt.plot(intervals, counts, marker='o', label='Watched videos')\n",
    "    mean_amount = mean(counts)\n",
    "    median_amount = np.median(counts)\n",
    "    plt.axhline(y = mean_amount, color='g', linestyle = '--', label='Mean: {:.2f}'.format(mean_amount))\n",
    "    plt.axhline(y = median_amount, color='b', linestyle = '--', label='Median: {:.2f}'.format(median_amount))\n",
    "    plt.title(f\"Number of watched videos in {title}\")\n",
    "    plt.xlabel('Period')\n",
    "    plt.ylabel('Number of watched videos')\n",
    "    plt.xticks(rotation=45, ha='right')\n",
    "    plt.grid(True)\n",
    "    plt.legend()\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### By day (last 30)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 485,
   "metadata": {},
   "outputs": [],
   "source": [
    "start = last_time.replace(hour=0, minute=0, second=0, microsecond=0)\n",
    "end = last_time.replace(hour=23, minute=59, second=59, microsecond=59)\n",
    "thirty_days_ago = start - timedelta(days=30)\n",
    "\n",
    "intervals, counts = get_count_by_interval(start, end, thirty_days_ago, timedelta(days=1))\n",
    "\n",
    "plot_amount_by_interval(intervals, counts, \"last 30 days\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### By week (last 20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "start = last_time.replace(hour=0, minute=0, second=0, microsecond=0) - timedelta(days=6)\n",
    "end = last_time.replace(hour=23, minute=59, second=59, microsecond=59)\n",
    "twenty_weeks_ago = start - timedelta(weeks=19)\n",
    "\n",
    "intervals, counts = get_count_by_interval(start, end, twenty_weeks_ago, timedelta(weeks=1))\n",
    "\n",
    "plot_amount_by_interval(intervals, counts, \"last 20 weeks\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### By month (last 24)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "start = last_time.replace(hour=0, minute=0, second=0, microsecond=0) - relativedelta(months=1)\n",
    "end = last_time.replace(hour=23, minute=59, second=59, microsecond=59)\n",
    "two_years_ago = start - relativedelta(months=23)\n",
    "\n",
    "intervals, counts = get_count_by_interval(start, end, two_years_ago, relativedelta(months=1), date_format='%m/%Y')\n",
    "\n",
    "plot_amount_by_interval(intervals, counts, \"last 2 years\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc3_2_'></a>[Moments](#toc0_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Understand what are the days of the week and the hours of the day when you usually watch YouTube."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Distribution over days of week"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Utilities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 413,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot the distribution of the number of watched videos by day of the week \n",
    "# for videos watched between two dates\n",
    "\n",
    "def plot_amount_distribution_by_week_day(start = None, end = None):\n",
    "    days = ['Mon', 'Tue', 'Wed', 'Thu', 'Fri', 'Sat', 'Sun']\n",
    "    counts = [0, 0, 0, 0, 0, 0, 0]\n",
    "    \n",
    "    for video in processed_history:\n",
    "        time = datetime.fromisoformat(video['time'].replace(\"Z\", \"+00:00\"))\n",
    "        if (start is None or start <= time) and (end is None or time <= end):\n",
    "            counts[time.weekday()] += 1\n",
    "            \n",
    "    plt.bar(days, counts)\n",
    "    plt.title('Watched videos by day of the week')\n",
    "    plt.xlabel('Day of the Week')\n",
    "    plt.ylabel('Number of watched videos')\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Last week"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "start = last_time_midnight - timedelta(days=7)\n",
    "plot_amount_distribution_by_week_day(start)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Last month"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "start = last_time_midnight - relativedelta(months=1)\n",
    "plot_amount_distribution_by_week_day(start)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Last 6 months"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "start = last_time_midnight - relativedelta(months=6)\n",
    "plot_amount_distribution_by_week_day(start)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Last year"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "start = last_time_midnight - relativedelta(years=1)\n",
    "plot_amount_distribution_by_week_day(start)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### All time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_amount_distribution_by_week_day()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Distribution over hours of day"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Utilities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 419,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot the distribution of the number of watched videos by day of the week \n",
    "# for videos watched between two dates\n",
    "\n",
    "def plot_amount_distribution_by_hour(start = None, end = None):\n",
    "    hours = [str(i) for i in range(24)]\n",
    "    counts = [0] * 24\n",
    "\n",
    "    for video in processed_history:\n",
    "        time = datetime.fromisoformat(video['time'].replace(\"Z\", \"+00:00\"))\n",
    "        if (start is None or start <= time) and (end is None or time <= end):\n",
    "            counts[time.hour] += 1\n",
    "    \n",
    "    plt.bar(hours, counts)\n",
    "    plt.title('Watched videos by hour of the day')\n",
    "    plt.xlabel('Hour of the Day')\n",
    "    plt.ylabel('Number of watched videos')\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Last week"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "start = last_time_midnight - timedelta(days=7)\n",
    "plot_amount_distribution_by_hour(start)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Last month"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "start = last_time_midnight - relativedelta(months=1)\n",
    "plot_amount_distribution_by_hour(start)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Last 6 months"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "start = last_time_midnight - relativedelta(months=6)\n",
    "plot_amount_distribution_by_hour(start)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Last year"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "start = last_time_midnight - relativedelta(years=1)\n",
    "plot_amount_distribution_by_hour(start)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### All time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "plot_amount_distribution_by_hour()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a id='toc4_'></a>[Topic category frequency](#toc0_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Understand what topics your watched videos are usually about and how they have changed over time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc4_1_'></a>[Most frequent in period](#toc0_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Utilities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 425,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the number of watched videos by category between two dates\n",
    "\n",
    "def get_category_counts(start = None, end = None, max_most_common=max_most_common):\n",
    "    category_names = []\n",
    "\n",
    "    for video in processed_history:\n",
    "        time = datetime.fromisoformat(video['time'].replace(\"Z\", \"+00:00\"))\n",
    "\n",
    "        if (start is None or time >= start) and (end is None or time <= end):\n",
    "            for category in video['topicCategories']:\n",
    "                category_name = category.split('/')[-1]\n",
    "                category_names.append(category_name)\n",
    "            \n",
    "    category_counts = Counter(category_names)\n",
    "    \n",
    "    if not category_counts:\n",
    "        return [], []\n",
    "    else:\n",
    "        return zip(*category_counts.most_common(max_most_common))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 426,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot the distribution of the number of watched videos by category\n",
    "\n",
    "def draw_bar_chart(categories, counts, period, max_most_common=max_most_common):\n",
    "    plt.figure(figsize=(10, 6))\n",
    "    plt.barh(categories, counts)\n",
    "    plt.xlabel('Frequency')\n",
    "    plt.ylabel('Category Names')\n",
    "    plt.title(f\"{max_most_common} most frequent categories {period}\")\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Last week"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "start = last_time_midnight - timedelta(days=7)\n",
    "tags, counts = get_category_counts(start)\n",
    "draw_bar_chart(tags[::-1], counts[::-1], \"7 days\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Last month"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "thirty_days_ago = last_time_midnight - relativedelta(months=1)\n",
    "tags, counts = get_category_counts(thirty_days_ago)\n",
    "draw_bar_chart(tags[::-1], counts[::-1], \"last month\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Last 6 months"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "six_months_ago = last_time_midnight - relativedelta(months=6)\n",
    "tags, counts = get_category_counts(six_months_ago)\n",
    "draw_bar_chart(tags[::-1], counts[::-1], \"last 6 months\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Last year"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "one_year_ago = last_time_midnight - relativedelta(years=1)\n",
    "tags, counts = get_category_counts(one_year_ago)\n",
    "draw_bar_chart(tags[::-1], counts[::-1], \"last 1 year\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### All time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tags, counts = get_category_counts(None)\n",
    "draw_bar_chart(tags[::-1], counts[::-1], \"all time\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc4_2_'></a>[Most frequent by period](#toc0_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Utilities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 432,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print the frequency of the most common categories for intervals between two dates\n",
    "\n",
    "def print_category_frequency_per_period(start, end, min_start, interval, max_per_period = 5):\n",
    "    # Get all counts\n",
    "    periods, categories, counts = [], [], []\n",
    "    while start >= min_start:\n",
    "        _categories, _counts = get_category_counts(start, end)\n",
    "        categories.append(_categories)\n",
    "        counts.append(_counts)\n",
    "        periods.append(f\"{start.strftime('%d/%m/%Y')} - {end.strftime('%d/%m/%Y')}\")\n",
    "        start = start - interval\n",
    "        end = end - interval\n",
    "\n",
    "    # Print counts for most common categories\n",
    "    for i in range(len(periods)):\n",
    "        print(periods[i], end=\"\\t\\t\")\n",
    "        for j in range(min(max_per_period, len(categories[i]))):\n",
    "            print(f\"{categories[i][j]} ({counts[i][j]})\", end=\", \")\n",
    "        print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### By day (last 30)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "start = last_time.replace(hour=0, minute=0, second=0, microsecond=0)\n",
    "end = last_time.replace(hour=23, minute=59, second=59, microsecond=59)\n",
    "min_start = start - timedelta(days=30)\n",
    "interval = timedelta(days=1)\n",
    "\n",
    "print_category_frequency_per_period(start, end, min_start, interval)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### By week (last 20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "start = last_time.replace(hour=0, minute=0, second=0, microsecond=0) - timedelta(days=6)\n",
    "end = last_time.replace(hour=23, minute=59, second=59, microsecond=59)\n",
    "min_start = start - timedelta(weeks=19)\n",
    "interval = timedelta(days=7)\n",
    "\n",
    "print_category_frequency_per_period(start, end, min_start, interval)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### By month (last 24)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "start = last_time.replace(hour=0, minute=0, second=0, microsecond=0) - relativedelta(months=1)\n",
    "end = last_time.replace(hour=23, minute=59, second=59, microsecond=59)\n",
    "min_start = start - relativedelta(months=23)\n",
    "interval = relativedelta(months=1)\n",
    "\n",
    "print_category_frequency_per_period(start, end, min_start, interval)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc4_3_'></a>[Frequency over time](#toc0_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Utilities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 436,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot the frequency variation of selected categories for intervals between two dates\n",
    "\n",
    "def plot_category_frequency_variation(categories, start, end, min_start, interval, date_format='%d/%m'):\n",
    "    periods = []\n",
    "    counts = [[] for _ in range(len(categories))]\n",
    "\n",
    "    while start >= min_start:\n",
    "        _categories, _counts = get_category_counts(start, end)\n",
    "        \n",
    "        for i in range(len(categories)):\n",
    "            if categories[i] in _categories:\n",
    "                index = _categories.index(categories[i])\n",
    "                counts[i].append(_counts[index])\n",
    "            else:\n",
    "                counts[i].append(0)\n",
    "\n",
    "        periods.append(f\"{start.strftime(date_format)} - {end.strftime(date_format)}\")\n",
    "        start = start - interval\n",
    "        end = end - interval\n",
    "\n",
    "    plt.figure(figsize=(10, 5))\n",
    "\n",
    "    for i in range(len(categories)):\n",
    "        plt.plot(periods[::-1], counts[i][::-1], marker='o', label=categories[i])\n",
    "\n",
    "    plt.title('Frequency variation of topics')\n",
    "    plt.xlabel('Period')\n",
    "    plt.ylabel('Frequency')\n",
    "    plt.xticks(rotation=45, ha='right')\n",
    "    plt.grid(True)\n",
    "    plt.legend()\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 437,
   "metadata": {},
   "outputs": [],
   "source": [
    "# You can change the topics to the ones you want to analyze\n",
    "# By default it will use the top 5 most frequent topics in the last week\n",
    "\n",
    "start = last_time_midnight - timedelta(days=7)\n",
    "tags, _ = get_category_counts(start)\n",
    "selected_categories = list(tags[:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### By day (last 30)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "start = last_time.replace(hour=0, minute=0, second=0, microsecond=0)\n",
    "end = last_time.replace(hour=23, minute=59, second=59, microsecond=59)\n",
    "min_start = start - timedelta(days=30)\n",
    "interval = timedelta(days=1)\n",
    "\n",
    "plot_category_frequency_variation(selected_categories, start, end, min_start, interval)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### By week (last 20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "start = last_time.replace(hour=0, minute=0, second=0, microsecond=0) - timedelta(days=6)\n",
    "end = last_time.replace(hour=23, minute=59, second=59, microsecond=59)\n",
    "min_start = start - timedelta(weeks=19)\n",
    "interval = timedelta(days=7)\n",
    "\n",
    "plot_category_frequency_variation(selected_categories, start, end, min_start, interval)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### By month (lat 24)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "start = last_time.replace(hour=0, minute=0, second=0, microsecond=0) - relativedelta(months=1)\n",
    "end = last_time.replace(hour=23, minute=59, second=59, microsecond=59)\n",
    "min_start = start - relativedelta(months=23)\n",
    "interval = relativedelta(months=1)\n",
    "\n",
    "plot_category_frequency_variation(selected_categories, start, end, min_start, interval, date_format='%m/%Y')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a id='toc5_'></a>[Tag frequency](#toc0_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Understand what tags usually appear in your watched videos and how thay have changed over time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc5_1_'></a>[Most frequent in period](#toc0_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Utilities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 441,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the number of watched videos by tag between two dates\n",
    "\n",
    "def get_tag_counts(start = None, end = None, max_most_common=max_most_common):\n",
    "    tags = []\n",
    "\n",
    "    for video in processed_history:\n",
    "        time = datetime.fromisoformat(video['time'].replace(\"Z\", \"+00:00\"))\n",
    "        if (start is None or time >= start) and (end is None or time <= end):\n",
    "            tags.extend(video['tags'])\n",
    "\n",
    "    counts = Counter(tags)\n",
    "    \n",
    "    if not counts:\n",
    "        return [], []\n",
    "    else:\n",
    "        return zip(*counts.most_common(max_most_common))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 442,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot the distribution of the number of watched videos by tag\n",
    "\n",
    "def draw_bar_chart(tags, counts, period, max_most_common=max_most_common):\n",
    "    plt.figure(figsize=(10, 6))\n",
    "    plt.barh(tags, counts)\n",
    "    plt.xlabel('Frequency')\n",
    "    plt.ylabel('Tag')\n",
    "    plt.title(f\"{max_most_common} most frequent tags {period}\")\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Last week"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "start = last_time_midnight - timedelta(days=7)\n",
    "tags, counts = get_tag_counts(start)\n",
    "draw_bar_chart(tags[::-1], counts[::-1], \"7 days\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Last month"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "thirty_days_ago = last_time_midnight - relativedelta(months=1)\n",
    "tags, counts = get_tag_counts(thirty_days_ago)\n",
    "draw_bar_chart(tags[::-1], counts[::-1], \"last month\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Last 6 months"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "six_months_ago = last_time_midnight - relativedelta(months=6)\n",
    "tags, counts = get_tag_counts(six_months_ago)\n",
    "draw_bar_chart(tags[::-1], counts[::-1], \"last 6 months\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Last year"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "one_year_ago = last_time_midnight - relativedelta(years=1)\n",
    "tags, counts = get_tag_counts(one_year_ago)\n",
    "draw_bar_chart(tags[::-1], counts[::-1], \"last 1 year\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### All time"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tags, counts = get_tag_counts(None)\n",
    "draw_bar_chart(tags[::-1], counts[::-1], \"all time\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc5_2_'></a>[Most frequent by period](#toc0_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Utilities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 448,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print the frequency of the most common tags for intervals between two dates\n",
    "\n",
    "def print_tag_frequency_per_period(start, end, min_start, interval, max_per_period = 5):\n",
    "    # get all counts\n",
    "    periods, tags, counts = [], [], []\n",
    "    while start >= min_start:\n",
    "        _tags, _counts = get_tag_counts(start, end)\n",
    "        tags.append(_tags)\n",
    "        counts.append(_counts)\n",
    "        periods.append(f\"{start.strftime('%d/%m/%Y')} - {end.strftime('%d/%m/%Y')}\")\n",
    "        start = start - interval\n",
    "        end = end - interval\n",
    "\n",
    "    # print counts for most common tags\n",
    "    for i in range(len(periods)):\n",
    "        print(periods[i], end=\"\\t\\t\")\n",
    "        for j in range(min(max_per_period, len(tags[i]))):\n",
    "            print(f\"{tags[i][j]} ({counts[i][j]})\", end=\", \")\n",
    "        print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### By day (last 30)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "start = last_time.replace(hour=0, minute=0, second=0, microsecond=0)\n",
    "end = last_time.replace(hour=23, minute=59, second=59, microsecond=59)\n",
    "min_start = start - timedelta(days=30)\n",
    "interval = timedelta(days=1)\n",
    "\n",
    "print_tag_frequency_per_period(start, end, min_start, interval)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### By week (last 20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "start = last_time.replace(hour=0, minute=0, second=0, microsecond=0) - timedelta(days=6)\n",
    "end = last_time.replace(hour=23, minute=59, second=59, microsecond=59)\n",
    "min_start = start - timedelta(weeks=19)\n",
    "interval = timedelta(days=7)\n",
    "\n",
    "print_tag_frequency_per_period(start, end, min_start, interval)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### By month (last 24)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "start = last_time.replace(hour=0, minute=0, second=0, microsecond=0) - relativedelta(months=1)\n",
    "end = last_time.replace(hour=23, minute=59, second=59, microsecond=59)\n",
    "min_start = start - relativedelta(months=23)\n",
    "interval = relativedelta(months=1)\n",
    "\n",
    "print_tag_frequency_per_period(start, end, min_start, interval)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc5_3_'></a>[Frequency over time](#toc0_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Utilities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 452,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Plot the frequency variation of selected tags for intervals between two dates\n",
    "\n",
    "def plot_tag_frequency_variation(tags, start, end, min_start, interval, date_format='%d/%m'):\n",
    "    periods = []\n",
    "    counts = [[] for _ in range(len(tags))]\n",
    "\n",
    "    while start >= min_start:\n",
    "        _tags, _counts = get_tag_counts(start, end)\n",
    "        \n",
    "        for i in range(len(tags)):\n",
    "            if tags[i] in _tags:\n",
    "                index = _tags.index(tags[i])\n",
    "                counts[i].append(_counts[index])\n",
    "            else:\n",
    "                counts[i].append(0)\n",
    "\n",
    "        periods.append(f\"{start.strftime(date_format)} - {end.strftime(date_format)}\")\n",
    "        start = start - interval\n",
    "        end = end - interval\n",
    "\n",
    "    plt.figure(figsize=(10, 5))\n",
    "\n",
    "    for i in range(len(tags)):\n",
    "        plt.plot(periods[::-1], counts[i][::-1], marker='o', label=tags[i])\n",
    "\n",
    "    plt.title('Frequency variation of tags')\n",
    "    plt.xlabel('Period')\n",
    "    plt.ylabel('Frequency')\n",
    "    plt.xticks(rotation=45, ha='right')\n",
    "    plt.grid(True)\n",
    "    plt.legend()\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 453,
   "metadata": {},
   "outputs": [],
   "source": [
    "# You can change the tags to the ones you want to analyze\n",
    "# By default it will use the top 5 most frequent tags in the last week\n",
    "\n",
    "start = last_time_midnight - timedelta(days=7)\n",
    "tags, _ = get_tag_counts(start)\n",
    "selected_tags = list(tags[:5])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### By day (last 30)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "start = last_time.replace(hour=0, minute=0, second=0, microsecond=0)\n",
    "end = last_time.replace(hour=23, minute=59, second=59, microsecond=59)\n",
    "min_start = start - timedelta(days=30)\n",
    "interval = timedelta(days=1)\n",
    "\n",
    "plot_tag_frequency_variation(selected_tags, start, end, min_start, interval)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### By week (last 20)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "start = last_time.replace(hour=0, minute=0, second=0, microsecond=0) - timedelta(days=6)\n",
    "end = last_time.replace(hour=23, minute=59, second=59, microsecond=59)\n",
    "min_start = start - timedelta(weeks=19)\n",
    "interval = timedelta(days=7)\n",
    "\n",
    "plot_tag_frequency_variation(selected_tags, start, end, min_start, interval)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### By month (lat 24)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "start = last_time.replace(hour=0, minute=0, second=0, microsecond=0) - relativedelta(months=1)\n",
    "end = last_time.replace(hour=23, minute=59, second=59, microsecond=59)\n",
    "min_start = start - relativedelta(months=23)\n",
    "interval = relativedelta(months=1)\n",
    "\n",
    "plot_tag_frequency_variation(selected_tags, start, end, min_start, interval, date_format='%m/%Y')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a id='toc6_'></a>[Clustering utilities](#toc0_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Utilities to perform cluster analysis of sets of expressions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc6_1_'></a>[Visualization](#toc0_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cluster plotting and stats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 457,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Given the results of clustering, plot the clusters and the percentage of labels in each cluster\n",
    "\n",
    "def plot_clusters_and_info(labels, centroids, words, vectorized_words, num_clusters):\n",
    "    # Find the closest word to each centroid\n",
    "    closest_words = {}\n",
    "\n",
    "    for i in range(num_clusters):\n",
    "        # Get the indices of the points in the current cluster\n",
    "        cluster_indices = np.where(labels == i)[0]\n",
    "        cluster_vectors = vectorized_words[cluster_indices]\n",
    "\n",
    "        # Find the index of the closest point to the centroid and get the corresponding word\n",
    "        closest, _ = pairwise_distances_argmin_min(centroids[i].reshape(1, -1), cluster_vectors)\n",
    "        closest_word = words[cluster_indices[closest[0]]]\n",
    "        closest_words[i] = closest_word\n",
    "\n",
    "    # Reduce the dimensionality of the vectors to 2D for visualization\n",
    "    pca = PCA(n_components=2, random_state=42)\n",
    "    reduced_vectors = pca.fit_transform(vectorized_words)\n",
    "\n",
    "    # Plot the clusters\n",
    "    scatter = plt.scatter(reduced_vectors[:, 0], reduced_vectors[:, 1], c=labels, cmap='viridis')\n",
    "\n",
    "    handles, _ = scatter.legend_elements()\n",
    "    legend_labels = [closest_words[i] for i in range(num_clusters)]\n",
    "    plt.legend(handles, legend_labels)\n",
    "    plt.show()\n",
    "\n",
    "    # Compute the percentage of labels in each cluster\n",
    "    total_labels = len(labels)\n",
    "    cluster_counts = Counter(labels)\n",
    "\n",
    "    data = [{'Cluster': closest_words[k], 'Count': v, 'Percentage': f\"{v/total_labels:.2%}\"} for k, v in cluster_counts.items()]\n",
    "    data.append({'Cluster': 'Total', 'Count': total_labels, 'Percentage': '100.00%'})\n",
    "\n",
    "    df = pd.DataFrame(data)\n",
    "    print(df.to_string(index=False))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc6_2_'></a>[K-Means](#toc0_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 458,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Perform KMeans clustering on the words and display the results\n",
    "\n",
    "def k_means_clustering(words, vectorized_words, num_clusters):\n",
    "    # KMeans clustering\n",
    "    kmeans = KMeans(n_clusters=num_clusters, random_state=0).fit(vectorized_words)\n",
    "    labels = kmeans.labels_\n",
    "    centroids = kmeans.cluster_centers_\n",
    "\n",
    "    plot_clusters_and_info(labels, centroids, words, vectorized_words, num_clusters)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc6_3_'></a>[Agglomerative clustering](#toc0_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 459,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Perform agglomerative clustering on the words and display the results\n",
    "\n",
    "def agglomerative_clustering(words, vectorized_words, num_clusters):\n",
    "    # Clustering and dendrogram visualization\n",
    "    pca = PCA(n_components=min(10, len(vectorized_words)), random_state=42)\n",
    "    reduced_vectors = pca.fit_transform(vectorized_words)\n",
    "\n",
    "    linked = linkage(reduced_vectors, method='ward')\n",
    "    plt.figure(figsize=(20, 10))\n",
    "    dendrogram(linked,\n",
    "            orientation='top',\n",
    "            labels=words,\n",
    "            distance_sort='descending',\n",
    "            show_leaf_counts=True,\n",
    "            leaf_rotation=90,\n",
    "            leaf_font_size=10)\n",
    "    plt.title(\"Dendrogram\")\n",
    "    plt.show()\n",
    "\n",
    "    # Visualization\n",
    "    labels = np.array(fcluster(linked, t=num_clusters, criterion='maxclust')) - 1\n",
    "    centroids = np.array([vectorized_words[labels == label].mean(axis=0) for label in range(num_clusters)])\n",
    "\n",
    "    plot_clusters_and_info(labels, centroids, words, vectorized_words, num_clusters)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a id='toc7_'></a>[Topic category clustering](#toc0_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Understand how the video topics you usually watch relate to each other and can be grouped together."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc7_1_'></a>[Category embedding](#toc0_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 460,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the category names and their counts for a period\n",
    "# Change start (and end) to get different periods\n",
    "\n",
    "start = last_time_midnight - timedelta(days=7)\n",
    "unique_categories, counts = get_category_counts(start, end=None, max_most_common=100)\n",
    "categories = np.repeat(unique_categories, np.array(counts), axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 461,
   "metadata": {},
   "outputs": [],
   "source": [
    "nlp = spacy.load('en_core_web_lg')\n",
    "\n",
    "unique_cat_vectors = np.array([nlp(str(cat)).vector for cat in unique_categories])\n",
    "cat_vectors = np.array([nlp(str(cat)).vector for cat in categories])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc7_2_'></a>[K-Means](#toc0_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "k_means_clustering(categories, cat_vectors, num_clusters=7)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc7_3_'></a>[Agglomerative clustering](#toc0_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "agglomerative_clustering(unique_categories, unique_cat_vectors, num_clusters=7)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a id='toc8_'></a>[Tag clustering](#toc0_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Understand how the tags in your watched videos relate to each other and can be grouped together."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc8_1_'></a>[Tag embedding](#toc0_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 464,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get the tags and their counts for a period\n",
    "# Change start (and end) to get different periods\n",
    "\n",
    "start = last_time_midnight - timedelta(days=7)\n",
    "unique_tags, counts = get_tag_counts(start, end=None, max_most_common=100)\n",
    "tags = np.repeat(unique_tags, np.array(counts), axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 465,
   "metadata": {},
   "outputs": [],
   "source": [
    "nlp = spacy.load('en_core_web_lg')\n",
    "\n",
    "unique_tag_vectors = np.array([nlp(str(cat)).vector for cat in unique_tags])\n",
    "tag_vectors = np.array([nlp(str(cat)).vector for cat in tags])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc8_2_'></a>[K-Means](#toc0_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "k_means_clustering(tags, tag_vectors, num_clusters=7)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## <a id='toc8_3_'></a>[Agglomerative clustering](#toc0_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "agglomerative_clustering(unique_tags, unique_tag_vectors, num_clusters=7)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}