DataFrame_Demo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Demonstration of Arkouda DataFrame vs Pandas DataFrame\n",
    "Arkouda DataFrame should maintain the same functionality of pandas dataframes\n",
    "\n",
    "Functionality Currently Supported by Both \n",
    "- Drop\n",
    "- Index Reset\n",
    "- Renaming Columns\n",
    "- Head/Tail Functionality\n",
    "- Groupby\n",
    "- Sorting\n",
    "- Permutations\n",
    "\n",
    "Only supported by Arkouda\n",
    "- Filter by Range\n",
    "\n",
    "Pandas Features to add to Arkouda\n",
    "- Missing Groupby Functionality\n",
    "    - First\n",
    "    - Access Group By Name\n",
    "\n",
    "- Reset Index\n",
    "    - Track original indexes\n",
    "\n",
    "- Sorting\n",
    "    - Sort on Rows\n",
    "    - inplace argument - allowing for user to specify if action should update the calling object or return a new object\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import arkouda as ak\n",
    "ak.connect(connect_url=\"tcp://localhost:5555\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "from IPython.display import display, clear_output\n",
    "from ipywidgets import HTML, HBox, VBox, Output"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def reset_ak_df():\n",
    "    username = ak.array(['Alice', 'Bob', 'Alice', 'Carol', 'Bob', 'Alice'])\n",
    "    userid = ak.array([111, 222, 111, 333, 222, 111])\n",
    "    item = ak.array([0, 0, 1, 1, 2, 0])\n",
    "    day = ak.array([5, 5, 6, 5, 6, 6])\n",
    "    amount = ak.array([0.5, 0.6, 1.1, 1.2, 4.3, 0.6])\n",
    "    df = ak.DataFrame({'userName': username, 'userID': userid,\n",
    "                    'item': item, 'day': day, 'amount': amount})\n",
    "    return df\n",
    "\n",
    "def reset_pd_df():\n",
    "    username_pd = ['Alice', 'Bob', 'Alice', 'Carol', 'Bob', 'Alice']\n",
    "    userid_pd = [111, 222, 111, 333, 222, 111]\n",
    "    item_pd = [0, 0, 1, 1, 2, 0]\n",
    "    day_pd = [5, 5, 6, 5, 6, 6]\n",
    "    amount_pd = [0.5, 0.6, 1.1, 1.2, 4.3, 0.6]\n",
    "    pd_df = pd.DataFrame({'userName': username_pd, 'userID': userid_pd,\n",
    "                    'item': item_pd, 'day': day_pd, 'amount': amount_pd})\n",
    "    return pd_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create a DataFrame"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Arkouda DataFrame\n",
    "df = reset_ak_df()\n",
    "\n",
    "#Pandas DataFrame\n",
    "pd_df = reset_pd_df()\n",
    "\n",
    "#it is easy to access the shape in pandas\n",
    "#display(pd_df.shape)\n",
    "\n",
    "#display configuration\n",
    "ak_create_out = Output()\n",
    "pd_create_out = Output()\n",
    "with ak_create_out:\n",
    "    display(df)\n",
    "with pd_create_out:\n",
    "    display(pd_df)\n",
    "    display(pd_df.shape)\n",
    "ak_box = VBox([HTML(\"<h3>Arkouda</h3>\"), ak_create_out])\n",
    "pd_box = VBox([HTML(\"<h3>Pandas</h3>\"), pd_create_out])\n",
    "\n",
    "container = HBox([ak_box, pd_box])\n",
    "container.layout.display = 'flex'\n",
    "container.layout.justify_content = 'space-around'\n",
    "\n",
    "html = '''\n",
    "    <h1>Create DataFrame</h1>\n",
    "    <p>Creation of ak.DataFrame and pd.DataFrame is very similar. It is worth noting that Arkouda will automatically supply the shape when displaying, but Pandas does not. It is easy enough to view the shape in Pandas using DataFrame.shape. This is done below to show the same functionality for both.</p>\n",
    "'''\n",
    "\n",
    "parent_container = VBox([HTML(html), container])\n",
    "display(parent_container)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Drop Functionality"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Drop entire column - arkouda\n",
    "df.drop('userName', axis=1, inplace=True)\n",
    "\n",
    "#drop entire column - pandas\n",
    "pd_df.drop('userName', axis=1, inplace=True)\n",
    "\n",
    "#Display\n",
    "ak_drop_output = Output()\n",
    "pd_drop_output = Output()\n",
    "\n",
    "with ak_drop_output:\n",
    "    display(df)\n",
    "with pd_drop_output:\n",
    "    display(pd_df)\n",
    "    display(pd_df.shape)\n",
    "\n",
    "ak_drop_info = HTML('''\n",
    "    <h3>Arkouda</h3>\n",
    "    <ul>\n",
    "        <li>The \"index\" column cannot be droped</li>\n",
    "    </ul>\n",
    "''')\n",
    "\n",
    "pd_drop_info = HTML('''\n",
    "    <h3>Pandas</h3>\n",
    "''')\n",
    "\n",
    "ak_box = VBox([ak_drop_info, ak_drop_output])\n",
    "pd_box = VBox([pd_drop_info, pd_drop_output])\n",
    "\n",
    "container = HBox([ak_box, pd_box])\n",
    "container.layout.display = 'flex'\n",
    "container.layout.justify_content = 'space-around'\n",
    "\n",
    "html = '''\n",
    "    <h1>Drop Functionality</h1>\n",
    "    <p>In both examples below, the userName column has been dropped.</p>\n",
    "    <p>The axis parameter determines if we drop rows or columns. 0 - drop rows, 1 - drop columns. Defaults to 0.</p>\n",
    "    <p>The inplace parameter determines if the action is applied to the calling object or if the result returned as a new object.</p>\n",
    "'''\n",
    "\n",
    "parent_container = VBox([HTML(html), container])\n",
    "display(parent_container)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#create dataframe with duplicate rows - Arkouda\n",
    "username = ak.array(['Alice', 'Bob', 'Alice', 'Carol', 'Bob', 'Alice'])\n",
    "userid = ak.array([111, 222, 111, 333, 222, 111])\n",
    "item = ak.array([0, 1, 0, 2, 1, 0])\n",
    "day = ak.array([5, 5, 5, 5, 5, 5])\n",
    "df = ak.DataFrame({'userName': username, 'userID': userid,\n",
    "                    'item': item, 'day': day})\n",
    "\n",
    "#deduplicate the dataset - Arkouda\n",
    "dedup = df.drop_duplicates()\n",
    "\n",
    "#create DataFrame with duplicate rows - pandas\n",
    "username_pd = ['Alice', 'Bob', 'Alice', 'Carol', 'Bob', 'Alice']\n",
    "userid_pd = [111, 222, 111, 333, 222, 111]\n",
    "item_pd = [0, 1, 0, 2, 1, 0]\n",
    "day_pd = [5, 5, 5, 5, 5, 5]\n",
    "pd_df = pd.DataFrame({'userName': username_pd, 'userID': userid_pd,\n",
    "                    'item': item_pd, 'day': day_pd})\n",
    "\n",
    "dedup_pd = pd_df.drop_duplicates()\n",
    "\n",
    "#Configure display\n",
    "ak_dedup_before = Output()\n",
    "ak_dedup_after = Output()\n",
    "pd_dedup_before = Output()\n",
    "pd_dedup_after = Output()\n",
    "\n",
    "with ak_dedup_before:\n",
    "    display(df)\n",
    "with ak_dedup_after:\n",
    "    display(dedup)\n",
    "\n",
    "ak_dedup_compare = HBox([VBox([HTML(\"<h4>Before</h4>\"), ak_dedup_before]), VBox([HTML(\"<h4>After</h4>\"), ak_dedup_after])])\n",
    "ak_dedup_info = '''\n",
    "    <h3>Arkouda</h3>\n",
    "'''\n",
    "\n",
    "with pd_dedup_before:\n",
    "    display(pd_df)\n",
    "    display(pd_df.shape)\n",
    "with pd_dedup_after:\n",
    "    display(dedup_pd)\n",
    "    display(dedup_pd.shape)\n",
    "\n",
    "pd_dedup_compare = HBox([VBox([HTML(\"<h4>Before</h4>\"), pd_dedup_before]), VBox([HTML(\"<h4>After</h4>\"), pd_dedup_after])])\n",
    "pd_dedup_info = '''\n",
    "    <h3>Pandas</h3>\n",
    "'''\n",
    "\n",
    "ak_box = VBox([HTML(ak_dedup_info), ak_dedup_compare])\n",
    "pd_box = VBox([HTML(pd_dedup_info), pd_dedup_compare])\n",
    "\n",
    "container = HBox([ak_box, pd_box])\n",
    "container.layout.display = 'flex'\n",
    "container.layout.justify_content = 'space-around'\n",
    "\n",
    "html = '''\n",
    "    <h1>De-duplication</h1>\n",
    "'''\n",
    "\n",
    "parent_container = VBox([HTML(html), container])\n",
    "clear_output()\n",
    "display(parent_container)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reset Index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#next line are for display purposes only\n",
    "ak_idx_before = Output()\n",
    "\n",
    "#taking a slice with keep indexes as they are \n",
    "slice_df = df[ak.array([5, 1, 3])]\n",
    "#display\n",
    "with ak_idx_before:\n",
    "    display(slice_df)\n",
    "\n",
    "#resetting the index will rebase at 0 - Arkouda\n",
    "ak_re_idx = slice_df.reset_index(inplace=False)\n",
    "\n",
    "#for pandas we reuse the dedup result from pandas in the previous step\n",
    "re_idx = dedup_pd.reset_index(inplace=False)\n",
    "\n",
    "\n",
    "#Configure display\n",
    "ak_idx_after = Output()\n",
    "pd_idx_before = Output()\n",
    "pd_idx_after = Output()\n",
    "with ak_idx_after:\n",
    "    display(ak_re_idx)\n",
    "\n",
    "ak_idx_compare = HBox([VBox([HTML(\"<h4>Before</h4>\"), ak_idx_before]), VBox([HTML(\"<h4>After</h4>\"), ak_idx_after])])\n",
    "ak_idx_info = '''\n",
    "    <h3>Arkouda</h3>\n",
    "    <ul>\n",
    "        <li>Arkouda will automatically reindex for you in some situations.</li>\n",
    "    </ul>\n",
    "'''\n",
    "\n",
    "with pd_idx_before:\n",
    "    display(dedup_pd)\n",
    "    display(dedup_pd.shape)\n",
    "with pd_idx_after:\n",
    "    display(re_idx)\n",
    "    display(re_idx.shape)\n",
    "\n",
    "pd_idx_compare = HBox([VBox([HTML(\"<h4>Before</h4>\"), pd_idx_before]), VBox([HTML(\"<h4>After</h4>\"), pd_idx_after])])\n",
    "pd_idx_info = '''\n",
    "    <h3>Pandas</h3>\n",
    "    <ul>\n",
    "        <li>Generates an additional column named \"index\" to store the original index</li>\n",
    "    </ul>\n",
    "'''\n",
    "\n",
    "ak_box = VBox([HTML(ak_idx_info), ak_idx_compare])\n",
    "pd_box = VBox([HTML(pd_idx_info), pd_idx_compare])\n",
    "\n",
    "container = HBox([ak_box, pd_box])\n",
    "container.layout.display = 'flex'\n",
    "container.layout.justify_content = 'space-around'\n",
    "\n",
    "html = '''\n",
    "    <h1>Reset Indexes</h1>\n",
    "    <p>Resetting the index rebases the indexes at 0. This is useful in situations when you are dealing with subsets of the original data.</p>\n",
    "'''\n",
    "\n",
    "parent_container = VBox([HTML(html), container])\n",
    "clear_output()\n",
    "display(parent_container)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Renaming Columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#next 3 lines are for display purposes only\n",
    "ak_rename_before = Output()\n",
    "\n",
    "df = reset_ak_df()\n",
    "\n",
    "with ak_rename_before:\n",
    "    display(df)\n",
    "\n",
    "#rename userName to user_name and userID to user_id - Arkouda\n",
    "renamed_ak = df.rename({'userName':'user_name', 'userID': 'user_id'}, axis=1, inplace=False)\n",
    "\n",
    "#reset back to original pandas dataframe\n",
    "pd_df = reset_pd_df()\n",
    "\n",
    "#rename columns - Pandas\n",
    "renamed_pd = pd_df.rename(columns={'userName':'user_name', 'userID':'user_id'}, inplace=False)\n",
    "\n",
    "#Configure display\n",
    "ak_rename_after = Output()\n",
    "pd_rename_before = Output()\n",
    "pd_rename_after = Output()\n",
    "with ak_rename_after:\n",
    "    display(renamed_ak)\n",
    "\n",
    "with pd_rename_before:\n",
    "     display(pd_df)\n",
    "     display(pd_df.shape)\n",
    "with pd_rename_after:\n",
    "    display(renamed_pd)\n",
    "    display(renamed_pd.shape)\n",
    "\n",
    "ak_rename_compare = HBox([VBox([HTML(\"<h4>Before</h4>\"), ak_rename_before]), VBox([HTML(\"<h4>After</h4>\"), ak_rename_after])])\n",
    "ak_rename_info = '''\n",
    "    <h3>Arkouda</h3>\n",
    "'''\n",
    "\n",
    "pd_rename_compare = HBox([VBox([HTML(\"<h4>Before</h4>\"), pd_rename_before]), VBox([HTML(\"<h4>After</h4>\"), pd_rename_after])])\n",
    "pd_rename_info = '''\n",
    "    <h3>Pandas</h3>\n",
    "'''\n",
    "\n",
    "ak_box = VBox([HTML(ak_rename_info), ak_rename_compare])\n",
    "pd_box = VBox([HTML(pd_rename_info), pd_rename_compare])\n",
    "\n",
    "container = HBox([ak_box, pd_box])\n",
    "container.layout.display = 'flex'\n",
    "container.layout.justify_content = 'space-around'\n",
    "\n",
    "html = '''\n",
    "    <h1>Column Renaming</h1>\n",
    "'''\n",
    "\n",
    "parent_container = VBox([HTML(html), container])\n",
    "clear_output()\n",
    "display(parent_container)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Head/Tail Functionality"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#compute head - arkouda\n",
    "df_head = renamed_ak.head(3)\n",
    "\n",
    "#compute head - pandas\n",
    "pd_head = renamed_pd.head(3)\n",
    "\n",
    "#Display\n",
    "ak_head_output = Output()\n",
    "pd_head_output = Output()\n",
    "\n",
    "with ak_head_output:\n",
    "    display(df_head)\n",
    "with pd_head_output:\n",
    "    display(pd_head)\n",
    "    display(pd_head.shape)\n",
    "\n",
    "ak_head_info = HTML('''\n",
    "    <h3>Arkouda</h3>\n",
    "''')\n",
    "\n",
    "pd_head_info = HTML('''\n",
    "    <h3>Pandas</h3>\n",
    "''')\n",
    "\n",
    "ak_box = VBox([ak_head_info, ak_head_output])\n",
    "pd_box = VBox([pd_head_info, pd_head_output])\n",
    "\n",
    "container = HBox([ak_box, pd_box])\n",
    "container.layout.display = 'flex'\n",
    "container.layout.justify_content = 'space-around'\n",
    "\n",
    "html = '''\n",
    "    <h1>Head of Dataset</h1>\n",
    "'''\n",
    "\n",
    "parent_container = VBox([HTML(html), container])\n",
    "display(parent_container)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#tail - Arkouda\n",
    "df_tail = renamed_ak.tail(2)\n",
    "\n",
    "#tail - pandas\n",
    "pd_tail = renamed_pd.tail(2)\n",
    "\n",
    "#Configure Display\n",
    "ak_tail_output = Output()\n",
    "pd_tail_output = Output()\n",
    "\n",
    "with ak_tail_output:\n",
    "    display(df_tail)\n",
    "with pd_tail_output:\n",
    "    display(pd_tail)\n",
    "    display(pd_tail.shape)\n",
    "\n",
    "ak_tail_info = HTML('''\n",
    "    <h3>Arkouda</h3>\n",
    "''')\n",
    "\n",
    "pd_tail_info = HTML('''\n",
    "    <h3>Pandas</h3>\n",
    "''')\n",
    "\n",
    "ak_box = VBox([ak_tail_info, ak_tail_output])\n",
    "pd_box = VBox([pd_tail_info, pd_tail_output])\n",
    "\n",
    "container = HBox([ak_box, pd_box])\n",
    "container.layout.display = 'flex'\n",
    "container.layout.justify_content = 'space-around'\n",
    "\n",
    "html = '''\n",
    "    <h1>Tail of Dataset</h1>\n",
    "'''\n",
    "\n",
    "parent_container = VBox([HTML(html), container])\n",
    "display(parent_container)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## GroupBy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Single Column Group By - Arkouda\n",
    "gb = renamed_ak.GroupBy('user_name')\n",
    "keys, count = gb.count()\n",
    "\n",
    "#Single Column Group By - Pandas \n",
    "pd_gb = renamed_pd.groupby('user_name')\n",
    "\n",
    "#Configure Display\n",
    "ak_gb_output = Output()\n",
    "pd_gb_output = Output()\n",
    "\n",
    "with ak_gb_output:\n",
    "    display(HTML(f'<b>GroupBy Keys:</b> {keys}'))\n",
    "    display(HTML(f'<b>GroupBy Counts:</b> {count}'))\n",
    "with pd_gb_output:\n",
    "    display(pd_gb.first())\n",
    "\n",
    "ak_gb_info = HTML('''\n",
    "    <h3>Arkouda</h3>\n",
    "    <ul>\n",
    "        <li>Stores groupings as 2 lists. The first is the value from the column grouped on; this is the key. The second is the number of records with the associated key. These two lists are configured so that the indexes in each array correspond, ie keys[i] is the key with x number of records, where x is the value in count[i].</li>\n",
    "        <li>Currently, there is no visual representation equivalent to that of the pandas module.</li>\n",
    "    </ul>\n",
    "''')\n",
    "\n",
    "pd_gb_info = HTML('''\n",
    "    <h3>Pandas</h3>\n",
    "    <ul>\n",
    "        <li>Data is able to be viewed in a format that represents a DataFrame.</li>\n",
    "        <li>More robust access options: first(), last(), get_group()</li>\n",
    "        <li>Options to keep a specific occurence of the record.</li>\n",
    "    </ul>\n",
    "''')\n",
    "\n",
    "ak_box = VBox([ak_gb_info, ak_gb_output])\n",
    "pd_box = VBox([pd_gb_info, pd_gb_output])\n",
    "ak_box.layout.max_width = '50%'\n",
    "\n",
    "container = HBox([ak_box, pd_box])\n",
    "container.layout.display = 'flex'\n",
    "container.layout.justify_content = 'space-around'\n",
    "\n",
    "html = '''\n",
    "    <h1>GroupBy - Single Column</h1>\n",
    "'''\n",
    "\n",
    "parent_container = VBox([HTML(html), container])\n",
    "clear_output()\n",
    "display(parent_container)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Multiple Column Grouping - Arkouda\n",
    "gb = renamed_ak.GroupBy(['user_name', 'day'])\n",
    "keys, count = gb.count()\n",
    "\n",
    "#Multiple Column Grouping - Pandas\n",
    "pd_gb = renamed_pd.groupby(['user_name', 'day'])\n",
    "\n",
    "#Configure Display\n",
    "ak_gb_output = Output()\n",
    "pd_gb_output = Output()\n",
    "\n",
    "with ak_gb_output:\n",
    "    display(HTML(f'<b>GroupBy Keys:</b> {keys}'))\n",
    "    display(HTML(f'<b>GroupBy Counts:</b> {count}'))\n",
    "with pd_gb_output:\n",
    "    display(pd_gb.first())\n",
    "\n",
    "ak_gb_info = HTML('''\n",
    "    <h3>Arkouda</h3>\n",
    "    <ul>\n",
    "        <li>Here, the keys list will return a list of ak.pdarray. Each of these arrays contain the keys for the columns specified in the groupby call. Think of this as a tuple for your key - [array1, array2, ..., arrayN], the key at index 0 is equivalent to (array1[0], array2[0[, ..., arrayN[0])</li>\n",
    "    </ul>\n",
    "''')\n",
    "\n",
    "pd_gb_info = HTML('''\n",
    "    <h3>Pandas</h3>\n",
    "''')\n",
    "\n",
    "ak_box = VBox([ak_gb_info, ak_gb_output])\n",
    "pd_box = VBox([pd_gb_info, pd_gb_output])\n",
    "ak_box.layout.max_width = '50%'\n",
    "\n",
    "container = HBox([ak_box, pd_box])\n",
    "container.layout.display = 'flex'\n",
    "container.layout.justify_content = 'space-around'\n",
    "\n",
    "html = '''\n",
    "    <h1>GroupBy - Multiple Columns</h1>\n",
    "    <p>This works the same as a single column, but the result is different in Arkouda</p>\n",
    "'''\n",
    "\n",
    "parent_container = VBox([HTML(html), container])\n",
    "clear_output()\n",
    "display(parent_container)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Sorting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#arg/coarg sorting in Arkouda\n",
    "arg_s = renamed_ak.argsort(key='user_name')\n",
    "\n",
    "coarg_s = renamed_ak.coargsort(keys=['user_id', 'amount'])\n",
    "\n",
    "ak_arg_output = Output()\n",
    "with ak_arg_output:\n",
    "    display(HTML(f'<b>Arg Sort Permutation:</b> {arg_s}'))\n",
    "    display(HTML(f'<b>Coarg Sort Permutation:</b> {coarg_s}'))\n",
    "\n",
    "html = HTML('''\n",
    "    <h1>Arkouda Arg/Coarg Sorting</h1>\n",
    "    <p>Returns the permutation of indices that would sort the dataframe.<p>\n",
    "    <p><b>Coargsort only guaranteed grouping for string values, not lexicographical sorting.<b></p>\n",
    "''')\n",
    "\n",
    "parent_container = VBox([html, ak_arg_output])\n",
    "clear_output()\n",
    "display(parent_container)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Sort values - arkouda\n",
    "s = renamed_ak.sort_values(by='user_id')\n",
    "s_mult = renamed_ak.sort_values(by=['user_id', 'amount'])\n",
    "\n",
    "#sort values pandas\n",
    "s_pd = renamed_pd.sort_values(by=['user_id'])\n",
    "s_pd_mult = renamed_pd.sort_values(by=['user_id', 'amount'])\n",
    "\n",
    "ak_sort_output = Output()\n",
    "pd_sort_output = Output()\n",
    "\n",
    "with ak_sort_output:\n",
    "    display(HTML('<b>Single Column</b>'), s)\n",
    "    display(HTML('<b>Multiple Column</b>'), s_mult)\n",
    "\n",
    "with pd_sort_output:\n",
    "    display(HTML('<b>Single Column</b>'), s_pd, s_pd.shape)\n",
    "    display(HTML('<b>Multiple Column</b>'), s_pd_mult, s_pd_mult.shape)\n",
    "\n",
    "ak_sort_info = HTML('''\n",
    "    <h3>Arkouda</h3>\n",
    "    <ul>\n",
    "        <li>This functionality relies on argsort and coargsort. However, it goes one step further in applying the permutation to the DataFrame allowing for visualization similar to pandas.</li>\n",
    "    </ul>\n",
    "''')\n",
    "\n",
    "pd_sort_info = HTML('''\n",
    "    <h3>Pandas</h3>\n",
    "    <ul>\n",
    "        <li>Sorting can be done on both the rows and columns. Use the axis parameter to specify.</li>\n",
    "    </ul>\n",
    "''')\n",
    "\n",
    "ak_box = VBox([ak_sort_info, ak_sort_output])\n",
    "pd_box = VBox([pd_sort_info, pd_sort_output])\n",
    "ak_box.layout.max_width = '50%'\n",
    "\n",
    "container = HBox([ak_box, pd_box])\n",
    "container.layout.display = 'flex'\n",
    "container.layout.justify_content = 'space-around'\n",
    "\n",
    "html = '''\n",
    "    <h1>Sorting</h1>\n",
    "    <p>This functionality automatically selects to use argsort or coargsort.</p>\n",
    "    <p>Single column sorting performed on the user_id column.</p>\n",
    "    <p>Multiple column sorting is performed on the user_id and amount columns.</p>\n",
    "'''\n",
    "\n",
    "parent_container = VBox([HTML(html), container])\n",
    "clear_output()\n",
    "display(parent_container)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Permutations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#apply permutaion - Arkouda\n",
    "df.apply_permutation(ak.array([1, 0, 2, 3, 4, 5]))\n",
    "\n",
    "#apply permutation - Pandas\n",
    "perm = renamed_pd.reindex([1, 0, 2, 3, 4, 5])\n",
    "\n",
    "ak_perm_output = Output()\n",
    "pd_perm_output = Output()\n",
    "\n",
    "with ak_perm_output:\n",
    "    display(df)\n",
    "\n",
    "with pd_perm_output:\n",
    "    display(perm)\n",
    "\n",
    "ak_perm_info = HTML('''\n",
    "    <h3>Arkouda</h3>\n",
    "''')\n",
    "\n",
    "pd_perm_info = HTML('''\n",
    "    <h3>Pandas</h3>\n",
    "    <ul>\n",
    "        <li>Can perform on the calling object or return a new object with the permutation applied.</li>\n",
    "    </ul>\n",
    "''')\n",
    "\n",
    "ak_box = VBox([ak_perm_info, ak_perm_output])\n",
    "pd_box = VBox([pd_perm_info, pd_perm_output])\n",
    "ak_box.layout.max_width = '50%'\n",
    "\n",
    "container = HBox([ak_box, pd_box])\n",
    "container.layout.display = 'flex'\n",
    "container.layout.justify_content = 'space-around'\n",
    "\n",
    "html = '''\n",
    "    <h1>Permutations</h1>\n",
    "    <p>Order the data based on the order of the indexes supplied. Pandas refers to this a 'reindexing'.</p>\n",
    "'''\n",
    "\n",
    "parent_container = VBox([HTML(html), container])\n",
    "clear_output()\n",
    "display(parent_container)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Filter By Range - Arkouda\n",
    "- Filter on the count of the groupby. Returns boolean pdarray "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gb = renamed_ak.GroupBy('user_id')\n",
    "keys, count = gb.count()\n",
    "filter = renamed_ak.filter_by_range(keys=['user_id'], low=1, high=2)\n",
    "\n",
    "ak_filter_output = Output()\n",
    "with ak_filter_output:\n",
    "    display(HTML(f'<b>GroupBy Keys:</b> {keys}'))\n",
    "    display(HTML(f'<b>GroupBy Counts:</b> {count}'))\n",
    "    display(HTML(f'<b>Filter Results:</b> {filter}'))\n",
    "\n",
    "html = HTML('''\n",
    "    <h1>Arkouda Filter By Range</h1>\n",
    "    <p>Run groupby on the keys specified and return True/False if the corresponding count falls in the range specified. Returns a list of booleans indicating the status of the key at each index.<p>\n",
    "''')\n",
    "\n",
    "parent_container = VBox([html, ak_filter_output])\n",
    "clear_output()\n",
    "display(parent_container)"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "c9843ec7bf92a92e5648c61892a56b72546cb6f7f783cc4fa72865bc7befe91a"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}