Registration_Example.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Arkouda Registration Example\n",
    "In this example we will:\n",
    "- Create random pdarrays, Strings, Categoricals, SegArrays, and GroupBys\n",
    "- Register these objects\n",
    "- Verify their registration status using `ak.list_registry`, `.is_registered`, and `.info`\n",
    "- Convert `ak.information`'s JSON return string into a python object \n",
    "- Remove all non-registered objects from the symbol table using `ak.clear`\n",
    "- Disconnect from the arkouda server\n",
    "- Reconnect to the arkouda server\n",
    "- Attach to all registered objects\n",
    "- Unregister everything\n",
    "- Shutdown the server\n",
    "\n",
    "Arkouda functions used:\n",
    "- `ak.connect`\n",
    "- `ak.randint`\n",
    "- `ak.random_strings_uniform`\n",
    "- `ak.pdarray.register`\n",
    "- `ak.Strings.register`\n",
    "- `ak.Categorical.register`\n",
    "- `ak.SegArray.register`\n",
    "- `ak.GroupBy.register`\n",
    "- `ak.list_registry`\n",
    "- `ak.pdarray.is_registered`\n",
    "- `ak.Strings.is_registered`\n",
    "- `ak.Categorical.is_registered`\n",
    "- `ak.SegArray.is_registered`\n",
    "- `ak.GroupBy.is_registered`\n",
    "- `ak.Strings.info`\n",
    "- `ak.Strings.pretty_print_info`\n",
    "- `ak.infomation`\n",
    "- `ak.pretty_print_information`\n",
    "- `ak.clear`\n",
    "- `ak.disconnect`\n",
    "- `ak.attach`\n",
    "- `ak.GroupBy.attach`\n",
    "- `ak.pdarray.unregister`\n",
    "- `ak.Strings.unregister`\n",
    "- `ak.Categorical.unregister`\n",
    "- `ak.SegArray.unregister`\n",
    "- `ak.GroupBy.unregister`\n",
    "- `ak.shutdown`\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Launching and Connecting to Arkounda Server\n",
    "Be sure to follow the installation instructions on the [Arkouda README](https://github.com/mhmerrill/arkouda#readme) before running this notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import arkouda as ak"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The arkouda server must be running before connecting with `ak.connect`. The server can be launched by navigating to the arkouda directory and running `./arkouda_server -nl 1`. For more info, refer to [Running arkouda_server](https://github.com/mhmerrill/arkouda#running-arkouda_server-toc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# connect to the arkouda server using the connect_url which the server prints out\n",
    "ak.connect(connect_url=\"tcp://localhost:5555\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Intializing and Registering Variables\n",
    "We create random `pdarray`, `Strings`, `Categorical`, `SegArray`, and `GroupBy` objects and register them using in place `.register` functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pda1 = ak.randint(0,10,100)\n",
    "pda2 = ak.randint(0,10,100)\n",
    "str1 = ak.random_strings_uniform(2, 5, 100)\n",
    "str2 = ak.random_strings_uniform(2, 5, 100)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`Categorical` is built from `Strings` objects, so here we're using our `Strings` objects to create our `Categorical` objects"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat1 = ak.Categorical(str1)\n",
    "cat2 = ak.Categorical(str2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`SegArray` is a segmented array built from multiple `pdarray`s of varying length"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# First, we generate three lists that are then flattened into a single list\n",
    "a = [10, 11, 12, 13, 14, 15]\n",
    "b = [20, 21]\n",
    "c = [30, 31, 32, 33]\n",
    "flat = a + b + c\n",
    "\n",
    "# Then, we turn this flattened list into a pdarray\n",
    "akflat = ak.array(flat)\n",
    "\n",
    "# Next, we create a pdarray for the segments, which can be thought of as the starting index in the flattened array for each of the three intial lists\n",
    "# In this case, 0 is the start of the first segment, list a; the length of list a, 6, is the start of the second segment, list b; and the combined length\n",
    "# of lists a and b, 8, is the starting index of list c\n",
    "segments = ak.array([0, len(a), len(a)+len(b)])\n",
    "\n",
    "# Finally, we create our SegArry using the segments and akflat pdarrays\n",
    "segarr = ak.SegArray(segments, akflat)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`GroupBy` is a grouping object that can group together multiple `groupable` objects (`pdarray`, `Strings`, and `Categorical`) or a list of these types. Each component can have multiple registered entries, so to keep things simple we will create two `GroupBy`s using a `pdarray` and `Strings` object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "group1 = ak.GroupBy(ak.randint(0,10,100))\n",
    "group2 = ak.GroupBy(str1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pda1.register('pda1')\n",
    "str1.register('str1')\n",
    "cat1.register('cat1')\n",
    "segarr.register('segarr')\n",
    "group1.register('group1')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Verifying Registration and Using Information Methods\n",
    "We have just registered `pdarray pda1`, `String str1`, `Categorical cat1`, `SegArray segarr`, and `GroupBy group1`. Note that `pdarray pda2`, `String str2`, `Categorical cat2`, and `GroupBy group2` have not been registered. We can verify this using `ak.list_registry`, `.is_registered`, or `.info`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ak.list_registry returns a python list of all registered object names\n",
    "print(ak.list_registry())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that `SegArray` does not currently have a `.is_registered` method, so it is not included here"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Class level .is_registered() returns a boolean indicating the object's registration status\n",
    "print(f'pda1 is registered: {pda1.is_registered()}')\n",
    "print(f'pda2 is registered: {pda2.is_registered()}')\n",
    "print(f'str1 is registered: {str1.is_registered()}')\n",
    "print(f'str2 is registered: {str2.is_registered()}')\n",
    "print(f'cat1 is registered: {cat1.is_registered()}')\n",
    "print(f'cat2 is registered: {cat2.is_registered()}')\n",
    "print(f'group1 is registered: {group1.is_registered()}')\n",
    "print(f'group2 is registered: {group2.is_registered()}')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Class level .info returns all attributes of an object in a JSON formatted string\n",
    "\n",
    "print(\"str1.info():\")\n",
    "print(str1.info())\n",
    "\n",
    "print(\"cat1.info():\")\n",
    "print(cat1.info())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Class level .pretty_print_info returns all the attributes from .info in human readable form\n",
    "\n",
    "print(\"str1.pretty_print_info():\")\n",
    "str1.pretty_print_info()\n",
    "\n",
    "print(\"cat1.pretty_print_info():\")\n",
    "print(cat1.pretty_print_info())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using `ak.information` and `ak.pretty_print_information` it's easy to get attributes of all objects in the registry/symbol table"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# ak.information returns all attributes of an object in a JSON formatted string\n",
    "# ak.information can be called with a single object, all registered objects, or all objects\n",
    "\n",
    "print(\"ak.information('registered_object_name'):\")\n",
    "print(ak.information('pda1'))\n",
    "\n",
    "print('\\nak.information(ak.RegisteredSymbols):')\n",
    "print(ak.information(ak.RegisteredSymbols))\n",
    "\n",
    "print('\\nak.information(ak.AllSymbols):')\n",
    "print(ak.information(ak.AllSymbols))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# all the same arguments can be passed into ak.pretty_print_information for human readable output\n",
    "print(\"ak.pretty_print_information('registered_object_name'):\")\n",
    "ak.pretty_print_information('pda1')\n",
    "\n",
    "print('\\nak.pretty_print_information(ak.RegisteredSymbols):')\n",
    "ak.pretty_print_information(ak.RegisteredSymbols)\n",
    "\n",
    "print('\\nak.pretty_print_information(ak.AllSymbols):')\n",
    "ak.pretty_print_information(ak.AllSymbols)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see `ak.pretty_print_information(ak.RegisteredSymbols)` only contains references to objects related to `pda1` and `str1`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The JSON formmated string that's returned by `ak.information` and class level `.info` functions can be turned into a list of dictionaries in python using `json.loads` from the `json` library"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The JSON output of ak.information and .info can be parsed into python using the JSON library\n",
    "import json\n",
    "\n",
    "uint8_list = [symbol for symbol in json.loads(ak.information(ak.AllSymbols)) if symbol['dtype'] == 'uint8']\n",
    "for sym in uint8_list:\n",
    "    print(sym)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`ak.clear()` removes all non-registered objects from the symbol table, so `ak.pretty_print_information(ak.AllSymbols)` is different after a clear"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Before clear:')\n",
    "ak.pretty_print_information(ak.AllSymbols)\n",
    "\n",
    "ak.clear()\n",
    "\n",
    "print('\\nAfter clear:')\n",
    "ak.pretty_print_information(ak.AllSymbols)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Disconnecting from Arkouda Server and Attaching Registered Objects\n",
    "Users can `attach` to objects registered with the server after a disconnect. This enables access to registered objects even if the original python object is lost  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We are going to simulate the python client dying. To do this, we set references for `pda1` and `str1` to `None` and disconnect from the server"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pda1 = None\n",
    "str1 = None\n",
    "cat1 = None\n",
    "segarr = None\n",
    "group1 = None"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ak.disconnect()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we come back and reconnect to the server. We attempt to access `pda1` and `str1` but no longer have the objects in python"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# connect to the arkouda server using the connect_url which the server prints out\n",
    "ak.connect(\"localhost\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f'pda1:{pda1}')\n",
    "print(f'str1:{str1}')\n",
    "print(f'cat1:{cat1}')\n",
    "print(f'segarr:{segarr}')\n",
    "print(f'group1:{group1}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "However the server still has these objects registered"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('After reconnect to server:')\n",
    "# ak.pretty_print_information and ak.information without arguments defaults to ak.RegisteredSymbols\n",
    "ak.pretty_print_information()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We want to attach to these objects on the server to regain access to them"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pda1 = ak.util.attach('pda1')\n",
    "str1 = ak.util.attach('str1')\n",
    "cat1 = ak.util.attach('cat1', 'categorical')\n",
    "segarr = ak.util.attach('segarr', 'segarray')\n",
    "group1 = ak.GroupBy.attach('group1')\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(f'pda1:{pda1}')\n",
    "print(f'str1:{str1}')\n",
    "print(f'cat1:{cat1}')\n",
    "print(f'segarr:{segarr}')\n",
    "print(f'group1:{group1}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We restored our access to `pda1`, `str1`, `cat1`, `segarr`, and `group1` using the `attach` functionality. Now we unregister everything and shutdown the arkouda server "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pda1.unregister()\n",
    "str1.unregister()\n",
    "cat1.unregister()\n",
    "segarr.unregister()\n",
    "group1.unregister()\n",
    "ak.clear()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(ak.information(ak.AllSymbols))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ak.shutdown()"
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "602ef4e43bc5e522b9fa8254db2ce3bcd48b16c5cd430020a77bd45828b6d245"
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}