JavaZone2022/Delta-Demo.ipynb

{
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Data Lakes on Acid - Delta Demo\r\n",
        "![javazone](https://datalandsbyen.norge.no/assets/uploads/files/1654846045873-9019c0fa-4dfb-4849-b897-6b31a2679ed3-image.png)"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "data_path = 'abfss://home@stjz2022demo.dfs.core.windows.net/oslobysykkel'"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.livy.statement-meta+json": {
              "spark_pool": "pool01",
              "session_id": "10",
              "statement_id": 2,
              "state": "finished",
              "livy_statement_state": "available",
              "queued_time": "2022-09-08T07:43:22.9099133Z",
              "session_start_time": "2022-09-08T07:43:22.9527215Z",
              "execution_start_time": "2022-09-08T07:46:58.4668175Z",
              "execution_finish_time": "2022-09-08T07:46:58.6170955Z",
              "spark_jobs": null
            },
            "text/plain": "StatementMeta(pool01, 10, 2, Finished, Available)"
          },
          "metadata": {}
        }
      ],
      "execution_count": 1,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# API V1 - 2016 - 2018\r\n",
        "Here we read trip data from the \"v1\" API, covering the years from 2016 - 2018.\r\n",
        "\r\n",
        "The schema here is very simple, with only 4 fields."
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "v1Data = (spark.read\r\n",
        "        .format(\"csv\")\r\n",
        "        .option(\"header\", \"true\")\r\n",
        "        .option(\"inferSchema\", \"true\")\r\n",
        "        .load(f\"{data_path}/raw/csv/v1/*.csv\")\r\n",
        ")\r\n",
        "\r\n",
        "print(f\"Total number of records: {v1Data.count()}\")\r\n",
        "print(v1Data.printSchema())\r\n",
        "display(v1Data.limit(5))"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.livy.statement-meta+json": {
              "spark_pool": "pool01",
              "session_id": "10",
              "statement_id": 3,
              "state": "finished",
              "livy_statement_state": "available",
              "queued_time": "2022-09-08T07:43:22.911536Z",
              "session_start_time": null,
              "execution_start_time": "2022-09-08T07:46:58.7104933Z",
              "execution_finish_time": "2022-09-08T07:47:46.2388006Z",
              "spark_jobs": null
            },
            "text/plain": "StatementMeta(pool01, 10, 3, Finished, Available)"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Total number of records: 7747817\nroot\n |-- Start station: integer (nullable = true)\n |-- Start time: string (nullable = true)\n |-- End station: integer (nullable = true)\n |-- End time: string (nullable = true)\n\nNone\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.synapse.widget-view+json": {
              "widget_id": "a08e9a3c-749a-4ccf-b38f-895fb5d0ca3c",
              "widget_type": "Synapse.DataFrame"
            },
            "text/plain": "SynapseWidget(Synapse.DataFrame, a08e9a3c-749a-4ccf-b38f-895fb5d0ca3c)"
          },
          "metadata": {}
        }
      ],
      "execution_count": 2,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "collapsed": false
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Legacy Station Ids\r\n",
        "\r\n",
        "The station_ids used in the v1 API have been updated, so we need to map the legacy id to the new id values.\r\n",
        "\r\n",
        "First we'll inspect the \"dimension table\" that provides the mapping."
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "stationIdDimDf = (spark.read\r\n",
        "    .format(\"csv\")\r\n",
        "    .option(\"header\", \"true\")\r\n",
        "    .option(\"inferSchema\", \"true\")\r\n",
        "    .load(f\"{data_path}/raw/csv/legacy_new_station_id_mapping.csv\")\r\n",
        ")\r\n",
        "display(stationIdDimDf.limit(5))"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.livy.statement-meta+json": {
              "spark_pool": "pool01",
              "session_id": "10",
              "statement_id": 4,
              "state": "finished",
              "livy_statement_state": "available",
              "queued_time": "2022-09-08T07:43:22.913073Z",
              "session_start_time": null,
              "execution_start_time": "2022-09-08T07:47:46.3351439Z",
              "execution_finish_time": "2022-09-08T07:47:48.1858632Z",
              "spark_jobs": null
            },
            "text/plain": "StatementMeta(pool01, 10, 4, Finished, Available)"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.synapse.widget-view+json": {
              "widget_id": "4acd3209-b1a8-4a62-ad2d-658231901889",
              "widget_type": "Synapse.DataFrame"
            },
            "text/plain": "SynapseWidget(Synapse.DataFrame, 4acd3209-b1a8-4a62-ad2d-658231901889)"
          },
          "metadata": {}
        }
      ],
      "execution_count": 3,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "collapsed": false
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Map legacy id to a current id\r\n",
        "\r\n",
        "Now we will:\r\n",
        "1. update the column names to lower_snake_case\r\n",
        "2. add new columns for the year and the month from the start date, so that we can use this for logical partitioning\r\n",
        "3. add a new column for the API version (\"1\")\r\n",
        "4. join the v1 data with the dimension table for mapping both start_station_id and end_station_id\r\n"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import pyspark.sql.functions as F\r\n",
        "\r\n",
        "v1Data = (v1Data\r\n",
        "    .withColumnRenamed(\"Start station\", \"start_station_id\")\r\n",
        "    .withColumnRenamed(\"End station\", \"end_station_id\")\r\n",
        "    .withColumnRenamed(\"Start time\", \"started_at\")\r\n",
        "    .withColumnRenamed(\"End time\", \"ended_at\")\r\n",
        "    .withColumn(\"start_year\", F.year(\"started_at\"))\r\n",
        "    .withColumn(\"start_month\", F.month(\"started_at\"))\r\n",
        "    .withColumn(\"api_version\", F.lit(1))\r\n",
        ")\r\n",
        "\r\n",
        "v1Data = (v1Data.join(stationIdDimDf, v1Data.start_station_id ==  stationIdDimDf.legacy_id, \"inner\")\r\n",
        "    .drop(\"start_station_id\", \"legacy_id\")\r\n",
        "    .withColumnRenamed(\"new_id\", \"start_station_id\")\r\n",
        "    .join(stationIdDimDf, v1Data.end_station_id ==  stationIdDimDf.legacy_id, \"inner\")\r\n",
        "    .drop(\"end_station_id\", \"legacy_id\")\r\n",
        "    .withColumnRenamed(\"new_id\", \"end_station_id\"))\r\n",
        "\r\n",
        "display(v1Data.limit(5))"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.livy.statement-meta+json": {
              "spark_pool": "pool01",
              "session_id": "10",
              "statement_id": 5,
              "state": "finished",
              "livy_statement_state": "available",
              "queued_time": "2022-09-08T07:43:22.9145332Z",
              "session_start_time": null,
              "execution_start_time": "2022-09-08T07:47:48.2874238Z",
              "execution_finish_time": "2022-09-08T07:48:07.050637Z",
              "spark_jobs": null
            },
            "text/plain": "StatementMeta(pool01, 10, 5, Finished, Available)"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.synapse.widget-view+json": {
              "widget_id": "6818c452-136a-4f43-bd83-242655307f81",
              "widget_type": "Synapse.DataFrame"
            },
            "text/plain": "SynapseWidget(Synapse.DataFrame, 6818c452-136a-4f43-bd83-242655307f81)"
          },
          "metadata": {}
        }
      ],
      "execution_count": 4,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "collapsed": false
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Write to Delta Table\r\n",
        "\r\n",
        "Now we'll write the data to a Delta table!\r\n",
        "\r\n",
        "Note that we can tell Spark to automatically partition the data\r\n",
        "by `start_year` and `start_month`."
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "(v1Data.write\r\n",
        "    .mode(\"append\")\r\n",
        "    .partitionBy(\"start_year\", \"start_month\")\r\n",
        "    .format(\"delta\")\r\n",
        "    .save(f\"{data_path}/processed/delta\")\r\n",
        ")"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.livy.statement-meta+json": {
              "spark_pool": "pool01",
              "session_id": "10",
              "statement_id": 6,
              "state": "finished",
              "livy_statement_state": "available",
              "queued_time": "2022-09-08T07:43:22.9162151Z",
              "session_start_time": null,
              "execution_start_time": "2022-09-08T07:48:07.1408715Z",
              "execution_finish_time": "2022-09-08T07:48:30.0605462Z",
              "spark_jobs": null
            },
            "text/plain": "StatementMeta(pool01, 10, 6, Finished, Available)"
          },
          "metadata": {}
        }
      ],
      "execution_count": 5,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# API V2 (2019 - 2022)\r\n",
        "\r\n",
        "1. Read the raw data (CSV format)\r\n",
        "2. Extract `start_year` and `start_month`\r\n",
        "3. Add `api_version` = 3\r\n",
        "4. Review the data"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "v2Data = (spark.read\r\n",
        "        .format(\"csv\")\r\n",
        "        .option(\"header\", \"true\")\r\n",
        "        .option(\"inferSchema\", \"true\")\r\n",
        "        .load(f\"{data_path}/raw/csv/v2/*.csv\")\r\n",
        ")\r\n",
        "\r\n",
        "v2Data = (v2Data\r\n",
        "    .withColumn(\"start_year\", F.year(\"started_at\"))\r\n",
        "    .withColumn(\"start_month\", F.month(\"started_at\"))\r\n",
        "    .withColumn(\"api_version\", F.lit(2))\r\n",
        ")\r\n",
        "\r\n",
        "print(f\"Total number of records: {v2Data.count()}\")\r\n",
        "print(v2Data.printSchema())\r\n",
        "display(v2Data.limit(5))"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.livy.statement-meta+json": {
              "spark_pool": "pool01",
              "session_id": "10",
              "statement_id": 7,
              "state": "finished",
              "livy_statement_state": "available",
              "queued_time": "2022-09-08T07:43:22.9203087Z",
              "session_start_time": null,
              "execution_start_time": "2022-09-08T07:48:30.1556971Z",
              "execution_finish_time": "2022-09-08T07:49:03.8193741Z",
              "spark_jobs": null
            },
            "text/plain": "StatementMeta(pool01, 10, 7, Finished, Available)"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Total number of records: 6233450\nroot\n |-- started_at: string (nullable = true)\n |-- ended_at: string (nullable = true)\n |-- duration: integer (nullable = true)\n |-- start_station_id: integer (nullable = true)\n |-- start_station_name: string (nullable = true)\n |-- start_station_description: string (nullable = true)\n |-- start_station_latitude: double (nullable = true)\n |-- start_station_longitude: double (nullable = true)\n |-- end_station_id: integer (nullable = true)\n |-- end_station_name: string (nullable = true)\n |-- end_station_description: string (nullable = true)\n |-- end_station_latitude: double (nullable = true)\n |-- end_station_longitude: double (nullable = true)\n |-- start_year: integer (nullable = true)\n |-- start_month: integer (nullable = true)\n |-- api_version: integer (nullable = false)\n\nNone\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.synapse.widget-view+json": {
              "widget_id": "afc67d23-6436-49ce-8f34-8f1ce43322b3",
              "widget_type": "Synapse.DataFrame"
            },
            "text/plain": "SynapseWidget(Synapse.DataFrame, afc67d23-6436-49ce-8f34-8f1ce43322b3)"
          },
          "metadata": {}
        }
      ],
      "execution_count": 6,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "collapsed": false
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Write v2 to Delta\r\n",
        "\r\n",
        "Now we append the v2 data to the same Delta table!... or do we?\r\n",
        "\r\n"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "(v2Data.write\r\n",
        "    .mode(\"append\")\r\n",
        "    .partitionBy(\"start_year\", \"start_month\")\r\n",
        "    .format(\"delta\")\r\n",
        "    .save(f\"{data_path}/processed/delta\")\r\n",
        ")"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.livy.statement-meta+json": {
              "spark_pool": "pool01",
              "session_id": "10",
              "statement_id": 9,
              "state": "finished",
              "livy_statement_state": "available",
              "queued_time": "2022-09-08T08:02:28.6819589Z",
              "session_start_time": null,
              "execution_start_time": "2022-09-08T08:02:28.7933813Z",
              "execution_finish_time": "2022-09-08T08:02:57.8190859Z",
              "spark_jobs": null
            },
            "text/plain": "StatementMeta(pool01, 10, 9, Finished, Available)"
          },
          "metadata": {}
        }
      ],
      "execution_count": 8,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Alternatively we can configure the spark context to _always_ allow for drift.\r\n",
        "\r\n",
        "```\r\n",
        "spark.conf.set(\"spark.databricks.delta.schema.autoMerge.enabled\", \"true\")\r\n",
        "```"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Delta Table API\r\n",
        "\r\n",
        "We use the API to create a variable named `dt` pointing to our DeltaTable.alias"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from delta.tables import DeltaTable\r\n",
        "\r\n",
        "dt = DeltaTable.forPath(spark, f\"{data_path}/processed/delta\")"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.livy.statement-meta+json": {
              "spark_pool": "pool01",
              "session_id": "10",
              "statement_id": 10,
              "state": "finished",
              "livy_statement_state": "available",
              "queued_time": "2022-09-08T08:02:55.4192601Z",
              "session_start_time": null,
              "execution_start_time": "2022-09-08T08:02:57.9093316Z",
              "execution_finish_time": "2022-09-08T08:02:58.0647858Z",
              "spark_jobs": null
            },
            "text/plain": "StatementMeta(pool01, 10, 10, Finished, Available)"
          },
          "metadata": {}
        }
      ],
      "execution_count": 9,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "collapsed": false
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "We have a quick look at the table to confirm that both v1 and v2 data are in the table."
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "display(dt.toDF().where(\"api_version = 1\").limit(3))\r\n",
        "display(dt.toDF().where(\"api_version = 2\").limit(3))"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.livy.statement-meta+json": {
              "spark_pool": "pool01",
              "session_id": "10",
              "statement_id": 21,
              "state": "finished",
              "livy_statement_state": "available",
              "queued_time": "2022-09-08T08:09:57.8651669Z",
              "session_start_time": null,
              "execution_start_time": "2022-09-08T08:09:58.0143152Z",
              "execution_finish_time": "2022-09-08T08:10:00.7915185Z",
              "spark_jobs": null
            },
            "text/plain": "StatementMeta(pool01, 10, 21, Finished, Available)"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.synapse.widget-view+json": {
              "widget_id": "dbc6fdfb-27df-4b48-a85b-45b167744413",
              "widget_type": "Synapse.DataFrame"
            },
            "text/plain": "SynapseWidget(Synapse.DataFrame, dbc6fdfb-27df-4b48-a85b-45b167744413)"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.synapse.widget-view+json": {
              "widget_id": "4042975a-612f-45b9-9f30-0761dd826ebf",
              "widget_type": "Synapse.DataFrame"
            },
            "text/plain": "SynapseWidget(Synapse.DataFrame, 4042975a-612f-45b9-9f30-0761dd826ebf)"
          },
          "metadata": {}
        }
      ],
      "execution_count": 20,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "collapsed": false
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Delta History\r\n",
        "\r\n"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "display(dt.history())\r\n",
        "total_rows = dt.toDF().count()\r\n",
        "print(f\"Total rows: {total_rows}\")"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.livy.statement-meta+json": {
              "spark_pool": "pool01",
              "session_id": "10",
              "statement_id": 12,
              "state": "finished",
              "livy_statement_state": "available",
              "queued_time": "2022-09-08T08:02:55.6486165Z",
              "session_start_time": null,
              "execution_start_time": "2022-09-08T08:03:02.1797719Z",
              "execution_finish_time": "2022-09-08T08:03:04.0505735Z",
              "spark_jobs": null
            },
            "text/plain": "StatementMeta(pool01, 10, 12, Finished, Available)"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.synapse.widget-view+json": {
              "widget_id": "2e4f74ac-6c6a-48fd-8120-9f81fc2b22d5",
              "widget_type": "Synapse.DataFrame"
            },
            "text/plain": "SynapseWidget(Synapse.DataFrame, 2e4f74ac-6c6a-48fd-8120-9f81fc2b22d5)"
          },
          "metadata": {}
        }
      ],
      "execution_count": 11,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "collapsed": false
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Oh no! A sophisticated hacker has infiltrated the notebook\r\n",
        "# and executed the delete() command, removing all data!  \r\n",
        "dt.delete()"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.livy.statement-meta+json": {
              "spark_pool": "pool01",
              "session_id": "10",
              "statement_id": 14,
              "state": "finished",
              "livy_statement_state": "available",
              "queued_time": "2022-09-08T08:02:55.9825179Z",
              "session_start_time": null,
              "execution_start_time": "2022-09-08T08:03:09.267469Z",
              "execution_finish_time": "2022-09-08T08:03:13.8643837Z",
              "spark_jobs": null
            },
            "text/plain": "StatementMeta(pool01, 10, 14, Finished, Available)"
          },
          "metadata": {}
        }
      ],
      "execution_count": 13,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Count the table again\r\n",
        "total_rows = dt.toDF().count()\r\n",
        "print(f\"Total rows: {total_rows}\")"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.livy.statement-meta+json": {
              "spark_pool": "pool01",
              "session_id": "10",
              "statement_id": 15,
              "state": "finished",
              "livy_statement_state": "available",
              "queued_time": "2022-09-08T08:02:56.0868962Z",
              "session_start_time": null,
              "execution_start_time": "2022-09-08T08:03:13.9560054Z",
              "execution_finish_time": "2022-09-08T08:03:16.2148475Z",
              "spark_jobs": null
            },
            "text/plain": "StatementMeta(pool01, 10, 15, Finished, Available)"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Total rows: 0\n"
          ]
        }
      ],
      "execution_count": 14,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Time to Travel with Time Travel!\r\n",
        "![Delorean at Rebel](https://www.kode24.no/images/74140596.jpg?imageId=74140596&width=320&height=190)\r\n",
        "\r\n",
        "First have a peek a the history in the Delta log"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "display(dt.history())"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.livy.statement-meta+json": {
              "spark_pool": "pool01",
              "session_id": "10",
              "statement_id": 16,
              "state": "finished",
              "livy_statement_state": "available",
              "queued_time": "2022-09-08T08:02:56.1936704Z",
              "session_start_time": null,
              "execution_start_time": "2022-09-08T08:03:16.3100687Z",
              "execution_finish_time": "2022-09-08T08:03:18.3948814Z",
              "spark_jobs": null
            },
            "text/plain": "StatementMeta(pool01, 10, 16, Finished, Available)"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.synapse.widget-view+json": {
              "widget_id": "ead63b09-9bde-40cd-83f4-f60262aea043",
              "widget_type": "Synapse.DataFrame"
            },
            "text/plain": "SynapseWidget(Synapse.DataFrame, ead63b09-9bde-40cd-83f4-f60262aea043)"
          },
          "metadata": {}
        }
      ],
      "execution_count": 15,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "collapsed": false
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Restore to version `1` and count the data again"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Alternatively: dt.restoreToTimestamp\r\n",
        "dt.restoreToVersion(1)\r\n",
        "print(f\"Total rows: {dt.toDF().count()}\")"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.livy.statement-meta+json": {
              "spark_pool": "pool01",
              "session_id": "10",
              "statement_id": 17,
              "state": "finished",
              "livy_statement_state": "available",
              "queued_time": "2022-09-08T08:02:56.3001687Z",
              "session_start_time": null,
              "execution_start_time": "2022-09-08T08:03:18.487218Z",
              "execution_finish_time": "2022-09-08T08:03:33.1796894Z",
              "spark_jobs": null
            },
            "text/plain": "StatementMeta(pool01, 10, 17, Finished, Available)"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Total rows: 11539308\n"
          ]
        }
      ],
      "execution_count": 16,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "display(dt.history())"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.livy.statement-meta+json": {
              "spark_pool": "pool01",
              "session_id": "10",
              "statement_id": 32,
              "state": "finished",
              "livy_statement_state": "available",
              "queued_time": "2022-09-08T08:25:37.388005Z",
              "session_start_time": null,
              "execution_start_time": "2022-09-08T08:25:37.4976907Z",
              "execution_finish_time": "2022-09-08T08:25:44.3668738Z",
              "spark_jobs": null
            },
            "text/plain": "StatementMeta(pool01, 10, 32, Finished, Available)"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.synapse.widget-view+json": {
              "widget_id": "c0c75224-c791-456b-a2b6-22f61cd2a3b1",
              "widget_type": "Synapse.DataFrame"
            },
            "text/plain": "SynapseWidget(Synapse.DataFrame, c0c75224-c791-456b-a2b6-22f61cd2a3b1)"
          },
          "metadata": {}
        }
      ],
      "execution_count": 31,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "collapsed": false
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Structured Streaming demo (if there's time)\r\n",
        "\r\n",
        "\"Structured Streaming is a scalable and fault-tolerant stream processing engine built on the Spark SQL engine.\"\r\n",
        "\r\n",
        "The general pattern is to read from a \"source\", perform any transformations, and write to a \"sink\".\r\n",
        "\r\n",
        "Here we use the \"rate\" source to generate fake data for October 2022 and tagged as `api_version = 3`\r\n",
        "and then write the data to the Delta table sink."
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from pyspark.sql.functions import col, lit, rand, round\r\n",
        "\r\n",
        "# Change to True to enable the stream\r\n",
        "there_is_still_time = False\r\n",
        "\r\n",
        "if there_is_still_time:\r\n",
        "    # Let's generate some fake data as \"api_version = 3\"\r\n",
        "    delta_checkpoint_path= f\"{data_path}/processed/deltacheckpoint\"\r\n",
        "\r\n",
        "    df = (spark.readStream\r\n",
        "        .format(\"rate\")\r\n",
        "        .option(\"rowsPerSecond\", 10)\r\n",
        "        .load()\r\n",
        "        .withColumn(\"start_station_id\", lit(round(rand()*500)).cast(\"int\"))\r\n",
        "        .withColumn(\"end_station_id\", lit(round(rand()*500)).cast(\"int\"))\r\n",
        "        .withColumn(\"started_at\", col(\"timestamp\").cast(\"string\"))\r\n",
        "        .withColumn(\"ended_at\", col(\"timestamp\").cast(\"string\"))\r\n",
        "        .withColumn(\"start_year\", lit(2022))\r\n",
        "        .withColumn(\"start_month\", lit(10))\r\n",
        "        .withColumn(\"api_version\", lit(3))\r\n",
        "    )\r\n",
        "\r\n",
        "    df = df.select(\"start_station_id\", \"end_station_id\", \"started_at\", \"ended_at\", \"start_year\", \"start_month\", \"api_version\")\r\n",
        "\r\n",
        "    query = (df.writeStream\r\n",
        "        .format(\"delta\")\r\n",
        "        .option(\"checkpointLocation\", delta_checkpoint_path)\r\n",
        "        .start(f\"{data_path}/processed/delta\")\r\n",
        "    )"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.livy.statement-meta+json": {
              "spark_pool": "pool01",
              "session_id": "10",
              "statement_id": 34,
              "state": "finished",
              "livy_statement_state": "available",
              "queued_time": "2022-09-08T08:26:11.3746822Z",
              "session_start_time": null,
              "execution_start_time": "2022-09-08T08:26:11.4592498Z",
              "execution_finish_time": "2022-09-08T08:26:11.9638362Z",
              "spark_jobs": null
            },
            "text/plain": "StatementMeta(pool01, 10, 34, Finished, Available)"
          },
          "metadata": {}
        }
      ],
      "execution_count": 33,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Verify that the stream is active\r\n",
        "query.isActive"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.livy.statement-meta+json": {
              "spark_pool": "pool01",
              "session_id": "10",
              "statement_id": 44,
              "state": "finished",
              "livy_statement_state": "available",
              "queued_time": "2022-09-08T08:37:10.554123Z",
              "session_start_time": null,
              "execution_start_time": "2022-09-08T08:37:10.6645066Z",
              "execution_finish_time": "2022-09-08T08:37:10.8192579Z",
              "spark_jobs": null
            },
            "text/plain": "StatementMeta(pool01, 10, 44, Finished, Available)"
          },
          "metadata": {}
        },
        {
          "output_type": "execute_result",
          "execution_count": 133,
          "data": {
            "text/plain": "False"
          },
          "metadata": {}
        }
      ],
      "execution_count": 43,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Count the fake new \"version 3\" records that are being streamed in\r\n",
        "count = dt.toDF().where(\"api_version = 3\").count()\r\n",
        "print(count)"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.livy.statement-meta+json": {
              "spark_pool": "pool01",
              "session_id": "10",
              "statement_id": 37,
              "state": "finished",
              "livy_statement_state": "available",
              "queued_time": "2022-09-08T08:26:22.6366959Z",
              "session_start_time": null,
              "execution_start_time": "2022-09-08T08:26:22.7165687Z",
              "execution_finish_time": "2022-09-08T08:26:25.5014945Z",
              "spark_jobs": null
            },
            "text/plain": "StatementMeta(pool01, 10, 37, Finished, Available)"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "434\n"
          ]
        }
      ],
      "execution_count": 36,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Medallion Architecture \r\n",
        "\r\n",
        "Using Structured Streams and Delta tables we can set up an automated data pipeline that processes\r\n",
        "data through incremental stages from a \"raw\" (AKA \"bronze\") table to a final data product (AKA \"gold\").crc\r\n",
        "\r\n",
        "Databricks refers to this as a [\"Medallion architecture\"](https://www.databricks.com/glossary/medallion-architecture)\r\n",
        "\r\n",
        "![Medallion architecture](https://www.databricks.com/wp-content/uploads/2022/03/delta-lake-medallion-architecture-2.jpeg)"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Note that here we rebuild the table every time (`.outputMode(\"complete\")`).\r\n",
        "\r\n",
        "You generally would not want to do this on a frequently-updated stream."
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "if there_is_still_time:\r\n",
        "    groupedQuery = (spark.readStream\r\n",
        "        .format(\"delta\")\r\n",
        "        .load(f\"{data_path}/processed/delta\")\r\n",
        "        .groupBy(\"start_year\", \"start_month\")\r\n",
        "        .count()\r\n",
        "        .writeStream\r\n",
        "        .format(\"delta\")\r\n",
        "        .outputMode(\"complete\")\r\n",
        "        .option(\"checkpointLocation\", f\"{data_path}/processed/groupedTripsCheckpoint\")\r\n",
        "        .start(f\"{data_path}/processed/groupedTrips\")\r\n",
        "    )"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.livy.statement-meta+json": {
              "spark_pool": "pool01",
              "session_id": "10",
              "statement_id": 38,
              "state": "finished",
              "livy_statement_state": "available",
              "queued_time": "2022-09-08T08:26:28.3426136Z",
              "session_start_time": null,
              "execution_start_time": "2022-09-08T08:26:28.4353201Z",
              "execution_finish_time": "2022-09-08T08:26:29.0509013Z",
              "spark_jobs": null
            },
            "text/plain": "StatementMeta(pool01, 10, 38, Finished, Available)"
          },
          "metadata": {}
        }
      ],
      "execution_count": 37,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Let's have a look at the streaming data (October 2022)"
      ],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from pyspark.sql.functions import col\r\n",
        "\r\n",
        "if there_is_still_time and groupedQuery.isActive:\r\n",
        "    df = (spark.read\r\n",
        "        .format(\"delta\")\r\n",
        "        .load(f\"{data_path}/processed/groupedTrips\")\r\n",
        "        .where(\"start_year = 2022 and start_month = 10\")\r\n",
        "        .orderBy(col(\"count\").desc())\r\n",
        "    )\r\n",
        "    display(df)"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.livy.statement-meta+json": {
              "spark_pool": "pool01",
              "session_id": "10",
              "statement_id": 40,
              "state": "finished",
              "livy_statement_state": "available",
              "queued_time": "2022-09-08T08:27:10.5180902Z",
              "session_start_time": null,
              "execution_start_time": "2022-09-08T08:27:10.6231406Z",
              "execution_finish_time": "2022-09-08T08:27:15.9223017Z",
              "spark_jobs": null
            },
            "text/plain": "StatementMeta(pool01, 10, 40, Finished, Available)"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.synapse.widget-view+json": {
              "widget_id": "dbc2790a-62cb-4227-9857-9803a2ead697",
              "widget_type": "Synapse.DataFrame"
            },
            "text/plain": "SynapseWidget(Synapse.DataFrame, dbc2790a-62cb-4227-9857-9803a2ead697)"
          },
          "metadata": {}
        }
      ],
      "execution_count": 39,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "collapsed": false
      }
    },
    {
      "cell_type": "markdown",
      "source": [],
      "metadata": {
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    },
    {
      "cell_type": "code",
      "source": [
        "display(dt.history().limit(5))"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.livy.statement-meta+json": {
              "spark_pool": "pool01",
              "session_id": "10",
              "statement_id": 42,
              "state": "finished",
              "livy_statement_state": "available",
              "queued_time": "2022-09-08T08:27:40.1435756Z",
              "session_start_time": null,
              "execution_start_time": "2022-09-08T08:27:40.2502893Z",
              "execution_finish_time": "2022-09-08T08:27:48.8961484Z",
              "spark_jobs": null
            },
            "text/plain": "StatementMeta(pool01, 10, 42, Finished, Available)"
          },
          "metadata": {}
        },
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.synapse.widget-view+json": {
              "widget_id": "a389131f-b1d3-4caf-87a8-41e1cfb399dd",
              "widget_type": "Synapse.DataFrame"
            },
            "text/plain": "SynapseWidget(Synapse.DataFrame, a389131f-b1d3-4caf-87a8-41e1cfb399dd)"
          },
          "metadata": {}
        }
      ],
      "execution_count": 41,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        },
        "collapsed": false
      }
    },
    {
      "cell_type": "code",
      "source": [
        "if query.isActive:\r\n",
        "    query.stop()\r\n",
        "if groupedQuery.isActive:\r\n",
        "    groupedQuery.stop()"
      ],
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "application/vnd.livy.statement-meta+json": {
              "spark_pool": "pool01",
              "session_id": "10",
              "statement_id": 43,
              "state": "finished",
              "livy_statement_state": "available",
              "queued_time": "2022-09-08T08:29:08.6492368Z",
              "session_start_time": null,
              "execution_start_time": "2022-09-08T08:29:08.7568301Z",
              "execution_finish_time": "2022-09-08T08:29:08.949455Z",
              "spark_jobs": null
            },
            "text/plain": "StatementMeta(pool01, 10, 43, Finished, Available)"
          },
          "metadata": {}
        }
      ],
      "execution_count": 42,
      "metadata": {
        "jupyter": {
          "source_hidden": false,
          "outputs_hidden": false
        },
        "nteract": {
          "transient": {
            "deleting": false
          }
        }
      }
    }
  ],
  "metadata": {
    "kernelspec": {
      "name": "synapse_pyspark",
      "display_name": "Synapse PySpark"
    },
    "language_info": {
      "name": "python"
    },
    "description": null,
    "save_output": true,
    "synapse_widget": {
      "version": "0.1",
      "state": {
        "a08e9a3c-749a-4ccf-b38f-895fb5d0ca3c": {
          "type": "Synapse.DataFrame",
          "sync_state": {
            "table": {
              "rows": [
                {
                  "0": "159",
                  "1": "2016-06-19 14:02:57 +0200",
                  "2": "229",
                  "3": "2016-06-19 14:23:29 +0200"
                },
                {
                  "0": "253",
                  "1": "2016-06-19 14:03:06 +0200",
                  "2": "222",
                  "3": "2016-06-19 16:15:48 +0200"
                },
                {
                  "0": "210",
                  "1": "2016-06-19 14:03:06 +0200",
                  "2": "246",
                  "3": "2016-06-19 14:48:24 +0200"
                },
                {
                  "0": "238",
                  "1": "2016-06-19 14:03:23 +0200",
                  "2": "191",
                  "3": "2016-06-19 14:11:54 +0200"
                },
                {
                  "0": "201",
                  "1": "2016-06-19 14:03:26 +0200",
                  "2": "249",
                  "3": "2016-06-19 14:13:44 +0200"
                }
              ],
              "schema": [
                {
                  "key": "0",
                  "name": "Start station",
                  "type": "int"
                },
                {
                  "key": "1",
                  "name": "Start time",
                  "type": "string"
                },
                {
                  "key": "2",
                  "name": "End station",
                  "type": "int"
                },
                {
                  "key": "3",
                  "name": "End time",
                  "type": "string"
                }
              ],
              "truncated": false
            },
            "isSummary": false,
            "language": "scala"
          },
          "persist_state": {
            "view": {
              "type": "details",
              "tableOptions": {},
              "chartOptions": {
                "chartType": "bar",
                "aggregationType": "sum",
                "categoryFieldKeys": [
                  "1"
                ],
                "seriesFieldKeys": [
                  "0"
                ],
                "isStacked": false
              }
            }
          }
        },
        "4acd3209-b1a8-4a62-ad2d-658231901889": {
          "type": "Synapse.DataFrame",
          "sync_state": {
            "table": {
              "rows": [
                {
                  "0": "532",
                  "1": "212"
                },
                {
                  "0": "823",
                  "1": "134"
                },
                {
                  "0": "824",
                  "1": "131"
                },
                {
                  "0": "825",
                  "1": "175"
                },
                {
                  "0": "826",
                  "1": "186"
                }
              ],
              "schema": [
                {
                  "key": "0",
                  "name": "new_id",
                  "type": "int"
                },
                {
                  "key": "1",
                  "name": "legacy_id",
                  "type": "int"
                }
              ],
              "truncated": false
            },
            "isSummary": false,
            "language": "scala"
          },
          "persist_state": {
            "view": {
              "type": "details",
              "tableOptions": {},
              "chartOptions": {
                "chartType": "bar",
                "aggregationType": "sum",
                "categoryFieldKeys": [
                  "1"
                ],
                "seriesFieldKeys": [
                  "0"
                ],
                "isStacked": false
              }
            }
          }
        },
        "6818c452-136a-4f43-bd83-242655307f81": {
          "type": "Synapse.DataFrame",
          "sync_state": {
            "table": {
              "rows": [
                {
                  "0": "2016-10-01 06:00:08 +0200",
                  "1": "2016-10-01 06:09:47 +0200",
                  "2": "2016",
                  "3": "10",
                  "4": "1",
                  "5": "783",
                  "6": "782"
                },
                {
                  "0": "2016-10-01 06:00:41 +0200",
                  "1": "2016-10-01 06:11:07 +0200",
                  "2": "2016",
                  "3": "10",
                  "4": "1",
                  "5": "625",
                  "6": "825"
                },
                {
                  "0": "2016-10-01 06:01:02 +0200",
                  "1": "2016-10-01 06:15:52 +0200",
                  "2": "2016",
                  "3": "10",
                  "4": "1",
                  "5": "510",
                  "6": "442"
                },
                {
                  "0": "2016-10-01 06:01:20 +0200",
                  "1": "2016-10-01 06:15:04 +0200",
                  "2": "2016",
                  "3": "10",
                  "4": "1",
                  "5": "514",
                  "6": "464"
                },
                {
                  "0": "2016-10-01 06:02:16 +0200",
                  "1": "2016-10-01 06:04:29 +0200",
                  "2": "2016",
                  "3": "10",
                  "4": "1",
                  "5": "506",
                  "6": "464"
                }
              ],
              "schema": [
                {
                  "key": "0",
                  "name": "started_at",
                  "type": "string"
                },
                {
                  "key": "1",
                  "name": "ended_at",
                  "type": "string"
                },
                {
                  "key": "2",
                  "name": "start_year",
                  "type": "int"
                },
                {
                  "key": "3",
                  "name": "start_month",
                  "type": "int"
                },
                {
                  "key": "4",
                  "name": "api_version",
                  "type": "int"
                },
                {
                  "key": "5",
                  "name": "start_station_id",
                  "type": "int"
                },
                {
                  "key": "6",
                  "name": "end_station_id",
                  "type": "int"
                }
              ],
              "truncated": false
            },
            "isSummary": false,
            "language": "scala"
          },
          "persist_state": {
            "view": {
              "type": "details",
              "tableOptions": {},
              "chartOptions": {
                "chartType": "bar",
                "aggregationType": "sum",
                "categoryFieldKeys": [
                  "0"
                ],
                "seriesFieldKeys": [
                  "2"
                ],
                "isStacked": false
              }
            }
          }
        },
        "afc67d23-6436-49ce-8f34-8f1ce43322b3": {
          "type": "Synapse.DataFrame",
          "sync_state": {
            "table": {
              "rows": [
                {
                  "0": "2019-04-02 22:18:47.926000+00:00",
                  "1": "2019-04-02 22:24:25.427000+00:00",
                  "2": "337",
                  "3": "401",
                  "4": "Schous plass",
                  "5": "nærmest Thorvald Meyers gate",
                  "6": "59.920259",
                  "7": "10.760629",
                  "8": "381",
                  "9": "Grønlands torg",
                  "10": "ved Tøyenbekken",
                  "11": "59.91252",
                  "12": "10.76224",
                  "13": "2019",
                  "14": "4",
                  "15": "2"
                },
                {
                  "0": "2019-04-02 22:25:54.203000+00:00",
                  "1": "2019-04-02 22:27:36.114000+00:00",
                  "2": "101",
                  "3": "556",
                  "4": "Pilestredet",
                  "5": "ved Sporveisgata",
                  "6": "59.9262242",
                  "7": "10.7296246",
                  "8": "556",
                  "9": "Pilestredet",
                  "10": "ved Sporveisgata",
                  "11": "59.9262242",
                  "12": "10.7296246",
                  "13": "2019",
                  "14": "4",
                  "15": "2"
                },
                {
                  "0": "2019-04-02 22:26:39.308000+00:00",
                  "1": "2019-04-02 22:31:59.445000+00:00",
                  "2": "320",
                  "3": "561",
                  "4": "Kirkeveien",
                  "5": "nord for Gørbitz' gate",
                  "6": "59.9335582",
                  "7": "10.7264261",
                  "8": "471",
                  "9": "Marienlyst",
                  "10": "i Suhms gate",
                  "11": "59.932454",
                  "12": "10.721769",
                  "13": "2019",
                  "14": "4",
                  "15": "2"
                },
                {
                  "0": "2019-04-03 03:09:58.992000+00:00",
                  "1": "2019-04-03 03:12:00.589000+00:00",
                  "2": "121",
                  "3": "393",
                  "4": "Hans Nielsen Hauges plass",
                  "5": "Mellom Åsengata og Nordkappgata",
                  "6": "59.939244",
                  "7": "10.774319",
                  "8": "393",
                  "9": "Hans Nielsen Hauges plass",
                  "10": "Mellom Åsengata og Nordkappgata",
                  "11": "59.939244",
                  "12": "10.774319",
                  "13": "2019",
                  "14": "4",
                  "15": "2"
                },
                {
                  "0": "2019-04-03 03:10:02.415000+00:00",
                  "1": "2019-04-03 03:14:44.537000+00:00",
                  "2": "282",
                  "3": "597",
                  "4": "Fredensborg",
                  "5": "ved rundkjøringen",
                  "6": "59.920995",
                  "7": "10.750358",
                  "8": "557",
                  "9": "Akersgata",
                  "10": "ved Grensen",
                  "11": "59.9141971",
                  "12": "10.7428828",
                  "13": "2019",
                  "14": "4",
                  "15": "2"
                }
              ],
              "schema": [
                {
                  "key": "0",
                  "name": "started_at",
                  "type": "string"
                },
                {
                  "key": "1",
                  "name": "ended_at",
                  "type": "string"
                },
                {
                  "key": "2",
                  "name": "duration",
                  "type": "int"
                },
                {
                  "key": "3",
                  "name": "start_station_id",
                  "type": "int"
                },
                {
                  "key": "4",
                  "name": "start_station_name",
                  "type": "string"
                },
                {
                  "key": "5",
                  "name": "start_station_description",
                  "type": "string"
                },
                {
                  "key": "6",
                  "name": "start_station_latitude",
                  "type": "double"
                },
                {
                  "key": "7",
                  "name": "start_station_longitude",
                  "type": "double"
                },
                {
                  "key": "8",
                  "name": "end_station_id",
                  "type": "int"
                },
                {
                  "key": "9",
                  "name": "end_station_name",
                  "type": "string"
                },
                {
                  "key": "10",
                  "name": "end_station_description",
                  "type": "string"
                },
                {
                  "key": "11",
                  "name": "end_station_latitude",
                  "type": "double"
                },
                {
                  "key": "12",
                  "name": "end_station_longitude",
                  "type": "double"
                },
                {
                  "key": "13",
                  "name": "start_year",
                  "type": "int"
                },
                {
                  "key": "14",
                  "name": "start_month",
                  "type": "int"
                },
                {
                  "key": "15",
                  "name": "api_version",
                  "type": "int"
                }
              ],
              "truncated": false
            },
            "isSummary": false,
            "language": "scala"
          },
          "persist_state": {
            "view": {
              "type": "details",
              "tableOptions": {},
              "chartOptions": {
                "chartType": "bar",
                "aggregationType": "sum",
                "categoryFieldKeys": [
                  "0"
                ],
                "seriesFieldKeys": [
                  "2"
                ],
                "isStacked": false
              }
            }
          }
        },
        "2e4f74ac-6c6a-48fd-8120-9f81fc2b22d5": {
          "type": "Synapse.DataFrame",
          "sync_state": {
            "table": {
              "rows": [
                {
                  "0": "1",
                  "1": "2022-09-08 08:02:53.792",
                  "4": "WRITE",
                  "5": {
                    "mode": "Append",
                    "partitionBy": "[\"start_year\",\"start_month\"]"
                  },
                  "9": "0",
                  "10": "Serializable",
                  "11": "true",
                  "12": {
                    "numFiles": "47",
                    "numOutputRows": "6233450",
                    "numOutputBytes": "188214696"
                  },
                  "14": "Apache-Spark/3.2.1.5.0-70548303 Delta-Lake/1.2.1.4"
                },
                {
                  "0": "0",
                  "1": "2022-09-08 07:48:23.311",
                  "4": "WRITE",
                  "5": {
                    "mode": "Append",
                    "partitionBy": "[\"start_year\",\"start_month\"]"
                  },
                  "10": "Serializable",
                  "11": "true",
                  "12": {
                    "numFiles": "34",
                    "numOutputRows": "5305858",
                    "numOutputBytes": "65874381"
                  },
                  "14": "Apache-Spark/3.2.1.5.0-70548303 Delta-Lake/1.2.1.4"
                }
              ],
              "schema": [
                {
                  "key": "0",
                  "name": "version",
                  "type": "bigint"
                },
                {
                  "key": "1",
                  "name": "timestamp",
                  "type": "timestamp"
                },
                {
                  "key": "2",
                  "name": "userId",
                  "type": "string"
                },
                {
                  "key": "3",
                  "name": "userName",
                  "type": "string"
                },
                {
                  "key": "4",
                  "name": "operation",
                  "type": "string"
                },
                {
                  "key": "5",
                  "name": "operationParameters",
                  "type": "MapType(StringType,StringType,true)"
                },
                {
                  "key": "6",
                  "name": "job",
                  "type": "StructType(StructField(jobId,StringType,true), StructField(jobName,StringType,true), StructField(runId,StringType,true), StructField(jobOwnerId,StringType,true), StructField(triggerType,StringType,true))"
                },
                {
                  "key": "7",
                  "name": "notebook",
                  "type": "StructType(StructField(notebookId,StringType,true))"
                },
                {
                  "key": "8",
                  "name": "clusterId",
                  "type": "string"
                },
                {
                  "key": "9",
                  "name": "readVersion",
                  "type": "bigint"
                },
                {
                  "key": "10",
                  "name": "isolationLevel",
                  "type": "string"
                },
                {
                  "key": "11",
                  "name": "isBlindAppend",
                  "type": "boolean"
                },
                {
                  "key": "12",
                  "name": "operationMetrics",
                  "type": "MapType(StringType,StringType,true)"
                },
                {
                  "key": "13",
                  "name": "userMetadata",
                  "type": "string"
                },
                {
                  "key": "14",
                  "name": "engineInfo",
                  "type": "string"
                }
              ],
              "truncated": false
            },
            "isSummary": false,
            "language": "scala"
          },
          "persist_state": {
            "view": {
              "type": "details",
              "tableOptions": {},
              "chartOptions": {
                "chartType": "bar",
                "aggregationType": "sum",
                "categoryFieldKeys": [
                  "1"
                ],
                "seriesFieldKeys": [
                  "0"
                ],
                "isStacked": false
              }
            }
          }
        },
        "ead63b09-9bde-40cd-83f4-f60262aea043": {
          "type": "Synapse.DataFrame",
          "sync_state": {
            "table": {
              "rows": [
                {
                  "0": "2",
                  "1": "2022-09-08 08:03:10.105",
                  "4": "DELETE",
                  "5": {
                    "predicate": "[]"
                  },
                  "9": "1",
                  "10": "Serializable",
                  "11": "false",
                  "12": {
                    "numRemovedFiles": "81",
                    "executionTimeMs": "576",
                    "scanTimeMs": "560",
                    "rewriteTimeMs": "0"
                  },
                  "14": "Apache-Spark/3.2.1.5.0-70548303 Delta-Lake/1.2.1.4"
                },
                {
                  "0": "1",
                  "1": "2022-09-08 08:02:53.792",
                  "4": "WRITE",
                  "5": {
                    "mode": "Append",
                    "partitionBy": "[\"start_year\",\"start_month\"]"
                  },
                  "9": "0",
                  "10": "Serializable",
                  "11": "true",
                  "12": {
                    "numFiles": "47",
                    "numOutputRows": "6233450",
                    "numOutputBytes": "188214696"
                  },
                  "14": "Apache-Spark/3.2.1.5.0-70548303 Delta-Lake/1.2.1.4"
                },
                {
                  "0": "0",
                  "1": "2022-09-08 07:48:23.311",
                  "4": "WRITE",
                  "5": {
                    "mode": "Append",
                    "partitionBy": "[\"start_year\",\"start_month\"]"
                  },
                  "10": "Serializable",
                  "11": "true",
                  "12": {
                    "numFiles": "34",
                    "numOutputRows": "5305858",
                    "numOutputBytes": "65874381"
                  },
                  "14": "Apache-Spark/3.2.1.5.0-70548303 Delta-Lake/1.2.1.4"
                }
              ],
              "schema": [
                {
                  "key": "0",
                  "name": "version",
                  "type": "bigint"
                },
                {
                  "key": "1",
                  "name": "timestamp",
                  "type": "timestamp"
                },
                {
                  "key": "2",
                  "name": "userId",
                  "type": "string"
                },
                {
                  "key": "3",
                  "name": "userName",
                  "type": "string"
                },
                {
                  "key": "4",
                  "name": "operation",
                  "type": "string"
                },
                {
                  "key": "5",
                  "name": "operationParameters",
                  "type": "MapType(StringType,StringType,true)"
                },
                {
                  "key": "6",
                  "name": "job",
                  "type": "StructType(StructField(jobId,StringType,true), StructField(jobName,StringType,true), StructField(runId,StringType,true), StructField(jobOwnerId,StringType,true), StructField(triggerType,StringType,true))"
                },
                {
                  "key": "7",
                  "name": "notebook",
                  "type": "StructType(StructField(notebookId,StringType,true))"
                },
                {
                  "key": "8",
                  "name": "clusterId",
                  "type": "string"
                },
                {
                  "key": "9",
                  "name": "readVersion",
                  "type": "bigint"
                },
                {
                  "key": "10",
                  "name": "isolationLevel",
                  "type": "string"
                },
                {
                  "key": "11",
                  "name": "isBlindAppend",
                  "type": "boolean"
                },
                {
                  "key": "12",
                  "name": "operationMetrics",
                  "type": "MapType(StringType,StringType,true)"
                },
                {
                  "key": "13",
                  "name": "userMetadata",
                  "type": "string"
                },
                {
                  "key": "14",
                  "name": "engineInfo",
                  "type": "string"
                }
              ],
              "truncated": false
            },
            "isSummary": false,
            "language": "scala"
          },
          "persist_state": {
            "view": {
              "type": "details",
              "tableOptions": {},
              "chartOptions": {
                "chartType": "bar",
                "aggregationType": "sum",
                "categoryFieldKeys": [
                  "1"
                ],
                "seriesFieldKeys": [
                  "0"
                ],
                "isStacked": false
              }
            }
          }
        },
        "dbc6fdfb-27df-4b48-a85b-45b167744413": {
          "type": "Synapse.DataFrame",
          "sync_state": {
            "table": {
              "rows": [
                {
                  "0": "2017-05-01 00:07:00 +0200",
                  "1": "2017-05-01 00:14:55 +0200",
                  "2": "2017",
                  "3": "5",
                  "4": "1",
                  "5": "612",
                  "6": "547"
                },
                {
                  "0": "2017-05-01 02:29:23 +0200",
                  "1": "2017-05-01 02:34:16 +0200",
                  "2": "2017",
                  "3": "5",
                  "4": "1",
                  "5": "777",
                  "6": "509"
                },
                {
                  "0": "2017-05-01 06:00:07 +0200",
                  "1": "2017-05-01 06:06:22 +0200",
                  "2": "2017",
                  "3": "5",
                  "4": "1",
                  "5": "782",
                  "6": "825"
                }
              ],
              "schema": [
                {
                  "key": "0",
                  "name": "started_at",
                  "type": "string"
                },
                {
                  "key": "1",
                  "name": "ended_at",
                  "type": "string"
                },
                {
                  "key": "2",
                  "name": "start_year",
                  "type": "int"
                },
                {
                  "key": "3",
                  "name": "start_month",
                  "type": "int"
                },
                {
                  "key": "4",
                  "name": "api_version",
                  "type": "int"
                },
                {
                  "key": "5",
                  "name": "start_station_id",
                  "type": "int"
                },
                {
                  "key": "6",
                  "name": "end_station_id",
                  "type": "int"
                },
                {
                  "key": "7",
                  "name": "duration",
                  "type": "int"
                },
                {
                  "key": "8",
                  "name": "start_station_name",
                  "type": "string"
                },
                {
                  "key": "9",
                  "name": "start_station_description",
                  "type": "string"
                },
                {
                  "key": "10",
                  "name": "start_station_latitude",
                  "type": "double"
                },
                {
                  "key": "11",
                  "name": "start_station_longitude",
                  "type": "double"
                },
                {
                  "key": "12",
                  "name": "end_station_name",
                  "type": "string"
                },
                {
                  "key": "13",
                  "name": "end_station_description",
                  "type": "string"
                },
                {
                  "key": "14",
                  "name": "end_station_latitude",
                  "type": "double"
                },
                {
                  "key": "15",
                  "name": "end_station_longitude",
                  "type": "double"
                }
              ],
              "truncated": false
            },
            "isSummary": false,
            "language": "scala"
          },
          "persist_state": {
            "view": {
              "type": "details",
              "tableOptions": {},
              "chartOptions": {
                "chartType": "bar",
                "aggregationType": "sum",
                "categoryFieldKeys": [
                  "0"
                ],
                "seriesFieldKeys": [
                  "2"
                ],
                "isStacked": false
              }
            }
          }
        },
        "4042975a-612f-45b9-9f30-0761dd826ebf": {
          "type": "Synapse.DataFrame",
          "sync_state": {
            "table": {
              "rows": [
                {
                  "0": "2019-05-01 02:34:27.956000+00:00",
                  "1": "2019-05-01 02:42:55.441000+00:00",
                  "2": "2019",
                  "3": "5",
                  "4": "2",
                  "5": "557",
                  "6": "493",
                  "7": "507",
                  "8": "Akersgata",
                  "9": "ved Grensen",
                  "10": "59.9141971",
                  "11": "10.7428828",
                  "12": "Sofienbergparken nordvest",
                  "13": "ved Toftes gate",
                  "14": "59.9243639",
                  "15": "10.7615858"
                },
                {
                  "0": "2019-05-01 03:00:34.783000+00:00",
                  "1": "2019-05-01 03:09:42.148000+00:00",
                  "2": "2019",
                  "3": "5",
                  "4": "2",
                  "5": "486",
                  "6": "744",
                  "7": "547",
                  "8": "Det Teologiske fakultet",
                  "9": "i Blindernveien",
                  "10": "59.935342",
                  "11": "10.726472",
                  "12": "Hallénparken",
                  "13": "ved Vogts gate",
                  "14": "59.9315302",
                  "15": "10.762169"
                },
                {
                  "0": "2019-05-01 03:00:45.403000+00:00",
                  "1": "2019-05-01 03:14:29.612000+00:00",
                  "2": "2019",
                  "3": "5",
                  "4": "2",
                  "5": "561",
                  "6": "446",
                  "7": "824",
                  "8": "Kirkeveien",
                  "9": "nord for Gørbitz' gate",
                  "10": "59.9335582",
                  "11": "10.7264261",
                  "12": "Bislett Stadion",
                  "13": "ved rundkjøringen",
                  "14": "59.92549",
                  "15": "10.731288"
                }
              ],
              "schema": [
                {
                  "key": "0",
                  "name": "started_at",
                  "type": "string"
                },
                {
                  "key": "1",
                  "name": "ended_at",
                  "type": "string"
                },
                {
                  "key": "2",
                  "name": "start_year",
                  "type": "int"
                },
                {
                  "key": "3",
                  "name": "start_month",
                  "type": "int"
                },
                {
                  "key": "4",
                  "name": "api_version",
                  "type": "int"
                },
                {
                  "key": "5",
                  "name": "start_station_id",
                  "type": "int"
                },
                {
                  "key": "6",
                  "name": "end_station_id",
                  "type": "int"
                },
                {
                  "key": "7",
                  "name": "duration",
                  "type": "int"
                },
                {
                  "key": "8",
                  "name": "start_station_name",
                  "type": "string"
                },
                {
                  "key": "9",
                  "name": "start_station_description",
                  "type": "string"
                },
                {
                  "key": "10",
                  "name": "start_station_latitude",
                  "type": "double"
                },
                {
                  "key": "11",
                  "name": "start_station_longitude",
                  "type": "double"
                },
                {
                  "key": "12",
                  "name": "end_station_name",
                  "type": "string"
                },
                {
                  "key": "13",
                  "name": "end_station_description",
                  "type": "string"
                },
                {
                  "key": "14",
                  "name": "end_station_latitude",
                  "type": "double"
                },
                {
                  "key": "15",
                  "name": "end_station_longitude",
                  "type": "double"
                }
              ],
              "truncated": false
            },
            "isSummary": false,
            "language": "scala"
          },
          "persist_state": {
            "view": {
              "type": "details",
              "tableOptions": {},
              "chartOptions": {
                "chartType": "bar",
                "aggregationType": "sum",
                "categoryFieldKeys": [
                  "0"
                ],
                "seriesFieldKeys": [
                  "2"
                ],
                "isStacked": false
              }
            }
          }
        },
        "c0c75224-c791-456b-a2b6-22f61cd2a3b1": {
          "type": "Synapse.DataFrame",
          "sync_state": {
            "table": {
              "rows": [
                {
                  "0": "34",
                  "1": "2022-09-08 08:25:32.943",
                  "4": "STREAMING UPDATE",
                  "5": {
                    "outputMode": "Append",
                    "queryId": "9a0b548c-2c6f-47d2-ada0-d7fb6eb2e87f",
                    "epochId": "30"
                  },
                  "9": "33",
                  "10": "Serializable",
                  "11": "true",
                  "12": {
                    "numRemovedFiles": "0",
                    "numOutputRows": "11",
                    "numOutputBytes": "13950",
                    "numAddedFiles": "8"
                  },
                  "14": "Apache-Spark/3.2.1.5.0-70548303 Delta-Lake/1.2.1.4"
                },
                {
                  "0": "33",
                  "1": "2022-09-08 08:25:28.828",
                  "4": "STREAMING UPDATE",
                  "5": {
                    "outputMode": "Append",
                    "queryId": "9a0b548c-2c6f-47d2-ada0-d7fb6eb2e87f",
                    "epochId": "29"
                  },
                  "9": "32",
                  "10": "Serializable",
                  "11": "true",
                  "12": {
                    "numRemovedFiles": "0",
                    "numOutputRows": "2",
                    "numOutputBytes": "3513",
                    "numAddedFiles": "2"
                  },
                  "14": "Apache-Spark/3.2.1.5.0-70548303 Delta-Lake/1.2.1.4"
                },
                {
                  "0": "32",
                  "1": "2022-09-08 08:25:19.415",
                  "4": "STREAMING UPDATE",
                  "5": {
                    "outputMode": "Append",
                    "queryId": "9a0b548c-2c6f-47d2-ada0-d7fb6eb2e87f",
                    "epochId": "28"
                  },
                  "9": "31",
                  "10": "Serializable",
                  "11": "true",
                  "12": {
                    "numRemovedFiles": "0",
                    "numOutputRows": "10",
                    "numOutputBytes": "13987",
                    "numAddedFiles": "8"
                  },
                  "14": "Apache-Spark/3.2.1.5.0-70548303 Delta-Lake/1.2.1.4"
                },
                {
                  "0": "31",
                  "1": "2022-09-08 08:25:09.292",
                  "4": "STREAMING UPDATE",
                  "5": {
                    "outputMode": "Append",
                    "queryId": "9a0b548c-2c6f-47d2-ada0-d7fb6eb2e87f",
                    "epochId": "27"
                  },
                  "9": "30",
                  "10": "Serializable",
                  "11": "true",
                  "12": {
                    "numRemovedFiles": "0",
                    "numOutputRows": "10",
                    "numOutputBytes": "13995",
                    "numAddedFiles": "8"
                  },
                  "14": "Apache-Spark/3.2.1.5.0-70548303 Delta-Lake/1.2.1.4"
                },
                {
                  "0": "30",
                  "1": "2022-09-08 08:24:59.053",
                  "4": "STREAMING UPDATE",
                  "5": {
                    "outputMode": "Append",
                    "queryId": "9a0b548c-2c6f-47d2-ada0-d7fb6eb2e87f",
                    "epochId": "26"
                  },
                  "9": "29",
                  "10": "Serializable",
                  "11": "true",
                  "12": {
                    "numRemovedFiles": "0",
                    "numOutputRows": "3",
                    "numOutputBytes": "5271",
                    "numAddedFiles": "3"
                  },
                  "14": "Apache-Spark/3.2.1.5.0-70548303 Delta-Lake/1.2.1.4"
                },
                {
                  "0": "29",
                  "1": "2022-09-08 08:24:56.781",
                  "4": "STREAMING UPDATE",
                  "5": {
                    "outputMode": "Append",
                    "queryId": "9a0b548c-2c6f-47d2-ada0-d7fb6eb2e87f",
                    "epochId": "25"
                  },
                  "9": "28",
                  "10": "Serializable",
                  "11": "true",
                  "12": {
                    "numRemovedFiles": "0",
                    "numOutputRows": "9",
                    "numOutputBytes": "14021",
                    "numAddedFiles": "8"
                  },
                  "14": "Apache-Spark/3.2.1.5.0-70548303 Delta-Lake/1.2.1.4"
                },
                {
                  "0": "28",
                  "1": "2022-09-08 08:24:47.479",
                  "4": "STREAMING UPDATE",
                  "5": {
                    "outputMode": "Append",
                    "queryId": "9a0b548c-2c6f-47d2-ada0-d7fb6eb2e87f",
                    "epochId": "24"
                  },
                  "9": "27",
                  "10": "Serializable",
                  "11": "true",
                  "12": {
                    "numRemovedFiles": "0",
                    "numOutputRows": "10",
                    "numOutputBytes": "13986",
                    "numAddedFiles": "8"
                  },
                  "14": "Apache-Spark/3.2.1.5.0-70548303 Delta-Lake/1.2.1.4"
                },
                {
                  "0": "27",
                  "1": "2022-09-08 08:24:44.781",
                  "4": "STREAMING UPDATE",
                  "5": {
                    "outputMode": "Append",
                    "queryId": "9a0b548c-2c6f-47d2-ada0-d7fb6eb2e87f",
                    "epochId": "23"
                  },
                  "9": "26",
                  "10": "Serializable",
                  "11": "true",
                  "12": {
                    "numRemovedFiles": "0",
                    "numOutputRows": "2",
                    "numOutputBytes": "3513",
                    "numAddedFiles": "2"
                  },
                  "14": "Apache-Spark/3.2.1.5.0-70548303 Delta-Lake/1.2.1.4"
                },
                {
                  "0": "26",
                  "1": "2022-09-08 08:24:35.31",
                  "4": "STREAMING UPDATE",
                  "5": {
                    "outputMode": "Append",
                    "queryId": "9a0b548c-2c6f-47d2-ada0-d7fb6eb2e87f",
                    "epochId": "22"
                  },
                  "9": "25",
                  "10": "Serializable",
                  "11": "true",
                  "12": {
                    "numRemovedFiles": "0",
                    "numOutputRows": "10",
                    "numOutputBytes": "13987",
                    "numAddedFiles": "8"
                  },
                  "14": "Apache-Spark/3.2.1.5.0-70548303 Delta-Lake/1.2.1.4"
                },
                {
                  "0": "25",
                  "1": "2022-09-08 08:24:25.663",
                  "4": "STREAMING UPDATE",
                  "5": {
                    "outputMode": "Append",
                    "queryId": "9a0b548c-2c6f-47d2-ada0-d7fb6eb2e87f",
                    "epochId": "21"
                  },
                  "9": "24",
                  "10": "Serializable",
                  "11": "true",
                  "12": {
                    "numRemovedFiles": "0",
                    "numOutputRows": "10",
                    "numOutputBytes": "13986",
                    "numAddedFiles": "8"
                  },
                  "14": "Apache-Spark/3.2.1.5.0-70548303 Delta-Lake/1.2.1.4"
                },
                {
                  "0": "24",
                  "1": "2022-09-08 08:24:22.963",
                  "4": "STREAMING UPDATE",
                  "5": {
                    "outputMode": "Append",
                    "queryId": "9a0b548c-2c6f-47d2-ada0-d7fb6eb2e87f",
                    "epochId": "20"
                  },
                  "9": "23",
                  "10": "Serializable",
                  "11": "true",
                  "12": {
                    "numRemovedFiles": "0",
                    "numOutputRows": "3",
                    "numOutputBytes": "5271",
                    "numAddedFiles": "3"
                  },
                  "14": "Apache-Spark/3.2.1.5.0-70548303 Delta-Lake/1.2.1.4"
                },
                {
                  "0": "23",
                  "1": "2022-09-08 08:24:12.375",
                  "4": "STREAMING UPDATE",
                  "5": {
                    "outputMode": "Append",
                    "queryId": "9a0b548c-2c6f-47d2-ada0-d7fb6eb2e87f",
                    "epochId": "19"
                  },
                  "9": "22",
                  "10": "Serializable",
                  "11": "true",
                  "12": {
                    "numRemovedFiles": "0",
                    "numOutputRows": "14",
                    "numOutputBytes": "13860",
                    "numAddedFiles": "8"
                  },
                  "14": "Apache-Spark/3.2.1.5.0-70548303 Delta-Lake/1.2.1.4"
                },
                {
                  "0": "22",
                  "1": "2022-09-08 08:23:59.118",
                  "4": "STREAMING UPDATE",
                  "5": {
                    "outputMode": "Append",
                    "queryId": "9a0b548c-2c6f-47d2-ada0-d7fb6eb2e87f",
                    "epochId": "18"
                  },
                  "9": "21",
                  "10": "Serializable",
                  "11": "true",
                  "12": {
                    "numRemovedFiles": "0",
                    "numOutputRows": "3",
                    "numOutputBytes": "5271",
                    "numAddedFiles": "3"
                  },
                  "14": "Apache-Spark/3.2.1.5.0-70548303 Delta-Lake/1.2.1.4"
                },
                {
                  "0": "21",
                  "1": "2022-09-08 08:23:56.023",
                  "4": "STREAMING UPDATE",
                  "5": {
                    "outputMode": "Append",
                    "queryId": "9a0b548c-2c6f-47d2-ada0-d7fb6eb2e87f",
                    "epochId": "17"
                  },
                  "9": "20",
                  "10": "Serializable",
                  "11": "true",
                  "12": {
                    "numRemovedFiles": "0",
                    "numOutputRows": "4",
                    "numOutputBytes": "7026",
                    "numAddedFiles": "4"
                  },
                  "14": "Apache-Spark/3.2.1.5.0-70548303 Delta-Lake/1.2.1.4"
                },
                {
                  "0": "20",
                  "1": "2022-09-08 08:23:51.095",
                  "4": "STREAMING UPDATE",
                  "5": {
                    "outputMode": "Append",
                    "queryId": "9a0b548c-2c6f-47d2-ada0-d7fb6eb2e87f",
                    "epochId": "16"
                  },
                  "9": "19",
                  "10": "Serializable",
                  "11": "true",
                  "12": {
                    "numRemovedFiles": "0",
                    "numOutputRows": "3",
                    "numOutputBytes": "5268",
                    "numAddedFiles": "3"
                  },
                  "14": "Apache-Spark/3.2.1.5.0-70548303 Delta-Lake/1.2.1.4"
                },
                {
                  "0": "19",
                  "1": "2022-09-08 08:23:48.845",
                  "4": "STREAMING UPDATE",
                  "5": {
                    "outputMode": "Append",
                    "queryId": "9a0b548c-2c6f-47d2-ada0-d7fb6eb2e87f",
                    "epochId": "15"
                  },
                  "9": "18",
                  "10": "Serializable",
                  "11": "true",
                  "12": {
                    "numRemovedFiles": "0",
                    "numOutputRows": "2",
                    "numOutputBytes": "3512",
                    "numAddedFiles": "2"
                  },
                  "14": "Apache-Spark/3.2.1.5.0-70548303 Delta-Lake/1.2.1.4"
                },
                {
                  "0": "18",
                  "1": "2022-09-08 08:23:46.695",
                  "4": "STREAMING UPDATE",
                  "5": {
                    "outputMode": "Append",
                    "queryId": "9a0b548c-2c6f-47d2-ada0-d7fb6eb2e87f",
                    "epochId": "14"
                  },
                  "9": "17",
                  "10": "Serializable",
                  "11": "true",
                  "12": {
                    "numRemovedFiles": "0",
                    "numOutputRows": "2",
                    "numOutputBytes": "3514",
                    "numAddedFiles": "2"
                  },
                  "14": "Apache-Spark/3.2.1.5.0-70548303 Delta-Lake/1.2.1.4"
                },
                {
                  "0": "17",
                  "1": "2022-09-08 08:23:44.307",
                  "4": "STREAMING UPDATE",
                  "5": {
                    "outputMode": "Append",
                    "queryId": "9a0b548c-2c6f-47d2-ada0-d7fb6eb2e87f",
                    "epochId": "13"
                  },
                  "9": "16",
                  "10": "Serializable",
                  "11": "true",
                  "12": {
                    "numRemovedFiles": "0",
                    "numOutputRows": "2",
                    "numOutputBytes": "3514",
                    "numAddedFiles": "2"
                  },
                  "14": "Apache-Spark/3.2.1.5.0-70548303 Delta-Lake/1.2.1.4"
                },
                {
                  "0": "16",
                  "1": "2022-09-08 08:23:42.136",
                  "4": "STREAMING UPDATE",
                  "5": {
                    "outputMode": "Append",
                    "queryId": "9a0b548c-2c6f-47d2-ada0-d7fb6eb2e87f",
                    "epochId": "12"
                  },
                  "9": "15",
                  "10": "Serializable",
                  "11": "true",
                  "12": {
                    "numRemovedFiles": "0",
                    "numOutputRows": "2",
                    "numOutputBytes": "3514",
                    "numAddedFiles": "2"
                  },
                  "14": "Apache-Spark/3.2.1.5.0-70548303 Delta-Lake/1.2.1.4"
                },
                {
                  "0": "15",
                  "1": "2022-09-08 08:23:40.022",
                  "4": "STREAMING UPDATE",
                  "5": {
                    "outputMode": "Append",
                    "queryId": "9a0b548c-2c6f-47d2-ada0-d7fb6eb2e87f",
                    "epochId": "11"
                  },
                  "9": "14",
                  "10": "Serializable",
                  "11": "true",
                  "12": {
                    "numRemovedFiles": "0",
                    "numOutputRows": "4",
                    "numOutputBytes": "7028",
                    "numAddedFiles": "4"
                  },
                  "14": "Apache-Spark/3.2.1.5.0-70548303 Delta-Lake/1.2.1.4"
                },
                {
                  "0": "14",
                  "1": "2022-09-08 08:23:37.676",
                  "4": "STREAMING UPDATE",
                  "5": {
                    "outputMode": "Append",
                    "queryId": "9a0b548c-2c6f-47d2-ada0-d7fb6eb2e87f",
                    "epochId": "10"
                  },
                  "9": "13",
                  "10": "Serializable",
                  "11": "true",
                  "12": {
                    "numRemovedFiles": "0",
                    "numOutputRows": "2",
                    "numOutputBytes": "3514",
                    "numAddedFiles": "2"
                  },
                  "14": "Apache-Spark/3.2.1.5.0-70548303 Delta-Lake/1.2.1.4"
                },
                {
                  "0": "13",
                  "1": "2022-09-08 08:23:34.083",
                  "4": "STREAMING UPDATE",
                  "5": {
                    "outputMode": "Append",
                    "queryId": "9a0b548c-2c6f-47d2-ada0-d7fb6eb2e87f",
                    "epochId": "9"
                  },
                  "9": "12",
                  "10": "Serializable",
                  "11": "true",
                  "12": {
                    "numRemovedFiles": "0",
                    "numOutputRows": "3",
                    "numOutputBytes": "5271",
                    "numAddedFiles": "3"
                  },
                  "14": "Apache-Spark/3.2.1.5.0-70548303 Delta-Lake/1.2.1.4"
                },
                {
                  "0": "12",
                  "1": "2022-09-08 08:23:31.883",
                  "4": "STREAMING UPDATE",
                  "5": {
                    "outputMode": "Append",
                    "queryId": "9a0b548c-2c6f-47d2-ada0-d7fb6eb2e87f",
                    "epochId": "8"
                  },
                  "9": "11",
                  "10": "Serializable",
                  "11": "true",
                  "12": {
                    "numRemovedFiles": "0",
                    "numOutputRows": "3",
                    "numOutputBytes": "5271",
                    "numAddedFiles": "3"
                  },
                  "14": "Apache-Spark/3.2.1.5.0-70548303 Delta-Lake/1.2.1.4"
                },
                {
                  "0": "11",
                  "1": "2022-09-08 08:23:29.257",
                  "4": "STREAMING UPDATE",
                  "5": {
                    "outputMode": "Append",
                    "queryId": "9a0b548c-2c6f-47d2-ada0-d7fb6eb2e87f",
                    "epochId": "7"
                  },
                  "9": "10",
                  "10": "Serializable",
                  "11": "true",
                  "12": {
                    "numRemovedFiles": "0",
                    "numOutputRows": "3",
                    "numOutputBytes": "5270",
                    "numAddedFiles": "3"
                  },
                  "14": "Apache-Spark/3.2.1.5.0-70548303 Delta-Lake/1.2.1.4"
                },
                {
                  "0": "10",
                  "1": "2022-09-08 08:23:25.592",
                  "4": "STREAMING UPDATE",
                  "5": {
                    "outputMode": "Append",
                    "queryId": "9a0b548c-2c6f-47d2-ada0-d7fb6eb2e87f",
                    "epochId": "6"
                  },
                  "9": "9",
                  "10": "Serializable",
                  "11": "true",
                  "12": {
                    "numRemovedFiles": "0",
                    "numOutputRows": "2",
                    "numOutputBytes": "3513",
                    "numAddedFiles": "2"
                  },
                  "14": "Apache-Spark/3.2.1.5.0-70548303 Delta-Lake/1.2.1.4"
                },
                {
                  "0": "9",
                  "1": "2022-09-08 08:23:23.118",
                  "4": "STREAMING UPDATE",
                  "5": {
                    "outputMode": "Append",
                    "queryId": "9a0b548c-2c6f-47d2-ada0-d7fb6eb2e87f",
                    "epochId": "5"
                  },
                  "9": "8",
                  "10": "Serializable",
                  "11": "true",
                  "12": {
                    "numRemovedFiles": "0",
                    "numOutputRows": "3",
                    "numOutputBytes": "5270",
                    "numAddedFiles": "3"
                  },
                  "14": "Apache-Spark/3.2.1.5.0-70548303 Delta-Lake/1.2.1.4"
                },
                {
                  "0": "8",
                  "1": "2022-09-08 08:23:20.758",
                  "4": "STREAMING UPDATE",
                  "5": {
                    "outputMode": "Append",
                    "queryId": "9a0b548c-2c6f-47d2-ada0-d7fb6eb2e87f",
                    "epochId": "4"
                  },
                  "9": "7",
                  "10": "Serializable",
                  "11": "true",
                  "12": {
                    "numRemovedFiles": "0",
                    "numOutputRows": "2",
                    "numOutputBytes": "3513",
                    "numAddedFiles": "2"
                  },
                  "14": "Apache-Spark/3.2.1.5.0-70548303 Delta-Lake/1.2.1.4"
                },
                {
                  "0": "7",
                  "1": "2022-09-08 08:23:18.348",
                  "4": "STREAMING UPDATE",
                  "5": {
                    "outputMode": "Append",
                    "queryId": "9a0b548c-2c6f-47d2-ada0-d7fb6eb2e87f",
                    "epochId": "3"
                  },
                  "9": "6",
                  "10": "Serializable",
                  "11": "true",
                  "12": {
                    "numRemovedFiles": "0",
                    "numOutputRows": "3",
                    "numOutputBytes": "5271",
                    "numAddedFiles": "3"
                  },
                  "14": "Apache-Spark/3.2.1.5.0-70548303 Delta-Lake/1.2.1.4"
                },
                {
                  "0": "6",
                  "1": "2022-09-08 08:23:15.542",
                  "4": "STREAMING UPDATE",
                  "5": {
                    "outputMode": "Append",
                    "queryId": "9a0b548c-2c6f-47d2-ada0-d7fb6eb2e87f",
                    "epochId": "2"
                  },
                  "9": "5",
                  "10": "Serializable",
                  "11": "true",
                  "12": {
                    "numRemovedFiles": "0",
                    "numOutputRows": "3",
                    "numOutputBytes": "5271",
                    "numAddedFiles": "3"
                  },
                  "14": "Apache-Spark/3.2.1.5.0-70548303 Delta-Lake/1.2.1.4"
                },
                {
                  "0": "5",
                  "1": "2022-09-08 08:23:13.131",
                  "4": "STREAMING UPDATE",
                  "5": {
                    "outputMode": "Append",
                    "queryId": "9a0b548c-2c6f-47d2-ada0-d7fb6eb2e87f",
                    "epochId": "1"
                  },
                  "9": "4",
                  "10": "Serializable",
                  "11": "true",
                  "12": {
                    "numRemovedFiles": "0",
                    "numOutputRows": "3",
                    "numOutputBytes": "5271",
                    "numAddedFiles": "3"
                  },
                  "14": "Apache-Spark/3.2.1.5.0-70548303 Delta-Lake/1.2.1.4"
                },
                {
                  "0": "4",
                  "1": "2022-09-08 08:23:09.322",
                  "4": "STREAMING UPDATE",
                  "5": {
                    "outputMode": "Append",
                    "queryId": "9a0b548c-2c6f-47d2-ada0-d7fb6eb2e87f",
                    "epochId": "0"
                  },
                  "9": "3",
                  "10": "Serializable",
                  "11": "true",
                  "12": {
                    "numRemovedFiles": "0",
                    "numOutputRows": "0",
                    "numOutputBytes": "0",
                    "numAddedFiles": "0"
                  },
                  "14": "Apache-Spark/3.2.1.5.0-70548303 Delta-Lake/1.2.1.4"
                },
                {
                  "0": "3",
                  "1": "2022-09-08 08:03:27.637",
                  "4": "RESTORE",
                  "5": {
                    "version": "1",
                    "timestamp": null
                  },
                  "9": "2",
                  "10": "Serializable",
                  "11": "false",
                  "12": {
                    "numRestoredFiles": "81",
                    "removedFilesSize": "0",
                    "numRemovedFiles": "0",
                    "restoredFilesSize": "254089077",
                    "numOfFilesAfterRestore": "81",
                    "tableSizeAfterRestore": "254089077"
                  },
                  "14": "Apache-Spark/3.2.1.5.0-70548303 Delta-Lake/1.2.1.4"
                },
                {
                  "0": "2",
                  "1": "2022-09-08 08:03:10.105",
                  "4": "DELETE",
                  "5": {
                    "predicate": "[]"
                  },
                  "9": "1",
                  "10": "Serializable",
                  "11": "false",
                  "12": {
                    "numRemovedFiles": "81",
                    "executionTimeMs": "576",
                    "scanTimeMs": "560",
                    "rewriteTimeMs": "0"
                  },
                  "14": "Apache-Spark/3.2.1.5.0-70548303 Delta-Lake/1.2.1.4"
                },
                {
                  "0": "1",
                  "1": "2022-09-08 08:02:53.792",
                  "4": "WRITE",
                  "5": {
                    "mode": "Append",
                    "partitionBy": "[\"start_year\",\"start_month\"]"
                  },
                  "9": "0",
                  "10": "Serializable",
                  "11": "true",
                  "12": {
                    "numFiles": "47",
                    "numOutputRows": "6233450",
                    "numOutputBytes": "188214696"
                  },
                  "14": "Apache-Spark/3.2.1.5.0-70548303 Delta-Lake/1.2.1.4"
                },
                {
                  "0": "0",
                  "1": "2022-09-08 07:48:23.311",
                  "4": "WRITE",
                  "5": {
                    "mode": "Append",
                    "partitionBy": "[\"start_year\",\"start_month\"]"
                  },
                  "10": "Serializable",
                  "11": "true",
                  "12": {
                    "numFiles": "34",
                    "numOutputRows": "5305858",
                    "numOutputBytes": "65874381"
                  },
                  "14": "Apache-Spark/3.2.1.5.0-70548303 Delta-Lake/1.2.1.4"
                }
              ],
              "schema": [
                {
                  "key": "0",
                  "name": "version",
                  "type": "bigint"
                },
                {
                  "key": "1",
                  "name": "timestamp",
                  "type": "timestamp"
                },
                {
                  "key": "2",
                  "name": "userId",
                  "type": "string"
                },
                {
                  "key": "3",
                  "name": "userName",
                  "type": "string"
                },
                {
                  "key": "4",
                  "name": "operation",
                  "type": "string"
                },
                {
                  "key": "5",
                  "name": "operationParameters",
                  "type": "MapType(StringType,StringType,true)"
                },
                {
                  "key": "6",
                  "name": "job",
                  "type": "StructType(StructField(jobId,StringType,true), StructField(jobName,StringType,true), StructField(runId,StringType,true), StructField(jobOwnerId,StringType,true), StructField(triggerType,StringType,true))"
                },
                {
                  "key": "7",
                  "name": "notebook",
                  "type": "StructType(StructField(notebookId,StringType,true))"
                },
                {
                  "key": "8",
                  "name": "clusterId",
                  "type": "string"
                },
                {
                  "key": "9",
                  "name": "readVersion",
                  "type": "bigint"
                },
                {
                  "key": "10",
                  "name": "isolationLevel",
                  "type": "string"
                },
                {
                  "key": "11",
                  "name": "isBlindAppend",
                  "type": "boolean"
                },
                {
                  "key": "12",
                  "name": "operationMetrics",
                  "type": "MapType(StringType,StringType,true)"
                },
                {
                  "key": "13",
                  "name": "userMetadata",
                  "type": "string"
                },
                {
                  "key": "14",
                  "name": "engineInfo",
                  "type": "string"
                }
              ],
              "truncated": false
            },
            "isSummary": false,
            "language": "scala"
          },
          "persist_state": {
            "view": {
              "type": "details",
              "tableOptions": {},
              "chartOptions": {
                "chartType": "bar",
                "aggregationType": "sum",
                "categoryFieldKeys": [
                  "1"
                ],
                "seriesFieldKeys": [
                  "0"
                ],
                "isStacked": false
              }
            }
          }
        },
        "dbc2790a-62cb-4227-9857-9803a2ead697": {
          "type": "Synapse.DataFrame",
          "sync_state": {
            "table": {
              "rows": [
                {
                  "0": "2022",
                  "1": "10",
                  "2": "834"
                }
              ],
              "schema": [
                {
                  "key": "0",
                  "name": "start_year",
                  "type": "int"
                },
                {
                  "key": "1",
                  "name": "start_month",
                  "type": "int"
                },
                {
                  "key": "2",
                  "name": "count",
                  "type": "bigint"
                }
              ],
              "truncated": false
            },
            "isSummary": false,
            "language": "scala"
          },
          "persist_state": {
            "view": {
              "type": "details",
              "tableOptions": {},
              "chartOptions": {
                "chartType": "bar",
                "aggregationType": "sum",
                "categoryFieldKeys": [
                  "1"
                ],
                "seriesFieldKeys": [
                  "0"
                ],
                "isStacked": false
              }
            }
          }
        },
        "a389131f-b1d3-4caf-87a8-41e1cfb399dd": {
          "type": "Synapse.DataFrame",
          "sync_state": {
            "table": {
              "rows": [
                {
                  "0": "51",
                  "1": "2022-09-08 08:27:37.12",
                  "4": "STREAMING UPDATE",
                  "5": {
                    "outputMode": "Append",
                    "queryId": "9a0b548c-2c6f-47d2-ada0-d7fb6eb2e87f",
                    "epochId": "47"
                  },
                  "9": "50",
                  "10": "Serializable",
                  "11": "true",
                  "12": {
                    "numRemovedFiles": "0",
                    "numOutputRows": "120",
                    "numOutputBytes": "15973",
                    "numAddedFiles": "8"
                  },
                  "14": "Apache-Spark/3.2.1.5.0-70548303 Delta-Lake/1.2.1.4"
                },
                {
                  "0": "50",
                  "1": "2022-09-08 08:27:26.09",
                  "4": "STREAMING UPDATE",
                  "5": {
                    "outputMode": "Append",
                    "queryId": "9a0b548c-2c6f-47d2-ada0-d7fb6eb2e87f",
                    "epochId": "46"
                  },
                  "9": "49",
                  "10": "Serializable",
                  "11": "true",
                  "12": {
                    "numRemovedFiles": "0",
                    "numOutputRows": "100",
                    "numOutputBytes": "15573",
                    "numAddedFiles": "8"
                  },
                  "14": "Apache-Spark/3.2.1.5.0-70548303 Delta-Lake/1.2.1.4"
                },
                {
                  "0": "49",
                  "1": "2022-09-08 08:27:22.831",
                  "4": "STREAMING UPDATE",
                  "5": {
                    "outputMode": "Append",
                    "queryId": "9a0b548c-2c6f-47d2-ada0-d7fb6eb2e87f",
                    "epochId": "45"
                  },
                  "9": "48",
                  "10": "Serializable",
                  "11": "true",
                  "12": {
                    "numRemovedFiles": "0",
                    "numOutputRows": "30",
                    "numOutputBytes": "14075",
                    "numAddedFiles": "8"
                  },
                  "14": "Apache-Spark/3.2.1.5.0-70548303 Delta-Lake/1.2.1.4"
                },
                {
                  "0": "48",
                  "1": "2022-09-08 08:27:12.59",
                  "4": "STREAMING UPDATE",
                  "5": {
                    "outputMode": "Append",
                    "queryId": "9a0b548c-2c6f-47d2-ada0-d7fb6eb2e87f",
                    "epochId": "44"
                  },
                  "9": "47",
                  "10": "Serializable",
                  "11": "true",
                  "12": {
                    "numRemovedFiles": "0",
                    "numOutputRows": "100",
                    "numOutputBytes": "15515",
                    "numAddedFiles": "8"
                  },
                  "14": "Apache-Spark/3.2.1.5.0-70548303 Delta-Lake/1.2.1.4"
                },
                {
                  "0": "47",
                  "1": "2022-09-08 08:27:10.193",
                  "4": "STREAMING UPDATE",
                  "5": {
                    "outputMode": "Append",
                    "queryId": "9a0b548c-2c6f-47d2-ada0-d7fb6eb2e87f",
                    "epochId": "43"
                  },
                  "9": "46",
                  "10": "Serializable",
                  "11": "true",
                  "12": {
                    "numRemovedFiles": "0",
                    "numOutputRows": "20",
                    "numOutputBytes": "13903",
                    "numAddedFiles": "8"
                  },
                  "14": "Apache-Spark/3.2.1.5.0-70548303 Delta-Lake/1.2.1.4"
                }
              ],
              "schema": [
                {
                  "key": "0",
                  "name": "version",
                  "type": "bigint"
                },
                {
                  "key": "1",
                  "name": "timestamp",
                  "type": "timestamp"
                },
                {
                  "key": "2",
                  "name": "userId",
                  "type": "string"
                },
                {
                  "key": "3",
                  "name": "userName",
                  "type": "string"
                },
                {
                  "key": "4",
                  "name": "operation",
                  "type": "string"
                },
                {
                  "key": "5",
                  "name": "operationParameters",
                  "type": "MapType(StringType,StringType,true)"
                },
                {
                  "key": "6",
                  "name": "job",
                  "type": "StructType(StructField(jobId,StringType,true), StructField(jobName,StringType,true), StructField(runId,StringType,true), StructField(jobOwnerId,StringType,true), StructField(triggerType,StringType,true))"
                },
                {
                  "key": "7",
                  "name": "notebook",
                  "type": "StructType(StructField(notebookId,StringType,true))"
                },
                {
                  "key": "8",
                  "name": "clusterId",
                  "type": "string"
                },
                {
                  "key": "9",
                  "name": "readVersion",
                  "type": "bigint"
                },
                {
                  "key": "10",
                  "name": "isolationLevel",
                  "type": "string"
                },
                {
                  "key": "11",
                  "name": "isBlindAppend",
                  "type": "boolean"
                },
                {
                  "key": "12",
                  "name": "operationMetrics",
                  "type": "MapType(StringType,StringType,true)"
                },
                {
                  "key": "13",
                  "name": "userMetadata",
                  "type": "string"
                },
                {
                  "key": "14",
                  "name": "engineInfo",
                  "type": "string"
                }
              ],
              "truncated": false
            },
            "isSummary": false,
            "language": "scala"
          },
          "persist_state": {
            "view": {
              "type": "details",
              "tableOptions": {},
              "chartOptions": {
                "chartType": "bar",
                "aggregationType": "sum",
                "categoryFieldKeys": [
                  "1"
                ],
                "seriesFieldKeys": [
                  "0"
                ],
                "isStacked": false
              }
            }
          }
        }
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 2
}