Add JupySQL, for running SQL in Jupyter/IPython

https://jupysql.ploomber.io/
crate · Jan 24, 2025 · dadd00a · dadd00a
1 parent 554a093
commit dadd00a
Show file tree

Hide file tree

Showing 6 changed files with 244 additions and 16 deletions.
diff --git a/.gitignore b/.gitignore
@@ -9,6 +9,8 @@
 .installed.cfg
 .style
 .utils
+*.csv
+*.db
 *.egg-info
 *.lint
 *.pyc

diff --git a/CHANGES.rst b/CHANGES.rst
@@ -6,7 +6,9 @@ CHANGES
 Unreleased
 ----------
 - Added `MyST-NB`_, for rendering Jupyter notebooks
+- Added `JupySQL`_, for running SQL in Jupyter/IPython
 
+.. _JupySQL: https://jupysql.ploomber.io/
 .. _MyST-NB: https://myst-nb.readthedocs.io/
 
 2024/12/13 0.37.2

diff --git a/docs/myst/notebook-text.md b/docs/myst/notebook-text.md
@@ -11,6 +11,8 @@ file_format: mystnb
 The documentation can include [text-based notebooks] using [MyST-NB],
 effectively bringing Jupyter technologies to Markdown.
 
+## Basics
+
 ```{code-cell} ipython3
 import sys
 print("this is some stdout")
@@ -21,6 +23,39 @@ print("this is some stderr", file=sys.stderr)
 See also {ref}`notebook-traditional` and {ref}`cells`.
 :::
 
+## SQL Magics
+
+[JupySQL], the successor of [ipython-sql], enables running SQL in Jupyter/IPython
+via `%sql` and `%%sql` magics.
+
+```{code-cell} ipython3
+# Acquire data.
+!pip --quiet install csvkit
+!curl -s -L -O https://github.com/wireservice/csvkit/raw/refs/heads/master/examples/realdata/acs2012_5yr_population.csv
+!rm -f population.db
+!csvsql --db sqlite:///population.db --insert acs2012_5yr_population.csv
+```
+```{code-cell} ipython3
+# Run query using JupySQL.
+%reload_ext sql
+%sql sqlite:///population.db
+%sql SELECT * FROM acs2012_5yr_population ORDER BY total_population DESC LIMIT 10;
+```
+
+:::{note}
+Here, we are using SQLite, in order not to make `sqlalchemy-cratedb` a
+dependency of the documentation theme. An example using CrateDB can be
+explored at [CrateDB Examples: notebook/jupyter].
+:::
+
+:::{todo}
+Rendering the result table has unfortunate output when using dark mode.
+Please switch to light mode instead.
+:::
+
 
+[CrateDB Examples: notebook/jupyter]: https://github.com/crate/cratedb-examples/tree/main/notebook/jupyter
+[ipython-sql]: https://github.com/catherinedevlin/ipython-sql
+[JupySQL]: https://jupysql.ploomber.io/
 [MyST-NB]: https://myst-nb.readthedocs.io/
 [text-based notebooks]: https://myst-nb.readthedocs.io/en/latest/authoring/text-notebooks.html
diff --git a/docs/myst/notebook-traditional.ipynb b/docs/myst/notebook-traditional.ipynb
@@ -14,44 +14,229 @@
     "[MyST-NB]: https://myst-nb.readthedocs.io/\n"
    ]
   },
+  {
+   "metadata": {},
+   "cell_type": "markdown",
+   "source": "## Basics"
+  },
+  {
+   "metadata": {},
+   "cell_type": "code",
+   "source": [
+    "import sys\n",
+    "print(\"this is some stdout\")\n",
+    "print(\"this is some stderr\", file=sys.stderr)"
+   ],
+   "outputs": [],
+   "execution_count": null
+  },
+  {
+   "metadata": {},
+   "cell_type": "markdown",
+   "source": [
+    ":::{tip}\n",
+    "See also {ref}`notebook-text` and {ref}`cells`.\n",
+    ":::\n"
+   ]
+  },
+  {
+   "metadata": {},
+   "cell_type": "markdown",
+   "source": [
+    "## SQL Magics\n",
+    "\n",
+    "[JupySQL], the successor of [ipython-sql], enables running SQL in Jupyter/IPython\n",
+    "via `%sql` and `%%sql` magics.\n",
+    "\n",
+    "[ipython-sql]: https://github.com/catherinedevlin/ipython-sql\n",
+    "[JupySQL]: https://jupysql.ploomber.io/"
+   ]
+  },
   {
    "metadata": {
     "ExecuteTime": {
-     "end_time": "2025-01-11T21:09:38.185018Z",
-     "start_time": "2025-01-11T21:09:38.179652Z"
+     "end_time": "2025-01-11T23:45:54.796565Z",
+     "start_time": "2025-01-11T23:45:47.313805Z"
     }
    },
    "cell_type": "code",
    "source": [
-    "import sys\n",
-    "print(\"this is some stdout\")\n",
-    "print(\"this is some stderr\", file=sys.stderr)"
+    "# Acquire data.\n",
+    "!pip --quiet install csvkit\n",
+    "!curl -s -L -O https://github.com/wireservice/csvkit/raw/refs/heads/master/examples/realdata/acs2012_5yr_population.csv\n",
+    "!rm -f population.db\n",
+    "!csvsql --db sqlite:///population.db --insert acs2012_5yr_population.csv"
+   ],
+   "outputs": [],
+   "execution_count": 2
+  },
+  {
+   "metadata": {
+    "ExecuteTime": {
+     "end_time": "2025-01-11T23:45:55.786600Z",
+     "start_time": "2025-01-11T23:45:55.003819Z"
+    }
+   },
+   "cell_type": "code",
+   "source": [
+    "# Run query using JupySQL.\n",
+    "%reload_ext sql\n",
+    "%sql sqlite:///population.db\n",
+    "%sql SELECT * FROM acs2012_5yr_population ORDER BY total_population DESC LIMIT 10;"
    ],
    "outputs": [
     {
-     "name": "stdout",
+     "name": "stderr",
      "output_type": "stream",
      "text": [
-      "this is some stdout\n"
+      "/Users/amo/dev/crate/docs/crate-docs-theme/.venv/lib/python3.12/site-packages/sql/parse.py:338: SyntaxWarning: invalid escape sequence '\\:'\n",
+      "  \"\"\"\n",
+      "/Users/amo/dev/crate/docs/crate-docs-theme/.venv/lib/python3.12/site-packages/sql/parse.py:368: SyntaxWarning: invalid escape sequence '\\:'\n",
+      "  \"\"\"\n"
      ]
     },
     {
-     "name": "stderr",
-     "output_type": "stream",
-     "text": [
-      "this is some stderr\n"
-     ]
+     "data": {
+      "text/plain": [
+       "Connecting to 'sqlite:///population.db'"
+      ],
+      "text/html": [
+       "<span style=\"None\">Connecting to &#x27;sqlite:///population.db&#x27;</span>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/plain": [
+       "Running query in 'sqlite:///population.db'"
+      ],
+      "text/html": [
+       "<span style=\"None\">Running query in &#x27;sqlite:///population.db&#x27;</span>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    },
+    {
+     "data": {
+      "text/plain": [
+       "+---------+-------------------------+------------------+-----------------+\n",
+       "|   fips  |           name          | total_population | margin_of_error |\n",
+       "+---------+-------------------------+------------------+-----------------+\n",
+       "| 31055.0 |    Douglas County, NE   |     518271.0     |       0.0       |\n",
+       "| 31109.0 |   Lancaster County, NE  |     286425.0     |       0.0       |\n",
+       "| 31153.0 |     Sarpy County, NE    |     159413.0     |       0.0       |\n",
+       "| 31079.0 |     Hall County, NE     |     58681.0      |       0.0       |\n",
+       "| 31019.0 |    Buffalo County, NE   |     46330.0      |       0.0       |\n",
+       "| 31157.0 | Scotts Bluff County, NE |     36835.0      |       0.0       |\n",
+       "| 31053.0 |     Dodge County, NE    |     36590.0      |       0.0       |\n",
+       "| 31111.0 |    Lincoln County, NE   |     36212.0      |       0.0       |\n",
+       "| 31119.0 |    Madison County, NE   |     34766.0      |       0.0       |\n",
+       "| 31141.0 |    Platte County, NE    |     32195.0      |       0.0       |\n",
+       "+---------+-------------------------+------------------+-----------------+\n",
+       "Truncated to displaylimit of 10."
+      ],
+      "text/html": [
+       "<table>\n",
+       "    <thead>\n",
+       "        <tr>\n",
+       "            <th>fips</th>\n",
+       "            <th>name</th>\n",
+       "            <th>total_population</th>\n",
+       "            <th>margin_of_error</th>\n",
+       "        </tr>\n",
+       "    </thead>\n",
+       "    <tbody>\n",
+       "        <tr>\n",
+       "            <td>31055.0</td>\n",
+       "            <td>Douglas County, NE</td>\n",
+       "            <td>518271.0</td>\n",
+       "            <td>0.0</td>\n",
+       "        </tr>\n",
+       "        <tr>\n",
+       "            <td>31109.0</td>\n",
+       "            <td>Lancaster County, NE</td>\n",
+       "            <td>286425.0</td>\n",
+       "            <td>0.0</td>\n",
+       "        </tr>\n",
+       "        <tr>\n",
+       "            <td>31153.0</td>\n",
+       "            <td>Sarpy County, NE</td>\n",
+       "            <td>159413.0</td>\n",
+       "            <td>0.0</td>\n",
+       "        </tr>\n",
+       "        <tr>\n",
+       "            <td>31079.0</td>\n",
+       "            <td>Hall County, NE</td>\n",
+       "            <td>58681.0</td>\n",
+       "            <td>0.0</td>\n",
+       "        </tr>\n",
+       "        <tr>\n",
+       "            <td>31019.0</td>\n",
+       "            <td>Buffalo County, NE</td>\n",
+       "            <td>46330.0</td>\n",
+       "            <td>0.0</td>\n",
+       "        </tr>\n",
+       "        <tr>\n",
+       "            <td>31157.0</td>\n",
+       "            <td>Scotts Bluff County, NE</td>\n",
+       "            <td>36835.0</td>\n",
+       "            <td>0.0</td>\n",
+       "        </tr>\n",
+       "        <tr>\n",
+       "            <td>31053.0</td>\n",
+       "            <td>Dodge County, NE</td>\n",
+       "            <td>36590.0</td>\n",
+       "            <td>0.0</td>\n",
+       "        </tr>\n",
+       "        <tr>\n",
+       "            <td>31111.0</td>\n",
+       "            <td>Lincoln County, NE</td>\n",
+       "            <td>36212.0</td>\n",
+       "            <td>0.0</td>\n",
+       "        </tr>\n",
+       "        <tr>\n",
+       "            <td>31119.0</td>\n",
+       "            <td>Madison County, NE</td>\n",
+       "            <td>34766.0</td>\n",
+       "            <td>0.0</td>\n",
+       "        </tr>\n",
+       "        <tr>\n",
+       "            <td>31141.0</td>\n",
+       "            <td>Platte County, NE</td>\n",
+       "            <td>32195.0</td>\n",
+       "            <td>0.0</td>\n",
+       "        </tr>\n",
+       "    </tbody>\n",
+       "</table>\n",
+       "<span style=\"font-style:italic;text-align:center;\">Truncated to <a href=\"https://jupysql.ploomber.io/en/latest/api/configuration.html#displaylimit\">displaylimit</a> of 10.</span>"
+      ]
+     },
+     "execution_count": 3,
+     "metadata": {},
+     "output_type": "execute_result"
     }
    ],
-   "execution_count": 1
+   "execution_count": 3
   },
   {
    "metadata": {},
    "cell_type": "markdown",
    "source": [
-    ":::{tip}\n",
-    "See also {ref}`notebook-text` and {ref}`cells`.\n",
-    ":::\n"
+    ":::{note}\n",
+    "Here, we are using SQLite, in order not to make `sqlalchemy-cratedb` a\n",
+    "dependency of the documentation theme. An example using CrateDB can be\n",
+    "explored at [CrateDB Examples: notebook/jupyter].\n",
+    ":::\n",
+    "\n",
+    ":::{todo}\n",
+    "Rendering the result table has unfortunate output when using dark mode.\n",
+    "Please switch to light mode instead.\n",
+    ":::\n",
+    "\n",
+    "[CrateDB Examples: notebook/jupyter]: https://github.com/crate/cratedb-examples/tree/main/notebook/jupyter\n"
    ]
   }
  ],

diff --git a/setup.py b/setup.py
@@ -60,6 +60,7 @@
     install_requires=[
         "furo==2024.8.6",
         "jinja2>=3,<4",
+        "jupysql<0.11",
         "myst-nb<1.2",
         "myst-parser[linkify]<5",
         "sphinx>=7.1,<9",

diff --git a/src/crate/theme/rtd/conf/__init__.py b/src/crate/theme/rtd/conf/__init__.py
@@ -196,6 +196,9 @@
     "tasklist",
 ]
 
+# -- Options for MyST-NB ----------------------------------------------
+nb_execution_raise_on_error = True
+
 
 def setup(app):
-Original file line number
+Diff line change
@@ Expand Up / @@ -9,6 +9,8 @@ @@
     .installed.cfg
     .style
     .utils
+    *.csv
+    *.db
     *.egg-info
     *.lint
     *.pyc
@@ Expand Down @@