posts/a-deep-dive-into-myst-part-1-the-myst-parser-python-api-usage-in-nikola.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> *This is a series of blog posts inviting you to join me in a little journey I have\n",
    "experienced in the last few weeks to figure out a nice story for **MyST**\n",
    "inside the **Nikola** world.*\n",
    "\n",
    "Hey, a lot of time without writing, you know... (lack of) time, but let's go back to the\n",
    "important things... or at least, some interesting stuff to share with you!\n",
    "\n",
    "<!-- TEASER_END -->\n",
    "\n",
    "Recently, I have been involved with the [Executable Books](https://executablebooks.org/en/latest/)\n",
    "project and community (you can learn more about them [here](https://executablebooks.org/en/latest/about.html)).\n",
    "One of the major projects inside that community is the [MyST-Parser](https://github.com/executablebooks/MyST-Parser),\n",
    "a *rich* and *extensible* flavor of Markdown meant for technical documentation and\n",
    "publishing. It supports all the syntax of CommonMark Markdown at its base but, in\n",
    "addition, it includes several extensions to CommonMark adding extra syntax features for\n",
    "technical writing, such as the roles and directives used by Sphinx in the rST world!\n",
    "\n",
    "The whole Executable Books project is about providing an excellent pipeline for\n",
    "publishing Jupyter Notebooks, with the MyST-Parser being one of the central building\n",
    "blocks. They are even [interested](https://github.com/executablebooks/jupyter-book/issues/900)\n",
    "in a sane blog story for one of the key pieces the community has, the\n",
    "[Jupyter Book](https://jupyterbook.org/intro.html) project that indeed uses the\n",
    "MyST-Parser!\n",
    "\n",
    "When I looked into MyST, since I am always interested in how we can improve the\n",
    "\"blogging story\" for those using Jupyter Notebooks and [Nikola]((https://getnikola.com/))\n",
    "(a static site generator that loves Jupyter Notebooks and has a long story supporting\n",
    "multiple flavors of Markdown as Nikola plugins), I realized it would interesting to\n",
    "explore how this flavor of Markdown could potentially integrate with the Nikola\n",
    "ecosystem!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## First impressions (someone has beaten me to it...)\n",
    "\n",
    "The first idea was to implement MyST support in Nikola. The second one was just a search\n",
    "to see if the first idea was not previously explored by others. And indeed that was the\n",
    "case! Roberto, the Nikola creator, wrote a plugin for MyST about a year ago (there is\n",
    "[a brief blog post talking about it](https://ralsina.me/tr/es/weblog/posts/new-plugins-for-nikola.html)).\n",
    "That's great, isn't it? Obviously, I quickly jumped into looking at the codebase for\n",
    "that plugin and the [core of the implementation](https://github.com/getnikola/plugins/blob/master/v8/myst/myst.py#L66) was actually quite simple:\n",
    "\n",
    "```python\n",
    "output = myst_parser.main.to_html(new_data)\n",
    "```\n",
    "\n",
    "Roberto used the MyST Python API to generate the HTML output the Nikola machinery was\n",
    "going to use to build the blog post.\n",
    "\n",
    "Problem solved! Well, actually, it was not that simple..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Current limitations (there is always a but...)\n",
    "\n",
    "One of the key concepts in the MyST (and rST) world is the support for [roles](https://myst-parser.readthedocs.io/en/latest/sphinx/intro.html#reference-a-section-label-with-a-role)\n",
    "and [directives](https://myst-parser.readthedocs.io/en/latest/sphinx/intro.html#extend-markdown-with-a-directive).\n",
    "People using rST would be tempted to use MyST and the MyST plugin for Nikola if they\n",
    "have support for the main pieces they use on a daily basis to save them some time.\n",
    "\n",
    "So, what happens when you try to use the MyST Python API to render a simple directive:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from myst_parser.main import to_html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = \"\"\"\n",
    "```{admonition} This is my admonition\n",
    "This is my note\n",
    "```\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'<pre><code class=\"language-{admonition}\">This is my note\\n</code></pre>\\n'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "to_html(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That HTML code does not look like an admonition!\n",
    "When I searched into the MyST and Nikola issue trackers, I found a [report](https://github.com/executablebooks/MyST-Parser/issues/307)\n",
    "about this problem.\n",
    "\n",
    "One of the answers from a MyST core contributor was particularly interesting:\n",
    "\n",
    "> > I don't know if it's intended to behave this way.\n",
    "\n",
    "> Heya, yes it is; you have to use sphinx to fully parse to HTML\n",
    "\n",
    "That surprised me, as I assumed that roles and directives would be supported by the\n",
    "Python API, but I learnt things are a little bit more complex than that!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Understanding the limitations (fun/not fun dive, actually fun...)\n",
    "\n",
    "I started digging into the codebase and some MyST documentation.\n",
    "\n",
    "If you look at the `to_html` function I showed before: \n",
    "\n",
    "```python\n",
    "def to_html(text: str, env=None, config: Optional[MdParserConfig] = None):\n",
    "    config = config or MdParserConfig()\n",
    "    config.renderer = \"html\"\n",
    "    md = default_parser(config)\n",
    "    return md.render(text, env)\n",
    "```\n",
    "\n",
    "We have a `default_parser` rendering stuff (we will talk about that \"stuff\" later).\n",
    "Let's now get into the some pieces of the default parser implementation:\n",
    "\n",
    "```python\n",
    "def default_parser(config: MdParserConfig) -> MarkdownIt:\n",
    "    \"\"\"Return the default parser configuration for MyST\"\"\"\n",
    "    renderer_cls: Callable[[MarkdownIt], RendererProtocol]\n",
    "\n",
    "    if config.renderer == \"sphinx\":\n",
    "        from myst_parser.sphinx_renderer import SphinxRenderer\n",
    "\n",
    "        renderer_cls = SphinxRenderer\n",
    "    elif config.renderer == \"html\":\n",
    "        renderer_cls = RendererHTML\n",
    "    elif config.renderer == \"docutils\":\n",
    "        from myst_parser.docutils_renderer import DocutilsRenderer\n",
    "\n",
    "        renderer_cls = DocutilsRenderer\n",
    "    else:\n",
    "        raise ValueError(\"unknown renderer type: {0}\".format(config.renderer))\n",
    "```\n",
    "\n",
    "Well, this is quite interesting! We have a `SphinxRenderer` and `DocutilsRenderer`. You\n",
    "can learn about those [here](https://myst-parser.readthedocs.io/en/latest/api/renderers.html#).\n",
    "Those are an integral part of the MyST-Parser project. But when you look at the `html`\n",
    "renderer, the [RendererHTML class](https://markdown-it-py.readthedocs.io/en/latest/api/markdown_it.renderer.html#markdown_it.renderer.RendererHTML)\n",
    "is coming from another Executable Books project called `markdown-it-py`. This is a Python\n",
    "port of the well-used [markdown-it](https://github.com/markdown-it/markdown-it), plus\n",
    "some of its associated plugins.\n",
    "\n",
    "If you indeed keep looking at the `default_parser` implementation, you will find the\n",
    "MyST-Parser (among other things) is actually a collection of `markdown-it-py` plugins:\n",
    "\n",
    "```python\n",
    "    md = (\n",
    "        MarkdownIt(\"commonmark\", renderer_cls=renderer_cls)\n",
    "        .enable(\"table\")\n",
    "        .use(front_matter_plugin)\n",
    "        .use(myst_block_plugin)\n",
    "        .use(myst_role_plugin)\n",
    "        .use(footnote_plugin)\n",
    "        .use(wordcount_plugin, per_minute=config.words_per_minute)\n",
    "        .disable(\"footnote_inline\")\n",
    "        .disable(\"footnote_tail\")\n",
    "    )\n",
    "```\n",
    "\n",
    "Notice there are some MyST specific plugins such as the `myst_block_plugin` and the\n",
    "`myst_role_plugin`.\n",
    "\n",
    "So, it seems the main idea is to have a `MystParser`, a Sphinx parser, that parses stuff\n",
    "into a token stream using a set of general and some specific markdown-it-py plugins,\n",
    "creates a Docutils objects-based representation of the token stream and render those\n",
    "objects using the Sphinx machinery.\n",
    "\n",
    "In this model, it actually makes sense to see roles and directives supported when you go\n",
    "through the Docutils/Sphinx path (those concepts are core-supported in the\n",
    "Docutils/Sphinx ecosystem) and those unsupported when you use the MyST Python API which\n",
    "does not understand how to parse nor render these concepts as we've previously shown."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Overcome the limitations (don't worry I will stop here... for now)\n",
    "\n",
    "OK, this is long enough for now, the next post will deep dive a little bit more in the\n",
    "MyST-Parser and the markdown-it-py codebases in the context of potentially supporting\n",
    "roles and directives through the MyST-Parser Python API. That will led us into some\n",
    "fundamental question for the MyST ecosystem. We will also introduce some exciting new\n",
    "projects from the Executable Books community that are closely related with these ideas.\n",
    "\n",
    "I hope you enjoyed the ride and I will see you soon with the second part!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "PS: Thanks Chris Holdgraf for your suggestions at the time to write this blog post!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  },
  "nikola": {
   "category": "",
   "date": "2021-08-23 16:00:19 UTC-03:00",
   "description": "",
   "link": "",
   "slug": "a-deep-dive-into-myst-part-1-the-myst-parser-python-api-usage-in-nikola",
   "tags": "Jupyter, python, nikola, MyST, opensource",
   "title": "A deep dive into MyST, Part 1: The MyST-Parser Python API usage in Nikola",
   "type": "text"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}