posts/a-deep-dive-into-myst-part-2-the-myst-parser-docutils-and-sphinx.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "> *This is a series of blog posts inviting you to join me in a little journey I have\n",
    "experienced in the last few weeks to figure it out a nice story for **MyST**\n",
    "inside the **Nikola** world.*\n",
    "\n",
    "In the previous [blog post](https://damianavila.github.io/blog/posts/a-deep-dive-into-myst-part-1-the-myst-parser-python-api-usage-in-nikola.html),\n",
    "we have identified some limitations in the MyST-Parser Python API and we started to\n",
    "understand why roles and directives were not supported by the API as we expected.\n",
    "\n",
    "In this post, we will explore the machinery underneath the MyST-Parser with the goal to\n",
    "deepen our understanding and being able to propose some alternatives to provide the\n",
    "expected support.\n",
    "<!-- TEASER_END -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Markdown-it-py and its plugins\n",
    "\n",
    "Coming back to the MyST-Parser `default_parser` implementation, you can see the Parser\n",
    "as a [collection of `markdown-it-py` plugins](https://mdit-py-plugins.readthedocs.io/en/latest/):\n",
    "\n",
    "```python\n",
    "    md = (\n",
    "        MarkdownIt(\"commonmark\", renderer_cls=renderer_cls)\n",
    "        .enable(\"table\")\n",
    "        .use(front_matter_plugin)\n",
    "        .use(myst_block_plugin)\n",
    "        .use(myst_role_plugin)\n",
    "        .use(footnote_plugin)\n",
    "        .use(wordcount_plugin, per_minute=config.words_per_minute)\n",
    "        .disable(\"footnote_inline\")\n",
    "        .disable(\"footnote_tail\")\n",
    "    )\n",
    "```\n",
    "\n",
    "The `MarkdownIt` class is instantiated with some parsing configuration options, dictating\n",
    "the syntax rules and additional options for the parser and renderer. In addition,\n",
    "several plugins are activated to load a collection of additional syntax rules and render\n",
    "methods into the parser.\n",
    "\n",
    "When the input data is parsed via nested chains of rules, it generates a list (stream)\n",
    "of tokens, that will be eventually passed to the renderer to generate a Docutils object.\n",
    "\n",
    "In the previous post, we highlighted some [MyST specific markdown-it-py plugins](https://mdit-py-plugins.readthedocs.io/en/latest/#myst-plugins),\n",
    "such as the `myst_block_plugin` and the `myst_role_plugin`. Let's take a brief look at\n",
    "the latest one for the sake of simplicity:\n",
    "\n",
    "```python\n",
    "def myst_role_plugin(md: MarkdownIt):\n",
    "    \"\"\"Parse ``{role-name}`content```\"\"\"\n",
    "    md.inline.ruler.before(\"backticks\", \"myst_role\", myst_role)\n",
    "    md.add_render_rule(\"myst_role\", render_myst_role)\n",
    "```\n",
    "\n",
    "The `myst_role_plugin` is essentially adding a new syntax rule to the parser that now is\n",
    "able to the parse roles from the input and a new method for the renderer to render\n",
    "the now role-associated tokens:\n",
    "\n",
    "```python\n",
    "def render_myst_role(self, tokens, idx, options, env):\n",
    "    token = tokens[idx]\n",
    "    name = token.meta.get(\"name\", \"unknown\")\n",
    "    return (\n",
    "        '<code class=\"sphinx-role\">'\n",
    "        f\"{{{name}}}[{escapeHtml(token.content)}]\"\n",
    "        \"</code>\"\n",
    "    )\n",
    "```\n",
    "\n",
    "We now understand the output the MyST-Parser Python API is giving us at the time to\n",
    "parse and render a simple role input:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from myst_parser.main import to_html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = \"\"\"\n",
    "{emphasis}`content`\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'<p><code class=\"sphinx-role\">{emphasis}[content]</code></p>\\n'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "to_html(text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We previously showed the `to_html` function using the `default_parser` configured with\n",
    "the `html` renderer (provided by the `markdown-it-py` RendererHTML class).\n",
    "But, what happens when we use other renderers provided by the MyST-Parser Python API?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The Docutils and Sphinx renderers\n",
    "\n",
    "Let's first explore the `to_docutils` function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from myst_parser.main import to_docutils"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = \"\"\"\n",
    "{emphasis}`content`\n",
    "\n",
    "```{admonition} This is my admonition\n",
    "This is my note\n",
    "```\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<document source=\"notset\">\n",
      "    <paragraph>\n",
      "        <emphasis>\n",
      "            content\n",
      "    <admonition classes=\"admonition-this-is-my-admonition\">\n",
      "        <title>\n",
      "            This is my admonition\n",
      "        <paragraph>\n",
      "            This is my note\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(to_docutils(text).pformat())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The [DocutilsRenderer](https://github.com/executablebooks/MyST-Parser/blob/master/myst_parser/docutils_renderer.py#L72)\n",
    "converts a token directly to the docutils.document representation of the document,\n",
    "converting roles and directives to a docutils.nodes if a converter can be found for the\n",
    "given name.\n",
    "\n",
    "The [SphinxRenderer](https://github.com/executablebooks/MyST-Parser/blob/master/myst_parser/sphinx_renderer.py#L31)\n",
    "builds on the DocutilsRenderer to add sphinx specific nodes, e.g. for cross-referencing\n",
    "between documents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "from myst_parser.main import to_docutils"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = \"\"\"\n",
    "{ref}`target`\n",
    "\"\"\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<document source=\"notset\">\n",
      "    <paragraph>\n",
      "        <pending_xref refdoc=\"mock_docname\" refdomain=\"std\" refexplicit=\"False\" reftarget=\"target\" reftype=\"ref\" refwarn=\"True\">\n",
      "            <inline classes=\"xref std std-ref\">\n",
      "                target\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(to_docutils(text, in_sphinx_env=True).pformat())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice the sphinx-specific roles (and directives) needs the `in_sphinx_env` option enabled."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The MystParser class\n",
    "\n",
    "Previously, we presented the `MystParser` as a Sphinx parser:\n",
    "\n",
    "```python\n",
    "from sphinx.parsers import Parser as SphinxParser\n",
    "...\n",
    "class MystParser(SphinxParser):\n",
    "    \"\"\"Sphinx parser for Markedly Structured Text (MyST).\"\"\"\n",
    "```\n",
    "\n",
    "using a set of general and some specific `markdown-it-py` plugins (notice the\n",
    "`default_parser` configured - by default - with the \"sphinx\" renderer):\n",
    "\n",
    "```python\n",
    "    def parse(self, inputstring: str, document: nodes.document) -> None:\n",
    "        \"\"\"Parse source text.\n",
    "        :param inputstring: The source string to parse\n",
    "        :param document: The root docutils node to add AST elements to\n",
    "        \"\"\"\n",
    "        config = document.settings.env.myst_config\n",
    "        parser = default_parser(config)\n",
    "        parser.options[\"document\"] = document\n",
    "        env = AttrDict()\n",
    "```\n",
    "\n",
    "to parse the input text into a token stream and then rendering those (via the\n",
    "SphinxRenderer which is a subclass of the DocutilsRenderer):\n",
    "\n",
    "```python\n",
    "        tokens = parser.parse(inputstring, env)\n",
    "        if not tokens or tokens[0].type != \"front_matter\":\n",
    "            # we always add front matter, so that we can merge it with global keys,\n",
    "            # specified in the sphinx configuration\n",
    "            tokens = [Token(\"front_matter\", \"\", 0, content=\"{}\", map=[0, 0])] + tokens\n",
    "        parser.renderer.render(tokens, parser.options, env)\n",
    "```\n",
    "\n",
    "into a Docutils' document representation: \n",
    "\n",
    "```python\n",
    "def parse(app: Sphinx, text: str, docname: str = \"index\") -> nodes.document:\n",
    "    \"\"\"Parse a string as MystMarkdown with Sphinx application.\"\"\"\n",
    "    app.env.temp_data[\"docname\"] = docname\n",
    "    app.env.all_docs[docname] = time.time()\n",
    "    reader = SphinxStandaloneReader()\n",
    "    reader.setup(app)\n",
    "    parser = MystParser()\n",
    "    parser.set_application(app)\n",
    "    with sphinx_domains(app.env):\n",
    "        return publish_doctree(\n",
    "            text,\n",
    "            path.join(app.srcdir, docname + \".md\"),\n",
    "            reader=reader,\n",
    "            parser=parser,\n",
    "            parser_name=\"markdown\",\n",
    "            settings_overrides={\"env\": app.env, \"gettext_compact\": True},\n",
    "        )\n",
    "```\n",
    "\n",
    "Finally, those objects are passed to the built-in Sphinx machinery to write the html output."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## The big questions\n",
    "\n",
    "This is great! We now understand why we need Sphinx to generate the expected HTML output\n",
    "for roles and directives. Several questions now arise:\n",
    "\n",
    "* Do we want to have Sphinx-free support for rendering roles and directives in the MyST-Parser \n",
    "\n",
    "    Nikola (among other projects) already have their own machinery (based in Docutils) to\n",
    "    build the final HTML output. Getting the docutils object from the Python API would be\n",
    "    a super interesting way to easily expose and provide that object to the downstream\n",
    "    projects!\n",
    "\n",
    "    One caveat with this approach would be missing some Sphinx features (ie. cross-linking\n",
    "    between documents) based on custom roles and directives that we may need to re-implement.\n",
    "\n",
    "* Do we want to have docutils-free support for roles and directives in the MyST Python API?\n",
    "\n",
    "    Docutils actually introduces the roles and directives concept (that Sphinx extend) so if\n",
    "    we want to go docutils-free, then we will need to re-implement those concepts.\n",
    "\n",
    "* Does it makes sense to create a docutils alternative in Python? How much of its functionality would need to be replicated? How should it be extended or enhanced?\n",
    "\n",
    "    There is currently a nice example about an alternative implementation from the\n",
    "    Executable Books community, but in the Javascript world: https://github.com/executablebooks/markdown-it-docutils ;-).\n",
    "    Since we do not have Docutils there, it actually makes a lot of sense to write something\n",
    "    new. But, what is the value/need/place for an alternative implementation in Python?\n",
    "    Maybe, we do not need the whole Docutils stuff... but we maybe need some core\n",
    "    functionality?\n",
    "\n",
    "    If we decide to write some minimal support, what pieces are we interested to bring first?\n",
    "    Where those pieces should end up? The `markdown-it-docutils` package I referenced above is\n",
    "    actually a `markdown-it` (JS) plugin. If we follow that pattern, we should create a new\n",
    "    `markdown-it-py-docutils` plugin and we are no longer in the MyST-Parser territory.\n",
    "    But the MyST-Parser has, in fact, some [parsing directive functions](https://github.com/executablebooks/MyST-Parser/blob/master/myst_parser/parse_directives.py).\n",
    "    We may need to move that toward `markdown-it-py` as the [JS plugin does](https://github.com/executablebooks/markdown-it-docutils/blob/main/src/directives/main.ts).\n",
    "    That sounds nice, but... is there any other suitable (simpler) alternatives besides the\n",
    "    one I proposed above?\n",
    "\n",
    "Finally,in the MyST community, there are some ongoing discussions about developing a\n",
    "MyST specification that represent what we understand as the MyST language. Having one\n",
    "specification could be interesting and super useful because different actors interested\n",
    "in writing a MyST parser for their own space can check their stuff against that\n",
    "specification. Those actors could be different programming languages such as JS or Python\n",
    "or even more, it could be different flavors in the same programming language, such as\n",
    "Docutils, Sphinx, or a theoretical `markdown-it-py-docutils` ;-)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Show time... not really ;-)\n",
    "\n",
    "OK, again, this is long enough! In the next post I will try to start answering some of\n",
    "these questions showcasing alternative approaches for the Nikola use case.\n",
    "\n",
    "I hope you enjoyed the ride and I will see you soon with the third part!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  },
  "nikola": {
   "category": "",
   "date": "2021-08-31 11:35:28 UTC-03:00",
   "description": "",
   "link": "",
   "slug": "a-deep-dive-into-myst-part-2-the-myst-parser-docutils-and-sphinx",
   "tags": "Jupyter, python, nikola, MyST, docutils, sphinx, opensource",
   "title": "A deep dive into MyST, Part 2: The MyST-Parser, Docutils and Sphinx",
   "type": "text"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}