Port user guide to Sphinx + Markdown

[kinow]

Part of #219

This pull request is only a port of the user guide as-is to Sphinx + Markdown (structure). Other PR's will be created to modify the documents and work on the existing issues (content.)

Based on the documentation of JupyterHub, which renders both Markdown and rst, but also produces an epub, has spellchecking, and other goodies that we can copy if needed (I'll drop a thank you in their chat room once it's done.)

I used the same Sphinx theme too, as it's used in several PyData projects, so I guess many users of CWL might be already familiar and comfortable with this theme. It's also not too different than the original, but its look n' feel looks better I think? Will add a preview in the comment below.

• Add Sphinx
• Add minimal dependencies
• Add a theme
• Render existing Markdown files
  • Adjust changes in Liquid & MyST Markdown versions
    • comments (% text in MyST)
    • notes/alerts/warnings (```{note|warning}``` in MyST)
    • includes (```{include} dir/file.md``` in MyST)
    • links (no directives for page root in MyST/sphinx, I think)
      • The link in the format text need to be updated to [text][ref]
    • layouts (instead of Jekyll layouts, we need to use Sphinx templates)
    • fix responsive top menu (see comment below "On Firefox for Android the nav tree takes up the first few screens when browsing any one of the episodes")
    • pygments syntax highlight (we have several code blocks with ````cwl`, but that's not available in pygments; we will need a custom lexer; doesn't have to be perfect at this point)
      • the error in the build appeared as WARNING: Pygments lexer name 'cwl' is not known, and I fixed it for now using the YamlLexer for CWL files 😐
    • redirects for pages that changed URL, or even better, try to fix so that we keep using the old URLs (e.g. see the episodes' URLs)
• Build and display somewhere, probably RTD, and/or Netlify, or GH actions, etc (users must see a preview too)
  • GH - No easy preview in GH Actions ¹²
  • RTD - https://docs.readthedocs.io/en/stable/pull-requests.html ✔️
    • https://common-workflow-languageuser-guide.readthedocs.io/en/latest/
  • Netlify (it's kind of working, so might be the easiest to pick, see their bot's comment here)
  • Can we build and preview with RTD, and then push/deploy in GitHub Pages?

Footnotes

1. https://github.com/github/feedback/discussions/7730 ↩

2. https://github.com/rossjrw/pr-preview-action ↩

[kinow]

CI will be broken for a while. I can use that Git commit message that skips CI (don't recall if it's different per CI check?), or subscribers to this issue can ignore it until it's squashed and ready for review.

I rendered the existing Markdown files, but the menu and parsing of Markdown files need some adjusting. Here's what the index look like in both desktop & mobile viewports (pydata theme is responsive), and also the setup for (random example.)

[kinow]

To build it, either make singlehtml then serve the _build/html/index.html somehow (e.g. python -m SimpleHttpServer or similar) or use make watch, which will use sphinx-autobuild to watch and rebuild the project when any files change.

Markdown comments fixed. Now let's think about these includes....

[kinow]

I think the Python version in the Netlify build needs to be updated. I don't have much experience managing Netlify, but their docs seem helpful. @mr-c I've created a user on Netlify with my GitHub. Do you know how can I access the config for the builds in this repository, please?

[mr-c]

See https://docs.netlify.com/configure-builds/file-based-configuration/ ?

[mr-c]

Or better yet, just use readthedocs.org to do the rendering?

[kinow]

Tested on RTD, built successfully, will look at adding the webhooks later, so I don't have to re-build it after each commit.

https://common-workflow-languageuser-guide.readthedocs.io/en/latest/?

[kinow]

For the RTD webhook, I'd need someone to grant me karma to the settings of the project. At the moment that option is not showing for my user.

Ref: https://docs.readthedocs.io/en/stable/integrations.html

As workaround I can just ask RTD to re-publish the docs manually.

[mr-c]

@kinow I upped your permissions; is that enough?

[kinow]

@kinow I upped your permissions; is that enough?

I can see the Settings option now, but I think I'm still missing some permission. Here's my view of Settings on this repo, without Webhooks:

Here's when I open one of my own repositories:

I can fork and try to enable/use RTD on my branch for now if that's easier, @mr-c .

Thanks!
Bruno

[mr-c]

@kinow try again 🙂

[kinow]

@kinow try again slightly_smiling_face

Webhook created! Now the RTD build should happen automagically. Thanks @mr-c !

[mr-c]

Getting there!

On Firefox for Android the nav tree takes up the first few screens when browsing any one of the episodes

https://common-workflow-languageuser-guide.readthedocs.io/en/latest/_episodes/21-1st-workflow.html

[kinow]

On Firefox for Android the nav tree takes up the first few screens when browsing any one of the episodes

Ah, good catch! Let me add to the top comment so I don't forget to fix this one. Thanks @mr-c!

[kinow]

BTW, ReadTheDocs shows the build commands, so we can copy it and run locally. The build process is normally super fast, so troubleshooting build errors is extremely easy 🙂 Happy with RTD so far.

[kinow]

In order to maintain the old links, I've moved the episode files from _episodes to the root directory. Sphinx doesn't have a good way to define the URL of documents like in Jekyll, except for a redirect plug-in.

After moving the episodes, I've also updated the toctree. This caused all the items, the links to each episodes, were added to the top navigation bar.

That forced me to read about the Sphinx toctree (I've used it, but never really had to really read the docs), and also look at issues.

It looks like it's hard to customize the navigation bar, or to use a template to dynamically create a menu out of a toctree. Difficult, but not impossible:

• Customisable navigation pradyunsg/sphinx-basic-ng#16
• Make toctree and toc available in themes for templating sphinx-doc/sphinx#6676
• https://github.com/schettino72/sphinx_press_theme/blob/master/sphinx_press_theme/__init__.py#L11

@mr-c, @tetron should I try that, so that we have a Sphinx user guide with 100% of what we have in the user guide?

Asking because if we won't use this Carpentries episodes, and instead will re-write the user guide, then this effort would then be in vain, I think.

WDYT?

[mr-c]

Thanks for the progress update! Let's not try to recreate the same layout as the carpentries template, no.

Keep it simple and functional for now

[kinow]

I think most of the current user guide has been ported to Sphinx now. I've squashed the changes, but I think it might be complicated to review as the PR has 119 files changed.

Here's a summary of what's done or pending:

• Added the dependencies for Sphinx (see requirements.txt)
• Updated the Makefile to build with Sphinx (e.g. make html, or make watch for dev)
• Tried to keep the previous URL's
• Sphinx doesn't have a native way to create drop down menus. It uses the toctree to create the navigation menus. Instead of a drop down, I created a single page for Episodes and another one for Extras.
• I used the PyData theme, same theme used by Pandas, Numpy, JupyterHub, and others. It uses more of the display area to show extra sidebars. Initially it might look confusing, but I got used after a few minutes (first time I've seen it was with JupyterHub, used in my previous $work)
• Some links moved around, like Edit Page is by default on the sidebar, the version changer is at the top (there was none before), and other links are gone for now, like the Google Translate link. I left a comment for the footer links, but we can use some templating and/or Python to customized the theme. It only depends on what others prefer.
• The Front Matter content is allowed in Sphinx, but we are not using the parts for the lessons. In theory we could extend Sphinx (Python) to use it and create an almost identical layout, but that'd take a little longer. I decided to leave it for now since we might re-write most of the current content, and adopt a different structure too.

I created a RTD project, https://common-workflow-languageuser-guide.readthedocs.io/en/latest/, but we can rename it or create another one. I'm using it just to build & validate the changes (e.g. the version changer only works in RTD as locally browsers will block a request to fetch the JSON - CORS).

I think we can continue using it, and now start modifying the contents? Any thoughts? cc @mr-c, @tetron

Cheers
Bruno

[kinow]

I think we can add a link to the generated CONTRIBUTING.md here, and Cite and Contact as in the user guide. Or other links? WDYT?

[kinow]

Added a new item to the tasks list for checking how to build and preview it on RTD (as we are currently doing) and deploy to GitHub Pages 👍 I know we can simply use something like ghp-import or one of the GH Actions that do that, but I will think about pros/cons, and try to come up with something that looks simple & future-proof. So if anyone reviews this PR before, just consider that it can be pushed to a GH repository and used as GH Pages 👍

[kinow]

@mr-c I cannot push to this branch any more. Should I close and open it from another branch from my fork? Thanks!

[kinow]

@mr-c , @tetron , I spent some time looking at the links and content, and it looks to me like the links are the same as the existing user guide. I've added the footer links, except the one to edit the page (it's always displayed on the side bar) 👍

[kinow]

The last commit removes the version switchers at the top menu and at the right-sidebar. It also fixes a Markdown link in the “Tip” box on the main page 👍

[kinow]

@mr-c, @tetron, I'm reviewing some technical writing docs and looking at other guides, so had a video playing while I set up a test branch to experiment with GitHub Pages + Sphinx.

The branch is here, with a single commit adding a workflow: https://github.com/kinow/user_guide/tree/now-with-gh-pages

The workflow does the following:

• check out
• set up python
• use a cache for pip
• install dependencies (using cache if possible)
• make html to build docs
• uses a GH action to publish it to a branch, gh-pages-2

I realized half-way through writing the workflow that the User Guide source is actually in our gh-pages (markdown files), so had to use another branch (side-note: there are other actions to publish sphinx direclty to gh-pages, but they have gh-pages hard-coded, even though GitHub allows for users to choose the branch to use.)

You can look at the result here: https://kinow.github.io/user_guide/

If you get a page that looks like a personal blog, it could be my blog cached in your browser (assuming you ever opened kinow.github.io). In which case, you can try to force-refresh the page, or add a ?v=1234 so that the browser skips the cache and fetches a new page 👍

Let me know if I should add this action to this pull request, or if we should do it later. And whether we should switch branches, etc (I had to manually point GH Pages to use gh-pages-2 in my fork).

-Bruno

[kinow]

Rebased. Note that the Netlify build won't work as it is not configured for Sphinx (it's probably configured for Jekyll).

[mr-c]

@kinow thanks for the updates!

https://kinow.github.io/user_guide/reference.html has some template errors

But honestly, it is not really useful. Maybe better to remove from this version?

https://kinow.github.io/user_guide//yaml/#yaml-guide has a link error

Other than that, this looks almost ready to publish!

[kinow]

Thanks for the review @mr-c ! Will update it in the morning (in ~8 hours) 👍

[kinow]

@mr-c I removed the Reference page, and then thought that YAML link would be quick to fix…

I can confirm the _includes/links.md is being included in that page. But reference links like [YAML][yaml] are not working when referencing links from the included page — they work if you have the referenced link defined in the same Markdown document.

Then I had a look, and there are links like that in lessons too 😥 My guesses now are

• It could be a dependency that just needs updating, maybe a bug introduced when I upgraded it before
• It could be a setting or extension that I enabled, because I am quite sure these links worked at some point in this PR
• Or... it could be a bug in MyST

If it's a bug in MyST, we can probably work around it by defining a function for parsing links (that's doable in Sphinx, even when using something like MyST for parsing Markdown).

Doing some detective work now 🔍

Thanks!

[kinow]

But honestly, it is not really useful. Maybe better to remove from this version?

References page and menu entries pointing to it (and another resource that's not used I think) removed 👍

[kinow]

Fixed reference links (the [YAML][yaml]). Also fixed admonitions (tip/note/hint/etc), where we had some text with > causing it to have a blockquote inside.

Finally, also fixed a couple typos, and added the syntax highlighting to commands. Note that we can use cwl. That's an alias for YAML (see conf.py), but doing that now, later if we have a custom lexer then we can have a better syntax highlight for CWL in our docs.

Pending now is fix the long menu that appears in responsive mode, when browsing lessons. Then push everything to my fork to get it deployed to GH Pages. Compare with Read The Docs, and then hand over to @mr-c 😬

@mr-c I think others have the same complaint/issue that we have here: pydata/pydata-sphinx-theme#726

Will continue in the morrow.

[kinow]

@mr-c , I re-read the PyData sphinx theme docs, and from what I understood, there are three columns in the theme by default.

The left sidebar contains, by default, the search and the list of pages in the same level of the currently visualized page (defined by a toctree).

And the sidebar contains the section links (headings) within the page. Their docs use that format: https://pydata-sphinx-theme.readthedocs.io/en/latest/index.html

In the Episodes page (episodes.md), we have a toctree so that we can create a dynamic listing of episodes. So when we open an episode, the sidebar displays every other page in the toctree (which was defined in episodes.md). The problem being that, as you pointed, it's a bad user experience in mobile. My guess is that the best option here will be waiting for them to fix in the next release — hopefully — the issue to collapse that toctree listing in mobile pydata/pydata-sphinx-theme#726

Then, in the meantime, I added the last commit that has a work around to hide the left sidebar when we open on mobile 😬 WDYT? That's definitely not perfect, but that should allow users to navigate the user guide without being too distracted by that long list. Mobile users will have to resort to clicking on Previous / Next, or toggling the menu and clicking on Episodes, but I guess that should be fine?

If that's OK, then I believe this is ready for review. I've fixed all the Sphinx build errors today (we cannot use headings like ## inside admonitions as MyST logs a warning and replaces by a rst rubric; no way to call that in MyST, so I simply typed <p class="rubric">Text</p> to fix the warning and get the same HTML output).

RTD has built the latest commit, and I pushed it to my fork, so GH-pages is up to date as well.

• https://common-workflow-languageuser-guide.readthedocs.io/en/latest/
• https://kinow.github.io/user_guide/

I went through every page in the navigation, checking if it "looked" good, then opened the source code, confirmed everything looked OK.

Feel free to take over if you'd like, and change the branch names, deployment options, netlify, RTD, etc 🙂 And also let me know if you need any help, please.

Thank you!
-Bruno

[mr-c]

Accepted and merged to the new main branch; thank you @kinow !

[kinow]

Accepted and merged to the new main branch; thank you @kinow !

💥 thank you @mr-c !!!