Option to use readthedocs to auto build pull requests

[tkknight]

How we build docs now

We only build the docs as part of the CI to ensure a clean build - not to view the docs. Meaning if there is a change to the docs the build version cannot be viewed without building the docs yourself as a reviewer.

Workaround to showing built docs

As a workaround I have in the past used a different readthedocs (rtd) project to share build docs. I created (for free) the rtd project named https://readthedocs.org/projects/tkknight-iris-test-doc/ specifically for this purpose. This project is then manually configured to point to a branch in my iris forked repo and build it when the branch is updated. I then add a ink to the build docs in any pull request. Note that the build docs are only present until another docs pr comes along and I nuke it. The screenshot below shows it is configueed to render the theme_play branch in my forked repo atm (still a working in progress):

A Better way of doing this (maybe)

There is a setting on rtd's to automatically create a build of the pull request, the docs explain this well: https://docs.readthedocs.io/en/latest/pull-requests.html. There is a single checkbox to tick and that's it

I configured this on my forked repo and the rtd build then appears on the pull checks:

And the docs are now built and viewable: https://tkknight-iris-test-doc--6.org.readthedocs.build/en/6/, note the unique url specific to the rtd namespace and the pull request number (6).

Advantages to this...

• Super simple config to setup
• Automate build of the docs for each pr on creation and new commit - making reviewing docs changes easier

Disadvantages to this...

• The rtd build takes time to build as it is limited in compute resource (see https://docs.readthedocs.io/en/stable/builds.html?highlight=build%20limit#build-process), including only 2 concurrent builds (I assume for a given project, in this case for iris). Only having 2 builds may be a bottleneck for other pr's or even for the same pr if you do a commit in quick succession.

• A single build with no queuing took 786s (13mins)
  

• The delay in building the docs will make the pr checks take longer to complete (thus delaying the merge)

Final thoughts

We could enable this to see how well it works, and it it doesn't work out simple disable it.

Brain dump complete, I welcome any thoughts on this. I'n not attached to using this new approach but thought I would share as this has come up in recent chats.

[rcomer]

I think this would be good to have. Even when there is no substantial documentation change, it's still good to check the whatsnew renders nicely 👍

In the longer term, there has been talk of moving our CI to GH Actions to be consistent with other scientific python projects (#4193). Both Cartopy and Matplotlib have links from the PR to built docs which appear to have come out of Circle-CI. These can be viewed by clicking "Details" by "View the built docs" within the checks section of any of their PRs. Maybe worth looking into?

[tkknight]

I think you are right using GH Actions is probably the best way forward. A quick search for sphinx shows many options: https://github.com/marketplace?type=actions&query=sphinx+

I have added a comment to #4193 to this discussion for completeness.

[trexfeathers]

If we do end up using GHA, it would be good if any documentation related runs (doctests, link checks) could share the docs build with the docs render. There's potential for a lot of wasteful duplication but I get the impression GHA could solve that.

[trexfeathers]

This would be great [understatement], but the runtime puts me off. That could almost double our CI run time, and I don't know about anyone else but I'm reluctant to merge before a full set of CI passes.

[pp-mo]

As it is, I don't think the RTD build time would affect the running of the other checks ?
Personally, I'd be happy to wait for the "green button", as long as the main test runs are not delayed. Since that may often show up a problem requiring further work.
However, I'm not clear how adoption of GHA for everything might affect that -- could that even mean that 'main' tests wind up waiting for the docs build ? The existing Cirrus workflow does not seem to prioritise the test runs ideally, sometimes the thing you really want will be waiting for resource.

[trexfeathers]

I was indeed referring to the time taken to get a green tick - 13mins is longer than any of our other runs.

Regarding GHA: I had assumed that we could do a docs build using the GHA machine, which has at least as much resource as a Cirrus machine, and that takes much less than 13mins to run a docs build.

[pp-mo]

Just re-considering this.
Is it possible to enable this, as a useful resource when discussing documentation changes, but not have it used as a "check" from the GitHub green-tick point of view ?

[tkknight]

ReadTheDocs only allows a single checkbox to enable building the docs for a PR, there is no option I can see to not have it part of the PR checks.

@bjlittle overhauled the build approach for the docs and got the build time down to <7 mins or less which is great but it is still time to wait for the PR to have all checks done.

We could enable this as a trial and then turn or off if need be. Github Actions may provide an option too as @rcomer suggested, I have not had time to investigate.