feat: auto-mount edx-platform python requirements

These changes make to possible to run:

    tutor mounts add /path/to/my-xblock

The xblock directory with then be auto-magically bind-mounted in the
"openedx" image at build time, and the lms*/cms* containers at run time.

This makes it effectively possible to work as a developer on
edx-platform requirements.

We take the opportunity to move some openedx-specific code to a
dedicated module.

Close openedx-unsupported/wg-developer-experience#177

changelog.d/20231009_164751_regis_mount_edx_platform_packages.md

@@ -0,0 +1,2 @@
+- [Feature] Make it easy to work on 3rd-party edx-platform Python packages with `tutor mounts add /path/to/my/package`. (by @regisb)
+- [Feature] The ``iter_mounts`` template function can now take multiple image names as argument. This should concern only very advanced users. (by @regisb)

docs/reference/api/hooks/catalog.rst

@@ -16,3 +16,9 @@ The underlying Python hook classes and API are documented :ref:`here <hooks_api>
 
 .. autoclass:: tutor.hooks.Contexts
     :members:
+
+Open edX hooks
+--------------
+
+.. automodule:: tutor.plugins.openedx.hooks
+    :members:

docs/tutorials/edx-platform.rst

@@ -0,0 +1,130 @@
+.. _edx_platform:
+
+Working on edx-platform as a developer
+======================================
+
+Tutor supports running in development with ``tutor dev`` commands. Developers frequently need to work on a fork of some repository. The question then becomes: how to make their changes available within the "openedx" Docker container? 
+
+For instance, when troubleshooting an issue in `edx-platform <https://github.com/openedx/edx-platform>`__, we would like to make some changes to a local fork of that repository, and then apply these changes immediately in the "lms" and the "cms" containers (but also "lms-worker", "cms-worker", etc.)
+
+Similarly, when developing a custom XBlock, we would like to hot-reload any change we make to the XBlock source code within the containers.
+
+Tutor provides a simple solution to these questions. In both cases, the solution takes the form of a ``tutor mounts add ...`` command.
+
+Working on the "edx-platform" repository
+----------------------------------------
+
+Download the code from the upstream repository::
+
+    cd /my/workspace/edx-plaform
+    git clone https://github.com/openedx/edx-platform .
+
+Check out the right version of the upstream repository. If you are working on the `current "zebulon" release <https://docs.openedx.org/en/latest/community/release_notes/index.html>`__ of Open edX, then you should checkout the corresponding branch::
+
+    # "zebulon" is an example. You should put the actual release name here.
+    # I.e: aspen, birch, cypress, etc.
+    git checkout open-release/zebulon.master
+
+On the other hand, if you are working on the Tutor :ref:`"nightly" <nightly>` branch then you should checkout the master branch::
+
+    git checkout master
+
+Then, mount the edx-platform repository with Tutor::
+
+    tutor mounts add /my/workspace/edx-plaform
+
+This command does a few "magical" things 🧙 behind the scenes:
+
+1. Mount the edx-platform repository in the image at build-time. This means that when you run ``tutor images build openedx``, your custom repository will be used instead of the upstream. In particular, any change you've made to the installed requirements, static assets, etc. will be taken into account.
+2. Mount the edx-platform repository at run time. Thus, when you run ``tutor dev start``, any change you make to the edx-platform repository will be hot-reloaded.
+
+You can get a glimpse of how these auto-mounts work by running ``tutor mounts list``. It should output something similar to the following::
+
+    $ tutor mounts list
+    - name: /home/data/regis/projets/overhang/repos/edx/edx-platform
+    build_mounts:
+    - image: openedx
+        context: edx-platform
+    - image: openedx-dev
+        context: edx-platform
+    compose_mounts:
+    - service: lms
+        container_path: /openedx/edx-platform
+    - service: cms
+        container_path: /openedx/edx-platform
+    - service: lms-worker
+        container_path: /openedx/edx-platform
+    - service: cms-worker
+        container_path: /openedx/edx-platform
+    - service: lms-job
+        container_path: /openedx/edx-platform
+    - service: cms-job
+        container_path: /openedx/edx-platform
+
+Working on edx-platform Python dependencies
+-------------------------------------------
+
+Quite often, developers don't want to work on edx-platform directly, but on a dependency of edx-platform. For instance: an XBlock. This works the same way as above. Let's take the example of the `"edx-ora2" <https://github.com/openedx/edx-ora2>`__ package, for open response assessments. First, clone the Python package::
+
+    cd /my/workspace/edx-ora2
+    git clone https://github.com/openedx/edx-ora2 .
+
+Then, check out the right version of the package. This is the version that is indicated in the ``edx-platform/requirements/edx/base.txt``. Be careful that the version that is currently in use in your version of edx-platform is **not necessarily the latest version**::
+
+    git checkout <my-version-tag-or-branch>
+
+Then, mount this repository::
+
+    tutor mounts add /my/workspace/edx-ora2
+
+Verify that your repository is properly bind-mounted by running ``tutor mounts list``::
+
+    $ tutor mounts list
+    - name: /my/workspace/edx-ora2
+    build_mounts:
+    - image: openedx
+        context: mnt-edx-ora2
+    - image: openedx-dev
+        context: mnt-edx-ora2
+    compose_mounts:
+    - service: lms
+        container_path: /mnt/edx-ora2
+    - service: cms
+        container_path: /mnt/edx-ora2
+    - service: lms-worker
+        container_path: /mnt/edx-ora2
+    - service: cms-worker
+        container_path: /mnt/edx-ora2
+    - service: lms-job
+        container_path: /mnt/edx-ora2
+    - service: cms-job
+        container_path: /mnt/edx-ora2
+
+It is quite possible that your package is not automatically recognized and bind-mounted by Tutor. In such a case, you will need to create a :ref:`Tutor plugin <plugin_development_tutorial>` that implements the :py:data:`tutor.hooks.Filters.MOUNTED_DIRECTORIES` filter::
+
+    import tutor import hooks
+    hooks.Filters.MOUNTED_DIRECTORIES.add_item(("openedx", "my-package"))
+
+After you implement and enable that plugin, ``tutor mounts list`` should display your directory among the bind-mounted directories.
+
+You should then re-build the "openedx" Docker image to pick up your changes::
+
+    tutor images build openedx-dev
+
+Then, whenever you run ``tutor dev start``, the "lms" and "cms" container should automatically hot-reload your changes.
+
+To push your changes in production, you should do the same with ``tutor local`` and the "openedx" image::
+
+    tutor images build openedx
+    tutor local start -d
+
+Do I have to re-build the "openedx" Docker image after every change?
+--------------------------------------------------------------------
+
+No, you don't. Re-building the "openedx" Docker image may take a while, and you don't want to run this command every time you make a change to your local repositories. Because your host directory is bind-mounted in the containers at runtime, your changes will be automatically applied to the container. If you run ``tutor dev`` commands, then your changes will be automatically picked up.
+
+If you run ``tutor local`` commands (for instance: when debugging a production instance) then your changes will *not* be automatically picked up. In such a case you should manually restart the containers::
+
+    tutor local restart lms cms lms-worker cms-worker
+
+Re-building the "openedx" image should only be necessary when you want to push your changes to a Docker registry, then pull them on a remote server.

docs/tutorials/index.rst

@@ -9,6 +9,7 @@ Open edX customization
 
    plugin
    theming
+   edx-platform
    edx-platform-settings
    google-smtp
    nightly

tests/test_bindmount.py

@@ -53,13 +53,6 @@ def test_parse_implicit(self) -> None:
         import tutor.commands.compose
 
         self.assertEqual(
-            [
-                ("lms", "/path/to/edx-platform", "/openedx/edx-platform"),
-                ("cms", "/path/to/edx-platform", "/openedx/edx-platform"),
-                ("lms-worker", "/path/to/edx-platform", "/openedx/edx-platform"),
-                ("cms-worker", "/path/to/edx-platform", "/openedx/edx-platform"),
-                ("lms-job", "/path/to/edx-platform", "/openedx/edx-platform"),
-                ("cms-job", "/path/to/edx-platform", "/openedx/edx-platform"),
-            ],
+            [("openedx", "/path/to/edx-platform", "/openedx/edx-platform")],
             bindmount.parse_implicit_mount("/path/to/edx-platform"),
         )

tutor/bindmount.py

@@ -12,7 +12,7 @@ def get_mounts(config: types.Config) -> list[str]:
     return types.get_typed(config, "MOUNTS", list)
 
 
-def iter_mounts(user_mounts: list[str], name: str) -> t.Iterable[str]:
+def iter_mounts(user_mounts: list[str], *names: str) -> t.Iterable[str]:
     """
     Iterate on the bind-mounts that are available to any given compose service. The list
     of bind-mounts is parsed from `user_mounts` and we yield only those for service
@@ -23,7 +23,7 @@ def iter_mounts(user_mounts: list[str], name: str) -> t.Iterable[str]:
     """
     for user_mount in user_mounts:
         for service, host_path, container_path in parse_mount(user_mount):
-            if service == name:
+            if service in names:
                 yield f"{host_path}:{container_path}"
 
 

tutor/commands/compose.py

@@ -426,38 +426,6 @@ def dc_command(
     context.job_runner(config).docker_compose(command, *args)
 
 
-@hooks.Filters.COMPOSE_MOUNTS.add()
-def _mount_edx_platform(
-    volumes: list[tuple[str, str]], name: str
-) -> list[tuple[str, str]]:
-    """
-    When mounting edx-platform with `tutor mounts add /path/to/edx-platform`,
-    bind-mount the host repo in the lms/cms containers.
-    """
-    if name == "edx-platform":
-        path = "/openedx/edx-platform"
-        volumes += [
-            ("lms", path),
-            ("cms", path),
-            ("lms-worker", path),
-            ("cms-worker", path),
-            ("lms-job", path),
-            ("cms-job", path),
-        ]
-    return volumes
-
-
-@hooks.Filters.APP_PUBLIC_HOSTS.add()
-def _edx_platform_public_hosts(
-    hosts: list[str], context_name: t.Literal["local", "dev"]
-) -> list[str]:
-    if context_name == "dev":
-        hosts += ["{{ LMS_HOST }}:8000", "{{ CMS_HOST }}:8001"]
-    else:
-        hosts += ["{{ LMS_HOST }}", "{{ CMS_HOST }}"]
-    return hosts
-
-
 hooks.Filters.ENV_TEMPLATE_VARIABLES.add_item(("iter_mounts", bindmount.iter_mounts))
 
 

tutor/commands/images.py

@@ -274,20 +274,6 @@ def get_image_build_contexts(config: Config) -> dict[str, list[tuple[str, str]]]
     return build_contexts
 
 
-@hooks.Filters.IMAGES_BUILD_MOUNTS.add()
-def _mount_edx_platform(
-    volumes: list[tuple[str, str]], path: str
-) -> list[tuple[str, str]]:
-    """
-    Automatically add an edx-platform repo from the host to the build context whenever
-    it is added to the `MOUNTS` setting.
-    """
-    if os.path.basename(path) == "edx-platform":
-        volumes.append(("openedx", "edx-platform"))
-        volumes.append(("openedx-dev", "edx-platform"))
-    return volumes
-
-
 @click.command(short_help="Pull images from the Docker registry")
 @click.argument("image_names", metavar="image", type=PullImageNameParam(), nargs=-1)
 @click.pass_obj

tutor/hooks/catalog.py

@@ -338,6 +338,7 @@ def your_filter_callback(some_data):
     #: - ``is_buildkit_enabled``: a boolean function that indicates whether BuildKit is available on the host.
     #: - ``iter_values_named``: a function to iterate on variables that start or end with a given string.
     #: - ``iter_mounts``: a function that yields compose-compatible bind-mounts for any given service.
+    #: - ``iter_mounted_directories``: iterate on bind-mounted directory names.
     #: - ``patch``: a function to incorporate extra content into a template.
     #:
     #: :parameter filters: list of (name, value) tuples.
@@ -398,6 +399,53 @@ def your_filter_callback(some_data):
     #: Parameters are the same as for :py:data:`IMAGES_PULL`.
     IMAGES_PUSH: Filter[list[tuple[str, str]], [Config]] = Filter()
 
+    #: List of directories that will be automatically bind-mounted in an image (at
+    #: build-time) and a container (at run-time).
+    #:
+    #: Whenever a user runs: ``tutor mounts add /path/to/name``, "name" will be matched to
+    #: the regular expressions in this filter. If it matches, then the directory will be
+    #: automatically bind-mounted in the matching Docker image at build time and run
+    #: time. At build-time, they will be added to a layer named "mnt-{name}". At
+    #: run-time, they wll be mounted in ``/mnt/<name>``.
+    #:
+    #: In the case of edx-platform, ``pip install : -e .`` will be run in this directory
+    #: at build-time. And the : same host directory will be bind-mounted in that location
+    #: at run time. This : allows users to transparently work on edx-platform
+    #: dependencies, such as Python packages.
+    #:
+    #: By default, xblocks and some common edx-platform packages are already present in
+    #: this : filter, and associated to the "openedx image". Add your own Python
+    #: dependencies to this filter to make it easier for : users to work on the
+    #: dependencies of your app.
+    #:
+    #: See the list of all edx-platform base requirements here:
+    #: https://github.com/openedx/edx-platform/blob/master/requirements/edx/base.txt
+    #:
+    #: This filter was mostly designed for edx-platform, but it can be used by any
+    #: Python-based Docker image as well. The Dockerfile must declare mounted layers::
+    #:
+    #:     {% for name in iter_mounted_directories(MOUNTS, "yourimage") %}
+    #:     FROM scratch as mnt-{{ name }}
+    #:     {% endfor %}
+    #:
+    #: Then, Python packages are installed with::
+    #:
+    #:     {% for name in iter_mounted_directories(MOUNTS, "yourimage") %}
+    #:     COPY --from=mnt-{{ name }} --chown=app:app / /mnt/{{ name }}
+    #:     RUN pip install -e "/mnt/{{ name }}"
+    #:     {% endfor %}
+    #:
+    #: And the docker-compose service must include the following::
+    #:
+    #:    volumes:
+    #:      {%- for mount in iter_mounts(MOUNTS, "yourimage") %}
+    #:      - {{ mount }}
+    #:      {%- endfor %}
+    #:
+    #: :parameter list[tuple[str, str]] name_regex: Each tuple is the name of an image and a
+    #:   regular expression. For instance: ``("openedx", r".*xblock.*")``.
+    MOUNTED_DIRECTORIES: Filter[list[tuple[str, str]], []] = Filter()
+
     #: List of plugin indexes that are loaded when we run ``tutor plugins update``. By
     #: default, the plugin indexes are stored in the user configuration. This filter makes
     #: it possible to extend and modify this list with plugins.

tutor/plugins/__init__.py

@@ -10,7 +10,7 @@
 from tutor.types import Config, get_typed
 
 # Import modules to trigger hook creation
-from . import v0, v1
+from . import openedx, v0, v1
 
 # Cache of plugin patches, for efficiency
 ENV_PATCHES_DICT: dict[str, list[str]] = {}