feat: update conda pinnings and build docs (xref #736) (#740)

* Update local building instructions. Xref #736 * Base image has been updated to python 3.9 it seems * oops * needs with * Update conda for support of newer python versions * Update bioconda_utils-requirements.txt Co-authored-by: Devon Ryan <[email protected]>
bioconda · Feb 20, 2022 · 53db163 · 53db163
1 parent d657694
commit 53db163
Show file tree

Hide file tree

Showing 6 changed files with 32 additions and 80 deletions.
diff --git a/.circleci/config.yml b/.circleci/config.yml
@@ -32,7 +32,8 @@ variables:
     macos:
       xcode: "9.4.1"
   linux: &linux
-    machine: true
+    machine:
+      image: ubuntu-2004:202201-02
   install_bioconda_utils: &install_bioconda_utils
     run:
       name: Install bioconda-utils

diff --git a/.github/workflows/build-image.yml b/.github/workflows/build-image.yml
@@ -13,7 +13,8 @@ jobs:
 
     steps:
     - uses: actions/checkout@v2
-      fetch-depth: 0
+      with:
+        fetch-depth: 0
 
     - id: get-tag
       run: |
@@ -36,4 +37,4 @@ jobs:
         image='${{ steps.buildah-build.outputs.image }}'
         for tag in ${{ steps.buildah-build.outputs.tags }} ; do
           podman run --rm "${image}:${tag}" bioconda-utils --version
-        done
+        done
diff --git a/Dockerfile b/Dockerfile
@@ -28,6 +28,7 @@ RUN . /opt/conda/etc/profile.d/conda.sh && \
 FROM base as build
 WORKDIR /tmp/repo
 COPY . ./
+RUN . /opt/conda/etc/profile.d/conda.sh  && conda list
 RUN . /opt/conda/etc/profile.d/conda.sh  && conda activate base && \
     pip wheel . && \
     mkdir - /opt/bioconda-utils && \

diff --git a/bioconda_utils/bioconda_utils-requirements.txt b/bioconda_utils/bioconda_utils-requirements.txt
@@ -4,7 +4,7 @@
 
 # basics
 python>=3.7
-conda=4.8.4
+conda=4.11.0
 conda-build=3.21.4
 conda-verify=3.1.*
 argh=0.26.*          # CLI
@@ -20,7 +20,7 @@ boltons=18.*
 jsonschema=2.6.*     # JSON schema verification
 
 # pinnings
-conda-forge-pinning=2022.01.13.01.57.35
+conda-forge-pinning=2022.02.18.13.24.38
 
 # tools
 anaconda-client=1.6.*  # anaconda_upload

diff --git a/docs/source/contributor/build-system.rst b/docs/source/contributor/build-system.rst
@@ -3,15 +3,15 @@ Build system
 
 The build system for Bioconda takes recipes and converts them into conda
 packages that are uploaded to anaconda.org as well as Docker containers that
-are uploaded to quay.io. All of this happens in a transparent way, with all
+are uploaded to quay.io as part of the Biocontainers project. All of this happens in a transparent way, with all
 build logs available for inspection. The code for the build system can be found
 in `bioconda-utils <https://github.com/bioconda/bioconda-utils>`_, but parts
 are also with the ``bioconda-recipes`` repo. This document serves as
 a high-level overview; the code remains the authoritative source on exactly
 what happens during a build.
 
 Why so complicated? We have to work within the constraints of conda-build,
-Docker, and CircleCI, while simultaneously supporting the same build system on
+Docker, and Azure, while simultaneously supporting the same build system on
 a local machine so contributors can test. We also have isolated bioconda-utils
 from bioconda-recipes to better facilitate testing of the infrastructure, and
 to (one day!) make it general enough that others can use the framework for
@@ -22,28 +22,29 @@ simplify, and do so where we can.
 Stages of a bioconda build
 --------------------------
 A GitHub pull request, or any pushed changes to a GitHub pull request, triggers
-a new build on CircleCI. One build can contain mulitple recipes, limited only
-by the time limit imposed by CircleCI (they have generously given us an
-extended build time).  Each build on CircleCI starts with a fresh VM, so we
+a new build on Azure DevOps. One build can contain mulitple recipes, limited only
+by the time limit imposed by Azure.
+Each build on Azure starts with a fresh VM, so we
 need to create the entire bioconda-build system environment from scratch for
 each build.
 
-When testing locally with ``circleci build``, we use the
+When testing locally, we use the
 ``quay.io/bioconda/bioconda-utils-build-env-cos7`` Docker container to avoid changing the
 local system. This container is defined by `this Dockerfile
 <https://github.com/bioconda/bioconda-utils/blob/master/Dockerfile>`_.
 
 
-Otherwise, when running on CircleCI, a new Linux or MacOS VM is created for
+Otherwise, when running on Azure, a new Linux or MacOS VM is created for
 each build.
 
-The steps are orchestrated by the `circleci config file
-<https://github.com/bioconda/bioconda-recipes/blob/master/.circleci/config.yml>`_.
+The steps are orchestrated by the `Azure config file <https://github.com/bioconda/bioconda-recipes/blob/master/azure-pipeline.yml>`_.
 
 
 Configure the environment
 ~~~~~~~~~~~~~~~~~~~~~~~~~
 
+N.B., due to transitioning to Azure, the remainder of this section is no longer relevant and requires rewritting.
+
 - Configure the CI environment:
     - ``bioconda-recipes: .circleci/config.yml`` is the primary configuration
       file for the steps that are run. See https://circleci.com/docs/2.0/ for

diff --git a/docs/source/contributor/building-locally.rst b/docs/source/contributor/building-locally.rst
@@ -1,69 +1,28 @@
 Testing Recipes Locally
 =======================
 
-Queue times on CircleCI may sometimes make it more convenient and
+Queue times on Azure DevOps may sometimes make it more convenient and
 faster to work on complex packages locally. There are several ways to
 do so, each with their own caveats.
 
 .. contents::
    :local:
 
 
-.. _circleci-client:
+.. _bioconda_utils:
 
-Using the Circle CI Client
-~~~~~~~~~~~~~~~~~~~~~~~~~~
+Using bioconda-utils
+~~~~~~~~~~~~~~~~~~~~
 
-You can execute an almost exact copy of our Linux build pipeline by
-`installing the CircleCI client locally
-<https://circleci.com/docs/2.0/local-cli>`_ and running it from the
-folder where your copy of ``bioconda-recipes`` resides::
+Whether on a CI node or locally, Bioconda packages are built and tested using ``bioconda-utils``.
+You can install ``bioconda-utils`` locally by creating a new conda environment:
 
-    # Ensure the build container is up-to-date
-    docker pull quay.io/bioconda/bioconda-utils-build-env:latest
+.. code-block::
 
-    # Run the build locally
-    circleci build
+    # You can use "conda create" here instead, if you don't have mamba installed
+    mamba create -n bioconda -c conda-forge -c bioconda bioconda-utils
 
-You will have to have "commited" some changes as described above. The
-command will run our build system just as it is run online in a docker
-container. It will detect which recipes where modified using the
-``git`` history and proceed with linting and building those.
-
-Please note that this will only run the Linux build steps.
-
-You can use Docker volume bind-mounts to capture the local package
-channel with the newly built packages::
-
-    mkdir /tmp/conda-bld
-    circleci build --volume /tmp/conda-bld:/opt/conda/conda-bld
-
-After a successful build, you can then install from the local channel by
-providing the path to it::
-
-    conda install -c file:///tmp/conda-bld your-package
-
-Note that if the package also exists with the same version or a newer
-version in remote channels, then you may need to specify the exact local
-build string in your package spec to ensure your local build is
-installed, e.g.  ``your-package ==x.y.z build_string``.  See
-`conda-build:resources/package-spec` for more details on how to specify
-packages.
-
-
-.. _bootstrap:
-
-Using the "Bootstrap" Method
-~~~~~~~~~~~~~~~~~~~~~~~~~~~~
-
-Due to technical limitations of the Circle CI client, the above test does
-**not** run the more stringent ``mulled-build`` tests. To do so, use the
-following commands:
-
-.. code-block:: bash
-
-    ./bootstrap.py /tmp/miniconda
-    source ~/.config/bioconda/activate
+    conda activate bioconda
 
     # optional linting
     bioconda-utils lint --git-range master
@@ -73,14 +32,9 @@ following commands:
 
 The above commands do the following:
 
-- Install a separate miniconda installation in a temporary directory:
-   - Set up bioconda channels,
-   - install bioconda-utils dependencies into the root environment of
-     that installation,
-   - and write the file ``~/.config/bioconda/activate``
-- Source that new file to specifically activate the root environment
-  of the new installation
-- Run ``bioconda-utils`` in the new installation:
+- Creates a new environment with bioconda-utils.
+- Activates the new environment. You can later just start at ``conda activate bioconda``
+- Run ``bioconda-utils`` in the new environment:
    - The ``lint`` command will run the lint checks on your recipes
    - The ``build`` command will run the build pipeline
 
@@ -99,13 +53,7 @@ The above commands do the following:
      to be included in the build container.
 
 If you do not have access to Docker, you can still run the basic test by
-telling the bootstrap setup to not use docker, and by excluding the
-``--docker`` and ``--mulled-test`` arguments in the last command::
-
-    ./bootstrap.py --no-docker /tmp/miniconda
-    source ~/.config/bioconda/activate
-    bioconda-utils build --git-range master
-
+omitting the ``--docker`` and ``--mulled-test`` options.
 
 Using the "Debug" Method
 ~~~~~~~~~~~~~~~~~~~~~~~~