build-and-upload-conda-packages

If a repository contains a meta.yaml file to build conda packages, the process of building and uploading packages to an Anaconda user or organization is automatized by this GitHub Action. The user has only to be sure that the repository have access to the corresponding Anaconda token.

In summary, this GitHub Action does the following:

It is suggested that the action is triggered by a release or a prerelease publication.
The action checks if the meta.yaml file exists in the directory specified by the user.
It compiles then the list of packages to the platform and Python version specified by the user.
Finally, the action uploads all packages built to the Anaconda user or organization specified by the user, with the label specified by the user (if no label is specified, the default label main is used).
The paths of the built packages are preserved as an output, to be passed to later jobs.

This GitHub Action was developed by the Computational Biology and Drug Design Research Unit -UIBCDF- at the Mexico City Children's Hospital Federico Gómez. Other GitHub Actions can be found at the UIBCDF GitHub site.

Requirements

In addition to define the workflow to use this action in the '.github/workflow' directory of your repository. There are three things you have to solve: having an Anaconda token to authorize the access to your conda repository or organization, using Git tags as package version numbers in the metadata file with the conda build instructions, and a Yaml file to create a conda environment to work with conda-build.

Anaconda token as GitHub secret

In order to upload a package to your anaconda user or organization, you have to define an Anaconda API token. There are two ways you can easily do it. From the command line you can execute:

# In case of uploading to a user profile
# Replace 'MyToken' with the name you want for your token
TOKEN=$(anaconda auth --create --name MyToken)

or

# Replace 'MyToken' with the name you want for your token
# Replace 'MyOrg' with the name of your organization
TOKEN=$(anaconda auth --create --name MyToken --org MyOrg)

After that, You can access to the value of your token:

echo $TOKEN

There is another way to create and manage your tokens. You can do it with the web page of your Anaconda user or organization. In that page, the menu entrie 'Settings--> Access' shows the following form:

In this form mark the option 'Allow write access to the API site', choose a name and an expiration date for your new token, and click 'Create'. Below this form, you will find the list of your tokens to view their value or remove them.

Now, the last step. The repository where you want to include this GitHub Action has to have access, quietly, to the value of this token. But you have to protect it, including the token string explictly in the workflow is far from being a good idea. To solve this problem, GitHub includes for your GitHub repositories or organizations, the possibility to store 'encrypted secrets'. You can find documentation about it in the GitHub docs site. Let's see here how to store the Anaconda token accesible for any public repository of an organization. Look for the section 'Settings -> Secrets' in the organization main web page in GitHub and click on the button 'New organization scret'. Briefly, choose a new name -this way for the variable storing the token value in GitHub- and copy the value of your token (a long string of "random" characters). As you can see in the secret creation form, you can give access to all public repositories in the organization or just a selected set of them (also to your private ones depending on your plan).

After all this process, what matters to use this GitHub Action is the name of the secret with the Anaconda token value.

Git Tag as package version

The instructions to build the new conda packages are defined by means of a YAML file usually named meta.yaml. It is your work having this file well implemented. Here you can find a guide to decide the building parameters in this file.

In this metadata file, the conda-build recipe includes setting the version number of the new package. To avoid adding it manually, use {{ environ['GIT_DESCRIBE_TAG'] }} in the package: version: field of the meta.yaml file. Therefore, the first lines or your meta.yaml file should look like this:

package:
  name: package_name # write your library name
  version: "{{ environ['GIT_DESCRIBE_TAG'] }}"

source:
  path: ../../

build:
  number: 1

Remember that version numbers including the dash character "-" are not supported by conda-build as package versions. If you want to release a version like "1.0.0-beta.1", replace it with something like "1.0.0b1". Check PEPE-86 verlib conventions for further details.

A Yaml file to create a conda environment to work with conda-build

Building your conda packages requires dependencies that can be solved with a temporary conda environment. You already know that the list of dependencies of your library are specified in your meta.yaml file with the compilation instructions, but the required channels needed to find them need to be provided. In the case of this GitHub Action, these channels are taken from the Yaml file with the details to execute the packages compilation (see the section "Documentation conda environment"). This Yaml file will look like:

channels:    # write here the list of channels to look for your library dependencies
  - uibcdf
  - conda-forge
  - default

dependencies:   # Keep this block with only these two packages
  - anaconda-client
  - conda-build

How to use it

To include this GitHub Action, put a Yaml file (named, for instance, build_and_upload_conda_packages.yaml) with the following content in the .github/workflows directory of your repository. An example of the structure of your build_and_upload_conda_packages.yaml is shown below:

name: Build and upload conda packages

on:
  release:
    types: [released, prereleased]
# workflow_dispatch:        # Un-comment line if you also want to trigger action manually

jobs:
  conda_deployment_with_new_tag:
    name: Conda deployment of package with Python ${{ matrix.python-version }}
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: [3.9, 3.10, 3.11]
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Conda environment creation and activation
        uses: conda-incubator/setup-miniconda@v3
        with:
          python-version: ${{ matrix.python-version }}
          environment-file: path/to/build/env.yaml    # Path to the build conda environment
          auto-update-conda: false
          auto-activate-base: false
          show-channel-urls: true
      - name: Set label
        id: set-label
        shell: bash
        run: |
          if [[ "${{ github.event.action }}" == "prereleased" ]]; then
            label=dev
          else
            label=main
          echo "label=$label" >> $GITHUB_OUTPUT
      - name: Build and upload the conda packages
        uses: uibcdf/[email protected]
        with:
          meta_yaml_dir: path/to/meta_yaml/directory
          python-version: ${{ matrix.python-version }} # Values previously defined in `matrix`
          platform_linux-64: true
          platform_osx-64: true
          platform_win-64: true
          label: ${{ steps.set-label.outputs.label }}
          user: uibcdf # Replace with the right user
          token: ${{ secrets.ANACONDA_TOKEN }} # Replace with the right name of your secret

Host platform package and conversion to other platforms

When the package is built from a pure Python library, conda convert can produce packages for a long list of platforms. Instead, if you have to build a package that contains compiled code, conda convert does not work and only the package for the host platform will be built. In this former case, if you want to produce packages for multiple platforms (for instance, 'linux-64', 'osx-64' and 'win-64`), you can use something like the following:

name: Build and upload conda packages

on:
  release:
    types: [released, prereleased]
# workflow_dispatch:        # Un-comment line if you also want to trigger action manually

jobs:
  conda_deployment_with_new_tag:
    name: Conda deployment of package for platform ${{ matrix.os }} with Python ${{ matrix.python-version }}
    runs-on: ${{ matrix.os }}
    strategy:
      matrix:
        os: [macOS-latest, ubuntu-latest, windows-latest]
        python-version: [3.9, 3.10, 3.11]
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Conda environment creation and activation
        uses: conda-incubator/setup-miniconda@v3
        with:
          python-version: ${{ matrix.python-version }}
          environment-file: path/to/build/env.yaml    # Path to the build conda environment
          auto-update-conda: false
          auto-activate-base: false
          show-channel-urls: true
      - name: Set label
        id: set-label
        shell: bash
        run: |
          if [[ "${{ github.event.action }}" == "prereleased" ]]; then
            label=dev
          else
            label=main
          echo "label=$label" >> $GITHUB_OUTPUT
      - name: Build and upload the conda packages
        uses: uibcdf/[email protected]
        with:
          meta_yaml_dir: path/to/meta_yaml/directory
          python-version: ${{ matrix.python-version }} # Values previously defined in `matrix`
          platform_linux-64: true
          platform_osx-64: true
          platform_win-64: true
          label: ${{ steps.set-label.outputs.label }}
          user: uibcdf # Replace with the right user
          token: ${{ secrets.ANACONDA_TOKEN }} # Replace with the right name of your secret

Note

In this case, is likely that your meta.yaml file need preprocessing selectors to differenciate the build among the different platforms.

Additional command line arguments

This action, internally, calls the following commands:

conda build (or conda mambabuild, if mambabuild is set to true)
conda convert (if any platform conversion is specified)
anaconda upload (if upload is set to true)

The above commands have multiple command-line arguments that can be passed, for each of the respective command, by using the conda_build_args, conda_convert_args and anaconda_upload_args input parameters.

For example, the build_and_upload_conda_packages.yaml file below builds a package with the --override-chanels and --channel my_channel conda build options, to avoid searching for a default or .condarc channel, but only looking at the specified channel. Furthermore, it also passes the --show-imports argument to the conda convert command, to show Python imports for the compiled parts of the package.

name: Build and upload conda packages

on:
  release:
    types: [released, prereleased]
# workflow_dispatch:        # Un-comment line if you also want to trigger action manually

jobs:
  conda_deployment_with_new_tag:
    name: Conda deployment of package with Python ${{ matrix.python-version }}
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: [3.9, 3.10, 3.11]
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Conda environment creation and activation
        uses: conda-incubator/setup-miniconda@v3
        with:
          python-version: ${{ matrix.python-version }}
          environment-file: path/to/build/env.yaml    # Path to the build conda environment
          auto-update-conda: false
          auto-activate-base: false
          show-channel-urls: true
      - name: Set label
        id: set-label
        shell: bash
        run: |
          if [[ "${{ github.event.action }}" == "prereleased" ]]; then
            label=dev
          else
            label=main
          echo "label=$label" >> $GITHUB_OUTPUT
      - name: Build and upload the conda packages
        uses: uibcdf/[email protected]
        with:
          meta_yaml_dir: path/to/meta_yaml/directory
          python-version: ${{ matrix.python-version }} # Values previously defined in `matrix`
          platform_linux-64: true
          platform_osx-64: true
          platform_win-64: true
          label: ${{ steps.set-label.outputs.label }}
          user: uibcdf # Replace with the right user
          token: ${{ secrets.ANACONDA_TOKEN }} # Replace with the right name of your secret
          conda_build_args: --override-chanels --channel my_channel
          conda_convert_args: --show-imports

Warning

Some command line arguments in conda_build_args, conda_convert_args or anaconda_upload_args cannot be declared, as they are already used internally by the action, or if they clash with specific input parameters: conda_build_args

--output-folder
--no-anaconda-upload
--python

conda_convert_args

--output-folder

anaconda_upload_args

--label/-l together with label input
--user/-u together with user input
--force together with overwrite input

Input parameters

Name	Description	Required/Optional	Default value
`meta_yaml_dir`	Path to the directory where the `meta.yaml` file with building instructions is located	Required
`python-version`	Python version of the built packages	Required
`upload`	Upload the built package to Anaconda. Setting to false is useful to test build correctness in CI	Optional	true
`overwrite`	Do not cancel the uploading if a package with the same name is found already in Anaconda	Optional	false
`mambabuild`	Uses boa (mambabuild) to build the packages.	Optional	false
`user`	Name of the Anaconda.org channel to upload the package to	Optional
`token`	Anaconda token for the package uploading (more info here)	Optional
`label`	Label for the uploaded package (`main`, `dev`, ...)	Optional	main
`platform_host`	Build packages for the host platform	Optional	true
`platform_all`	Build packages for all supported platforms	Optional	false
`platform_linux-64`	Build a package for the platform: linux-64	Optional	false
`platform_linux-32`	Build a package for the platform: linux-32	Optional	false
`platform_osx-64`	Build a package for the platform: osx-64	Optional	false
`platform_osx-arm64`	Build a package for the platform: osx-arm64	Optional	false
`platform_linux-ppc64`	Build a package for the platform: linux-ppc64	Optional	false
`platform_linux-ppc64le`	Build a package for the platform: linux-ppc64le	Optional	false
`platform_linux-s390x`	Build a package for the platform: linux-s390x	Optional	false
`platform_linux-armv6l`	Build a package for the platform: linux-armv6l	Optional	false
`platform_linux-armv7l`	Build a package for the platform: linux-armv7l	Optional	false
`platform_linux-aarch64`	Build a package for the platform: linux-aarch64	Optional	false
`platform_win-32`	Build a package for the platform: linux-win-32	Optional	false
`platform_win-64`	Build a package for the platform: linux-win-64	Optional	false
`conda_build_args`	Command line arguments to pass to the `conda build` command.	Optional
`conda_convert_args`	Command line arguments to pass to the `conda convert` command.	Optional
`anaconda_upload_args`	Command line arguments to pass to the `anaconda upload` command.	Optional

They are placed in the with: subsection of the step where the uibcdf/action-build-and-upload-conda-packages action is used:

      - name: Build and upload the conda packages
        uses: uibcdf/[email protected]
        with:
          meta_yaml_dir: devtools/conda-build
          python-version: ${{ matrix.python-version }} # Values previously defined in `matrix`
          platform_linux-64: true
          platform_osx-64: true
          platform_win-64: true
          user: uibcdf
          label: dev
          token: ${{ secrets.ANACONDA_TOKEN }}

At this point, you can probably wonder: but the python version of the package is specified in the meta.yaml file, isn't it?

`python_version`

To define the python versions of the packages created by the GitHub Action you should use the input parameter of the workflow named strategy: matrix: python-version.

    strategy:
      matrix:
        python-version: [3.9, 3.10, 3.11]

No matter the python version found in the meta.yaml file, this action will make all the work to convert the building instructions to the python versions you specify in your GitHub workflow.

Outputs

Name	Description
paths	Space-separated paths for the built packages, in the format `path1 path2 ... pathN`.

The output paths can be useful for later jobs, for example to make a release with the built packages as artifacs.

Testing build correctness

Typically this workflow is called in a deployment step, e.g. after a pull-request has succeeded and a new tag is added to the repository, or when a release is created. This means deployment generally occurs without checking if any errors to the build process were introduced in a pull request or code change.

To address this, the upload option is provided. Setting upload: false will run all the steps in the workflow except uploading to Anaconda.org. This can be useful in a CI workflow to test build correctness, and catch errors during the pull-request process.

For example, this is a CI workflow similar to the deployment workflows above, that only builds the package without uploading it to Anaconda.org:

name: Test Build of conda packages

# Controls when the action will run.
on:
  # Triggers the workflow on push or pull request events but only for the main branch
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

  # Allows you to run this workflow manually from the Actions tab
  workflow_dispatch:

jobs:
  conda_deployment_with_new_tag:
    name: Test conda build of package with Python ${{ matrix.python-version }}
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: [3.9, 3.10, 3.11]
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: Conda environment creation and activation
        uses: conda-incubator/setup-miniconda@v3
        with:
          python-version: ${{ matrix.python-version }}
          environment-file: path/to/build/env.yaml    # Path to the build conda environment
          auto-update-conda: false
          auto-activate-base: false
          show-channel-urls: true
      - name: Build and upload the conda packages
        uses: uibcdf/[email protected]
        with:
          meta_yaml_dir: path/to/meta_yaml/directory
          python-version: ${{ matrix.python-version }} # Values previously defined in `matrix`
          platform_linux-64: true
          platform_osx-64: true
          platform_win-64: true
          upload: false

Other tools like this one

This GitHub Action was developed to solve a need of the UIBCDF. And to be used, additionally, as example of house-made GitHub Action for researchers and students in this lab.

Below, you can find a list of other tools in the market that carry out these same tasks. We recognize and thank the work of their developers. Many of those GitHub Actions were used by us to learn how to set up our own one.

If you think that your GitHub Action should be mentioned here, fell free to open a PR with a new line.

Github Actions for Conda package build/deployment

https://github.com/fdiblen/anaconda-action https://github.com/MichaelsJP/conda-package-publish-action https://github.com/Jegp/conda-package-publish-action https://github.com/amauryval/publish_conda_package_action https://github.com/elbeejay/conda-publish-action https://github.com/maxibor/conda-package-publish-action https://github.com/m0nhawk/conda-package-publish-action https://github.com/fcakyon/conda-publish-action

Name		Name	Last commit message	Last commit date
Latest commit History 90 Commits
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
action.yml		action.yml
create_token.png		create_token.png

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Repository files navigation

build-and-upload-conda-packages

Requirements

Anaconda token as GitHub secret

Git Tag as package version

A Yaml file to create a conda environment to work with conda-build

How to use it

Host platform package and conversion to other platforms

Additional command line arguments

Input parameters

`python_version`

Outputs

Testing build correctness

Other tools like this one

Github Actions for Conda package build/deployment

About

Releases 7

Contributors 3

License

uibcdf/action-build-and-upload-conda-packages

Folders and files

Latest commit

History

Repository files navigation

build-and-upload-conda-packages

Requirements

Anaconda token as GitHub secret

Git Tag as package version

A Yaml file to create a conda environment to work with conda-build

How to use it

Host platform package and conversion to other platforms

Additional command line arguments

Input parameters

python_version

Outputs

Testing build correctness

Other tools like this one

Github Actions for Conda package build/deployment

About

Topics

Resources

License

Stars

Watchers

Forks

Releases 7

Contributors 3

`python_version`