Skip to content

Update pypi publishing #376

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 3 commits into
base: main
Choose a base branch
from
Draft

Update pypi publishing #376

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
9b87b32
first draft
f-hafner Feb 8, 2024
2326f3c
rewrite and add links
f-hafner Mar 21, 2024
83ba05a
add workflow file
f-hafner Mar 22, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
81 changes: 81 additions & 0 deletions {{cookiecutter.directory_name}}/.github/workflows/release.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
name: Publish Python distribution to PyPI and TestPyPI
# Based on https://packaging.python.org/en/latest/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/
# and https://github.com/dianna-ai/dianna/blob/main/.github/workflows/release.yml
on:
workflow_dispatch:
release:
types:
- published

jobs:
build:
name: Build distribution
runs-on: ubuntu-latest

steps:
- uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v4
with:
python-version: "3.x"
- name: Install pypa/build
run: >-
python3 -m
pip install
build
--user
- name: Build a binary wheel and a source tarball
run: python3 -m build
- name: Store the distribution packages
uses: actions/upload-artifact@v3
with:
name: python-package-distributions
path: dist/

publish-to-testpypi:
name: Publish Python distribution to TestPyPI
needs:
- build
if: github.event_name == 'workflow_dispatch'
runs-on: ubuntu-latest
environment:
name: testpypi
url: https://test.pypi.org/p/{{ cookiecutter.package_name }}
permissions:
id-token: write # IMPORTANT: mandatory for trusted publishing

steps:
- name: Download all the dists
uses: actions/download-artifact@v3
with:
name: python-package-distributions
path: dist/
- name: Publish distribution to TestPyPI
uses: pypa/gh-action-pypi-publish@release/v1
with:
repository-url: https://test.pypi.org/legacy/
verbose: true

publish-to-pypi:
name: >-
Publish Python distribution to PyPI
if: github.event_name == 'release' && github.event.action == 'published'
needs:
- build
runs-on: ubuntu-latest
environment:
name: pypi
url: https://pypi.org/p/{{ cookiecutter.package_name }}
permissions:
id-token: write

steps:
- name: Download all the dists
uses: actions/download-artifact@v3
with:
name: python-package-distributions
path: dist/
- name: Publish distribution to PyPI
uses: pypa/gh-action-pypi-publish@release/v1
with:
verbose: true
143 changes: 94 additions & 49 deletions {{cookiecutter.directory_name}}/README.dev.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -121,74 +121,119 @@ bump-my-version minor # bumps from e.g. 0.3.2 to 0.4.0
bump-my-version patch # bumps from e.g. 0.3.2 to 0.3.3
```

## Making a release
## Using Github Actions to make a release of the package

This section describes how to make a release in 3 parts:
This section describes how to make a release with a GitHub actions workflow.

1. preparation
1. making a release on PyPI
1. making a release on GitHub
### Setup
The following steps set up the infrastructure for making releases easy via Github Actions. They are necessary once for your package.

### (1/3) Preparation

1. Update the <CHANGELOG.md> (don't forget to update links at bottom of page)
2. Verify that the information in [`CITATION.cff`](CITATION.cff) is correct.
3. Make sure the [version has been updated](#versioning).
4. Run the unit tests with `pytest -v`
#### (1/2) Define Github workflow

### (2/3) PyPI
This is necessary once per package.

In a new terminal:
You can set up a workflow file following instructions [here](https://packaging.python.org/en/latest/guides/publishing-package-distribution-releases-using-github-actions-ci-cd-workflows/).
The workflow consists of 3 jobs: build, publishing to testpypi, publishing to pypi. The workflow for build is as usual.

```shell
# OPTIONAL: prepare a new directory with fresh git clone to ensure the release
# has the state of origin/main branch
cd $(mktemp -d {{ cookiecutter.package_name }}.XXXXXX)
git clone {{ cookiecutter.repository }} .
The job for publishing on TestPyPI is as follows:

```yml
publish-to-testpypi:
name: Publish Python distribution to TestPyPI
needs:
- build
if: github.event_name == 'workflow_dispatch'
runs-on: ubuntu-latest

# make sure to have a recent version of pip and the publishing dependencies
python -m pip install --upgrade pip
python -m pip install .[publishing]
environment:
name: testpypi
url: https://test.pypi.org/p/{{ cookiecutter.package_name }}

# create the source distribution and the wheel
python -m build
permissions:
id-token: write # IMPORTANT: mandatory for trusted publishing

# upload to test pypi instance (requires credentials)
python -m twine upload --repository testpypi dist/*
steps:
- name: Download all the dists
uses: actions/download-artifact@v3
with:
name: python-package-distributions
path: dist/
- name: Publish distribution to TestPyPI
uses: pypa/gh-action-pypi-publish@release/v1
with:
repository-url: https://test.pypi.org/legacy/
verbose: true # optional; for debugging
```

Visit
[https://test.pypi.org/project/{{cookiecutter.package_name}}](https://test.pypi.org/project/{{cookiecutter.package_name}})
and verify that your package was uploaded successfully. Keep the terminal open, we'll need it later.

In a new terminal, without an activated virtual environment or an env directory:
The job for publishing on PyPI is as follows:

```yml
publish-to-pypi:
name: >-
Publish Python distribution to PyPI
if: github.event_name == 'release' && github.event.action == 'published'
needs:
- build
runs-on: ubuntu-latest
environment:
name: pypi
url: https://pypi.org/p/{{ cookiecutter.package_name }}
permissions:
id-token: write # IMPORTANT: mandatory for trusted publishing

steps:
- name: Download all the dists
uses: actions/download-artifact@v3
with:
name: python-package-distributions
path: dist/
- name: Publish distribution to PyPI
uses: pypa/gh-action-pypi-publish@release/v1
with:
verbose: true

```shell
cd $(mktemp -d {{ cookiecutter.package_name }}-test.XXXXXX)
```

# prepare a clean virtual environment and activate it
python -m venv env
source env/bin/activate
Notes
- The `id-token: write` item is important because it will enable GitHub to publish your package on PyPI (see below). You can read more about how this works [here](https://docs.pypi.org/trusted-publishers/).
- The `url` item for `environment` needs to match exactly the PyPI projecte name, defined in the next step.

# make sure to have a recent version of pip and setuptools
python -m pip install --upgrade pip
#### (2/2) PyPI and TestPyPI

# install from test pypi instance:
python -m pip -v install --no-cache-dir \
--index-url https://test.pypi.org/simple/ \
--extra-index-url https://pypi.org/simple {{ cookiecutter.package_name }}
```
This is necessary once per package.

Check that the package works as it should when installed from pypitest.
On your TestPyPI account, go to "Your Projects", and under "Publishing", fill the form for "Add a new pending publisher". You need to specify:
- {{ cookiecutter.package_name }} as the PyPI project name.
- The workflow name as the name of the `yml` file with the github workflow, for instance `release.yml`
- The environment name as defined in the workflow file, in the above examples this is `testpypi` and `pypi`

Then upload to pypi.org with:
Do the same on your PyPI account.

```shell
# Back to the first terminal,
# FINAL STEP: upload to PyPI (requires credentials)
python -m twine upload dist/*
```

### (3/3) GitHub
### Making a release

With the described setup, you can now make a new release in 3 steps.

#### (1/3) Preparation

1. Update the <CHANGELOG.md> (don't forget to update links at bottom of page)
2. Verify that the information in [`CITATION.cff`](CITATION.cff) is correct.
3. Make sure the [version has been updated](#versioning).
4. Run the unit tests with `pytest -v`


#### (2/3) Make a release to TestPyPI

In your web browser, visit [{{cookiecutter.repository}}/actions/workflows/release.yml]({{cookiecutter.repository}}/actions/workflows/release.yml). Click on "Run workflow".

If the workflow passes, you have successfully released a new version to TestPyPI. Verify this by visiting [https://test.pypi.org/project/{{cookiecutter.package_name}}](https://test.pypi.org/project/{{cookiecutter.package_name}})

If the workflow fails, you should investigate the bug. You can use a [manual upload with twine](https://docs.pypi.org/trusted-publishers/using-a-publisher/#the-manual-way) for this.

#### (3/3) Make a release on Github, triggering a release to PyPI

Don't forget to also make a [release on GitHub]({{cookiecutter.repository_url}}/releases/new). If your repository uses the GitHub-Zenodo integration this will also trigger Zenodo into making a snapshot of your repository and sticking a DOI on it.
If the release to TestPyPI worked, you can now release to PyPI.
Visit [{{cookiecutter.repository}}/releases/new]({{cookiecutter.repository}}/releases/new) and make a new release. When this is done, the github action workflow defined above runs. You can check that the workflow ran correctly, and visit [https://test.pypi.org/project/{{cookiecutter.package_name}}](https://test.pypi.org/project/{{cookiecutter.package_name}}) to make sure the release is on PyPI.
If your repository uses the GitHub-Zenodo integration this will also trigger Zenodo into making a snapshot of your repository and sticking a DOI on it.