Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(release): update pypi packaging #604

Open
wants to merge 46 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 38 commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
28359f4
feat: first pass at new packaging
thomasrockhu-codecov Jan 26, 2025
628be10
fix: skip pypy3.9
thomasrockhu-codecov Jan 26, 2025
cd25cc3
fix: add wildcard on pp39
thomasrockhu-codecov Jan 26, 2025
523cb96
fix: no pypy
thomasrockhu-codecov Jan 26, 2025
8ac380f
fix: add in separate publish step
thomasrockhu-codecov Jan 26, 2025
1bf1eb9
fix: update requirements.txt
thomasrockhu-codecov Jan 26, 2025
4f76ae7
fix: pipe
thomasrockhu-codecov Jan 26, 2025
4f8a082
fix: ls first
thomasrockhu-codecov Jan 26, 2025
70afd28
fix: wheelhouse
thomasrockhu-codecov Jan 26, 2025
e0501e9
fix: change the name
thomasrockhu-codecov Jan 26, 2025
5741400
fix: download all artifacts
thomasrockhu-codecov Jan 26, 2025
7440fc8
fix: update tool.setuptools
thomasrockhu-codecov Jan 26, 2025
c21c77b
fix: split the pypi and asset building
thomasrockhu-codecov Jan 26, 2025
a54b07c
fix: use CODECOV_ENV
thomasrockhu-codecov Jan 26, 2025
4e1317d
fix: do some more jiggling
thomasrockhu-codecov Jan 27, 2025
7ff9954
fix: remove log
thomasrockhu-codecov Jan 27, 2025
58adb2f
fix: make it a file
thomasrockhu-codecov Jan 27, 2025
5af56d3
fix: maybe like this
thomasrockhu-codecov Jan 27, 2025
64b4014
fix: remove setup.py cases
thomasrockhu-codecov Jan 27, 2025
0cccacf
fix: show all files
thomasrockhu-codecov Jan 27, 2025
29aefe9
fix: add push tags
thomasrockhu-codecov Jan 27, 2025
c8f1ee6
Update pyproject.toml
thomasrockhu-codecov Jan 27, 2025
18e1b6f
fix: address some comments
thomasrockhu-codecov Jan 27, 2025
84b3110
fix: more comments
thomasrockhu-codecov Jan 27, 2025
ae13da2
fix: versioning
thomasrockhu-codecov Jan 27, 2025
d1de10c
fix: versioning
thomasrockhu-codecov Jan 28, 2025
eed38e0
fix: remove dynamic
thomasrockhu-codecov Jan 28, 2025
c6db499
fix: change pattern
thomasrockhu-codecov Jan 28, 2025
7cabf3f
fix: update the patterns
thomasrockhu-codecov Jan 28, 2025
930fd02
fix: dist/
thomasrockhu-codecov Jan 28, 2025
90ac85c
fix: just dist
thomasrockhu-codecov Jan 28, 2025
1fcbe17
fix: ls
thomasrockhu-codecov Jan 28, 2025
6681711
fix: oops
thomasrockhu-codecov Jan 28, 2025
a8b0785
fix: whats in the dist
thomasrockhu-codecov Jan 28, 2025
19b0087
fix: how about wildcard now
thomasrockhu-codecov Jan 28, 2025
159cc0a
fix: now with codecov
thomasrockhu-codecov Jan 28, 2025
0c2f99d
fix: hack it
thomasrockhu-codecov Jan 28, 2025
63754d6
fix: use steps
thomasrockhu-codecov Jan 28, 2025
5c40814
fix: address comments
thomasrockhu-codecov Jan 30, 2025
badf918
fix: add env to cibuildwheel
thomasrockhu-codecov Jan 30, 2025
ca1386f
fix: formatting
thomasrockhu-codecov Jan 30, 2025
4656ba8
fix: nope
thomasrockhu-codecov Jan 30, 2025
b0706b6
fix: try with manifest
thomasrockhu-codecov Jan 31, 2025
4be3b24
fix: address comments
thomasrockhu-codecov Jan 31, 2025
3d16042
Update .github/workflows/create_release.yml
thomasrockhu-codecov Feb 1, 2025
beb3cca
Update pyproject.toml
thomasrockhu-codecov Feb 1, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 11 additions & 10 deletions .github/workflows/build_assets.yml
thomasrockhu-codecov marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,13 @@
name: Build Compiled Assets

on:
workflow_call:
inputs:
release:
type: boolean
default: false
description: "Attach artifacts to a release"
pull_request:
branches:
- main
push:
# Pattern matched against refs/tags
tags:
- 'v[0-9]+.[0-9]+.[0-9]+'

jobs:
build_assets:
Expand Down Expand Up @@ -68,12 +69,12 @@ jobs:
run: ${{matrix.CMD_BUILD}}
- name: Upload a Build Artifact
uses: actions/upload-artifact@v4
if: inputs.release == false
if: github.ref_type == 'tag'
with:
name: ${{ matrix.OUT_FILE_NAME }}
path: ./dist/${{ matrix.OUT_FILE_NAME }}
- name: Upload Release Asset
if: inputs.release == true
if: github.ref_type == 'tag'
id: upload-release-asset
uses: svenstaro/upload-release-action@v2
with:
Expand Down Expand Up @@ -118,12 +119,12 @@ jobs:
./scripts/build_${{ matrix.distro_name }}_arm.sh ${{ matrix.distro_name }}_${{ matrix.arch }}
- name: Upload a Build Artifact
uses: actions/upload-artifact@v4
if: inputs.release == false
if: github.ref_type == 'tag'
with:
name: codecovcli_${{ matrix.distro_name }}_${{ matrix.arch }}
path: ./dist/codecovcli_${{ matrix.distro_name }}_${{ matrix.arch }}
- name: Upload Release Asset
if: inputs.release == true
if: github.ref_type == 'tag'
id: upload-release-asset
uses: svenstaro/upload-release-action@v2
with:
Expand Down
98 changes: 69 additions & 29 deletions .github/workflows/build_for_pypi.yml
thomasrockhu-codecov marked this conversation as resolved.
Show resolved Hide resolved
Original file line number Diff line number Diff line change
@@ -1,47 +1,87 @@
---
name: Build and Optionally Publish to PyPi

on:
workflow_call:
inputs:
publish:
type: boolean
default: false
description: "Publish to PyPi"
pull_request:
branches:
- main
push:
# Pattern matched against refs/tags
tags:
- 'v[0-9]+.[0-9]+.[0-9]+'

jobs:
build_for_pypi:
build_src_for_pypi:
runs-on: ubuntu-latest
permissions:
id-token: write # This is required for requesting the JWT
contents: read # This is required for actions/checkout
steps:
- uses: actions/checkout@v4
with:
persist-credentials: false
submodules: true
- name: Set up Python 3.11
uses: actions/setup-python@v4
with:
python-version: "3.11"
- name: Install dependencies
run: |
pip install -r requirements.txt
python setup.py build
python setup.py develop
- name: Build distributions for different platforms
pip install build
- name: Build src dist
run: |
pip install wheel
python setup.py sdist bdist_wheel --plat-name=manylinux2014_x86_64
python setup.py bdist_wheel --plat-name=macosx-12.6-universal2
python setup.py bdist_wheel --plat-name=win_amd64
- name: Move src file to match PEP625
python -m build --sdist
env:
PIP_CONSTRAINT: requirements.txt
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Try also setting this somewhere in the cibuildwheel config to have the pins there too. This might need some experimentation, though.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hmm, I'm not sure that will be as easily possible if there is no checkout-like step for building the dist wheels. any advice?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@thomasrockhu-codecov the trick is to include all the source files into sdist. When not using setuptools-scm that does this automatically, you'll have to add them into MANIFEST.in manually. sdist is your checkout in this context.

Ideally, sdist contents should be as close to Git checkout as possible. Many downstreams use sdists as the source of truth, meaning that they don't just build wheels from sdist, but also run the tests and build the docs out of it. Well, if said sdist is well maintained and doesn't rip them out forcing working with Git, that is.

- name: Store the distribution packages
uses: actions/upload-artifact@v4
with:
name: cibw-sdist
path: ./**/*.tar.gz

build_dist_for_pypi:
needs:
- build_src_for_pypi
runs-on: ${{ matrix.os }}
strategy:
matrix:
os:
- macos-13
- macos-latest
- ubuntu-24.04-arm
- ubuntu-latest
- windows-latest
steps:
- name: Download the sdist
uses: actions/download-artifact@v4
with:
name: cibw-sdist
- name: Get sdist filename
id: get-sdist
run: |
cd dist
ls | grep *.tar.gz | xargs -I x mv -- x "$(ls | grep *.tar.gz | sed 's/codecov-cli/codecov_cli/')"
ls -al
cd ..
echo "sdist_filename=$(ls dist/)" >> $GITHUB_OUTPUT
thomasrockhu-codecov marked this conversation as resolved.
Show resolved Hide resolved
- name: Build wheels
uses: pypa/[email protected]
with:
package-dir: dist/${{ steps.get-sdist.outputs.sdist_filename }}
- name: Store the distribution packages
uses: actions/upload-artifact@v4
with:
name: cibw-wheels-${{ matrix.os }}
path: ./wheelhouse/*.whl


publish_to_pypi:
thomasrockhu-codecov marked this conversation as resolved.
Show resolved Hide resolved
needs:
- build_dist_for_pypi
- build_src_for_pypi
permissions:
id-token: write # This is required for OIDC
runs-on: ubuntu-latest
steps:
- name: Download source
uses: actions/download-artifact@v4
with:
pattern: ./**/*.tar.gz
- name: Download all wheels
uses: actions/download-artifact@v4
with:
pattern: ./**/*.whl
thomasrockhu-codecov marked this conversation as resolved.
Show resolved Hide resolved
- name: Publish package to PyPi
if: inputs.publish == true
if: github.ref_type == 'tag'
thomasrockhu-codecov marked this conversation as resolved.
Show resolved Hide resolved
uses: pypa/gh-action-pypi-publish@release/v1
with:
attestations: false
verbose: true
17 changes: 0 additions & 17 deletions .github/workflows/build_main.yml

This file was deleted.

2 changes: 2 additions & 0 deletions .github/workflows/ci-job.yml
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,8 @@ jobs:
- name: Test with pytest
run: |
pytest --cov --junitxml=3.12junit.xml
env:
CODECOV_ENV: test

- name: Upload coverage to Codecov
uses: codecov/codecov-action@v5
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,8 @@ jobs:
- name: Test with pytest
run: |
pytest --cov --junitxml=${{matrix.python-version}}junit.xml
env:
CODECOV_ENV: test
- name: Dogfooding codecov-cli
if: ${{ !github.event.pull_request.head.repo.fork && github.repository_owner == 'codecov' }}
run: |
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/create_release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ jobs:
- id: get-release-vars
name: Configure Release Vars
run: |
echo release_version=v$(grep -E "version=\"[0-9]+\.[0-9]+\.[0-9]+\"" setup.py | grep -Eo "[0-9]+\.[0-9]+\.[0-9]+") >> "$GITHUB_OUTPUT"
echo release_version=v$(grep -E "version=\"[0-9]+\.[0-9]+\.[0-9]+\"" pyproject.toml | grep -Eo "[0-9]+\.[0-9]+\.[0-9]+") >> "$GITHUB_OUTPUT"
thomasrockhu-codecov marked this conversation as resolved.
Show resolved Hide resolved
echo previous_version=$(git tag --sort=-creatordate | head -n 2 | tail -n 1) >> "$GITHUB_OUTPUT"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the Python version in this job is 3.11+, you can just use the stdlib module for reading TOML:

Suggested change
echo previous_version=$(git tag --sort=-creatordate | head -n 2 | tail -n 1) >> "$GITHUB_OUTPUT"
echo previous_version=$(git tag --sort=-creatordate | head -n 2 | tail -n 1) >> "$GITHUB_OUTPUT"
- name: Extract hardcoded dist version from PEP 621 config
id: dist-meta
run: |
from os import environ
from pathlib import Path
from tomllib import loads as convert_toml_text_to_dict
FILE_APPEND_MODE = 'a'
pyproject_toml_txt = Path("pyproject.toml").read_text(encoding="utf-8")
pyproject_toml = convert_toml_text_to_dict(pyproject_toml_txt)
version_string = pyproject_toml["project"]["version"]
with Path(environ['GITHUB_OUTPUT']).open(
mode=FILE_APPEND_MODE,
) as outputs_file:
print(f'release-version={version_string !s}', file=outputs_file)
shell: python

(if not — call actions/setup-python earlier to ensure it's modern enough)

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(note that I'm setting release-version over release_version, and a new step ID here because it's usually helpful to have kebab-case identifiers in GHA, making it easier to distinguish them from identifiers from other embedded scripts)


- name: Create GitHub Release
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/create_release_pr.yml
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,8 @@ jobs:
- name: Update version and push
id: make-commit
run: |
sed -i 's/version="[0-9]\+\.[0-9]\+\.[0-9]\+"/version="${{ github.event.inputs.versionName }}"/g' setup.py
git add setup.py
sed -i 's/version="[0-9]\+\.[0-9]\+\.[0-9]\+"/version="${{ github.event.inputs.versionName }}"/g' pyproject.toml
thomasrockhu-codecov marked this conversation as resolved.
Show resolved Hide resolved
git add pyproject.toml
git commit -S --message "Prepare release ${{ github.event.inputs.versionName }}"
echo "commit=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT"
git push origin release/${{ github.event.inputs.versionName }}
Expand Down
3 changes: 3 additions & 0 deletions .pip-tools.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[tool.pip-tools]
all-build-deps = true
all-extras = true
Loading
Loading