[infra] add testpypi nightly build

[kevinjqliu]

Closes #872

This PR adds the ability to run and publish pypi artifacts nightly to testpypi. It will publish as pyiceberg with new version under 0.9.0.dev${TIMESTAMP}, i.e. 0.9.0.dev20250206050843, adhering to PEP440 conversion.

Here's a test run on my fork which published to https://test.pypi.org/project/pyiceberg-kevinliu/

This requires setting up Publishing to PyPI with a Trusted Publisher on https://test.pypi.org/ and using example of publishing pyiceberg_core to testpypi

Note, this PR also refactors .github/workflows/python-release.yml, i've ran the workflow on my fork and manually verified the version for both pypi and svn artifacts.

Setup trusted publisher on testpypi for apache/iceberg-python repo https://test.pypi.org/manage/project/pyiceberg/settings/publishing/

[kevinjqliu]

@Fokko can you take a look at the general flow? I tested in my fork. WYDT about setting the version to 0.9.0rc20250203?

[kevinjqliu]

verified that workflow is skipped on my forked repo
https://github.com/kevinjqliu/iceberg-python/actions/runs/13123573377

[kevinjqliu]

new run https://github.com/kevinjqliu/iceberg-python/actions/runs/13172235318

[Fokko]

What's up with macos-15? :)

[kevinjqliu]

macos-14, macos-15 are the same hardware so produce the same artifacts. i was getting some error uploading to pypi due to

pypi zipfile.BadZipFile: Bad magic number for central directory

removing macos-15 worked 🤷

[Fokko]

I was hoping that we could leave out this kind of debug statements. But it looks like the merge job doesn't mention the actual files: https://github.com/kevinjqliu/iceberg-python/actions/runs/13119881319/job/36603748563

[Fokko]

@kevinjqliu This looks great! Thanks for working on it. Very nice how you re-use the build workflow. I left some small comments on how we can maybe simplify manual dispatch, apart from that it looks great. Let me know if you need any help with setting the credential, I think we need to do that through INFRA

[kevinjqliu]

@Fokko i refactor the code to remove RC, thanks for the suggestion!
I was able to rerun on my fork and published to pyiceberg-kevinliu 0.9.0rc20250203 (I guess pypi changes .dev to rc??)

For setting up trusted publisher, its configured on testpypi. I see you're the maintainer for testpypi pyiceberg, the setting is at https://test.pypi.org/manage/project/pyiceberg/settings/publishing/
Or you can add me and i can configure it

[kevinjqliu]

weird, i dont have Manage permission. let me ask sung to see how he did it for pyiceberg_core

[kevinjqliu]

oh i see, only "owner" can make this change (from pypi: What collaborator roles are available for a project on PyPI?)

This is what it looks like for pyiceberg-kevinliu

So we should fill it out for pyiceberg like so: Owner: apache(?) Repository name:apache/iceberg-pythonWorkflow name:nightly-pypi-build.ymlEnvironment name:testpypi`

@Fokko can you check if you have permission to do so? otherwise, i can reach out to apache infra

[Fokko]

@kevinjqliu I just bumped you to owner 👍

Let me check why it converts rc into dev for the artifacts

[kevinjqliu]

Thanks! I couldnt find anything about the dev -> rc conversion.

I just added trusted publisher pyiceberg on testpypi

[kevinjqliu]

oh wait @Fokko i think thats a false alarm. i was looking at a previous run which had the RC in the version. i yanked that one and now its back to normal https://test.pypi.org/project/pyiceberg-kevinliu/

the files also have dev in its name https://test.pypi.org/project/pyiceberg-kevinliu/#files