Skip to content

MAINT Unvendor tests at the lockfile creation step#116

Merged
ryanking13 merged 10 commits intopyodide:mainfrom
ryanking13:unvendor-test-split
Mar 22, 2025
Merged

MAINT Unvendor tests at the lockfile creation step#116
ryanking13 merged 10 commits intopyodide:mainfrom
ryanking13:unvendor-test-split

Conversation

@ryanking13
Copy link
Member

@ryanking13 ryanking13 commented Mar 6, 2025

This PR changes when test codes are unvendored.

Before:

  • it is unvendored right after the wheel file is built (== at the same step when build.post script runs).
  • when generating a lockfile, we check if the test codes are unvendored and update the lockfile entry.

After:

  • it is unvendored when we are creating a lockfile.

Why?

I would like to move the unvendoring logic to somewhere else like pyodide-pack or pyodide-lock (pyodide/pyodide-lock#30, pyodide/pyodide-pack#41), so package maintainers can use that feature when they are not using a recipe.


Related issues:

Copy link
Member

@agriyakhetarpal agriyakhetarpal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for making this change, @ryanking13! I agree that pyodide build as a build frontend should not alter the package to remove the test files; it should be the job of the recipe maintainers. This also makes things a bit easier for me with #109, so it is much appreciated.

I think pyodide/pyodide-pack#41 is the ideal location to move these tests unvendoring to, as it will help that repository gain more mobility.

I would suggest that we don't merge/release this change until we release a version of pyodide-pack with that issue resolved. What do you think? I could spend some time implementing it, or step back if you are interested. :)

@ryanking13
Copy link
Member Author

I think pyodide/pyodide-pack#41 is the ideal location to move these tests unvendoring to, as it will help that repository gain more mobility.

Yeah, at first, I thought we should put it in pyodide-lock, but I realized that the reason why we made a separate repository for pyodide-lock was to make the lockfile definition as small as possible (which was a request from JupyterLite team).

So instead of putting complex logic in the pyodide-lock. Maybe let's keep the pyodide-lock as small as possible, and use other packages to do the job.

@ryanking13
Copy link
Member Author

I would suggest that we don't merge/release this change until we release a version of pyodide-pack with that issue resolved. What do you think?

This change itself does not introduce any breaking changes (hence no changelog), so I think we can merge this as is. Then, I think you can start copying the logic to the pyodide-pack. After that, we can make pyodide-pack a dependency of pyodide-build.

I could spend some time implementing it, or step back if you are interested. :)

I would be happy if you could work on this. TBH, no one has recently worked on pyodide-pack after Romans retirement, so you would be the best person who understands the codebase.

@ryanking13
Copy link
Member Author

I am not going to make a release though, as "-wasm-sjlj" part needs to be updated after we update the Emscripten version in Pyodide (pyodide/pyodide#5332)

@agriyakhetarpal
Copy link
Member

I would suggest that we don't merge/release this change until we release a version of pyodide-pack with that issue resolved. What do you think?

This change itself does not introduce any breaking changes (hence no changelog), so I think we can merge this as is. Then, I think you can start copying the logic to the pyodide-pack. After that, we can make pyodide-pack a dependency of pyodide-build.

I could spend some time implementing it, or step back if you are interested. :)

I would be happy if you could work on this. TBH, no one has recently worked on pyodide-pack after Romans retirement, so you would be the best person who understands the codebase.

Oh, I see. Yes, please feel free to merge this! I'll close pyodide/pyodide-lock#30 for now, and proceed to work on pyodide-pack soon.

@ryanking13
Copy link
Member Author

I'll merge this after testing in pyodide/pyodide.

@ryanking13 ryanking13 marked this pull request as draft March 13, 2025 13:00
@ryanking13 ryanking13 marked this pull request as ready for review March 22, 2025 04:11
@ryanking13 ryanking13 merged commit c829ce2 into pyodide:main Mar 22, 2025
6 checks passed
ryanking13 added a commit that referenced this pull request Mar 29, 2025
After #116, I noticed that there are verbose logs printed at the very
end step of recipe build:

```
Building packages... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 12/12 100% Time elapsed: 0:05:13
Copying build logs to /src/packages/build-logs                                                                                                                                                                       
Copying built packages to /src/dist                                                                                                                                                                                  
Writing pyodide-lock.json to /src/dist/pyodide-lock.json                                                                                                                                                             
Unpacking to: /tmp/tmpbcmq2jas/lzma-1.0.0...OK
Repacking wheel as /src/dist/lzma-1.0.0-cp313-cp313-pyodide_2025_0_wasm32.whl...OK
Unpacking to: /tmp/tmp8h460lyt/sqlite3-1.0.0...OK
Repacking wheel as /src/dist/sqlite3-1.0.0-cp313-cp313-pyodide_2025_0_wasm32.whl...OK
Unpacking to: /tmp/tmpq4g8gu0_/pydoc_data-1.0.0...OK
Repacking wheel as /src/dist/pydoc_data-1.0.0-cp313-cp313-pyodide_2025_0_wasm32.whl...OK
Unpacking to: /tmp/tmp2w_726d9/hashlib-1.0.0...OK
Repacking wheel as /src/dist/hashlib-1.0.0-cp313-cp313-pyodide_2025_0_wasm32.whl...OK
Unpacking to: /tmp/tmp0kwjjwcv/micropip-0.9.0...OK
Repacking wheel as /src/dist/micropip-0.9.0-py3-none-any.whl...OK
Unpacking to: /tmp/tmp76vv54io/ssl-1.0.0...OK
Repacking wheel as /src/dist/ssl-1.0.0-cp313-cp313-pyodide_2025_0_wasm32.whl...OK
Unpacking to: /tmp/tmpw4r6k1d5/pydecimal-1.0.0...OK
Repacking wheel as /src/dist/pydecimal-1.0.0-cp313-cp313-pyodide_2025_0_wasm32.whl...OK
Unpacking to: /tmp/tmp2p_yb3sh/tblib-3.0.0...OK
Repacking wheel as /src/dist/tblib-3.0.0-py3-none-any.whl...OK
Unpacking to: /tmp/tmpwrj6kqd2/packaging-24.2...OK
Repacking wheel as /src/dist/packaging-24.2-py3-none-any.whl...OK
```

It is because `python -m wheel {pack, unpack}` prints these logs to
stdout. So, I added a verbose parameter to silence these outputs when we
don't want them.
ryanking13 added a commit that referenced this pull request Mar 29, 2025
I forgot this parameter in #116

---------

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

Comments