Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failing to load present METADATA #13

Open
mboisson opened this issue Aug 17, 2021 · 9 comments · May be fixed by #14
Open

Failing to load present METADATA #13

mboisson opened this issue Aug 17, 2021 · 9 comments · May be fixed by #14

Comments

@mboisson
Copy link

I am writing a script using wheelfile, and it fails on some wheels that seem otherwise correct. It fails with :

Traceback (most recent call last):
  File "../manipulate_wheels.py", line 115, in <module>
    main()
  File "../manipulate_wheels.py", line 57, in main
    with WheelFile(w) as wf:
  File "/cvmfs/soft.computecanada.ca/custom/python/envs/manipulate_wheels/lib/python3.8/site-packages/wheelfile.py", line 1224, in __init__
    self.validate()
  File "/cvmfs/soft.computecanada.ca/custom/python/envs/manipulate_wheels/lib/python3.8/site-packages/wheelfile.py", line 1730, in validate
    raise ValueError(
ValueError: METADATA file is not present in the archive or is corrupted.

yet, if I unzip the wheel, there is:

jupyterlab-3.1.7.dist-info/METADATA

I am not sure what causes this.
@mboisson
Copy link
Author

Ah, the issue is that wheelfile does not handle the License-File attribute.

That is in specific. In general, I think wheelfile barfs whenever it encounters an attribute it does not know about. Is this something expected ?

@mboisson mboisson linked a pull request Aug 17, 2021 that will close this issue
Open
@mboisson
Copy link
Author

Ah, I see that this is using
https://www.python.org/dev/peps/pep-0639/
which is still in draft.... but I find it is being used in many of the wheels we host already.

@MrMino
Copy link
Owner

MrMino commented Aug 17, 2021

Yea, I've just debugged it and was wondering when did PyPA specify the License-File field. The sad thing is that the wheel in question still uses Metadata-Version: 2.1...

Thanks for pointing this out. I'll probably add it provisionally into the next version even though it's not in the spec yet. Do you happen to have any other examples wheels that use this, just so I can make sure there aren't any other surprises?

@mboisson
Copy link
Author

I will run our script
https://github.com/ComputeCanada/wheels_builder/blob/main/manipulate_wheels.py
on a large sample of our ~6k wheels, see if anything else pops up.

I have patched our local installation of wheelfile.py to include the patch in PR #14, so it won't crash for License-File. I'll see if it crashes on something else.

@MrMino
Copy link
Owner

MrMino commented Aug 17, 2021

From the discussion it seems that the current text is not up to date. I'll have to either wait for the PEP to get updated or read through the messages to get the feel for the consesnus, both of which will take time.

Thanks for the PR. Next version will come after I'll get back from vacation (at the beginning of September).

@mboisson
Copy link
Author

Ok, I added some exception handling to wheelfile.py to be able to print a description of the error, and here is what I got:

$ ./manipulate_wheels.py --wheels /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/*{cp37,py3,any}*.whl --print_req | grep -B1 Exception
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/blist-1.3.6+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'provide'.
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/Bottleneck-1.3.2+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'require'.
--
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/dipy-1.3.0+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'require'.
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/dipy-1.4.0+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'require'.
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/dipy-1.4.1+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'require'.
--
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/gvar-11.9.2+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'require'.
--
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/immutables-0.15+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'provide'.
--
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/jaxlib-0.1.64+computecanada-cp37-none-linux_x86_64.whl:
Exception:Invalid version: '0.1.64+computecanada+cuda110'
--
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/louvain-0.7.0+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'provide'.
--
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/MDAnalysis-1.0.0+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'require'.
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/MDAnalysis-1.1.1+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'provide'.
--
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/phono3py-2.0.0+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'provide'.
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/phonopy-2.11.0+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'provide'.
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/phonopy-2.9.1+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'provide'.
--
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/pysam-0.15.4+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'require'.
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/pysam-0.16.0.1+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'require'.
--
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/scikit_image-0.18.2+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'require'.
--
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/spglib-1.16.1+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'provide'.
--
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/vegas-4.0.1+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'require'.
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/vegas-4.0.3+computecanada-cp37-cp37m-linux_x86_64.whl:
Exception:Unknown field: 'require'.
--
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/ete3-3.1.2+computecanada-py3-none-any.whl:
Exception:Unknown field: 'require'.
--
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/requests_ntlm-1.1.0+computecanada-py3-none-any.whl:
Exception:Unknown field: 'provide'.
--
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/ete3-3.1.2+computecanada-py3-none-any.whl:
Exception:Unknown field: 'require'.
--
Requirements for wheel /cvmfs/soft.computecanada.ca/custom/python/wheelhouse/gentoo/generic/requests_ntlm-1.1.0+computecanada-py3-none-any.whl:
Exception:Unknown field: 'provide'.

It looks like Require and Provide are somewhat commonly used.

The invalid version one is obviously our mess up when first patching our wheels without wheelfile, using our crude sed in a shell script.

@MrMino
Copy link
Owner

MrMino commented Aug 17, 2021

Alright, thanks! I'll work it out in the next version.

@ccoulombe
Copy link

How is this coming along?

@MrMino
Copy link
Owner

MrMino commented Oct 10, 2021

Not swiftly unfortunately, due to my private life. This is still on my mind though, don't worry.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

added support for license_file defined in PEP639 mboisson/wheelfile
3 participants
@MrMino @mboisson @ccoulombe